Skip to content

Instantly share code, notes, and snippets.

@davidcallen

davidcallen/using-aws-secrets-manager-with-sops.md

Last active June 19, 2024 04:16
Show Gist options
  • Save davidcallen/36e1d068fc352f4563297c7eb96f287d to your computer and use it in GitHub Desktop.
Save davidcallen/36e1d068fc352f4563297c7eb96f287d to your computer and use it in GitHub Desktop.
Download ZIP
Using AWS Secrets Manager with SOPS
Raw
using-aws-secrets-manager-with-sops.md

Using AWS Secrets Manager with SOPS

Dealing securely with secrets in your infrastructure is a difficult task. It is very easy to simply push the problem "further down the road". Hopefully this Gist can provide some insight.

One possible solution is using AWS Secrets Manager (ASM) and Terraform.

We can securely store our secrets in an encrypted file which can then be committed to our source repository. This encryption will use an AWS KMS Key.

Whilst one can use the AWS CLI to encrypt the file, this include the risk of the unencrypted file lingering on the terraform workspace local filesystem. Using the Mozilla Sops tool we can prevent this since it combines the decryption+editing+encryption workflow.

So to begin...

Terraform our AWS KMS Key

We will need a KMS Customer Managed CMK Key for encrypting/decrypting our Secrets files. We want this to only be usable by our "Cloud Admin" via IAM and by this AWS account (this is assuming secrets are silo-ed by account).

resource "aws_kms_key" "secrets" {
  description             = "example-kms-secrets"
  key_usage               = "ENCRYPT_DECRYPT"
  policy = jsonencode({
    Version = "2012-10-17"
    Id      = "KeyForEncryptingDecryptingOurSecretsFiles"
    Statement = [
      {
        Sid    = "Enable IAM policies"
        Effect = "Allow"
        Principal = {
          "AWS" = "arn:aws:iam::${var.environment.account_id}:root"
        }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "Allow access for Administrators"
        Effect = "Allow"
        Principal = {
          "AWS" = [
            aws_iam_role.admin.arn
          ]
        }
        Action   = "kms:*"
        Resource = "*"
      }
    ]
  })
  tags = merge(module.global_variables.default_tags, var.environment.default_tags, {
    Name        = "example-kms-secrets"
    Application = "Key for encrypting/decrypting our Secrets files"
  })
}
resource "aws_kms_alias" "secrets" {
  name          = "alias/example-kms-secrets"
  target_key_id = aws_kms_key.secrets.key_id
}
output "kms_secrets_arn" {
  value = aws_kms_key.secrets.arn
}

Use SOPS to create our encrypted file

Use sops to create a new encrypted file using the KMS key created as output from above Terraform like this :

cd terraform/core
SOPS_KMS_ARN="arn:aws:kms:eu-west-1:123456789:key/7ed3d239-xxxx-4eba-bfe0-c490bff8ff7e" aws-vault exec my-aws-profile -- sops secrets.encrypted.json

sops will open an editor (it honours your EDITOR env var). Paste in the secret information as below :

example-user:
  username: aaaaa
  password: 12345

Note that we are using aws-vault by 99designs to simplify and secure aws access. It will get our AWS credentials and role_arn and pass it through to the sops tool using environment variables.

sops will create a JSON format file that contains our encrypted data and looks like this redacted snippet :

{
    "data": "ENC[AES256_GCM,data:Zk5IK .... ,type:str]",
    "sops": {
        "kms": [
            {
                "arn": "arn:aws:kms:eu-west-1:123456789:key/7ed3d239-xxxx-4eba-bfe0-c490bff8ff7e",
                "created_at": "2022-02-09T10:00:57Z",
                "enc": "AQ .... ==",
                "aws_profile": ""
            }
        ]
    }
}

Note that the above "data" JSON attribute holds our encrypted secret information. In my scenario my encrypted info is in YAML format. If you use a JSON format then the sops terraform integration is slightly simpler.

Decrypting SOPS file in Terraform

To use the encrypted file from Terraform I use a sops terraform provider by carlpett.

terraform {
  required_providers {
    # ....
    sops = {
      source = "carlpett/sops"
      version = "= 0.6.3"
    }
  }
  required_version = "= 1.0.11"
}

This provider can read the encrypted file into a Terraform variable with :

provider "sops" {}
data "sops_file" "primary" {
  source_file = "${path.module}/secrets.encrypted.json"
  input_type = "raw"
}
locals {
  secrets_primary = yamldecode(data.sops_file.primary.raw)
}

Then use the secret to create a Secret within AWS Secrets Manager with :

resource "aws_secretsmanager_secret" "secret" {
  name_prefix = "example-"
  description = var.description
}
resource "aws_secretsmanager_secret_version" "secret" {
  secret_id     = aws_secretsmanager_secret.secret.id
  secret_string = local.secrets_primary.example-user.password
}

This secret needs to have its access secured so we add an IAM policy :

# Deny access to all except if from our list of IAM Role (Profiles) and their user ids (unique_ids)
locals {
  allowed_iam_user_ids = [
    123456789,      # our aws account id
    "${data.aws_iam_role.admin.unique_id}:*",   # lookup for our "Cloud Admin" role so then can administer the Secret.
    "${module.iam-nexus.nexus-role.unique_id}:*"  # Our EC2 instance profile (role) that wants the secret.
  ]
}
resource "aws_secretsmanager_secret_policy" "secret" {
  secret_arn = aws_secretsmanager_secret.secret.arn
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "GetSecretForRoles"
        Effect = "Deny"
        Principal = {
          "AWS" = "arn:aws:iam::${var.account_id}:root"
        }
        Action = [
          "secretsmanager:GetSecretValue"
        ]
        Resource = [
          aws_secretsmanager_secret.secret.arn
        ]
        Condition = {
          StringNotLike = {
            # Deny access to all except if from our list of IAM Role (Profiles) and their user ids (unique_ids)
            "aws:userId" = concat(local.allowed_iam_user_ids[*])
          }
        }
      }
    ]
  })
}

You could encapsulate the creation of the ASM Secret (above) into a convenient terraform module as in this one.

Using the ASM Secret

In our example we want to use the secret in an EC2 instance.

We could use Terraform to pass the secret (and other config) through the AWS User Data however this is a bit insecure.

Instead we assign the EC2 instance an IAM Instance Profile. This has been granted read access to that individual ASM Secret in its IAM Policy.

The EC2 instance uses cloud-init to invoke a boot-up script that can get the secret and apply it to its application configuration e.g.

ADMIN_USER_PASSWORD=$(aws secretsmanager get-secret-value --region=${AWS_REGION} --secret-id ${APP_ADMIN_USER_PASSWORD_SECRET_ID} --query SecretString --output text)
# now store the ${ADMIN_USER_PASSWORD} in our app config somewhere.

The awkward terraform state file

Of course with this solution the secret will still be present in plaintext in the terraform state file.

We can minimise this risk by using a remote state storage like an S3 bucket that has at-rest encryption enabled and a restricted IAM policy to ensure only the terraform/cloud admin can read it.

Using Random passwords to avoid secrets within terraform state file

If we were not particular about the passwords being pre-determined, then we could use ASM to generate a random password for us and avoid its presence in the terraform state file.

Currently ASM can generate new secrets with a random password using the CLI verb get-random-password here

However this feature is not supported as yet in terraform-provider-aws. There is an issue raised for it here and the pending Code fix

This feature will be compelling when available and could avoid the need to use sops.

(Although it will mean that the secrets become less determined which is slightly awkward in a fast build-up/tear-down Lab environment. More importantly it would probably not work well with Secrets that are not just passwords e.g. SSH private keys )

Summary : The Good, the Bad and the Ugly

Good :

  • Using the AWS KMS Key we avoid the recursive creation of more secrets to protect our secrets (since we already have a secured access to that AWS account where the key resides).
  • Using sops tool prevents risk of creating an unencrypted secrets file on an Admin's machine
  • Access to secret on ASM is audited
  • Secret in ASM can be fetched from an EC2 instance (less risky than via passing via user-data)
  • Using ASM may be improved once this Issue fixed will help avoid it :
  • Secrets can be protected by an IAM policy (unlike AWS SSM Parameter Store)
  • Secrets can be ring-fenced for each environment e.g. dev, staging, prod. They can be contained in their respective AWS accounts within ASM and the sop encrypted files are only decryptable by the KMS Key within that environment. This avoids a "dev" or "staging" admin having access to "prod" secrets.

Bad:

  • Cost of $0.40 per secret per month (unlike the free AWS SSM Parameter Store)
  • Rotation of secrets cannot be done by ASM since the encrypted secrets file is the golden source. So rotation would need to be a terraform-driven process

Credits

This gist was based on this excellent article by Gruntworks. Hopefully I have succeeded in adding value to it (for this use-case) by :

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment