Secret Management

Secrets like usernames, passwords, API keys, and private keys should be kept in the GitHub Secrets store. They should not be hard-coded and included in a GitHub repository, nor hard-coded into CI/CD pipelines or Docker images.

GitHub Secrets cannot share the credential store with other applications & services, but GitHub Actions pipelines may copy values to another service in AWS, to make them available for use. The will generally be published to either the AWS Systems Manager (SSM) Paramater Store as encrypted values, or the AWS Secrets Manager. Both services have similar capabilities, but size limits and integration with ECS/Lambda/etc differ.

The SSM paramater store securely store secrets for use by your apps and other infrastructure. It has first-class integration for some services -- for example, the Elastic Container Service (ECS) can decrypt and inject secrets from SSM automatically upon booting a containeropen in new window. For services that do not offer this integration, the AWS SDK can be used to access & decrypt secrets in your application code.

The SSM parameter store can hold other, not-secret config values too. That is beyond the scope of this article.

Strategy

All Jenkins jobs are sorted into folders. Folders provide a security boundry for credentials: only jobs inside a folder can access credentials scoped to the folder. This way, an 'HR Enterprise' job would not have access to a 'Student Enterprise' credential. Groups of developers can be assigned permission to create/update/delete credentials for their folders.

There is a working demo of this strategy available in the jenkins-credential-sync-test repositoryopen in new window.

GitHub Setup

Consult GitHub's guide on managing repository secrets and using them in GitHub Actions:

Using secrets in GitHub Actionsopen in new window

Terraform Setup

The SSM parameter resources should be created by Terraform as IaC. The easiest way to do this is specifying a template resource and putting your parameter names in an array. The parameter names should be emitted as an output from your Terraform modules.

You may optionally create an application-specific encryption key for your secrets. This is free and has no downsides, so it is recommended to do so.

# Make this a list of your secrets. These names should match the Jenkins credential IDs.
locals {
    parameters = ["jcst-demo-api-key", "jcst-password"]
}

# This will create two resources; one for each entry in the parameters array.
# You should customize the name for your app. 
# SSM is a a hierarchy, so "/your-app/env" is a good prefix!
resource "aws_ssm_parameter" "secure_param" {
  count = length(local.parameters)

  name        = "/github-ssm-sync/tech-demo/${local.parameters[count.index]}"
  description = "Demo secret: ${local.parameters[count.index]}"
  type        = "SecureString"
  value       = "SSM parameter store not populated from Jenkins"
  key_id      = aws_kms_key.key.arn

  tags = {
    environment = "tech-demo"
  }

  # The parameter will be created with a dummy value. Jenkins will update it with 
  # the final value in a subsequent pipeline step.
  #
  # TF will not override the parameter once it has been created.
  lifecycle {
    ignore_changes = [value]
  }
}

output "parameters" {
    value = zipmap(local.parameters, slice(aws_ssm_parameter.secure_param.*.name, 0, length(local.parameters)))
}

resource "aws_kms_key" "key" {
  description = "Key for encrypting demo config"
}

This will create two encrypted SSM parameters with dummy values.

Pipeline

To copy the values from GitHub Secrets to SSM, review the documentation for our AWS Secrets Sync action:

AWS Secrets Syncopen in new window

Using Secrets

Here are some notes on common ways to use SSM secrets.

In all cases, the service will need to be given access to the KMS key before it can decrypt parameters -- granting your application access to an encrypted SSM parameter without the key will result in errors when you attempt to access the secret!

Here is an example policy, based on the parameter example above:

data "aws_iam_policy_document" "lambda_secrets_policy" {
  statement {
    effect    = "Allow"
    actions   = ["ssm:GetParameters", "ssm:GetParameter"]
    resources = aws_ssm_parameter.secure_param.*.arn
  }

  statement {
    effect    = "Allow"
    actions   = ["kms:Decrypt"]
    resources = [aws_kms_key.key.arn]
  }
}

resource "aws_iam_role_policy" "lambda_secrets_policy" {
  name   = "SomeApp-Env-Secrets"
  policy = data.aws_iam_policy_document.lambda_secrets_policy.json

  # Your Lambda/ECS/etc execution role name
  role = ". . ."
}

In the Console

By default, the KMS encryption key that you create will not be usable. You will grant your Lambda/ECS/etc kms:Decrypt access, but if you want to review your secrets in the console or use them from the CLI, there is an additional step -- granting the developer role you access AWS with access.

To avoid exposing secrets to other teams in your department, this access it not automatically granted. The example IAM policy above could specify your login group's name instead of an execution role name, in order to enable decryption in the AWS Console.

data "aws_iam_role" "developers" {
  name = "as-ado-sbx-Devs-EACD"
}

resource "aws_iam_role_policy" "lambda_secrets_policy" {
  name   = "SomeApp-SomeEnv-DeveloperDecryptSecrets"
  role   = data.aws_iam_role.developers.name
  policy = data.aws_iam_policy_document.lambda_secrets_policy.json
}

The IAM article has a list of available roles.

ECS

In your ECS task definitionopen in new window, you can specify a secrets section with the ARNs for your parameters. They will automatically be decrypted and injected into your container as environment variables when it is started:

{
    "cpu": 1024,
    "memory": 2048,
    "name": "my-cool-task",
    "portMappings": [{
        "containerPort": 80,
        "hostPort": 80
    }],
    "essential": true,
    "image": "${aws_ecr_repository.my_ecr.repository_url}:latest",
    "secrets" : [
        {"name": "${local.ssm_params[0]}", "valueFrom": "${aws_ssm_parameter.secure_param.0.arn}"}
    ]
}

Lambda

Lambda does not have a native integration for SSM parameters. However, AWS offers SDKs for most languages: Python, Java, PHP, etc. that are all similar.

Here is an example using the AWS Javascript SDK:

const AWS = require('aws-sdk');
const ssm = new AWS.SSM({
  region: 'us-east-2',
});

const ssmResponse = await ssm.getParameter({
    Name: '/jenkins-ssm-sync/tech-demo/jcst-password',
    WithDecryption: true,
}).promise();

return ssmResponse.Parameter.Value;

If you are scheduling a Lambda to be run through CloudWatch or a Step Function, you can use Terraform to inject the full parameter name into the event that invokes the Lambda. Then you won't need to worry about keeping the parameter names in sync between SSM and your codebase.

Your Lambda should cache the retrieved SSM parameter values for its lifetime. There is an account-level rate limit for SSM of 40 requests per second, so avoid unnecessary GetParameter calls. There is a GetParameters call as well, which can be used to fetch a set of parameters in bulk.

# Secret Management

# Strategy

# GitHub Setup

# Terraform Setup

# Pipeline

# Using Secrets

# In the Console

# ECS

# Lambda