I've been describing our Hashicorp Vault journey here at ClueTrust in a number of posts. Chief among the reasons to use Vault is its ability to generate and rotate credentials with specific systems and services.

I've written before about PostgreSQL credential management using Vault, which has been quite successful. This has allowed for short-term, tightly-scoped credentials when interacting with our PostgreSQL servers, meaning that not only are we definitely not storing credentials in code, but the credentials we are using are only minimally potent if taken out of the environment.

This weekend, I began using the AWS Secrets Engine with the intention of creating a similar parttern for accessing our AWS resources.

Although ClueTrust mostly runs its own physical infrastructure, we do have some systems that we run in other environments, including geographically-diverse nameservers we run in AWS. As such, we use AWS credentials to deploy and maintain those (through Ansible, of course).

This weekend's effort was to move from using ansible-vault-stored secrets (which I would hand rotate every 3-6 months) to using Hashicorp-Vault-stored secrets which would be created as needed and would expire quickly.

STS Credentials

I have a fondness for minimized blast radius both in scope and time, which leads to a preference for using AWS STS tokens (see Temporary Security Tokens in AWS).

Using STS in Vault seems a bit of a trade-off. When you create your AWS Role in Vault, that role must have the ability to create and maintain STS tokens, but it must also contain all of the entitlements that you will be granting in that environment. This can be scoped per AWS Secrets Engine (so you can do multiple accounts each at their own endpoint), but the intention here is to trust Vault with as much privilege as all of the needs you have. Initiallly this may feel risky, as you're concentrating risk in that one set of credentials. However, if you were to use multiple static AWS roles and store them in the same Vault, you'd still have the same blast radius scope, but each of those credentials would likely have a larger temporal blast radius.

By using the STS credentials, you can scope minimally in time and specify (by existing AWS IAM groups, policy ARNs, or a specific policy document). This gives you flexibility from Vault to give tightly-scoped credentials when these are issued.

In my case, I used federation_token which provides maximium flexibility to the Vault management. Also available are assumed_role (which uses STS and assumes a role which the Vault-assigned AWS User is able to assume) and iam_user for which Vault creates new users for each request and then deletes those users after a TTL. Vault also supports Static Roles, which are similar to static credentials in databases.

AWS Policies and User

To enable the use of federation_token, your AWS user needs to have permission to use sts:GetFederationToken.

I created a Policy called Federator as such:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sts:GetFederationToken",
            "Resource": "*"
        }
    ]
}

A second policy (Change-self-access-keys) to allow for self-rotation of access keys:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:ListUsers",
                "iam:GetAccountPasswordPolicy"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:*AccessKey*",
                "iam:GetUser"
            ],
            "Resource": [
                "arn:aws:iam::*:user/${aws:username}"
            ]
        }
    ]
}

and then a user (vault-federation) which was assigned these two policies and the additional permissions required to do actual work.

You can assign these directly, although I did so through groups.

Establishing AWS secrets in Vault

The process of configuring Vault for AWS involves:

1. Enable your new secrets store in Vault:

   vault secrets enable aws
   

2. Create the IAM Role(s), Policies, and Users that you need in AWS (discussed below)

3. Register your User (for federation_token, you'll need an actual user; for assumed_role and iam_user you can authenticate to a Role) with Vault:

   vault write aws/config/root access_key=YYY \ 
     secret_key=XXX region=us-east-1 \
     sts_endpoint=https://sts.us-east-1.amazonaws.com sts_region=us-east-1
   

   (Note: for federation_token users, you'll want to assign an sts_endpoint and sts_region to enable use across non-default regions. It's usually best to choose a region that you frequently operate in)

4. Once you've registered the "root" user (poorly named, but it is what it is), you should rotate the credentials so that only vault has them. To do this:

   vault write -force aws/config/rotate-root
   

Creating AWS roles in Vault

Now the fun begins. Typical of use in Vault, you'll establish individual "roles" which can be used to retrieve credentials from AWS an through Vault policies you'll determine which users can access those roles.

I tend to create these roles through scripts and check them in to a git repository, so I will use a command like this:

vault write aws/roles/$role - < aws/roles/$role.json

in order to load my roles. An example of the JSON for a role I'm using is:

{
  "credential_type": "federation_token",
  "default_sts_ttl": 300,
  "iam_groups": null,
  "max_sts_ttl": 3600,
  "policy_arns": [
    "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
  ],
  "policy_document": ""
}

I could use the iam_groups list to use one or more groups as the policy, policy_document to pass a document containing the precise policy to assign (assuming it's a subset of the policies the role has), or (as I'm doing here) pass complete ARNs in a list.

All that remains is to make sure this role is accessible from your users or roles in Vault (left as an exercise to the reader).

Using aws credentials from Vault

Use of the credentials is straightforward:

1. Request the credential through CLI or API

   % vault write aws/sts/ec2-admin ttl=15m   
   Key                Value
   ---                -----
   lease_id           aws/sts/ec2-admin/NNN
   lease_duration     14m59s
   lease_renewable    false
   access_key         XXX
   secret_key         YYY
   security_token     ZZZ
   ttl                14m59s
   

2. Use these credentials for the next 15 minutes as necessary