Configuring Access Management in Amazon EKS with IAM Roles for Service Accounts
🔐

Configuring Access Management in Amazon EKS with IAM Roles for Service Accounts

19th April 2021 · 15 minute read

Mastering identity and access management is critical to running applications securely in the cloud. If you're using EKS, IAM Roles for Service Accounts are the best way to enable fine-grained access management for compute workloads in your cluster.

Before we Start

In this guide we will learn what IAM Roles for Service Accounts are, why they are important, and how to configure them on your EKS Cluster.

Building Blocks

Amazon EKS

This guide assumes you're using Amazon's flavour of Kubernetes known as Amazon EKS. Other cloud providers such as Azure and GCP will have different systems for managing authentication. For the sake of simplicity, this tutorial assumes you have already configured and are running an EKS Cluster in AWS.

CloudFormation

I always make an effort to use Declarative Infrastructure-as-Code for deploying resources into AWS. There are many tools available for this, but I tend to prefer CloudFormation for its simplicity and lack of upfront cost. Other tools such as Amazon CDK, Terraform or Pulumi can be used if preferred, though this specific tutorial may be limiting.

Configuration

ℹ️
The following steps are considered prerequisites, but can be skipped if you have already completed them.
‣
Install the AWS CLI

I will be using the AWS CLI in order to automate a lot of the steps involved in this guide and to avoid using the AWS Console.

For macOS, use Homebrew → brew install awscli.

For Windows, use Chocolatey → choco install awscli.

For other operating systems, follow this guide.

‣
Install eksctl

In order to automate some parts of this guide, there is a particular step in the install process that requires a utility called eksctl.

For macOS, use Homebrew → brew install eksctl.

For Windows, use Chocolatey → choco install eksctl.

For other operating systems, follow this guide.

‣
Install jq

In order to automate some parts of this process, we will use jq, a tool to parse JSON data in the terminal.

For macOS, use Homebrew → brew install jq.

For Windows, use Chocolatey → choco install jq.

For other operating systems, follow this guide.

‣
Provision an EKS Cluster

There are a large number of steps involved with provisioning a new EKS Cluster, especially if you know what you're doing and want to do it correctly.

If you don't have a cluster to work with, you can create a test cluster using the eksctl setup guide.

❗
Warning: Be mindful that an EKS Cluster can be expensive to keep running, even in its smallest configuration. I'd recommend setting a reminder ahead of time to dismantle any clusters created for testing purposes.

What are IAM Roles for Service Accounts?

As of Kubernetes 1.12, support was added for a new ProjectedServiceAccountToken feature, which is a JSON Web Token (JWT) that also contains the Service Account identity, and supports a configurable audience.

Service Accounts

A Service Account provides an identity for processes that run in a Pod.

When a Pod is created, if you do not specify a Service Account, it is automatically assigned the default Service Account in the same namespace.

IAM Roles

An IAM Role is an entity within your AWS account that has specific permissions. An IAM Role is similar to an IAM User, in that it is an AWS identity with permission policies that determine what the identity can and cannot do in AWS. However, instead of being uniquely associated with one person, an IAM Role is intended to be used by whoever needs it.

OpenID Connect

OpenID Connect (OIDC) is a simple identity layer on top of the OAuth 2.0 protocol. It allows clients to verify the identity of a Service Account with the use of a centralised authorisation server.

All EKS Clusters come with a OIDC Discovery Endpoint. This allows Pods to assume IAM Roles via the Secure Token Service, enabling authentication with an OIDC provider, receiving a JWT, which in turn can be used to assume an IAM Role.

Benefits of IAM Roles for Service Accounts

Using IAM Roles for Service Accounts has a few benefits;

  • First-Party Eliminates the need for a third-party solution like kaim or kube2iam.
  • Least Privilege Access You can scope permissions to individual Pods using the Service Account, rather than issuing extended permissions to the entire Node.
  • Credential Isolation Containers can only retrieve credentials for the IAM Role associated to the Service Account, and never have access to credentials that belong to other Pods.
  • Auditing Access to IAM Roles is augmented to event logging through AWS CloudTrail.

Setup Steps

Create an IAM OIDC Provider

Use the following command to create an IAM OIDC Provider, replacing the value for --cluster with the name of your EKS Cluster;

$ eksctl utils associate-iam-oidc-provider --approve --cluster my-cluster

Retrieve the Cluster OIDC Issuer

At the time of writing, CloudFormation does not support reading the OIDC Issuer as an output parameter of the EKS Cluster. To get around this, we can use the following CloudFormation template:

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Retrieves OIDC Issuer URL from an EKS Cluster'

Parameters:
  ClusterName:
    Type: String

Resources:

  # An IAM Role for retreiving information about the EKS Cluster.
	ClusterOIDCLambdaIAMRole:
	  Type: AWS::IAM::Role
	  Properties:
	    AssumeRolePolicyDocument:
	      Version: '2012-10-17'
	      Statement:
	        - Sid: AllowLambdaAssumeRole
            Effect: Allow
	          Principal:
	            Service:
	              - lambda.amazonaws.com
	          Action:
	            - 'sts:AssumeRole'
	    Path: /
	    Policies:
	      - PolicyName: ClusterOIDCPolicy
	        PolicyDocument:
	          Version: '2012-10-17'
	          Statement:
              - Sid: AllowDescribeCluster
	              Effect: Allow
	              Action:
	                - 'eks:DescribeCluster'
	              Resource: '*'
	            - Sid: AllowWriteLogs
                Effect: Allow
	              Action:
	                - 'logs:CreateLogGroup'
	                - 'logs:CreateLogStream'
	                - 'logs:PutLogEvents'
	              Resource: 'arn:aws:logs:*:*:*'

  # Lamdba function used to retrieve the OIDC issuer URL from the EKS Cluster.
	ClusterOIDCLambdaFunction:
	  Type: AWS::Lambda::Function
	  Properties:
	    Code:
	      ZipFile: !Sub |
	        import cfnresponse
	        import boto3
	        import json
	        def lambda_handler(event, context):
	          try:
	            oidc = {}
	            if event['RequestType'] == 'Create' or event['RequestType'] == 'Update':
	              client = boto3.client('eks')
	              response = client.describe_cluster(
	                name='${ClusterName}'
	              )
	              if response['ResponseMetadata']['HTTPStatusCode'] == 200:
	                oidc['issuer'] = (response['cluster']['identity']['oidc']['issuer']).split("https://")[1]
	            cfnresponse.send(event, context, cfnresponse.SUCCESS, oidc, "CustomResourcePhysicalID")
	            return oidc
	          except Exception as e:
	            print('Failed to fetch Cluster OIDC value for cluster', e)
	            cfnresponse.send(event, context, cfnresponse.FAILED, {}, "CustomResourcePhysicalID")
	    Handler: index.lambda_handler
	    MemorySize: 1024
	    Role: !GetAtt ClusterOIDCLambdaIAMRole.Arn
	    Runtime: python3.7
	    Timeout: 300

  # Resource to store the Cluster OIDC issuer URL.
	ClusterOIDC:
	  Type: Custom::ClusterOIDC
	  Properties:
	    ServiceToken: !GetAtt ClusterOIDCLambdaFunction.Arn

# Output the Cluster OIDC issuer URL for later use.
Outputs:
  ClusterOIDCIssuer:
    Value: !GetAtt ClusterOIDC.issuer
    Export:
      Name: !Sub ${ClusterName}OIDCIssuer

Copy the template and save it to your working directory as cluster-oidc.yml. Next, install the template by running the following, replacing the value for ClusterName with the name of your EKS Cluster:

$ aws cloudformation create-stack \
  --template-body file://cluster-oidc.yml \
  --stack-name cluster-oidc \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \
  --parameters ParameterKey=ClusterName,ParameterValue='my-cluster'

Create an IAM Role

The following CloudFormation template can be used to create an IAM Role for a Service Account;

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Creates an IAM Role for a Service Account'

Parameters:
  ClusterName:
    Type: String
  Namespace:
    Type: String
  ServiceAccount:
    Type: String

Resources:
  ClusterSelfDescribeRole:
    Type: AWS::IAM::Role
    DependsOn: ClusterOIDC
    Properties:
      RoleName: ClusterSelfDescribeRole
      Path: '/'
      # This policy document allows the service account to assume
      # the IAM Role through the federated OIDC provider.
      AssumeRolePolicyDocument: !Sub 
        - |
          {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Effect": "Allow",
                "Principal": {
                  "Federated": "arn:aws:iam::${AWS::AccountId}:oidc-provider/${Issuer}"
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                  "StringEquals": {
                    "${Issuer}:sub": "system:serviceaccount:${Namespace}:${ServiceAccount}"
                  }
                }
              }
            ]
          }
        - Issuer:
            Fn::ImportValue: !Sub ${ClusterName}OIDCIssuer
      # This policy document outlines the actual permissions of the
      # IAM Role. This is where you can add permissions for AWS
      # services such as S3, RDS or Secrets Manager. This policy
      # allows the pod to describe the cluster it is managed by for
      # testing purposes.
      Policies:
        - PolicyName: ClusterOIDCPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
	            - Sid: AllowDescribeCluster
                Effect: Allow
	              Action:
	                - 'eks:DescribeCluster'
	              Resource: '*

Copy the template and save it to your working directory as cluster-iam-role.yml. Next, install the template using the following command, replacing Namespace and ServiceAccount with the desired names (which do not exist yet, we will create them in the next step);

$ aws cloudformation create-stack \
  --template-body file://cluster-iam-role.yml \
  --stack-name cluster-iam-role \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \
  --parameters ParameterKey=ClusterName,ParameterValue='my-cluster' \
  --parameters ParameterKey=Namespace,ParameterValue='test' \
  --parameters ParameterKey=ServiceAccount,ParameterValue='my-service-account'

Once the CloudFormation template has been applied, you will need to find the ARN for the newly created IAM Role. Do this by running the following in your terminal;

$ aws iam list-roles jq -r 
  | '.Roles | map(select(.RoleName == "ClusterSelfDescribeRole" )) | .[0].Arn'

arn:aws:iam::123456789012:role/ClusterSelfDescribeRole

Install a Service Account

Now that we have IAM Roles for Service Accounts configured, we can start creating the resources corresponding inside Kubernetes that will consume them.

Using the newly created IAM Role ARN, as well as same values for Namespace and ServiceAccount from the previous step, run the following command in your terminal to create a new Service Account;

$ kubectl apply -n test -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ClusterSelfDescribeRole
EOF
ℹ️
Note: You may firstly need to run kubectl create namespace test to create the namespace if it does not exist.

You can confirm the Service Account was created successfully by running the following command in your terminal;

$ kubectl get sa -n test

NAME                SECRETS   AGE
default             1         13s
my-service-account  1         9s

Create a Pod with the Service Account

All that's left to do is spin up a Pod and assign it to the newly created Service Account. I've opted to use the official AWS CLI Docker image as it will make it easy to consume and test the IAM Role.

To install the Pod, run the following in your terminal;

$ kubectl apply -n test -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: my-service-account
  containers:
    - name: my-container
      image: amazon/aws-cli
      command: ["sh", "-c", "aws eks describe-cluster --name my-cluster"]
EOF

Test the Service Account

As configured, the Pod should automatically run the aws eks describe-cluster command and if configured successfully, output information about the cluster in which it is running.

Confirm this by running the following in your terminal to read the logs for the newly created Pod;

$ kubectl logs -n test my-pod

{
  "cluster": {
    "name": "my-cluster",
    "arn": "arn:aws:eks:us-east-1:123456789012:cluster/my-cluster",
    ...
  }
}

Resources

Consistent information about how to configure IAM Roles for Service Accounts is notoriously difficult to track down. This post aims to distill the desired outcomes down to the fewest number of steps.

The following is the official documentation for IAM Roles for Service Accounts which covers all of the required setup steps, although many of them involve using the AWS Console or AWS CLI.

For those like myself that want to configure infrastructure with Declarative Infrastructure-as-Code by default, this post provided a solution to reading the OIDC Issuer from the EKS Cluster using CloudFormation which I have appropriated for this post.