On Amazon EKS and ACK

The extensibility of Kubernetes itself through custom resource definitions and the popularity of the Operator and Controller pattern has created a broad range of wonderfully useful cluster add-ons. It is a long-established practice to deploy cluster add-ons to extend the capabilities of Amazon EKS clusters.

One category of cluster add-ons happily runs without the need to access cluster-external resources. Another type of cluster add-ons leverages external resources, as we want them to interact with AWS Services, for example. These include cluster add-ons for specific use cases such as node scalers that change Amazon EC2-backed worker node capacity of the Kubernetes data plane; add-ons such as Ingress Controllers that create AWS Elastic Load Balancers on our behalf to forward traffic into the cluster; add-ons that automatically add DNS records into Amazon Route53 hosted zones for Ingress objects that use hostname rules; or add-ons such as EBS CSI and EFS CSI drivers that dynamically provision Amazon storage services for us.

Utilising cluster add-ons to provision AWS Services from within a Kubernetes cluster dynamically is a powerful concept to leverage, as it helps (among many other things) reduce the amount of AWS Services provisioning that needs to happen outside the cluster.

When ready-made cluster add-ons are unavailable for particular use cases, then Platform teams might reach towards alternative ‘out of band’ options to provision AWS Services, only to tie them back into Amazon EKS clusters afterwards. A similar situation arises when developers want to build Kubernetes applications that leverage specific AWS Services. After all, writing cluster add-ons or Controllers for your use cases is not always possible for everyone.

It is therefore not uncommon to see an emerging landscape of diverse tool sets and pipelines being used at various layers of the stack, such as tools and pipelines that provision AWS Services, another set for Kubernetes cluster add-ons, and yet another set for handling application workloads.

AWS Services: AWS provisioning tools such as the AWS Cloud Development Kit (CDK), HashiCorp Terraform, or other command line interfaces such as the AWS CLI are often used to provision the Amazon EKS control plane, as well as many other essential AWS native services such Amazon S3 object storage; Amazon RDS and Amazon ElastiCache database services; networking services such as AWS PrivateLink, Security Groups and NACLs; and security and compliance services such as Amazon GuardDuty, AWS Config, and AWS Security Hub. AWS CloudFormation templates are then perhaps kept on AWS CodeCommit and deployed via AWS CodePipeline, for example.

Kubernetes Cluster Add-ons: Cluster add-ons can also be provisioned via CDK or CloudFormation templates, which is a typical pattern for those types of add-ons that interact with AWS Services and therefore require privileges to do so. Alternatively, only the AWS IAM resources might be provisioned via CDK or CloudFormation, and the cluster add-ons themselves are deployed and configured via GitOps Controllers such as FluxCD.

Kubernetes Applications: The application workloads can also be deployed via GitOps. For example, applications that are templated with Kustomize or packaged as Helm charts can be described in Git and reconciled into the Amazon EKS cluster runtime via FluxCD.

This can lead to an uncanny set of challenges, as we effectively develop and maintain a dedicated code-base and life-cycle for AWS resources and another for Kubernetes applications while constantly ensuring they are aligned.

To help us streamline the developer experience of Kubernetes clusters, we aim at increasing cloud-native workflows, and we look at reducing the amount of “out of band” provisioning of AWS resources.

Enter AWS Controllers for Kubernetes or ACK for short. ACK Service Controllers provide a Kubernetes experience for interacting with AWS Services. ACKs are open-source Controllers that equip Kubernetes clusters with generic access to many different AWS Services. This enables developers to describe the desired state of AWS resources using Kubernetes configuration language and the Kubernetes API they are already familiar with. The following diagram illustrates the target developer workflow.

ACK Service Controllers deploy via GitOps onto Amazon EKS and provision AWS Services for applications

ACKs can be a valuable addition to the existing line-up of cluster add-ons, especially in a context where application developers leverage and adopt a growing range of AWS Services. In response to this, platform teams can help improve the developer experience by adding “AWS Service abstractions” to their clusters that application developers can readily use.

With ACK, we can add another set of cluster add-ons that help Kubernetes application developers access AWS Services in familiar and cloud-native ways.

Kubernetes application developers would then need to understand the parameterisation details of the respective ACK API and define the corresponding custom resource yaml’s… that’s it!

Let’s do it…

For those on a tight time budget: The TL;DR of the following sections shows that by adding ACKs to our cluster add-on line-up, we can improve the adoption of AWS-managed services for Kubernetes applications developers and reduce the need to manage AWS resources with a separate code-base. At the same time, we can fit the ACK approach into a GitOps model that streamlines Amazon EKS cluster operations.

The first thing to do is to install our set of desired ACK Service Controllers into a Kubernetes cluster so that application developers can deploy their application workloads confidently, in the knowledge that the Controllers will create the requested set of AWS resources.

A dedicated ACK Service Controller is released per AWS Service, which is an important characteristic that helps minimise the AWS IAM privilege footprint of the Controller pods. Each ACK has the access it needs for the AWS Service it handles. ACK artefacts are published on Amazon ECR. The ECR registry stores the container images of the ACKs and their corresponding Helm chart packages. You’ll notice two artefacts per controller when you sort the registry alphabetically. All ACKs are effectively deployed in the same way, albeit independently. The installation has two building blocks: Defining AWS permissions for our ACKs and installing the actual ACKs as helm charts.

We need the ACKs to interact automatically with their corresponding AWS Service APIs on our behalf. Notice that the ACKs do not synthesise CloudFormation in the backend but rather place calls directly to the respective AWS Service APIs. For this to happen, we need IAM resources and update our Amazon EKS cluster with IAM Role details associated with ACK Service Accounts. These are called IAM Roles for Service Accounts, or IRSA for short. In this way, we associate AWS IAM credentials to the ACK pods.

Each Controller needs their IRSA resources created as described in the ACK docs available here. We templated the IRSA steps with CDK to avoid executing a manual process every time.

The installation of the ACK helm charts can then happen manually via the Helm CLI or automatically via GitOps and FluxCD. The available parameters for the helm charts are nicely documented in their respective values.yaml file. Some subtleties to observe here: The ACK Helm charts are stored on Amazon ECR as OCI-compliant artefacts, and you will need to authenticate.

$ aws ecr-public get-login-password --region $AWS_REGION --profile $MY_AWS_CRED_PROFILE_NAME | helm registry login --username AWS --password-stdin public.ecr.aws

You will require AWS credentials to retrieve the ecr-public password to complete the helm registry login. In other words: You need an AWS account to access the charts. Once logged in, we can install any ACK chart with the Helm CLI and our environment variables, for example.

$ helm install --create-namespace -n $ACK_SYSTEM_NAMESPACE ack-$SERVICE-controller \ oci://public.ecr.aws/aws-controllers-k8s/$SERVICE-chart --version=$RELEASE_VERSION --set=aws.region=$AWS_REGION

With our selected ACKs installed, we’re ready to start offering them to our application developers.

Amazon S3: A developer can request an Amazon S3 bucket from the kube-api server, and the ACK Service Controller will oblige and create the Amazon S3 bucket. The complete set of available fields is shown in the API reference here. The ACK thankfully applies sane defaults for things not explicitly defined in the Bucket yaml. The status fields of the resulting Bucket object will also contain supplementary information about the S3 bucket that was provisioned on AWS.

apiVersion: s3.services.k8s.aws/v1alpha1
kind: Bucket
metadata:
  name: test-bucket
spec:
  name: test-bucket
  tagging:
    tagSet:
    - key: myTagKey
      value: myTagValue

Developers can then modify already provisioned buckets by changing the resource yamls via GitOps. The ACK Service Controller will then affect the change for mutable fields. For example, a developer may want to enable versioning for a given bucket. To achieve that we need to amend the yaml as shown.

apiVersion: s3.services.k8s.aws/v1alpha1
kind: Bucket
metadata:
  name: test-bucket
spec:
  name: test-bucket
  tagging:
    tagSet:
    - key: myTagKey
      value: myTagValue
  versioning:
    status: Enabled

The amended yaml then gets committed to Git and the GitOps controller applies the yaml to the cluster. The ACK Service Controller for S3 will then place the appropriate call to the AWS S3 API that changes the versioning from Disabled to Enabled.

Amazon RDS: The following example shows how a managed RDS database service can be requested from the kube-api server, and the ACK Controller will provision the database on AWS for us. Akin to the example for S3, we need to know about the database service we want to use and understand the fields we need. In our example, we’ll need to prepare a few things: We need a Secret that contains the credentials for the database we’re about to create. We need a DBSubnetGroup that contains the VPC SubNet IDs into which the RDS instance will be deployed, and we also need SecurityGroup IDs for the database instance.

With these things in hand, we can then define the actual database instance we want to provision. The complete set of available fields is shown in the ACK API reference. Once created, the status fields of the resulting DBInstance object will contain supplementary information about the RDS service that was provisioned on AWS.

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBInstance
metadata:
  name: test-rds-oracle
spec:
  dbInstanceIdentifier: test-rds-oracle
  allocatedStorage: 100
  autoMinorVersionUpgrade: true
  backupRetentionPeriod: 7
  dbInstanceClass: db.m5.large
  dbName: my-db-name
  dbSubnetGroupName: ${RDS_SUBNET_GROUP_NAME}
  engine: oracle-ee
  engineVersion: 19.0.0.0.ru-2022-04.rur-2022-04.r1
  masterUsername: my-user-name
  masterUserPassword:
    name: oracle-rds-creds
    key: password
  multiAZ: false
  publiclyAccessible: false
  storageEncrypted: true
  storageType: gp2
  vpcSecurityGroupIDs:
    - ${RDS_SECURITY_GROUP_ID}

The above manifest provisions our test-rds-oracle database instance. Notice that the Kubernetes object DBInstance name and the .spec.dbInstanceIdentifier can be different. The dbInstanceIdentifier field is the AWS database name. The ACK for RDS can also provision database instances with other database engines. Each database engine has its own set of pre-requisite custom resources and DBInstance fields.

Developers can then modify already provisioned database instances by changing the resource yamls via GitOps. For example, a developer can modify the allocatedStorage field from 100 to 200, commit the change to Git, and let the GitOps controller apply the yaml to the cluster. The ACK will then place the appropriate call to the AWS RDS API and affect the change.

The same principles apply to provisioning AWS resources with other ACKs: Application developers will quickly become familiar with constructing the custom resource yamls for the AWS service they want to use. The ACK API Reference is organised by Service Controller and helps identify and work with the resource fields.

Provisioning AWS resources through ACK leaves us with the need to obtain specific fields from the resources we created. For example, the RDS database instance will have an endpoint URL on which it is reachable. This endpoint would need to be made known to Kubernetes applications that need to establish a connection to the database and use it.

Passing variables from one module to another is typical with tools such as AWS CloudFormation or AWS CDK. With ACKs, we can achieve this by writing out the variables into a ConfigMap or Secret for other workloads to consume. Using ConfigMaps or Secrets is very much in line with Kubernetes best practices.

This is done with the FieldExport resource.

The below snippet shows an example of the Amazon S3 bucket we provisioned earlier.

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-bucket-location
data: {}
---
apiVersion: services.k8s.aws/v1alpha1
kind: FieldExport
metadata:
  name: test-bucket-location
spec:  
  to:
    name: test-bucket-location # Matches the ConfigMap
    kind: configmap
  from:
    path: ".status.location"
    resource:
      group: s3.services.k8s.aws
      kind: Bucket
      name: test-bucket # Matches the Bucket

With these yamls, the location field of the example S3 bucket we provisioned is written into a ConfigMap. Notice that the ConfigMap itself needs to be created separately, as the FieldExport resource only updates an already existing ConfigMap.

Analogously, we can write out the database endpoint address into a ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-rds-oracle-url
data: {}
---
apiVersion: services.k8s.aws/v1alpha1
kind: FieldExport
metadata:
  name: test-rds-oracle-url
spec:  
  to:
    name: test-rds-oracle-url # Matches the ConfigMap
    kind: configmap
  from:
    path: ".status.endpoint.address"
    resource:
      group: rds.services.k8s.aws
      kind: DBInstance
      name: test-rds-oracle # Matches the DBInstance

These ConfigMaps can then be conveniently consumed by the applications that need them.

Conclusions

With ACKs, we enable application developers to provision AWS-managed services directly from within their Kubernetes manifests. We can then build, package, and deploy our cloud-native applications, including ACK resource definitions, resulting in a reduced need to provision AWS resources separately and through other tools. This can help us move at least some of the AWS provisioning logic away from less dynamic methods into the cloud-native sphere of Kubernetes.

Additionally, with ACKs being packaged as Helm charts, they can extend our array of cluster add-ons and fit into existing GitOps pipelines that can deploy them into our cluster fleet. Application developers and SREs can also interact with Git to provision the AWS Services they need.