Moving EKS add-ons from self-managed to managed

EKS add-ons are like plugins for your Kubernetes cluster. They provide operational capabilities support to Kubernetes applications. There are quite a number of add-ons provided by AWS and these can be either self-managed or managed. Examples of these are kube-proxy, coredns and vpc-cni.

https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html

For self-managed add-ons, you will be handling their maintenance and upgrade. You will have to update the manifest and deploy it to your cluster, whereas for managed add-ons, you just need to specify the version that you need and AWS will handle the rest for you behind the scene.

If you are using managed add-ons and want to transit to self-managed, you can also easily do so by deleting the add-on with the Preserve on the cluster flag (or --preserve if you are using CLI). AWS will then stop managing the add-on and the add-on will be retained in your cluster. However, if you want to transit from self-managed to managed add-ons, you will probably wonder how to do it as there is no documentation on this (at the point of writing).

This blog post will show you examples of how it can be done easily.

Example 1: kube-proxy

By default, kube-proxy is already installed in your EKS cluster in the kube-system namespace. In our test cluster, we are using the v1.24.7-eksbuild.1 version and we do not have any managed add-ons.

We can simply do the transition by adding the add-on to the cluster. Since this is the first time we are adding it, we MUST set the Conflict resolution method option to Overwrite. Otherwise, the transition will fail. Subsequently, for the upgrade, you can set to Preserve if you are using custom values. After awhile, you can see that the kube-proxy has been updated and the add-on is also being added to the cluster. Be sure to do some testings like scaling the nodes up and down, observing the behaviour and checking for any issues.

Example 2: aws-ebs-csi-driver

In the above example, the self-managed add-on is created in the same namespace as the managed add-on. What if the self-managed add-on is created in a different namespace? Using aws-ebs-csi-driver add-on as an example, we will show you how to do it.

In the same test cluster, we have installed the add-on in the ebs-csi-driver namespace.

To transit it to a managed add-on, firstly we need to remove the existing one to avoid having two drivers running simultaneously, which may potentially cause conflicts when managing the EBS lifecycle. Then, add the add-on to the cluster.

It can take as fast as a minute to complete. After that, you can see the new aws-ebs-csi-driver resources and that the add-on has also been added.

To ensure everything is working as expected, it is recommended to do some testings like creating and deleting a persistent volume claim (PVC) before and after the transition. You can follow the example provided by AWS below.

https://docs.aws.amazon.com/eks/latest/userguide/ebs-sample-app.html

Common Questions

1. How do I pass in my custom values to the managed add-ons? E.g. setting values for certain environment variables, replica count and resources (CPU and memory).
   You can make use of the configuration values option when adding the add-ons and must adhere to the JSON schema for that particular version as it will be validated.
2. What should I do if the JSON schema does not support the property I need?
   Check if the higher version of the add-on supports the property that you need and upgrade to it. If it still does not support, you will have to patch the resources yourself either manually or via a deployment pipeline (recommended).
3. When do I need to set the --preserve flag for conflict resolution?
   This flag should be used when you are upgrading the add-on and want to preserve the patched (custom) values on your Kubernetes resources.