Securing AWS EKS — Configure the VPC CNI plugin to use IRSA

4 min readJul 13, 2023

During the last year I have been working more and more with AWS EKS. Kubernetes have been totally new for me so a lot to learn. Kubernetes is very complex but also very competent. Even though EKS is a managed service by AWS, out of the box it still requires some work to keep secured. This article will guide you through the process of configuring the VPC CNI plugin to use IRSA (IAM Role for Service Account).

What is VPC CNI

The VPC CNI (Container networking interface) is responsible for allocating IP addresses to the Kubernetes nodes and provides networking to pods. The plugin manage network interfaces (ENIs) on the nodes and uses it to assign IP addresses to pods.

Why you should use IRSA for VPC CNI

By default, the VPC CNI plugin inherits the EKS node IAM role. This mean that the pods of the VPC CNI plugin gets the permission assigned to the node IAM role.

If the node role have the AmazonEKS_CNI_Plugin attached, it effectively allows all pods running on the node to attach and detach ENIs, assign and un-assign IP address. Since this is a security risk, it should be remediated by enabling IRSA för the VPC CNI plugin.

Configure IRSA for the VPC CNI plugin

First you will need to get your EKS clusters OIDC provider URL.

aws eks describe-cluster — name my-cluster — query “cluster.identity.oidc.issuer” — output text

Create a json file named vpc-cni-trust-policy.json with the following policy. Make sure to updated it with your OIDC URL and account ID:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com",
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:aws-node"
                }
            }
        }
    ]
}

Create a new role:

aws iam create-role \
  --role-name AmazonEKSVPCCNIRole \
  --assume-role-policy-document file://"vpc-cni-trust-policy.json"

Attach the AmazonEKS_CNI_Policy to the newly created role:

aws iam attach-role-policy \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy \
  --role-name AmazonEKSVPCCNIRole

Attach the role to the VPC CNI plugin:

kubectl annotate serviceaccount \
    -n kube-system aws-node \
    eks.amazonaws.com/role-arn=arn:aws:iam::111122223333:role/AmazonEKSVPCCNIRole

As a last step, when you have successfully configured the VPC CNI plugin to use the new role, you will have to remove the AmazonEKS_CNI_Policy from the node role:

aws iam detach-role-policy — role-name AmazonEKSNodeRole — policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy

If you use terraform, here is code you can use. Create the role and policy:

module "vpc_cni_irsa" {
  source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"

  role_name             = "AmazonEKSVPCCNIRole"
  attach_vpc_cni_policy = true
  vpc_cni_enable_ipv4   = true

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:aws-node"]
    }
  }

  tags = var.tags
}

If you are using the EKS module, make sure you specify to use to service_account_role_arn under the vpc-cni addon and not to attach the cni_policy to the node role:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"

  cluster_addons = {
  vpc-cni = {
      most_recent              = true
      service_account_role_arn = module.vpc_cni_irsa.iam_role_arn
    }
  }
  eks_managed_node_groups = {
    iam_role_attach_cni_policy = false
  }
}

Redeploy aws-node pods

To use the new service account you must redeploy the aws-node pods. If you don’t, no pods will be able to start, will you see various rpc error/connection refused in the logs:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “31a724aca33611d4c93310146ec2c074b6567dc5ab2051c2857a879ce44cc678”: plugin type=”aws-cni” name=”aws-cni” failed (add): add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused”

All aws-node pods will be unhealthy and report:

Warning Unhealthy 9 minutes ago kubelet (combined from similar events): Readiness probe failed: {“level”:”info”,”ts”:”2023–06–20T09:05:18.596Z”,”caller”:”/root/sdk/go1.19.6/src/runtime/proc.go:250",”msg”:”timeout: failed to connect service \”:50051\” within 5s”}
Normal Killing 13 minutes ago kubelet Container aws-node failed liveness probe, will be restarted

Just restarting the pods will not be enough, they will have to be redeployed. Run this command to fix the issue and get the pods redeploy:

kubectl delete Pods -n kube-system -l k8s-app=aws-node

Confirm all pods have been restarted:

kubectl get pods -n kube-system -l k8s-app=aws-node

References

Conclusion

This was my first post about EKS and how to secure the VPC CNI plugin. But more to come! If you want to continue securing your EKS environment I highly suggest going through the EKS Best Practice Guides.