Use Azure Key Vault for Secret Store With Azure CSI Driver

5 min readAug 19, 2022

In this demo, we will see how we can install and use the Azure CSI Driver with AKS cluster and use it to fetch the secrets to cluster.

Prerequisites:

Assuming that you have the following The names of the resources are going to be referred in the braces throughout the demo, replace them accordingly) .

Resource Group ( test )
AKS cluster with managed identity enabled ( incubation-lab )
Key vault with following secrets ( test-azure-csi )
-> Secret1
-> secret2

Installation:

You have to update your cluster with following add-on for CSI driver. Use the below command. The following add-on will create a user-assigned managed identity whose naming convention will be like azurekeyvaultsecretsprovider-*.
This can be used to connect the Azure Key Vault with AKS cluster.

az aks enable-addons --addons azure-keyvault-secrets-provider --name incubation-lab --resource-group test

Once you enable with add-on, it will create following pods in k8s cluster in kube-system namespace

NOTE: Make sure that a Secrets Store CSI Driver pod and a Secrets Store Provider Azure pod are running on each node.

Provide Access to The Managed Identity

Now, we have to provide access to the managed identity which got created earlier when we have enabled the add-on.
We can get the ID of the user-assigned identity with following cmd.

az aks show -g <resource-group> -n <cluster-name> --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId -o tsv

output of the above cmd

Note the ClientID from the output.
Now, we have to give access to this identity to read the contents from key-vault. Use following cmds for that.

az keyvault set-policy -n <keyvault-name> --key-permissions get --spn <clinet-id>az keyvault set-policy -n <keyvault-name> --secret-permissions get --spn <clinet-id>az keyvault set-policy -n <keyvault-name> --certificate-permissions get --spn <clinet-id>

Methods:

There are 2 methods how you can integrate the secrets.

Using Mounts
Using Secrets as env variables

Method 1 : Mounts

In this method, we attach secrets as volume mounts to a pod. The pod can refer the files created in the mount path for secret contents.

follow below steps to use this method

Step 1: Create SecretProviderClass

Now, we have user-assigned managed identity which can access the contents of the key-vault. we have to create the SecretProviderClass object in k8s cluster which will fetch the contents from the key-vault.

Once you created the above YAML file, apply it using the following command.

kubectl apply -f SecretProviderClass.yaml

Once you apply the Manifest file, the SecretProviderClass object will be created. To check, use the following cmd

kubectl get secretproviderclass

secret provider class object got created

Step 2: Create Pod Using SecretProviderClass Object

Now we will create a busy-box pod which will be using the above created SecretProviderClass Object and fetch the secrets.

Create the following yaml

Apply it using the following command

kubectl apply -f secretproviderclass-pod.yaml

You can see that pod is up and running in the namespace in which you have deployed.

To see, if the pod successfully fetched the details, either go to the mounted location (/mnt/secrets-store) inside the pod or use the below commands

kubectl exec busybox-secrets-store-inline-user-msi -- ls /mnt/secrets-store/
 kubectl exec busybox-secrets-store-inline-user-msi -- cat /mnt/secrets-store/secret1
kubectl exec busybox-secrets-store-inline-user-msi -- cat /mnt/secrets-store/secret2

Method 2: Secrets as env variables

In the above method we have seen that we can mount the key-vault secrets as volume mounts. We can use them as environment variables as well. But, one thing we should remember is We should mount the secrets as well. we can not directly use them as Kubernetes secrets.

Step 1: Create SecretProviderClass Object

Use the following Yaml file which will create the Kubernetes secret as well as mount the secrets.

Apply the above Yaml file using below command

kubectl apply -f EnvVariables_SecretProviderClass.yaml

Once the SecretProviderClass object got created, we have to create the pod.

Step 2: Create the Pod

Create the pod using following YAML file. Once you apply this, busy-box pod will be up and running.

busy-box pod is up and running

Now, we will go inside the pod to see the mounted contents and env variables

kubectl exec -it busybox-secrets-store sh

Once you are in the pod, Print the env variable using env cmd and verify the contents of the mounted path as shown in below image

You will be able to see the secret only once the pod is up and running.

These secrets will be alive until the pods which are using these secrets are up and running. Once they all got terminated/deleted, the secrets will automatically deleted.

To demonstrate this, I am going to delete the busy-box pod which is using this secret as env variables .

As you can see in the above image, As soon as I delete the pod, the secret also got deleted

Conclusion

For more information, follow these links

The default refresh interval to fetch and update the secret contents from the key-vault is set to 2 min. we can modify it. and we can enable auto rotation as well. For that, follow this link https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/main/docs/book/src/topics/secret-auto-rotation.md
Whenever the secrets got updated, the SecretProviderClass Object will get update within the default time interval. But we have to take care updating Secret in Pods/deployment. we have to restart the pods whenever that happens. For that we can configure reloader where it will restart the pods based on our requirement. Follow this link https://github.com/stakater/Reloader
Create SecretProviderClass Objects with Other managed identities / Pod Identity : https://docs.microsoft.com/en-us/azure/aks/csi-secrets-store-identity-access
Install Azure CSI Driver using helm : https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/charts/csi-secrets-store-provider-azure/README.md