The Power of Kustomize and Helm

If you are new to Kubernetes, one of the first things you might do after securing your cluster, setting up your local kubectl config and running “kubectl get nodes”, is google something like “How do I deploy an application into Kubernetes?”. Which might give you a tutorial result that tells you to run a command like this

kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1

This works for your first introduction, but it won’t be long before you start asking yourself the question, “How do i manage deployments into a dev and prod kubernetes environment?”. There are a lot of things that could and should be different across environments, such as the database you’re using or the dns address that maps to your ingress

Enter Kustomize and Helm

If you were to then google the question above, you might find that the best way to structure your manifests for deploying into kubernetes is to use a template engine. As of writing this the two most popular templating engines for kubernetes are Kustomize (though it’s not a full templating engine) and Helm. You may have seen a lot of articles asking the question which one is better or which one to use. In this article, I’m going to tell you that you may not need to make a choice because they can both work very well together.

Understanding Kustomize

Kustomize is a native Kubernetes configuration management tool that allows you to customize and manage Kubernetes manifests declaratively. It provides an elegant and scalable way to modify base Kubernetes manifests using patches and overlays. With Kustomize, you can define resources and transformations using simple YAML files, without the need for complex templating engines.

Kustomize operates with the ideology that all changes and applications should be stored in files in order to reduce unintended side effects, a.k. declarative. Everything should be stored in a file and tracked in source control, you should not be plugging any values into your manifests dynamically.

In my experience, I have found kustomize to be very easy to get going because you can just write manifests with a few minimal overlays to patch any environment specific values, such as database url or ingress. However, I have struggled due to some environmental constraints that sometimes require more dynamic values during deployment rather than kustomizes expectation of having them committed into source control.

You can find a whole list of official kustomize examples here in the kustomize repo

Understanding Helm

Helm, on the other hand, is a package manager for Kubernetes that simplifies the deployment and management of applications. At its core, Helm uses charts, which are collections of pre-configured Kubernetes resources, templates, and default values. These templates can be customized using Helm’s template engine, which allows for dynamic parameterization of resources.

Helm is a true template engine complete with defaults, conditionals and variable substitution. helm gives you the power to completely change the resulting manifests based on configuration values set in a values.yaml file.

Helm operates in a way where it attempts to abstract the complexity of kubernetes manifests and expose just a few values that can be used to determine the manifests that ultimately get installed into the cluster. You can find example charts here.

Combining Kustomize and Helm

Kustomize and helm can be combined together using a very useful feature built into kustomize called “helmChartInflationGenerator”. What this generator will do is allow you to use helm charts within your kustomize manifests, and when running the kustomize command it will expand the helm chart to include all of the files generated by helm.

For a simple example of using the helmChartInflationGenerator, paste the below into a kustomization.yaml file in your current directory

helmChartInflationGenerator:
  - chartRepoUrl: https://grafana.github.io/helm-charts
    chartName: grafana
    releaseName: grafana
    releaseNamespace: monitoring

The above kustomization.yaml file will now be set to install that grafana helm chart into the monitoring namespace. Simply run the following command to apply this to your cluster

kustomize --enable-helm . | kubectl apply -f -

NOTE: the version of kustomize that is in kubectl does not allow the enable-helm flag, which is why you first need to generate the manifests with kustomize and then apply the manifests with kubectl.

This was a very simple example, and probably doesn’t warrant using the two tools together, you could probably just use helm directly in this case. So let me explain a few use cases where this can be very helpful

Use Case #1 — adding files not defined in helm chart

Sometimes a helm chart that you want to use does not provide everything you need for the software to work. I will use my installation of cert-manager as an example. The helm chart for cert-manager provides almost everything you need to use it. However after installing you still need to create an issuer, without going into the details of what this does, it is basically a custom resource that needs to be applied into the cluster, defined like this:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: fake@email.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-dev
    solvers:
      - http01:
          ingress:
            class: nginx

The helm chart does not really give a good way to apply this, so if you were to use helm directly to install cert manager you would need to

1. install the cert-manager helm chart
2. apply the ClusterIssuer resource defined above

However, I am lazy, and I want to do this in one step, so what I did is create a directory with the following structure

~/cert-manager
├── clusterIssuer.yaml
├── helmChart.yaml
├── kustomization.yaml
└── values.yaml

clusterIssuer.yaml is the same as what was defined above, but if you look at the other files

helmChart.yaml

apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
  name: cert-manager
name: cert-manager
namespace: cert-manager
releaseName: cert-manager
repo: https://charts.jetstack.io
valuesFile: values.yaml

This file basically just tells kustomize which helm chart to use

kustomization.yaml

resources:
- clusterIssuer.yaml

generators:
- helmChart.yaml

namespace: cert-manager

This is the entrypoint for kustomize, you can see it is apply the clusterIssuer.yaml above as a resource, and the helmChart.yaml as a generator (helmChartInflationGenerator)

values.yaml

installCRDs: true
serviceAccount:
  create: false
  name: cert-manager

# https://github.com/cert-manager/cert-manager/issues/3237
webhook:
  hostNetwork: true
  securePort: 10260

global:
  leaderElection:
    namespace: cert-manager

This is the configuration used for the cert-manager helm chart, it’s being referred to using the “valuesFile” key in the helm chart generator above.

When you apply this kustomize directory using the command:

kustomize --enable-helm . | kubectl apply -f -

It will not only apply the cert-manager helm chart, but also the ClusterIssuer resource, which means after the installation I am completely ready to start using cert-manager.

Use Case #2 — Editing hardcoded values in helm charts

This is where I think the most value is, sometimes you want to use a third party helm chart to install software, but there might be a compatibility issue with your environment. An example that I have run into is a version issue in the resource, where I need to patch the apiVersion of a specific resource. The helm chart did not provide a config in values.yaml for me to do this, so I was able to use kustomize to achieve this.

For a concrete example, we can look at my installation of cilium into my k3s cluster which requires some changes to a configmap that the helm chart applies

file structure

~/cilium
├── helmChart.yaml
├── kustomization.yaml
└── values.yaml

This might look very similar to use case 1, and it is, but if you take a closer look at the kustomization.yaml you will see another powerful feature of kustomize.

kustomization.yaml

generators:
- helm-chart.yaml

# Need to remove some keys from the cilium configmap
patches:
  - target:
      kind: 'ConfigMap'
      name: 'cilium-config'
      version: v1
    patch: |-
      - op: remove
        path: "/data/ipam"
      - op: remove
        path: "/data/cluster-pool-ipv4-cidr"
      - op: remove
        path: "/data/cluster-pool-ipv4-mask-size"


namespace: kube-system

Everything in this example is almost the same as example 1 except for the patches section of the kustomization.yaml. You’ll see it’s attempting to patch a ConfigMap resource named cilium-config, but if you notice from the file structure above that resource is not defined anywhere in this file system. That ConfigMap is actually coming from the helm chart, and those values are hardcoded, the helm chart does not provide any way to remove or change these values. However, kustomize is able to break down the resulting manifests and run a patch on them to remove the keys defined under patches.

Summary

If you’re trying to figure out how you want to deploy things into kubernetes, you may find a lot of articles comparing kustomize and helm. Before making a decision consider whether you would benefit from using them together. By adopting these two tools together and following best practices, your Kubernetes deployment workflow will become streamlined, scalable, and more manageable, enabling you to focus on delivering high-quality applications to your end-users.

Note: You may have noticed that in the examples above that I moved the helmChartInflationGenerator definition into its own file, this is for a reason. I ran into issues related to this github issue where the releaseNamespace was not being respected when this was defined in the kustomization file, so my charts were being installed in the default namespace. putting it in a separate file allowed me to install the chart in the desired namespace.