Open in app

Sign in

Write

Sign in

The Comprehensive Guide to Developing High-Quality Helm Charts Faster: Tips and Tools

Piotr Kleban
18 min readJul 21, 2023

Helm is a popular tool for managing Kubernetes applications. It allows us to package our application as a Helm chart, which is a collection of files that describe the Kubernetes resources needed to run the application.

When developing a chart, we want to make sure that it is well-designed, error-free, and reliable. To achieve this, we can follow some tips and tools that can help us speed up the process and create a high-quality chart.

This article aims to help with helm chart development, by offering a comprehensive guide that covers the following topics:

  • Creating a repository in GitHub to store and share your Helm chart files. We will show you how to use the helm repo add command to create and access your repository from your local machine.
  • Briefly going through the creation of chart options, such as helm create --starter. We will explain the structure and purpose of each file in a Helm chart
  • To get faster feedback in development, we will discuss different tools to quickly find issues, such as kubeconform for syntax checking, tools like kube-linter for analysis and validation by using Helm JSON schema
  • Installing your chart in a local cluster and going through the most common troubleshooting ways, such as helm release status, helm test and kubectl

By following this guide, I hope you will be able to faster develop high-quality and reliable Helm charts for your Kubernetes applications.

Table of Contents:

· An Overview of Helm
∘ Command helm create
∘ History of Helm (v2 vs. v3)
∘ How does Helm v3 client interact with clusters without Tiller?
∘ How to inspect the helm release manually?
· Creating helm repository in GitHub
∘ Command helm repo add
∘ GitHub repository
∘ Creating GitHub repository
· How tools can help to develop Helm charts more easily and effectively
∘ The tools
∘ Command helm lint
∘ Helm JSON schema
· Executing helm chart in a cluster and troubleshooting
∘ helm install manifest validation issue
∘ Command helm release status
∘ Other helm commands:
∘ Best practices:
· Conclusions

An Overview of Helm

Let’s start by reviewing the structure and components of a helm chart. A helm chart consists of several files that define the configuration of a Kubernetes application. A helm chart structure looks like this:

mychart/
  Chart.yaml # A file with YAML format that has the metadata of the chart
  LICENSE # (optional) A file with plain text format that has the license of the chart
  README.md # (optional): A file with Markdown format that has the introduction and documentation of the chart
  values.yaml # A file with YAML format that has the default values for the chart configuration
  values.schema.json # (optional): A file with JSON format that has a JSON Schema to validate and document the values in the values.yaml file
  charts/ # A directory with any subcharts or dependencies that the chart needs
  crds/ # Custom Resource Definitions
  templates/ # A directory with template files that use values to create valid Kubernetes manifest files
    NOTES.txt # (optional): A file with plain text format that has some brief usage notes

Chart.yaml: This file contains the metadata about the chart, such as its name, version, description, dependencies, etc. This file is required for a valid chart.

A simple Chart.yaml file for a helm chart could look something like this:

apiVersion: v2 # The chart API version
name: mychart # The name of the chart
version: 0.1.0 # The version of the chart
description: A simple example chart # A short description of the chart
appVersion: "1.2.3" # (optional) application version

This example contains the minimum required fields for a valid chart (only appVersion field is an optional). Field appVersion specifies the version of the application that is packaged by the chart. The appVersion field can be used in templates to refer to the application version, such as in the image tag:

image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"

values.yaml: This file contains the default configuration values for the chart. These values can be overridden by the user at the time of installation or upgrade.

charts/: This directory contains any subcharts or dependencies that your chart relies on. These charts can be local directories or remote archives.

crds/: This directory contains any Custom Resource Definitions (CRDs) that your chart needs to install. These CRDs are installed before any other resources in the chart.

templates/: This directory contains the template files that generate the Kubernetes manifest files when combined with the values. These files use the Go template language.

The template directory usually contains files like NOTES.txt and Kubernetes objects such as:

  • NOTES.txt: This file contains some short usage notes that are displayed to the user after installing or upgrading the chart. This file is optional, but recommended if you want to provide some helpful tips or commands for using your chart.
  • deployment.yaml
  • service.yaml
  • ingress.yaml
  • configmap.yaml
  • pdb.yaml
  • hpa.yaml
  • serviceaccount.yaml
  • role.yaml
  • rolebinding.yaml
  • (…)

For more details about structure, see here.

Command helm create

By default, helm create creates a directory and uses the built-in starter template files:

.
├── charts
├── Chart.yaml
├── templates
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── NOTES.txt
│   ├── serviceaccount.yaml
│   ├── service.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml

However, we can create our own starter. We can then use the --starter flag to point to the location of the custom starter scaffold. Starter helps us create a new chart faster that has a specific structure or convention that differs from the default one.

A good example is creating a chart for an Istio microservice. Instead of writing Istio manifests manually every time, you can use the helm-starter-istio starter, which generates Istio resources for your service, based on the template. It not only creates the common Service and Deployment manifest, but also the necessary Istio resources for a service, such as VirtualService or DestinationRule. It saves time and effort enabling you to deploy your service easily and quickly.

To use a custom starter chart, we need to copy it to the $HELM_DATA_HOME/helm/starter directory. If this directory does not exist, we can create it first. The $HELM_DATA_HOME variable stores the location of the Helm data directory, which is usually /home/<user>/.local/share/. To check the exact value of this variable, we can use the helm env command.

One example of a custom starter chart is service-mesh in helm-starter-istio, which provides templates for creating 4 different starters. We can download it using the git clone command:

git clone https://github.com/salesforce/helm-starter-istio.git /home/<user>/.local/share/helm/starters/starter-istio

To create a new Istio mesh-service chart (which is one of the four starter charts for Istio) we can use the helm create command as follows:

helm create mychart --starter starter-istio/mesh-service

And from this point we can start customizing it to our needs.

Now let’s take a look at the history of helm and how it can manage releases without a server (Tiller).

History of Helm (v2 vs. v3)

In Helm v2, Tiller was the server-side component of Helm that ran inside your Kubernetes cluster and interacted with the Helm client. Tiller was used to save the release information. In Helm v3, Tiller is no longer used.

How does Helm v3 client interact with clusters without Tiller?

The current version of Helm communicates directly to your Kubernetes cluster via API server. Helm 3 saves each release as a secret within the cluster (by default, for more information, see here).

How to inspect the helm release manually?

If you we to get the release information manually, you can use the following commands:

  • helm install mychart ./mychart to install an example Helm chart
  • and then we can use kubectl get secrets

This will list all the secrets (including the one that contains the release information for mychart). We can use kubectl describe secret or kubectl get secret -o yaml to inspect more details about the release.

Example secret:

apiVersion: v1
data:
  release: <base64>
kind: Secret
metadata:
  creationTimestamp: "2023-07-18T10:46:10Z"
  labels:
    modifiedAt: "1689677170"
    name: mychart
    owner: helm
    status: deployed
    version: "1"
  name: sh.helm.release.v1.mychart.v1
  namespace: default
  resourceVersion: "346676"
  uid: 07acf3f9-8cd5-418e-9a5a-8207ff6f91d0
type: helm.sh/release.v1

The secret has data.release, which holds the release data as a base64-encoded string. However, the decoded string is not directly readable, because Helm 3 applies another layer of encoding and compression to the release data.

To extract template/ of the release, you need to perform a couple of commands:

  • Decode the base64 string from the secret. This is the default encoding for Kubernetes secrets.
  • Decode the base64 string again. This is the additional encoding that Helm 3 applies to the release data.
  • Decompress the gzip data. This is the compression that Helm 3 applies to reduce the size of the release data.
  • Extract base64 strings of .chart.templates
  • Decode the base64 template strings

The final command to get the template/ release data looks like this:

kubectl get secrets/sh.helm.release.v1.mychart.v1 -o json  \
    | jq -r .data.release  \
    | base64 -d  \
    | base64 -d  \
    | gunzip -c   \
    | jq -r '.chart.templates[].data' \
    | base64 -d

Creating helm repository in GitHub

After we have created a sample chart with the helm create command, then we can commit it to our own repository and use helm repo add commands.

Command helm repo add

helm repo add <name> <URL>

# for example:
helm repo add bitnami https://charts.bitnami.com/bitnami

The helm repo add command adds a chart repository to your local Helm client. It does not install anything in the cluster. The chart repository is a web server that hosts one or more charts and an index file that lists them. The helm repo add command downloads the index file and stores it in your local machine, under the repository/cache/ directory (if we want to see the location of Helm directories, you can run the helm env command). This is where the helm search command finds information about charts.

GitHub repository

A chart repository is a way to share your charts with others. It is just an HTTP server that has an index.yaml file and your charts (packaged as .tgz files). You can use any HTTP server that can serve these files. Some examples of chart repository hosts might be Amazon S3, GitHub Pages, or your own web server.

Here is an example structure of a chart repository:

my-chart-repo/
  index.yaml
  charts/
    my-chart-0.1.0.tgz
    my-chart-0.2.0.tgz
    my-chart-0.3.0.tgz

Creating GitHub repository

The process of creating a repository is well described here. It shows how to use GitHub Pages as a chart repository for Helm. GitHub Pages is a feature of GitHub that allows you to host static web pages.

To summarize, steps are:

  • Create a GitHub repository for your charts
  • Create an index.yaml file that lists charts and their metadata (automated via Chart Releaser Action — GitHub Action)
  • Package your charts as .tgz files (automated via Chart Releaser Action — GitHub Action)
  • Enable GitHub Pages for the repository and choose the branch where your charts are stored
  • Add your GitHub URL as a Helm repository using helm repo add

It is convenient to use GitHub Actions — Chart Releaser Action to automate the whole the process of updating your index.yaml file and packaging your charts whenever you push changes to your repository.

To add our chart repository to Helm, we need to get the URL of, for example index.yaml in gh-pages branch of our helm-charts repository from here:

The link will be https://raw.githubusercontent.com/<org>/helm-charts/gh-pages. Use it as URL for command helm repo add , but removing index.yaml from URL

helm repo add <my_repo> \
  "https://raw.githubusercontent.com/<org>/helm-charts/gh-pages"

To confirm that our charts are available, we can run helm search and see the list of charts.

To fetch latest index.html use helm repo update <my_repo>

Alternatively, we can use other commands to inspect our chart:

helm get all mychart : this will show all the information about the release mychart, including the kubernetes manifest, notes, values, and hooks.

helm get values mychart --all: this will show all the values that were used to render the templates for the release mychart, including the ones from the chart’s default values.yaml file and the ones that were supplied by the user.

helm show all ./charts/mychart: output will include the Chart.yaml file, the default values.yaml file, and the README.md file for the chart.

How tools can help to develop Helm charts more easily and effectively

There are some tools that can help us speed up the process, create a high-quality and reliable chart. Tools can help us with different aspects of the chart development process, such as:

  • Syntax validators for Kubernetes manifest: This ensures that our generated chart template is valid and conforms to the Kubernetes OpenAPI specifications.
  • Analysis tools: These tools check our chart template or our installed chart in a running cluster for potential issues or improvements, such as resource optimization, security risks, or configuration best practices. These tools can be roughly divided into groups: analysis tools for Kubernetes objects, analysis tools of running cluster, analysis tools for Helm files
  • Testing tools: like helm test that runs tests on our deployed chart by executing test pods that are defined in our chart
  • Helm JSON schema: This helps us and our users to customize values.yaml file by providing a JSON schema values.schema.json that defines the structure

The tools

Some syntax validators for Kubernetes manifest are:

A tool that performs code analysis of your Kubernetes object definitions

  • kube-score
  • kubesec

A tool that audits running Kubernetes clusters, but also can perform static code analysis on Kubernetes YAML files

  • polaris
  • kubeaudit

Here are some examples of how to use some of these tools:

# Render the Helm chart pipe the output to kube-score
helm template -f values.yaml my-chart | kube-score score -
# Render the Helm chart pipe the output to kubesec
helm template -f values.yaml ./chart | kubesec scan /dev/stdin
# Run an audit on the kubernetes manifest
polaris audit -f pretty -audit-path deployment.yaml
# Run an audit on the Kubernetes manifest file
kubeaudit -f deployment.yaml

# COMMAND for running cluster:
#  Run an audit on the current Kubernetes cluster using polaris
polaris audit -f pretty
#  Run an audit on all resources in running Kubernetes cluster 
kubeaudit all

Another group of tools can analyze Helm charts directly. Example tools:

  • kube-linter
  • terrascan
  • checkov
  • trivy
  • datree (installed as a helm plugin)
  • kubescape

However, it is important to know that these tools won’t perform any checks of not rendered objects. For example, when the default values.yaml file has deployment.enabled set to false:

{{- if .Values.deployment.enabled -}}
apiVersion: apps/v1
kind: Deployment
metadata:
(...)
{{- end }}

values.yaml:

deployment:
  enabled: false

As this means that the Deployment object will not be rendered by Helm, therefore will not be analyzed by the tools.

Therefore, it is advisable to have multiple test values.yaml files with different combinations of values in order to fully validate our chart and have more confidence in its quality. This way, we can test how our chart behaves with different configurations and ensure that it meets our expectations and requirements.

Some mentioned tools that can perform more than one thing. For example, some tools can run audits on the running cluster, generate Helm charts, and then analyze the Kubernetes objects. Other tools can perform static analysis on various infrastructure as code languages, such as Terraform. Terrascan and checkov are examples of such tools that can work with both Kubernetes and Terraform.

With arkade, we can easily install some of the mentioned Kubernetes tools, and we can also use the arkade GitHub action in our pipelines.

Alternatively, some of the tools can also be installed as kubectl plugins using krew, which is a tool that manages the installation and updates of kubectl plugins.

As we can see, there are different tools and options that can support us in the development and testing of our chart. Now, let’s explore the built-in features of the helm tool.

Command helm lint

Helm lint is a command that checks a chart for issues, validates its dependencies and values, and reports any warnings or errors.

We can use flags, such as --strict to fail on lint warnings, or --values to specify values in a YAML file:

helm lint . --strict --values <values.yaml>

For example, suppose we have a chart that contains a deployment.yaml file with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ incl "mychart.fullname" . }}
  labels:
(...)

Notice that there is a typo in the name field: it should be {{ include "mychart.fullname" . }}, not {{ incl "mychart.fullname" . }}. This will cause a parse error when helm tries to render the template. To catch this error, we can run:

To catch this error, we can run helm lint . :

==> Linting charts/mychart
[INFO] Chart.yaml: icon is recommended
[ERROR] templates/: parse error at (mychart/templates/deployment.yaml:4): function "incl" not defined

Error: 1 chart(s) linted, 1 chart(s) failed

Helm lint also performs schema checking, which means it verifies that the values provided for the chart match the expected types and formats defined in the schema. For example, suppose we have a values.schema.json file in our chart with the following content:

{
    "$schema": "http://json-schema.org/schema#",
    "type": "object",
    "properties": {
        "replicaCount": {
            "type": "integer"
        },
(...)

This schema specifies that the replicaCount value should be an integer. However, we have a values.yaml file that contains the following:


replicaCount: "2" # <- not an integer
(...)

This will cause a schema validation error, because the value is a string, not an integer. To catch this error, we can run:

[ERROR] templates/: values don't meet the specifications of the schema(s) in the following chart(s):
mychart:
- replicaCount: Invalid type. Expected: integer, given: string

As we can see, helm lint detected the error and showed which value and chart failed the validation. Other commands that do schema validation helm install, helm update, and helm template.

Helm JSON schema

Helm schema is a feature that allows chart authors to define the structure and constraints of the values that their charts accept. By using a schema, we ensure that the values provided by the users are valid and consistent, and avoid errors or unexpected behaviors during installation or upgrade.

A helm schema is a JSON file that follows the JSON Schema specification. It describes the properties, types, formats, and validations of the values that a chart expects. A helm schema file must be named values.schema.json

One of the ways to generate a schema file from your existing values.yaml file, is a Helm Schema Gen plugin. It can scan your values.yaml file and create a basic values.schema.json file that you can customize further. To use this tool, you need to install it as a helm plugin:

helm plugin install https://github.com/karuppiah7890/helm-schema-gen

Then, you can run it inside your chart directory:

helm schema-gen values.yaml > values.schema.json

This will create a values.schema.json file in the same directory. You can edit this file to add more constraints or validations to your values.

Generating values.schema.json is a good way to initiate further fine-tuning of your schema.

For existing values.schema.json— for example hashicorp/vault repository has a values.schema.json file that you can see for reference: https://github.com/hashicorp/vault-helm/blob/main/values.schema.json

For more information, see here.

Executing helm chart in a cluster and troubleshooting

Once we are ready to run our chart in a cluster, we can deploy it via helm intall <name> <local_dir>.

Let’s look into some basic methods of troubleshooting that can help us identify and resolve the issues. if we encounter any problems or errors during or after the deployment.

To verify our Helm chart locally in Kubernetes cluster, we can, for example, install K3s and then:

  • setup kubeconfig by cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
  • apply helm chart via helm install my_chart ./charts/my_chart/

If your chart is exposed by ingress, you can access it through the Traefik service, which is default ingress controller in K3s is Traefik, (its config is located at /var/lib/rancher/k3s/server/manifests/traefik.yaml)

To obtain the IP address we can run the following command:

kubectl get service -n kube-system traefik -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Then, we can manually check the connection to our chart sending a request to the obtained IP address:

curl http://<ip>/hello-world

Alternatively, we can also use helm test to verify that your chart works as expected. For more information about helm test, see here.

This sandbox kubernetes environment on your local machine is great for testing and verifying your chart in development. Very easily we can also use tools like polaris or kubeaudit that connect to the cluster and perform various checks on your objects, such as security, performance, or best practices.

To verify chart in CI we can use Github Action chart-testing-action. It is action based on ct tool (chart-testing). Example provided on this repository starts KinD cluster (Kubernetes IN Docker) and performs helm linting, testing.

helm install manifest validation issue

There are some things to pay attention to while developing Helm charts.

If you use helm install with the --dry-run flag, it will not warn you about any typos or errors in your chart. For instance, if you have a typo like this:

(...)
spec:
  containers:
    - name: <name>
      image: <image>
      ports:
        - name: http
          resource: {} # <- typo!

Our values.yaml:

(...)
resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi
(...)

The helm install mychart <mychart> --dry-run flag will not catch the typo and will show you the output.

It seems that helm install mychart <mychart> does something like this (--validate=false):

helm template mychart <mychart> | kubectl apply --validate=false -f -

If you use helm install without the --dry-run flag, it will install the chart to the cluster, but it will only show you a warning if there are any unknown fields. For example, if you have the same typo as above, you will see a warning like this:

> helm install mychart <mychart>
W0717 19:26:24.190440   57152 warnings.go:70] unknown field "spec.template.spec.containers[0].resource"
NAME: <name>
LAST DEPLOYED: Mon Jul 17 19:26:23 2023
NAMESPACE: default
STATUS: deployed
(...)

However, this means that your chart was not installed as you expected, and it may have unintended consequences. For instance, in this case, the resources for the container are not set, which could lead to resource starvation or over allocation.

kubectl get pods/<name> -o yaml
(...)
spec:
  containers:
  - image: <image>
    imagePullPolicy: IfNotPresent
    name: <name>
    ports:
    - containerPort: 80
      name: http
      protocol: TCP
    resources: {} # <- not set!
(...)

So, helm install with --dry-run does not catch those errors in the chart. helm install without --dry-run installs the chart but shows a warning in the logs. Command kubectl apply works as expected:

The request is invalid: patch: Invalid value: (...)
unknown field "spec.template.spec.containers[0].resource"

That’s why analysis tools are very helpful.

Command helm release status

After installing our helm chart, we can use various commands to inspect our chart and its resources. For example, we can run helm release status <name> --show-resources --show-desc:

NAME: mychart
LAST DEPLOYED: Mon Jan 24 15:23:45 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
RESOURCES:
==> v1/Deployment
NAME    READY  UP-TO-DATE  AVAILABLE  AGE
myapp   1/1    1           1          2m

==> v1/Pod(related)
NAME                     READY  STATUS   RESTARTS  AGE
myapp-7c8f9f9c6b-4xq5w   1/1    Running  0         2m

==> v1/Service
NAME    TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
myapp   ClusterIP  10.96.184.123  <none>       80/TCP    2m

==> v1/ConfigMap
NAME    DATA  AGE
myapp   1     2m

NOTES:

It may show for example that some pods are not in READY state. The deployment STATUS can be one of the unwanted statues like: failed, or unknown.

One thing which is worth knowing is --wait flag for helm install. It makes Helm wait until all resources are in a ready state before marking the release as successful.

helm install <mychart> <mychart> --wait

Similar to helm release status to see the resources of our chart, we can use kubectl with a label selector. For example, we can run:

kubectl get all -l app.kubernetes.io/instance=myapp

The result will be similar to the helm status command.

Another method of troubleshooting is by using kubectl get events. For example:

> kubectl get events
LAST SEEN   TYPE      REASON                OBJECT                                              MESSAGE
40m         Warning   Unhealthy             pod/<name>       Readiness probe failed: Get "http://10.42.0.237:80/": dial tcp 10.42.0.237:80: connect: connection refused
35m         Warning   BackOff               pod/<name>       Back-off restarting failed container <name> in pod <pod-name>

This will show us the events that occurred in our cluster.

And a good place to start debugging our helm chart is to use kubectl describe to get more details about a specific Pods. For example, we can run:

kubectl describe pod -l app.kubernetes.io/instance=myapp

For example, we might see something like this:

(...)
Liveness probe failed: HTTP probe failed with statuscode: 404
(...)

This indicates that the liveness probe of the pod is failing.

And, of course, we can also use kubectl logs to see the logs of the containers in our pod. For example, we can run:

kubectl logs --prefix=true -l app.kubernetes.io/instance=myapp

This will show us the logs of all the containers in the pod that matches the specified label . We can use the --prefix flag to add a prefix to each log line with the pod name and container name. This can help us identify which container is producing which log message.

By using these commands, we can examine our installed chart and troubleshoot any issues that arise.

Other helm commands:

Useful commands (but not covered in this article so far) for maintenance:

  • helm upgrade --atomic: Upgrades a release and performs a rollback if the upgrade fails.
  • helm upgrade --install: Installs a release (even if it does not exist yet).
  • helm history: Shows the revision history of a release.
  • helm uninstall (helm list --short): Removes all releases

Best practices:

Conclusions

Using various tools can help us to speed up the process, create a high-quality and reliable chart, and avoid common pitfalls and errors. We can use analysis tools to check different aspects of our chart, such as validity, performance, security, configuration, and customization. By using these tools, we can deliver better results to our users. It is best to use tools like kubeconform frequently to validate the syntax, and tools like kube-linter to analyze our chart for potential issues and improvements in the process.

It is also best to publish our helm chart with a Helm JSON schema that defines the structure and constraints of our values.yaml file. This will help our users to create well-formed applications using our chart.

If you find what I do valuable, please 👏 clap, follow and share with your friends. ❤️ The more people I can reach, the more I can help, completely free. Your support not only motivates me to continue my work, but it also benefits others who might find my work helpful. ❤️

Your support fuels my motivation to create more high-quality content. 🚀

Thanks! 🙏🏼

Kubernetes
Helm
DevOps
Github
Helm Chart

--

--

Piotr Kleban

Written by Piotr Kleban

68 Followers

Wizard of automation. Makes sure that code does not explode when it goes live. Obsessed with agile, cloud-native, and modern approaches.

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams