Why I’m a fan of `helm template` over `helm install`

Note: This is an opinion post, not a technical tutorial. If people like this, maybe I’ll share more of my helm template -f ... workflows. Also I apologize for the nautically themed jokes. I’m not normally like that, but hey…the things we do for Medium.

Navigating the tempestuous seas of technology is nothing new to us engineers. We’ve traversed the tumultuous waves, weathered countless storms and emerged victorious (albeit a bit jaded and sarcastically amused) more times than we care to count. Today, let’s steer our ship towards the discussion of Helm, the much-touted “captain” of Kubernetes package management, specifically its two commands: helm install and helm template. As someone who has gone toe-to-toe with Helm’s commands in the unforgiving battlefields of production, I think I’ve earned my right to a seasoned, if not a somewhat salt-sprayed, perspective on the matter.

For the uninitiated, Helm is a package manager for Kubernetes that helps developers define, install, and upgrade complex Kubernetes applications. Think of Helm charts as the Kubernetes equivalent of a yum or apt repository. Now, let’s dissect helm install and helm template.

helm install

UPGRADE FAILED: "java-maven-app" has no deployed releases

First off, helm install. On paper, it looks like the Clark Kent of Helm commands. It deploys Helm charts directly onto your Kubernetes cluster with a semblance of simplicity that makes you want to believe it’ll save your day. It offers management of releases and their versions, resource updates, and all those shiny things that are designed to distract you from the cold, hard truth: it’s a siren call leading your ship into the rocks.

Now, why would I say that? Well, one word: control. With helm install, you give away your right to it. The chart is rendered in memory and then changes are directly applied to the cluster with little warning, leaving you with little to no control over what is going into your cluster. (And before you say “What about --dry-run?” My answer is “C’mon man, you know that never works.”) I've seen enough deployments in my career to know that the directness of helm install isn't always a blessing, especially when things go sideways.

Case in point: I recall an incident at our company where a Helm chart, deployed using helm install, brought down an entire production environment. It was a seemingly innocuous upgrade with a minor version bump, but due to a lack of backward compatibility, it was a disaster. Glazing over the dirty details a bit, the new version of the chart introduced changes that weren’t compatible with our Kubernetes cluster, causing a cascade of failing deployments. If we'd had the chance to review the Kubernetes yaml files before deployment, which helm template facilitates, we could've avoided this chaos.

hooks

Let’s not forget the infamous Helm hooks. Helm hooks are meant to be the superpowers of helm install, allowing you to intervene at certain points in a release cycle like post-release upgrade testing or pre-release validation Sounds promising, right? Well, not so much. Here's the issue: hooks can end up being more of a hindrance than a help.

Consider the case of a pre-install hook that sets up some necessary CRDs. With helm install, if the hook fails for any reason (and oh, it will find a reason), the whole release fails. One time, we were trying to set up a fairly complex Helm chart with multiple dependencies (prometheus operator anyone?)

Go ahead, try your silly chart

Part of the process involved using a pre-install hook to create CRDs. Guess what? The hook failed due to <<insert any issue here>>, and the whole installation came tumbling down. After that, we had to manually clean up the partially installed components and then retry the installation. Had we been using helm template, we could have bypassed the install failure altogether. Keep reading to find out how.

That brings me to another pain point: CRDs and Helm’s management (or should I say mismanagement?) of them. The issue lies in the fact that CRDs are cluster-wide resources. Installing or upgrading a chart that contains a CRD might lead to conflicts or even overwrite existing CRDs in your cluster. And what’s worse, it’s not an easy thing to fix. There was a time when a Helm chart upgrade, done via helm install, overwrote a CRD that was shared among several applications. This triggered a domino effect, causing multiple applications to fail. It was an avoidable disaster if we had been using helm template, which would've given us a chance to examine the output before applying it.

Now, the really fun (should I say best?) part? Hooks don’t work with helm template. Not even a bit. helm template only renders the chart’s templates, which does not include hooks. So, if you've put some critical logic in hooks, good luck to (or shame on) you. But for the rest of us, it will be a salvation the likes of which hasn’t been seen since the Bible times.

But…wait…`helm install` definitely has good things

Now, to counter this narrative, you might be arguing that Helm’s three-way strategic merge patches can handle upgrades better. In theory, Helm compares the old manifest, the live state, and the new manifest during an upgrade, which should iron out any issues. But let’s not kid ourselves. This has limitations, especially when dealing with complex Helm charts or substantial changes.

Take an instance when we were upgrading an application in production using helm install. A new ConfigMap was introduced in the new chart version. Ideally, the three-way merge should have handled this gracefully. (Absent in the cluster, absent in the base64 release contents in the secret, present in the rendered chart about to be applied.) But, in reality, it didn’t `UPGRADE FAILED: No resource with the name “” found`. This led to the normal mayhem that happens in post-release production environments.

One could argue that the --atomic flag can help, as it automatically rolls back the deployment if it doesn't succeed within a timeout. But from experience, I can tell you that it's not all it's cracked up to be. During an installation, when a helm install --atomic command was issued, the chart failed to deploy within the timeout (because NetworkPolicies, and there’s nothing more to say.) Helm attempted to roll back the changes, but guess what? The rollback failed too. And there we were, stuck in the no man’s land of a partially installed, non-functioning application.

The real kicker here? None of these issues exist when you use helm template. Why? Because helm template doesn’t apply the resources directly. Instead, it renders the templates into Kubernetes YAML files, which you can then apply using kubectl apply -f.

Enter helm template

Phippy, the Kubernetes Plushy…reimagined as a badass military commander

This is more than just an additional step. This is your insurance against chaos. You can examine the generated files before applying them, checking for any issues or potential conflicts. No more being blindsided by changes that helm install would've applied directly.

Take a simple scenario: you’ve made changes to your chart, and you want to ensure these won’t disrupt your environment. With helm template, you can generate the Kubernetes YAML files, review them, and even compare them against your current environment using tools like diff. You see any red flags? You can fix them before they become a problem. This saved us more times than I care to count.

And then, there’s the three-way merge…oh wait. Nope. Nope it’s all just Kubernetes. With helm template, you're reducing the dependencies on any of the Helm state management in Kubernetes which is/has been a source of pain the process of managing a production-grade Helm environment. Just template your chart out and away you go.

You also get a more granular control over your deployments with helm template. You can break down your Helm charts and deploy parts of it at a time. This can be incredibly beneficial when you're dealing with large Helm charts or when you want to deploy different parts of your application to different namespaces or clusters.

There is of course one large downside of helm template, and that is the fact that if you delete something from the chart, it’s not automatically deleted from the cluster. And that part is annoying, but waaaaaaaaaay less annoying that everything else that came with helm install.

Ok, time to be done

To wrap it up, in this constant tug-of-war between helm template and helm install, the former emerges as the clear winner in my book. Not because it's inherently better or more sophisticated, but because it hands the reins back to you, the engineer, allowing you to maintain control over your environment.

After all, if you’re anything like me, you’ve already learned the hard way that in the realm of production deployments, a little caution and control are worth their weight in gold. In the end, isn’t that what we all want: to sail smoothly without having to put out fires all the time?

So, fellow battle-hardened engineers, next time you’re considering your Helm commands, think about helm template. It might not be the shiniest tool in the box, but it’ll stand by you when the waves get rough. Because the last thing you need in a storm is a mutiny by your own tools.