Charting the Course: Helm Dependencies Updates Made Easy 🗺️
Maximizing Security, Minimizing Version Gaps, and Simplifying Updates with Helm
NOTE: for clarity on the term “Helm Dependencies Updates” and the use cases addressed, kindly refer to the introduction of this article. This will provide you with a comprehensive understanding of the challenges it solves and the context of our discussion.
Before We Dive In: Introduction 🌊
NOTE: we will leverage the artifacthub.io API to identify and update Helm dependencies whenever changes are detected.
Picture this 🤔: you’re utilizing Helm ⚓️, but you’re not adding a Helm repository or installing the Helm chart to your cluster 🖥️. Instead, you find yourself resorting to commands like helm template . | kubectl apply -f — or, if you’re operating with Argo CD in a GitOps context, it looks pretty similar 🔄. Argo CD employs Helm natively to deploy the rendered manifest hinged on the override values you’ve supplied 📄.
Does this situation resonate with you? 🎯 If so, I suspect you’re facing a familiar challenge. How do you handle Helm dependencies updates when a newer Helm chart version comes into play? ❓ Tools like renovate or dependabot aren’t quite cutting it because you’re not installing the Helm chart 🚫. Renovate allows it, but it's not easy to config and it feels like WIP. Wouldn’t it be splendid to have an automated solution that checks, one or two times a day (or as often as you’d like), if a new Helm chart has been released? 🔄⏲️
Imagine this solution automatically creating a branch 🌿 with the new version. And now, think about it auto-generating a pull request 📩 containing the altered values between your deployed Helm chart version and the new one. How neat would that be? 🤩
We’ve made this possible with a nifty Bash script that operates in tandem with the artifacthub.io API 🖥️🐚.
Behind the Scenes: Our Strategy 💡
Before diving into the how, let me offer a glimpse into our approach to give you a clearer understanding of why we manage things the way we do 💡.
We’ve fashioned a kubernetes-service-catalog serving as a centralized default configuration for all our services, ready to be rolled out across all our clusters strewn across multiple clouds, through GitOps with Argo CD 🚀. Here, we employ Helm sub-charts to make this happen. In the past, updating these multi-charts involved scheduling a monthly meeting to update all charts, put them to the test, and then propagate them to production 📅. In the best-case scenario, this meant a one-month gap between updates. We found ourselves spending half, if not the whole day updating all services successfully 🕰️. But the worst part? We were missing out on numerous critical updates! 😱
At this juncture, we opted for an action from hckops. This gave us the leverage to check daily if a newer Helm chart version had hit the stands, create a branch, and generate a pull request against the development branch 🔄🌿. This turned out to be a game-changer for us 🎉. We now need just around 5 minutes a day to update the development stage, see if there’s a game-breaking change, and then propagate to production.
Our workflow resembles this: grab morning coffee ☕, review the PRs, head over to artifacthub.io, check if values have been tweaked, then approve the PR and merge it into development 💼. But this only works on GitHub. A segment of our service catalog is hosted on Azure DevOps for varying reasons. We required an identical solution for our Azure DevOps repository, and that’s where our script and the Docker tool came into being 🎈. We also realized that a fair share of teams grapple with the same challenge, so we thought, why not share a simple script that could possibly contribute to enhancing daily work improvement? 🤝.
Behind the Magic: The Process 🎩
First things first, I would advise you to fork the repository 🍴.
You’ll come across a file named dependencies.yaml that looks like this:
dependencies:
- name: "External DNS"
source:
file: dns/external-dns/Chart.yaml
path: .dependencies[0].version
repository:
type: artifacthub
name: bitnami/external-dnsThis file is used by the script to loop over to fetch the deployed version, the repo-name (in this case, bitnami), the package-name (external-dns), and to call the artifacthub.io API to obtain the current version 🔄. If a version delta is detected, the script spawns a new branch, overwrites the `.dependencies[0].version`, and acquires the package-id. The script then uses this package-id to fetch the values from the deployed version to the current version and creates a diff, or dyff. This result can be used to show the changes as a dry-run or to generate a pull request with the result 📤.
To illustrate this, I’ve crafted an example using dns/external-dns. You’ll find a file named Chart.yaml that looks like this:
apiVersion: v2
name: external-dns
version: 1.0.0
description: This Chart deploys external-dns.
dependencies:
- name: external-dns
version: 6.20.4
repository: https://charts.bitnami.com/bitnami
As you can see, the version 6.20.4 is currently deployed for the external-DNS Helm chart.
Furthermore, there’s a file called config.env that looks like this:
BRANCH='main'
DRY_RUN='true'
GITHUB='false'
AZURE_DEVOPS='false'
WITHOUT_PR='false'NOTE: You can only set one type to true!
This file is used to set the environment variables, and its usage is pretty self-explanatory 🧭.
Last but not least, there’s a script named check-helm-dep-updates.sh which will be put into action in the following steps 📜.
The Dry-Run Way:
Run the script with the config.env:
BRANCH='main'
DRY_RUN='true'
GITHUB='false'
AZURE_DEVOPS='false'
WITHOUT_PR='false'./check-helm-dep-updates.sh and get the following output:
####################### Begin #######################
Name: External DNS
Version in Chart.yaml: 6.20.4
Current Version: 6.21.0
There's a difference between the versions.
Differences:
_ __ __
_| |_ _ / _|/ _| between old_values.yaml
/ _' | | | | |_| |_ and new_values.yaml
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned two differences
|___/
(root level)
+ one map entry added:
## @param ingressClassFilters Filter sources managed by external-dns via IngressClass (optional)
##
ingressClassFilters: []
image.tag
± value change
- 0.13.4-debian-11-r19
+ 0.13.5-debian-11-r55
####################### End #######################The GitHub Way
NOTE: you will need have access to the repo, because of that you should fork this repo. You will also need have to login to GitHub over (gh auth login)
Run the script with the config.env:
BRANCH='main'
DRY_RUN='false'
GITHUB='true'
AZURE_DEVOPS='false'
WITHOUT_PR='false'./check-helm-dep-updates.sh and get the following output:
Name: External DNS
Version in Chart.yaml: 6.20.4
Current Version: 6.21.0
There's a difference between the versions.
Switched to a new branch 'update-helm-bitnami-external-dns-6.21.0'
[update-helm-bitnami-external-dns-6.21.0 81b6c65] Update External DNS version from 6.20.4 to 6.21.0
1 file changed, 1 insertion(+), 1 deletion(-)
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 10 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 431 bytes | 431.00 KiB/s, done.
Total 5 (delta 3), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
remote:
remote: Create a pull request for 'update-helm-bitnami-external-dns-6.21.0' on GitHub by visiting:
remote: https://github.com/la-cc/helm-dependencies-update-helper-script/pull/new/update-helm-bitnami-external-dns-6.21.0
remote:
To github.com:la-cc/helm-dependencies-update-helper-script.git
* [new branch] update-helm-bitnami-external-dns-6.21.0 -> update-helm-bitnami-external-dns-6.21.0
Warning: 1 uncommitted change
Creating pull request for update-helm-bitnami-external-dns-6.21.0 into main in la-cc/helm-dependencies-update-helper-script
https://github.com/la-cc/helm-dependencies-update-helper-script/pull/1
M config.env
Switched to branch 'main'
Your branch is up to date with 'origin/main'.Additionally, within GitHub, you’ll observe the creation of a new branch featuring version changes. A corresponding pull request, resembling the following, will also be generated:
The Azure DevOps Way
NOTE: in order to proceed, you’ll need access to the repository, so I recommend forking it to your own account. Additionally, you’ll need to create a Personal Access Token (PAT) in Azure DevOps.
Run the script with the config.env:
BRANCH='main'
DRY_RUN='false'
GITHUB='false'
AZURE_DEVOPS='true'
WITHOUT_PR='false'./check-helm-dep-updates.sh and get the following output:
Name: External DNS
Version in Chart.yaml: 6.20.4
Current Version: 6.21.0
There's a difference between the versions.
Switched to a new branch 'update-helm-bitnami-external-dns-6.21.0'
[update-helm-bitnami-external-dns-6.21.0 7b88413] Update External DNS version from 6.20.4 to 6.21.0
1 file changed, 1 insertion(+), 1 deletion(-)
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 10 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 433 bytes | 433.00 KiB/s, done.
Total 5 (delta 3), reused 0 (delta 0), pack-reused 0
remote: Analyzing objects... (5/5) (82 ms)
remote: Storing packfile... done (88 ms)
remote: Storing index... done (50 ms)
To ssh.dev.azure.com:v3/XY/XY/test-helm-du
* [new branch] update-helm-bitnami-external-dns-6.21.0 -> update-helm-bitnami-external-dns-6.21.0
Switched to branch 'main'
Your branch is up to date with 'origin/main'Additionally, within Azure DevOps, you’ll observe the creation of a new branch featuring version changes. A corresponding pull request, resembling the following, will also be generated:
Automating with CI/CD 🤖
Naturally, you’d want everything to be automated, and the only task on your plate would be to review the pull request. Here’s where CI/CD (Continuous Integration/Continuous Deployment) fits in like a glove 🧤.
As an illustration, I’ll walk you through the process using Azure DevOps Pipelines. Though the example pertains to Azure, the principles can be adapted and translated to other platforms like GitHub Actions.
Azure DevOps Pipelines
NOTE: keep in mind, Azure DevOps pull requests have a description length limit of 4000 characters. If the diff output exceeds this limit, it will be replaced with a notification message.
I created two files.
The first file includes the main part azure-pipeline.yaml looks like:
trigger: none
# at 0600 every day
schedules:
- cron: "0 6 * * *"
displayName: cronjob runs every day at 0600
branches:
include:
- main
always: true
pool:
vmImage: "ubuntu-latest"
jobs:
- job: check_helm_dependencies_updates
workspace:
clean: all
steps:
- checkout: self
- template: azure-pipelines/check-helm-dependencies-updates.yamlThe second file azure-pipelines/check-helm-dependencies-updates.yaml will be used as template from the main pipeline and looks like:
steps:
- task: Bash@3
displayName: Install yq
inputs:
script: |
curl -L https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -o /usr/local/bin/yq &&\
chmod +x /usr/local/bin/yq
displayName: Install yq
targetType: inline
- task: Bash@3
displayName: Install GitHub CLI (gh)
inputs:
script: |
curl -L https://github.com/cli/cli/releases/download/v2.32.1/gh_2.32.1_linux_amd64.tar.gz -o gh_2.32.1_linux_amd64.tar.gz && \
tar xfv gh_2.32.1_linux_amd64.tar.gz -C /usr/local/bin && \
rm gh_2.32.1_linux_amd64.tar.gz
displayName: Install GitHub CLI (gh)
targetType: inline
- task: Bash@3
displayName: Install dyff
inputs:
script: |
curl -L https://github.com/homeport/dyff/releases/download/v1.5.8/dyff_1.5.8_linux_amd64.tar.gz -o dyff_v1.5.8_linux_amd64.tar.gz && \
tar xfv dyff_v1.5.8_linux_amd64.tar.gz -C /usr/local/bin && \
rm dyff_v1.5.8_linux_amd64.tar.gz
displayName: Install dyff
targetType: inline
- task: Bash@3
displayName: Check Helm Dependencies Updates
env:
AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
inputs:
script: |
# Read the file
file="$(pwd)/dependencies.yaml" # use absolute path
# Get the number of dependencies
count=$(yq e '.dependencies | length' $file)
# Iterate over the list
for ((i = 0; i < $count; i++)); do
# Name of the dependency
name=$(yq e ".dependencies[$i].name" $file)
# Path to the Chart.yaml file
chart_file=$(yq e ".dependencies[$i].source.file" $file)
# Path to the version number in the Chart.yaml file
version_path=$(yq e ".dependencies[$i].source.path" $file)
# Change directory to the chart file directory
cd $(dirname $chart_file) || exit
# Read the version from the Chart.yaml file
version=$(yq e "$version_path" "$(basename $chart_file)")
# Repository name for the Artifact API
repo_name=$(yq e ".dependencies[$i].repository.name" $file)
# Get the current version with the Artifact API
current_version=$(curl -sSL "https://artifacthub.io/api/v1/packages/helm/$repo_name/feed/rss" | yq -p=xml '.rss.channel.item[0].title' | cut -d' ' -f2)
# Sanitize the repo name
sanitized_name=$(echo $repo_name | tr -d ' ' | tr '/' '-')
# Output
echo "Name: $name"
echo "Version in Chart.yaml: $version"
echo "Current Version: $current_version"
# If there's a difference between the versions
if [ "$version" != "$current_version" ]; then
if [ ! $(git branch --list update-helm-$sanitized_name-$current_version) ]; then
echo "There's a difference between the versions."
# Get the package_id
package_id=$(curl -sSL "https://artifacthub.io/api/v1/packages/helm/$repo_name/$current_version" | yq -r .package_id)
# Insert the package_id + version into the command
new_values=$(curl -sSL "https://artifacthub.io/api/v1/packages/$package_id/$current_version/values")
# Save the new values to a temporary file
echo "$new_values" >new_values.yaml
# Insert the package_id + current version into the command
old_values=$(curl -sSL "https://artifacthub.io/api/v1/packages/$package_id/$version/values")
# Save the old values to a temporary file
echo "$old_values" >old_values.yaml
# Perform a diff on the two files
diff_result=$(dyff between old_values.yaml new_values.yaml) || true
# Output differences
echo "$diff_result" >diff_result.txt
awk '{ printf "\t%s\n", $0 }' diff_result.txt >shift_diff_result.txt
shift_diff_result=$(cat shift_diff_result.txt)
# If the diff output is too large for display, overwrite it with a message
if ((${#shift_diff_result} > 4000)); then
shift_diff_result="The diff output is too large for display (>4000 characters). Please refer to ArtifactHub directly for a detailed comparison of changes between the $version and $current_version."
fi
# Delete the temporary files
rm old_values.yaml new_values.yaml
# Configure git
git config --global user.email "bot-helm-dep-sheriff@hpa.de"
git config --global user.name "bot-helm-dep-sheriff"
git config --global pull.ff only
# Replace the old version with the new version in the Chart.yaml file using sed
sed -i.bak "s/version: $version/version: $current_version/g" "$(basename $chart_file)" && rm "$(basename $chart_file).bak"
# Create a new branch for this change
git checkout -b update-helm-$sanitized_name-$current_version || true
# Add the changes to the staging area
git add "$(basename $chart_file)"
# Create a commit with a message indicating the changes
git commit -m "Update $name version from $version to $current_version [skip ci]"
# Push the new branch to GitHub
git push @dev.azure.com/<REPLACE_ME>/<REPLACE_ME>/_git/<REPLACE_ME>">https://$(System.AccessToken)@dev.azure.com/<REPLACE_ME>/<REPLACE_ME>/_git/<REPLACE_ME> HEAD:update-helm-$sanitized_name-$current_version
# Create a Azure DevOps Pull Request
az repos pr create --title "Update $name version from $version to $current_version" --description "$shift_diff_result" --target-branch "$BRANCH" --source-branch update-helm-$sanitized_name-$current_version || true
# Get back to the source branch
git checkout $BRANCH
else
echo "Branch already exists. Checking out to the existing branch." || true
fi
else
echo "There's no difference between the versions."
fi
# Return to the original directory
cd - 1>/dev/null || exit
echo ""
done
displayName: Check Helm Dependencies Updates
targetType: inlineTaking the Docker Route 🐳
If you’d rather not go through the process of installing all the tools, there’s another way. You can run the script inside a Docker container. I’ve already set up a Docker container and detailed instructions on how to use it. If that piques your interest, feel free to check out the repository.
Wrapping Up 🎁
This approach has saved us countless hours over the last half year, and we’re more than happy to share the solution with you. We acknowledge that it may not be perfect, as it’s a quick snapshot of the logic we use, so feel free to tailor it to your needs ✂️.
One crucial point to remember is the potential for lengthy descriptions if the version delta is significant. With Azure DevOps Pipelines, the description limit is capped at 4000 characters, while GitHub allows up to 65536 characters. Therefore, we strongly recommend keeping the delta as small as possible. This not only smooths the workflow but also helps in closing all security vulnerabilities in a timely manner 🔒.
Contact Information
If you have some Questions, would like to have a friendly chat or just network to not miss any topics, then don’t use the comment function at medium, just feel free to add me to your LinkedIn network!
References
- GitHub Helm Dep Helper Docker — https://github.com/la-cc/helm-dependencies-update-helper
- GitHub Helm Dep Helper Script — https://github.com/la-cc/helm-dependencies-update-helper-script
- Kubernetes Service Catalog — https://github.com/Hamburg-Port-Authority/kubernetes-service-catalog
