GitOps for Helm Users

Outline

1. Install Flux CLI and Kind
2. Make Personal Access Token for creating repositories
3. Export env vars locally
4. Create local demo cluster
5. Simple bootstrap
6. Clone the newly created git repo to your local workspace
7. Lets create a Helm release the most common way, using the Helm CLI
8. Now lets convert these to declarative CRs that Flux understands
9. Lets go ahead and push this to Git
10. Lets check out the magic
11. Change a new Helm release value through Git
12. Pause and resume
13. Cleanup demo cluster 🧹
14. Disaster recovery ⛑
15. Wrap up

1. Install Flux CLI and Kind

$ brew upgrade fluxcd/tap/flux kind
$ brew reinstall fluxcd/tap/flux kind
$ flux --version && kind --version
flux version 0.29.4
kind version 0.12.0

2. Make Personal Access Token for creating repositories

1. Generate new token in dev settings
2. Check all permissions under repo & save
3. Copy PAT to buffer

3. Export env vars locally

I've done this in advance for now.

💡 If you want to show during a demo, follow best security practices by making the PAT off camera - or copy from a secure password app on camera etc – then read to var silently read -s, then export var.

$ export GITHUB_TOKEN=[paste PAT]
$ echo -n $GITHUB_TOKEN | wc -c
  40

4. Create local demo cluster

$ kind create cluster
(took 40s)

5. Simple bootstrap:

💡 The more complex your org is, the more complex your directory structure and patterns usually are.

There is no gold standard.

Flux is not opinionated about how directories are structured, rather it tries to be as flexible as possible to accommodate different patterns.

$ flux bootstrap github \
  --interval 10s \
  --owner scottrigby --personal \
  --repository flux-for-helm-users \
  --branch main \
  --path=clusters/dev
► connecting to github.com
✔ repository "https://github.com/scottrigby/flux-for-helm-users" created
► cloning branch "main" from Git repository "https://github.com/scottrigby/flux-for-helm-users.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed sync manifests to "main" ("42a5e71e792cf3ca0393fefea4c4375e72d9fc47")
► pushing component manifests to "https://github.com/scottrigby/flux-for-helm-users.git"
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
✔ public key: ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBMbDpSb+E912hnXZWX/x9RFWPscqsTJ/8bzgYLgYEywpkWwQNZVCjdvhLiNEexXMqk5IO3JxF9ScAa76IB6kYRFZ8WlGwoBNINU2HcXmtJF/9LZgUKzF53ioK9esCO+rYw==
✔ configured deploy key "flux-system-main-flux-system-./clusters/dev" for "https://github.com/scottrigby/flux-for-helm-users"
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ committed sync manifests to "main" ("055e5edfbace022504101c763b65b1f7c2134187")
► pushing sync manifests to "https://github.com/scottrigby/flux-for-helm-users.git"
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy
(took 1m3s)

6. Clone the newly created git repo to your local workspace

$ cd ~/code/github.com/scottrigby \
  && git clone git@github.com:scottrigby/flux-for-helm-users.git \
  && cd flux-for-helm-users
$ tree
.
└── clusters
    └── dev
        └── flux-system
            ├── gotk-components.yaml
            ├── gotk-sync.yaml
            └── kustomization.yaml

3 directories, 3 files

7. Lets create a Helm release the most common way, using the Helm CLI

Remember that we set custom values. We will get back to this later.

helm repo add podinfo https://stefanprodan.github.io/podinfo

Lets set some values to make this fun.

💡 Helm CLI is great to show all the available options in a chart:

$ helm show values podinfo --repo https://stefanprodan.github.io/podinfo
# Default values for podinfo.

replicaCount: 1
logLevel: info

ui:
  color: "#34577c"
  message: ""
  logo: ""

etc…

$ helm upgrade -i my-release podinfo/podinfo \
  --set replicaCount=2 \
  --set logLevel=debug \
  --set ui.color='red'
Release "my-release" does not exist. Installing it now.
…

8. Now lets convert these to declarative CRs that Flux understands

Create a Source Custom Resource locally

💡 The Helm CLI reads your locally defined Helm repo info (created in step 7). But the Flux Helm controller in your cluster will also need this same info.

We'll tell Flux about the Helm repo info with a HelmRepository CR representing a Flux source. Instead of helm add repo you can use flux create source helm to export the CRD to a local file:

$ flux create source helm podinfo \
  --url=https://stefanprodan.github.io/podinfo \
  --namespace=default \
  --export > clusters/dev/source-helmrepo-podinfo.yaml
$ cat clusters/dev/source-helmrepo-podinfo.yaml
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: podinfo
  namespace: default
spec:
  interval: 1m0s
  url: https://stefanprodan.github.io/podinfo

Next we'll create a HelmRelease Custom Resource locally, using the same Helm values we earlier specified with the Helm CLI.

💡 Helm CLI makes it very easy to get the values we earlier set for the release. We'll first export these to a file then take a look at its contents:

$ helm get values my-release -oyaml > my-values.yaml
$ cat my-values.yaml
logLevel: debug
replicaCount: 2
ui:
  color: red

And again Flux CLI makes it easy to create the CR. You may also do this by hand, or with an IDE (for example with the VSCode Flux plugin), but the CLI command eases this:

$ flux create helmrelease my-release \
  --release-name=my-release \
  --source=HelmRepository/podinfo \
  --chart=podinfo \
  --chart-version=">4.0.0" \
  --namespace=default \
  --values my-values.yaml \
  --export > ./clusters/dev/podinfo-helmrelease.yaml
$ cat clusters/dev/podinfo-helmrelease.yaml
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: my-release
  namespace: default
spec:
  chart:
    spec:
      chart: podinfo
      sourceRef:
        kind: HelmRepository
        name: podinfo
      version: '>4.0.0'
  interval: 1m0s
  values:
    logLevel: debug
    replicaCount: 2
    ui:
      color: red

We now no longer need the temporary values file. Lets be tidy:

rm my-values.yaml

9. Lets go ahead and push this to Git

$ git add clusters/dev
$ git commit -m 'Configure podinfo Helm Repo source and Helm Release'
$ git push
…

💡 From this point on, you are now doing GitOps.

10. Lets check out the magic

We can verify that Flux is now managing this Helm release.

💡 If you want to immediately trigger reconciliation on a local demo cluster you can manually call flux reconcile. We shouldn't need to trigger that manually in this demo because we set the interval to 10s.

In real word clusters there are important use cases for setting up webhook receivers to automate this immediacy, and there are equally important use cases for letting your defined sync interval run its course.

flux reconcile helmrelease my-release

Flux will add labels

$ kubectl get deploy my-release-podinfo -oyaml | grep flux
helm.toolkit.fluxcd.io/name: my-release
helm.toolkit.fluxcd.io/namespace: default

11. Change a new Helm release value through Git

You believe me that we are now doing GitOps, but let's prove it.

Change a value in your HelmRelease CR:

$ yq -i '.spec.values.ui.color = "blue"' clusters/dev/podinfo-helmrelease.yaml
$ git add clusters/dev
$ git diff --staged

diff --git a/clusters/dev/podinfo-helmrelease.yaml b/clusters/dev/podinfo-helmrelease.yaml
index b58eed2..5e1dc10 100644
--- a/clusters/dev/podinfo-helmrelease.yaml
+++ b/clusters/dev/podinfo-helmrelease.yaml
@@ -17,5 +17,4 @@ spec:
    logLevel: debug
    replicaCount: 2
    ui:
-      color: red
+      color: blue

$ git commit -m "blue me"
$ git push

We can see our Helm release incremented the revision:

$ helm list
NAME      	NAMESPACE	REVISION	UPDATED                              	STATUS  	CHART        	APP VERSION
my-release	default  	3       	2022-02-17 06:16:42.2293519 +0000 UTC	deployed	podinfo-6.0.3	6.0.3 

And that the new release revision applied our change:

$ helm diff revision my-release 2 3

          env:
          - name: PODINFO_UI_COLOR
-           value: red
+           value: blue
          image: ghcr.io/stefanprodan/podinfo:6.0.3
          imagePullPolicy: IfNotPresent

Let's get visual:

$ kubectl -n default port-forward deploy/my-release-podinfo 8080:9898
Forwarding from 127.0.0.1:8080 -> 9898

Browse to http://localhost:8080

Er mah gerd, it's blue!

Let's pretend we're in incident management and want to use Helm rollback

12. Pause and resume

💡 It's worth noting that Flux HelmRelease retains Helm release metadata and Helm's ability to manage the releases directly.

There are various benefits to this, including the ability to continue using your favorite development tools that integrate with Helm releases (such as helm list, helm diff plugin, etc).

This is also helpful in production. For example, there are legitimate use cases for pausing GitOps operations and temporarily using the Helm CLI, such as incident management. Pausing and resuming GitOps reconciliation may be done on a per Custom Resource basis without affecting the others, for example a single HelmRelease:

$ flux suspend helmrelease my-release --namespace default
► suspending helmreleases my-release in default namespace
✔ helmreleases suspended

Flux CLI has a handy flux get feature, that gives additional info in output including whether or not reconciliation is suspended for a resource. Here we can see SUSPENDED is True.

$ flux get hr my-release --namespace default
NAME      	REVISION	SUSPENDED	READY	MESSAGE                          
my-release	6.1.1   	True     	True 	Release reconciliation succeeded

Let's rollback to red using the Helm CLI, to show that it works.

$ helm rollback my-release 2
Rollback was a success! Happy Helming!

We can port forward again see that it worked:

$ kubectl -n default port-forward deploy/my-release-podinfo 8080:9898
Forwarding from 127.0.0.1:8080 -> 9898

OK yay, back to red! 😅

Once we're finished with our incident management window and want to resume GitOps reconciliation on that resource, we just need to resume again:

$ flux resume helmrelease my-release --namespace default
► resuming helmreleases my-release in default namespace
✔ helmreleases resumed
◎ waiting for HelmRelease reconciliation
✔ HelmRelease reconciliation completed
✔ applied revision 6.0.3

We can see SUSPENDED is False, which means reconciliation has resumed.

flux get hr my-release --namespace default
NAME      	REVISION	SUSPENDED	READY	MESSAGE                          
my-release	6.1.1   	False    	True 	Release reconciliation succeeded

Port forward again, and take a look.

Back to blue as planned.

13. Cleanup demo cluster 🧹

kind delete cluster

And if you wish, feel free to delete your demo GitHub repo.

💡 Or not! These commands are idempotent, so you can feel free to keep your repo. In fact… let's try it!

14. Disaster recovery ⛑

Want to see how Flux handles your Helm release in a disaster recovery scenario?

Let's simulate total cluster failure by just deleting it 😵:

kind delete cluster

We can create a new one by repeating step 4 (kind create cluster).

Then just need to install Flux components into the new cluster by repeating the flux bootstrap command from step 5.

💡 Because we still have our desired state defined in the Git repo we specify in flux bootstrap, reconciliation will happen automatically. Our Helm release should now match what we've defined in Git, as the source of truth!

🔍 You'll notice the Helm metadata revision is back to 1, because that is only useful as in-cluster storage. New cluster, revisions start anew.

$ helm list
NAME      	NAMESPACE	REVISION	UPDATED                              	STATUS  	CHART        	APP VERSION
my-release	default  	1       	2022-02-17 06:56:15.5594991 +0000 UTC	deployed	podinfo-6.0.3	6.0.3 

15. Wrap up

And there we have it!

• On a local kind cluster, we simulated an existing Helm release using the Helm CLI you're already familiar with (helm install)
• We used Flux CLI to bootstrap Flux components into the cluster, and simultaneously define and create (if it didn't already exist) a properly formatted Git repo containing the bootstrap manifests
• Used Flux CLI to easily create Custom Resources for the Helm repo and release, along with our existing release's custom values
• Pushed the files to Git, and show Flux labels which means it has taken ownership of managing your existing Helm release
• Proved this by making changes to Git only, and watch Flux magically update your Helm release from Git
• Showed how to pause and resume the automated continuous reconciliation on a single HelmRelease, which you might use during in-cluster development or incident management
• Simulated disaster recovery of your Helm release by deleting your entire cluster. Bootstrapping Flux again was all we needed to get your system and Helm-released apps running again