Open in app

Sign up

Sign in

Write

Sign up

Sign in

How to Develop Helm based Kubernetes Operators locally

Tassilo Smola
Level Up Coding
Published in
11 min readFeb 14, 2023

--

Photo by Growtika on Unsplash

In my last project, we had a setup of multiple Kubernetes clusters on different sites, with user specific namespaces in it. Our product consisted of several helm charts which were wrapped via Subcharts in one main chart. That chart was to be deployed on the users namespaces.

Everything worked fine, we were able to release sprint wise and the helm chart was packaged and pushed via a CI / CD pipeline to an artifactory where it was ready to be installed to the users namespaces.

But here's the sticking point:
Most users never updated the helm chart in their namespace and weren't able to get the latest fixes or features. We always had to ask which chart version was currently installed in the namespace and had to deal with a manual application lifecycle.

Introducing Kubernetes Operators

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop

That sounds very abstract, doesn't it?
An operator simply takes care of an automated application lifecycle and is also able to run custom operations such as database backups / restores or other stateful operations which are important when upgrading the application.

In this tutorial, we are focusing on a local operator development using the operator-sdk and minikube.

Prerequisites

In order to run this example, you need to have the following tools installed:

If the environment is set up accordingly, it’s time to get our hands a little dirty

Init operator project

Open a terminal, create a folder where the operator should be set up and run the following commands:

# scaffold operator structure
$ operator-sdk init --plugins=helm --domain=example.com 
Writing kustomize manifests for you to edit...
Next: define a resource with:
operator-sdk create api

# create api for helm chart
$ operator-sdk create api --group operators --version v1 --kind Example-platform
Writing kustomize manifests for you to edit...
Created helm-charts/example-platform
WARN[0000] Using default RBAC rules: failed to get Kubernetes config: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

A folder structure should now be visible, with boilerplate code that has to be adjusted. We will adjust the necessary values for local k8s deployment in a later step.

Prepare local Kubernetes cluster

Make sure your cluster is up and running, and you're able to access the dashboard. If you use minikube, the dashboard comes already pre-configured.
In this tutorial, we focus on the minikube installation. Nonetheless, steps should be similar if you use the docker desktop Kubernetes installation.

Now we have to start the local minikube instance and open the dashboard:

# start minikube
$ minikube start
# check if minikube is running
$ minikube status
# open minikube dashboard
$ minikube dashboard

The dashboard should now open:

Kubernetes Dashboard

Use local docker cache

In order for minikube to make use of the local pulled docker images, an eval command has to be executed in the terminal to map the docker engine to minikube.

Otherwise, a proxy to an image registry needs to be configured.

# point minikube docker engine to terminal currently used
eval $(minikube -p minikube docker-env)

Build local controller image

In order to grant minikube access to the controller docker image, cd to the root of the project and run a docker build in the terminal the eval command was executed before. In the Dockerfile, the sources like helm charts and watches are copied, which will be available as a custom resource definition as soon as the operator is deployed.

docker build . -t controller:latest

Running the operator

You have two choices of developing operators.

  1. Deploy the operator on the minikube cluster:
    This will create a separate namespace where the operator image containing the helm chart is built. The Custom Resource definitions are then available in all namespaces.
  2. Run the operator locally:
    This will run the operator locally in the terminal and expose the Custom Resource Definitions to minikube.

1. Deploy the operator on minikube cluster

This is the more tortious way because all the docker images have to be available for minikube. You can either set up a proxy to an image registry or run a manual docker pull to have the images downloaded in the minikube docker cache.

Take the terminal you executed the eval command before to have access to minikubes docker cache and pull required images for the operator:

# login to docker registry. If you don´t have a redhat account, crate one via developer.redhead.com
docker login registry.redhat.io
Username: XXX
Password: ******
Login Succeeded

# pull docker image from redhat registry 
docker pull registry.redhat.io/openshift4/ose-kube-rbac-proxy:v4.11
v4.11: Pulling from openshift4/ose-kube-rbac-proxy
97da74cc6d8f: Pull complete 
d8190195889e: Pull complete 
ad44d7582bbb: Pull complete 
1e85b2b37152: Pull complete 
6e5d4e83cb3e: Pull complete 
Digest: sha256:ac54cb8ff880a935ea3b4b1efc96d35bbf973342c450400d6417d06e59050027
Status: Downloaded newer image for registry.redhat.io/openshift4/ose-kube-rbac-proxy:v4.11
registry.redhat.io/openshift4/ose-kube-rbac-proxy:v4.11

In Makefile, change IMG value to Dockerfile build and tag

# Image URL to use all building/pushing image targets
IMG ?= example-platform:latest

Adjust config/manager/manager.yaml to be compatible with minikube. ImagePullPolicy has to set to Never, otherwise the cluster would return an ImagePullBackOff, because no registry is configured currently.

        runAsNonRoot: true
        imagePullPolicy: Never

Execute the following command to deploy the operator image to a separate namespace:

$ make deploy
cd config/manager && /home/user/repos/example-operator/bin/kustomize edit set image controller=controller:latest
/home/user/repos/example-operator/bin/kustomize build config/default | kubectl apply -f -
namespace/example-operator-system created
customresourcedefinition.apiextensions.k8s.io/example-platforms.operators.example.com created
serviceaccount/example-operator-controller-manager created
role.rbac.authorization.k8s.io/example-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/example-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/example-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/example-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/example-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/example-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/example-operator-proxy-rolebinding created
service/example-operator-controller-manager-metrics-service created
deployment.apps/example-operator-controller-manager created

Now the Deployment should be visible in the minikube dashboard:

Also, the custom resource definition should be visible in the project:

2. Run the operator locally

This is the easier way to develop operators and explore the lifecycle behavior as there are good log outputs from the operator and also no docker image or image registry dependencies.

Execute the following command to run the operator locally and just expose the created CRD to minikube. In the terminal, the log output should be visible describing the kubectl apply commands and a message watching resources

make install run
/home/user/repos/example-operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/example-platforms.operators.example.com created
/home/user/repos/example-operator/bin/helm-operator run
{"level":"info","ts":1675928925.1594994,"logger":"cmd","msg":"Version","Go Version":"go1.19.5","GOOS":"linux","GOARCH":"amd64","helm-operator":"v1.27.0","commit":"5cbdad9209332043b7c730856b6302edc8996faf"}
{"level":"info","ts":1675928925.1611917,"logger":"cmd","msg":"Watch namespaces not configured by environment variable WATCH_NAMESPACE or file. Watching all namespaces.","Namespace":""}
{"level":"info","ts":1675928925.176321,"logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1675928925.177788,"logger":"helm.controller","msg":"Watching resource","apiVersion":"operators.example.com/v1","kind":"Example-platform","namespace":"","reconcilePeriod":"1m0s"}
{"level":"info","ts":1675928925.1786354,"msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":1675928925.178664,"msg":"Starting server","kind":"health probe","addr":"[::]:8081"}
{"level":"info","ts":1675928925.1790054,"msg":"Starting EventSource","controller":"example-platform-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":1675928925.1790779,"msg":"Starting Controller","controller":"example-platform-controller"}
{"level":"info","ts":1675928925.280455,"msg":"Starting workers","controller":"example-platform-controller","worker count":16}

A Custom Resource definition as shown above should also be visible in the default namespace.

Install Operator and check installed helm chart

Right now, our operator is deployed and ready to use. If you want to make use of the underlying helm chart, you have to create and apply a custom resource definition. As soon as the custom resource definition is applied, the operator recognizes it and installs the helm chart in the namespace.

An example yaml definition is provided by the operator-sdk and stored in config/samples/operators_v1_example-platform.yaml.

All values from your chart values.yaml are already included in the sample and can be customized:

apiVersion: operators.example.com/v2
kind: Example-platform
metadata:
  name: example-platform-sample
spec:
  # Default values copied from <project_dir>/helm-charts/example-platform/values.yaml
  affinity: {}
  autoscaling:
    enabled: false
    maxReplicas: 100
    minReplicas: 1
    ...

Let's apply the Custom Resource Definition to the default namespace in minikube:

kubectl apply -f operators_v1_example-platform.yaml

The deployment should now be visible in the default namespace.

If you check the log output from the running operator, it recognizes the apply of the Custom Resource definition and reconciles the release:

I0209 08:51:33.570619    6570 request.go:682] Waited for 1.033958682s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.docker.internal:6443/apis/batch/v1?timeout=32s
{"level":"info","ts":1675929093.73841,"logger":"helm.controller","msg":"Reconciled release","namespace":"default","name":"example-platform-sample","apiVersion":"operators.example.com/v1","kind":"Example-platform","release":"example-platform-sample"}

Also, the helm chart with revision should exist.

$ helm list 
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS       CHART                    APP VERSION
example-platform-sample default         1               2023-02-09 09:15:56.6603448 +0100 CET   deployed     example-platform-0.1.0   1.17.0

Apply changes to the resources

So the operator successfully installed the helm chart wrapped in the Custom Resource Definition of kind example-platform in revision 1 in our default namespace.

Let's make some changes to the custom resource.

In config/samples/operators_v1_example-platform.yaml the .spec values are the default values from the helm charts values.yaml file. To showcase the change of the values from your helm chart, change the .spec.service.port from 80 to 8080 and apply it:

$ kubectl apply -f operators_v1_example-platform.yaml
example-platform.operators.example.com/example-platform-sample configured

As you can see, the CRD gets reconfigured. The output from the running operator recognizes the change and applies it to the deployed objects.

Upgrade helm chart

Let's test if the content in watches.yaml is listening for changes, i.e. when upgrading the helm chart. To do this, we have to run the operator locally using the install run command:

$ make install run
/usr/local/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/example-platforms.operators.example created
/home/smolauserlo/repos/test/bin/helm-operator run
{"level":"info","ts":1675751292.9460497,"logger":"cmd","msg":"Version","Go Version":"go1.18.3","GOOS":"linux","GOARCH":"amd64","helm-operator":"v1.22.0","commit":"9e95050a94577d1f4ecbaeb6c2755a9d2c231289"}

The operator is now running locally with the currently configured minikube cluster and listening for changes in the helm-charts/example-platform directory.

There should already be an installed helm chart when running helm list. We modify the content of the field appVersion in Chart.yaml to simulate an upgrade of the helm chart and check the log output from the operator after saving. This is similar to a make deploy command explained in step 1 and an ImagePullPolicy to :latest
Note: it can take up to one minute for the watcher to check for updates

$ helm list
NAME                            NAMESPACE       REVISION        UPDATED                              STATUS   CHART                   APP VERSION
example-platform-sample        default         1               2023-02-07 07:29:34.0835967 +0100 CETdeployed example-platform-0.1.0 1.16.0

Now update the appVersion field in Chart.yaml.

Check the log output in the terminal the operator is running in:

{"level":"info","ts":1675929822.8833482,"logger":"helm.controller","msg":"Upgraded release","namespace":"default","name":"example-platform-sample","apiVersion":"operators.example.com/v1","kind":"Example-platform","release":"example-platform-sample","force":false}
I0209 09:03:43.957269    6570 request.go:682] Waited for 1.028116888s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.docker.internal:6443/apis/flowcontrol.apiserver.k8s.io/v1beta2?timeout=32s
{"level":"info","ts":1675929824.118914,"logger":"helm.controller","msg":"Reconciled release","namespace":"default","name":"example-platform-sample","apiVersion":"operators.example.com/v1","kind":"Example-platform","release":"example-platform-sample"}

As we can see from the log output above, the watcher is recognizing the changes and runs a helm upgrade from the changed helm resources. If we check the installed helm chart via a helm list command, we can see the revision was increased to 2 and the version was updated to the version we specified in the appVersion field of the Chart.yaml file.

/config/samples$ helm list
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
example-platform-sample default         2               2023-02-09 09:03:42.81794404 +0100 CET  deployed        example-platform-0.1.0  1.17.0

Dealing with breaking changes

Our complete application lifecycle for helm upgrades is handled now. But what about breaking changes which shouldn't applied automatically on deployed instances?

In order to do this, we have to add another version of our Custom Resource Definition, which has to be applied manually.

This can be done in config/bases/operators.example_example-platforms.yaml file. In .spec.versions[] we can copy the content from v1 and paste it to v2 to create a new API version.

  - name: v2
    schema:
      openAPIV3Schema:
        description: Example-platform is the Schema for the example-platforms API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: Spec defines the desired state of Example-platform
            type: object
            x-kubernetes-preserve-unknown-fields: true
          status:
            description: Status defines the observed state of Example-platform
            type: object
            x-kubernetes-preserve-unknown-fields: true
        type: object
    served: true
    storage: false
    subresources:
      status: {}

After doing this, you have to restart the local operator because the file content is not defined in watches.yaml:

$ CTRL + c
{"level":"info","ts":1675753456.3530438,"msg":"Stopping and waiting for leader election runnables"}
{"level":"info","ts":1675753456.353102,"msg":"Shutdown signal received, waiting for all workers to finish","controller":"example-platform-controller"}
{"level":"info","ts":1675753456.353133,"msg":"All workers finished","controller":"example-platform-controller"}
{"level":"info","ts":1675753456.3531446,"msg":"Stopping and waiting for caches"}
{"level":"info","ts":1675753456.3533165,"msg":"Stopping and waiting for webhooks"}
{"level":"info","ts":1675753456.3533404,"msg":"Wait completed, proceeding to shutdown the manager"}

$ make install run
/usr/local/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/example-platforms.operators.example configured
/home/smolauserlo/repos/test/bin/helm-operator run
{"level":"info","ts":1675753458.0622938,"logger":"cmd","msg":"Version","Go Version":"go1.18.3","GOOS":"linux","GOARCH":"amd64","helm-operator":"v1.22.0","commit":"9e95050a94577d1f4ecbaeb6c2755a9d2c231289"}

The new version should now be visible under Custom Resource Definitions on the minikube dashboard:

To use the breaking changes, we also have to apply a new Custom Resource definition in the cluster. To do this, copy the file /config/samples/operators_v1_example-platform.yaml and name it to operators_v2_example-platform.yaml.

Also, the content of the field apiVersion in the CRD has to be updated from apiextensions.k8s.io/v1 to apiextensions.k8s.io/v2.

After doing his, we can apply the file to our minikube cluster.
Note: We have to delete the v1 CRD first because it's not possible to deploy two helm charts with the same release name in a namespace.

/config/samples$ kubectl delete -f operators_v1_example-platform.yaml
example-platform.operators.example "example-platform-sample" deleted

/config/samples$ kubectl apply -f operators_v2_example-platform.yaml
example-platform.operators.example/example-platform-sample configured

The helm chart with the Custom Resource Definition of kind example-platform in version v2 is now available in the namespace with a fresh Revision:

$ helm list
NAME                            NAMESPACE       REVISION        UPDATED                              STATUS   CHART                   APP VERSION
example-platform-sample        default         1               2023-02-07 08:10:17.041081 +0100 CET deployed example-platform-0.1.0 1.17.0

Conclusion

As you can see, it's fairly easy to initialize and set up a helm based Kubernetes Operator. Of course, we didn't handle stuff like Role Based Access Control (rbac), monitoring etc., as this was just a showcase how you can handle and automate the complete Application Lifecycle.

There is often a need to because of the lack of cluster permissions (You need cluster-admin privileges) or the requirement to isolate the development from the remote cluster.

If you want to check out the code, you can go to my GitHub repository, where I documented everything:

GitHub - tassilosmola/tutorial-kubernetes-helm-operator: Tutorial for develop local helm based…

This repository initializes an existing helm chart that can be managed by a kubernetes operator. For testing purposes…

github.com

Level Up Coding

Thanks for being a part of our community! Before you go:

🚀👉 Join the Level Up talent collective and find an amazing job

DevOps
Kubernetes
Application Lifecycle
Helm
Kubernetes Operator

--

--

Tassilo Smola
Level Up Coding
Follow

Written by Tassilo Smola

59 Followers

Freelancer | DevOps Engineer | Solution Architect | Thinking about technology, human nature and society.

Follow

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams