Effortlessly Develop and Deploy Kubernetes Custom Resources with Operator SDK: A Step-by-Step Guide

17 min readMar 28, 2023

Introduction

Welcome to this comprehensive guide on effortlessly developing and deploying Kubernetes Custom Resources using the Operator SDK! Kubernetes has become the go-to container orchestration platform, offering powerful features and extensibility that enables developers to build and manage complex applications. One of the core strengths of Kubernetes is its ability to manage Custom Resources, which allow users to extend the Kubernetes API with custom functionality tailored to their needs.

Kubernetes Custom Resources

Custom Resources are extensions of the Kubernetes API that define new types of resources specific to your application. These new resources can be managed using kubectl and the Kubernetes API, just like built-in resources such as Pods and Services. Custom Resources empower you to model your application's domain-specific requirements and declaratively manage them within Kubernetes.

Kubernetes Operators

While Custom Resources extend the Kubernetes API, Operators are responsible for managing these resources and ensuring that the desired state specified by the user is met. Operators are custom controllers that watch for changes in Custom Resources and perform actions to reconcile the desired state with the current state. They automate complex tasks and provide higher-level abstractions to manage applications and infrastructure more effectively.

Purpose of This Blog

In this blog, we will guide you through the process of developing and deploying a custom resource using the Operator SDK, a powerful tool that simplifies the process of building Kubernetes Operators. We will cover the basics of Custom Resources, the Operator pattern, and how to implement them using the Operator SDK. By the end of this tutorial, you will have a solid understanding of how to create and manage your own custom resources using the Operator SDK and apply these concepts to your own projects. Let’s dive in!

Prerequisites

Before we start our journey to develop and deploy a custom resource using the Operator SDK, it’s essential to have a clear understanding of the required tools and the expected level of knowledge. This section outlines the prerequisites needed to follow along with this tutorial.

Required Software and Tools

To successfully complete this tutorial, you’ll need to have the following software and tools installed on your local machine or development environment:

Go: The Operator SDK uses the Go programming language for implementing custom controllers. Make sure you have Go installed (version 1.16 or later) and properly configured. You can download and install Go from the official website.
Docker: Docker is required to build and package the custom controller into a container image. Install Docker from the official website and make sure it’s properly set up.
Kubernetes: You’ll need access to a Kubernetes cluster (version 1.20 or later) to deploy the custom resource and the operator. You can use a local cluster like minikube or kind, or a managed Kubernetes service like IBM Kubernetes Service(IKS), Google Kubernetes Engine (GKE), Amazon EKS, or Azure AKS.
kubectl: The Kubernetes command-line tool, kubectl, is necessary for interacting with your cluster and managing resources. Install kubectl following the official instructions.
Operator SDK: The Operator SDK simplifies the process of building, testing, and deploying Kubernetes operators. Install the Operator SDK CLI from the official GitHub repository.

Expected Knowledge and Experience

To make the most out of this tutorial, you should have a basic understanding of the following concepts:

Kubernetes: Familiarity with Kubernetes concepts such as Pods, Deployments, Services, and ConfigMaps is crucial. You should be comfortable with using kubectl and creating and managing resources using YAML manifests.
Golang: A basic understanding of the Go programming language is required, as we’ll be writing the custom controller using Go. You should be familiar with Go syntax, functions, structs, and error handling.

With these prerequisites in place, you’re ready to start developing and deploying custom resources using the Operator SDK!

Setting Up the Environment

Before diving into developing and deploying custom resources using the Operator SDK, it’s crucial to set up your development environment correctly. This section will walk you through installing and configuring the required software and tools.

Installing Go

Visit the official Go download page and download the appropriate installer for your operating system.
Follow the installation instructions for your platform.
Verify the installation by opening a terminal and running go version. You should see the installed Go version (1.16 or later).

Installing Docker

Visit the official Docker download page and choose the appropriate version for your platform (Docker Desktop for Mac/Windows or Docker Engine for Linux).
Follow the installation instructions for your operating system.
Verify the installation by opening a terminal and running docker --version. You should see the installed Docker version.

Setting Up a Kubernetes Cluster

You can use a local Kubernetes cluster like minikube or kind, or a managed Kubernetes service from a cloud provider.

Minikube

Follow the minikube installation instructions for your operating system.
Start a local cluster by running minikube start.
Verify the cluster is running with kubectl get nodes.

Kind

Follow the kind installation instructions for your operating system.
Create a local cluster by running kind create cluster.
Verify the cluster is running with kubectl get nodes.

Installing kubectl

Visit the official kubectl installation page and follow the instructions for your platform.
Verify the installation by running kubectl version --client. You should see the installed kubectl version.

Installing the Operator SDK

Visit the Operator SDK GitHub repository and follow the installation instructions for your platform.
Verify the installation by running operator-sdk version. You should see the installed Operator SDK version.

Now that your development environment is set up, you’re ready to start building custom resources and operators using the Operator SDK!

Creating the Custom Resource Definition (CRD)

In this section, we’ll create the Custom Resource Definition (CRD) for our custom resource. We’ll implement a custom resource named OldPodList that lists all the Pods in a specified namespace that haven't been recreated or updated for a given number of days.

Custom Resource Definition Structure

A CRD is a YAML manifest that defines a new type of resource within the Kubernetes API. It consists of metadata, a specification that defines the structure of the custom resource, and some additional settings such as scope and names.

Here’s the YAML manifest for our OldPodList CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: oldpodlists.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                days:
                  type: integer
                  minimum: 1
                namespace:
                  type: string
                  minLength: 1
  scope: Namespaced
  names:
    plural: oldpodlists
    singular: oldpodlist
    kind: OldPodList
    shortNames:
      - opl

Let’s go through the main components of this CRD:

metadata.name: The unique name of the CRD, composed of the plural form of the custom resource and the group name.
spec.group: The API group for the custom resource. This should be a domain name under your control to avoid conflicts with other CRDs.
spec.versions: A list of versions supported by the custom resource. Each version has a name, a schema, and flags to indicate whether it's served and stored.
spec.scope: The scope of the custom resource. It can be either Namespaced or Cluster. In our case, it's Namespaced, as our custom resource is tied to a specific namespace.
spec.names: The naming configuration for the custom resource, including its plural, singular, kind, and short names.
spec.versions[*].schema.openAPIV3Schema: The schema that defines the structure of the custom resource. It uses the OpenAPI v3 specification to define the properties of the custom resource, their types, and any validation constraints.

In our OldPodList CRD, the custom resource has two properties: days and namespace. The days property is an integer with a minimum value of 1, while the namespace property is a non-empty string. Both properties are part of the spec object.

With the CRD in place, you can save it as a file named oldpodlists.crd.yaml and apply it to your cluster using kubectl apply -f oldpodlists.crd.yaml. This will create the OldPodList custom resource type in your cluster, allowing you to create instances of this resource and manage them using kubectl.

Initializing the Operator Project

In this section, we will use the Operator SDK to initialize a new project for our custom resource. This project will include the necessary files and directories to build and deploy the operator. We’ll also briefly describe the generated files and folders.

Creating a New Operator Project

To initialize a new operator project, open a terminal, navigate to your desired working directory, and run the following command:

operator-sdk init --domain example.com --repo github.com/your-username/oldpodlist-operatoroperator-sdk init --domain example.com --repo github.com/your-username/oldpodlist-operator

Replace your-username with your GitHub username or organization name, and oldpodlist-operator with your preferred repository name. This command will create a new directory named oldpodlist-operator with the required files and folders for your operator.

Generated Files and Folders

The operator-sdk init command generates a basic project structure. Let's take a look at the main files and folders:

config/: Contains Kubernetes manifests and kustomize configuration files to deploy the operator, RBAC rules, and the CRDs.
controllers/: Holds the custom controller implementation for your custom resources.
api/: (Not generated by default) This directory will be created when you define a new custom resource using the operator-sdk create api command. It will contain the Go code for the custom resource types and the CRD manifest.
Dockerfile: The Dockerfile to build the operator's container image.
Makefile: A Makefile with pre-defined targets to build, test, and deploy the operator.
go.mod and go.sum: Go modules files to manage dependencies.
.gitignore: A Git ignore file with common exclusions for Go projects.

With the project structure in place, we can now proceed to implement our custom controller and define our custom resource API.

In this section, we’ll explore the role of a custom controller in managing the custom resource and walk through the process of implementing the custom controller using Go. We’ll highlight important code snippets and explain their purpose.

Role of a Custom Controller

A custom controller is responsible for watching and managing the custom resource. It listens for events (e.g., creation, updates, or deletion) on the custom resource and takes appropriate actions to reconcile the current state of the resource with the desired state specified by the user. Custom controllers are part of the Kubernetes Operator pattern, which aims to automate complex tasks and provide higher-level abstractions for managing applications and infrastructure.

Implementing the Custom Controller for OldPodList

First, let’s create a new API for our custom resource using the Operator SDK. Run the following command in your project directory:

operator-sdk create api --group example --version v1 --kind OldPodList --resource --controller

This command generates the required files and directories in the api/ and controllers/ folders. Now, let's dive into the implementation of the custom controller.

Custom Controller Structure

Open the controllers/oldpodlist_controller.go file. You'll see a skeleton implementation of the custom controller. The primary structure is the OldPodListReconciler struct and its Reconcile method:

The OldPodListReconciler struct embeds the client.Client interface for interacting with the Kubernetes API and holds a logger and a scheme. The Reconcile method is the core of the custom controller, where we'll implement our reconciliation logic.

Reconciliation Logic

The goal of our custom controller is to list all the Pods in the specified namespace that haven’t been recreated or updated for a given number of days. We’ll use the Reconcile method to fetch the OldPodList custom resource, list the Pods in the specified namespace, filter out the ones that are too old, and update the OldPodList status with the list of old Pods.

Here’s a high-level outline of the reconciliation logic:

Fetch the OldPodList custom resource using the req object.
List all the Pods in the specified namespace.
Filter the Pods that haven’t been recreated or updated for more than the specified number of days.
Update the OldPodList status with the list of old Pods.

Lets now proceed with the code. Navigate to the controllers directory and open the oldpodlist_controller.go file. This file contains a skeleton implementation of the custom controller. We'll need to add our logic to watch and manage the old Pods.

First, add the following imports at the top of the file:

import (
 "context"
 "time"

 corev1 "k8s.io/api/core/v1"
 apierrors "k8s.io/apimachinery/pkg/api/errors"
 "k8s.io/apimachinery/pkg/runtime"
 ctrl "sigs.k8s.io/controller-runtime"
 "sigs.k8s.io/controller-runtime/pkg/client"

 examplev1 "github.com/your-username/oldpodlist-operator/api/v1"
)

Replace your-username with your GitHub username or organization name.

Next, update the Reconcile method in the OldPodListReconciler struct with the following logic:

func (r *OldPodListReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
 // Fetch the OldPodList custom resource
 oldPodList := &examplev1.OldPodList{}
 err := r.Get(ctx, req.NamespacedName, oldPodList)
 if err != nil {
  if apierrors.IsNotFound(err) {
   return ctrl.Result{}, nil
  }
  return ctrl.Result{}, err
 }

 // List all Pods in the specified namespace
 podList := &corev1.PodList{}
 err = r.List(ctx, podList, client.InNamespace(oldPodList.Spec.Namespace))
 if err != nil {
  return ctrl.Result{}, err
 }

 // Filter the Pods that haven't been updated for the specified number of days
 threshold := time.Now().AddDate(0, 0, -1*int(oldPodList.Spec.Days))
 oldPods := []corev1.Pod{}
 for _, pod := range podList.Items {
  if pod.CreationTimestamp.Time.Before(threshold) {
   oldPods = append(oldPods, pod)
  }
 }

 // Update the OldPodList's Status with the old Pods
 oldPodList.Status.OldPods = oldPods
 err = r.Status().Update(ctx, oldPodList)
 if err != nil {
  return ctrl.Result{}, err
 }

 return ctrl.Result{}, nil
}

This code fetches the OldPodList custom resource, lists all Pods in the specified namespace, filters the old Pods based on the days property, and updates the custom resource's status with the old Pods.

Finally, update the SetupWithManager method to watch the OldPodList custom resources:

func (r *OldPodListReconciler) SetupWithManager(mgr ctrl.Manager) error {
 return ctrl.NewControllerManagedBy(mgr).
  For(&examplev1.OldPodList{}).
  Complete(r)
}

Building the Operator

In this section, we’ll build the operator’s container image using Docker and discuss how to push the image to a container registry. The container image will include the custom controller implementation that we’ve created for our OldPodList custom resource.

Building the Operator’s Container Image

In this section we will go over building the container image of the operator.

Before building the container image, make sure you’ve installed Docker and that it’s running correctly on your machine. You can check this by running docker --version in your terminal.

To build the container image, navigate to the root of your operator project and run the following command:

make docker-build IMG=<your-image-repo>:<tag>

Replace <your-image-repo> with the repository name in a container registry where you want to store the image, and <tag> with the desired version tag for the image.

For example:

make docker-build IMG=your-username/oldpodlist-operator:v0.1.0

Replace your-username with your container registry username, and oldpodlist-operator with your preferred repository name.

This command uses the Dockerfile in your project directory to build the operator's container image. Once the build is complete, you'll see the newly created image in your local Docker image registry.

Here’s the content of the default Dockerfile generated by the Operator SDK when you run the operator-sdk init command:

# Build the manager binary
FROM golang:1.16 as builder

WORKDIR /workspace
# Copy the Go Modules manifests
COPY go.mod go.mod
COPY go.sum go.sum
# Cache dependencies
RUN go mod download

# Copy the go source
COPY main.go main.go
COPY api/ api/
COPY controllers/ controllers/

# Build
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go

# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
FROM gcr.io/distroless/static:nonroot
WORKDIR /
COPY --from=builder /workspace/manager .
USER nonroot:nonroot

ENTRYPOINT ["/manager"]

Pushing the Container Image to a Registry

After building the operator’s container image, you need to push it to a container registry to make it available for deployment. You can use popular container registries like Docker Hub, Google Container Registry (GCR), Amazon Elastic Container Registry (ECR), or Azure Container Registry (ACR).

Before pushing the image, make sure you’re logged in to your container registry. For example, if you’re using Docker Hub, run:

docker login

Enter your Docker Hub username and password when prompted.

To push the container image, run the following command:

make docker-push IMG=<your-image-repo>:<tag>

Replace <your-image-repo> and <tag> with the same values you used when building the image.

For example:

 make docker-push IMG=your-username/oldpodlist-operator:v0.1.0

This command will push the container image to the specified repository and tag.

Now that the operator’s container image is available in a container registry, we can proceed to deploy the operator to the Kubernetes cluster.

Deploying the Operator to a Kubernetes Cluster

In this section, we’ll discuss how to deploy the OldPodList operator to a Kubernetes cluster using the generated YAML manifests. We'll cover the role of each manifest and guide you through applying them using kubectl.

Kubernetes Manifests

The Operator SDK generates a set of YAML manifests under the config/ directory in your project. These manifests are used to deploy the operator, create the necessary RBAC rules, and manage the custom resources. Here's a brief overview of the main manifests:

config/manager/manager.yaml: This manifest deploys the operator as a Deployment, with a single replica running the container image built from the Dockerfile.
config/rbac/role.yaml: This manifest defines a ClusterRole containing the required permissions for the operator to manage the custom resources and interact with the Kubernetes API.
config/rbac/role_binding.yaml: This manifest binds the ClusterRole to a ServiceAccount used by the operator's Deployment.
config/rbac/service_account.yaml: This manifest creates a ServiceAccount used by the operator's Deployment.
config/crd/bases/: This directory contains the CRD YAML manifests for your custom resources.

Deploying the Operator

To deploy the operator, follow these steps:

Set the container image: Update the config/manager/manager.yaml file to use the container image you built and pushed to the container registry. Replace the image field value with the image repository and tag you used in the previous section:

spec:
  template:
    spec:
      containers:
      - name: manager
        image: <your-image-repo>:<tag>

2. Install the CRDs: Apply the CRD manifest for the OldPodList custom resource to your Kubernetes cluster. From the root of your operator project, run:

kubectl apply -f config/crd/bases/example.com_oldpodlists.yaml

This command will create the OldPodList custom resource type in your cluster.

3. Deploy the operator: To deploy the operator with the necessary RBAC rules and service account, run the following command from the root of your operator project:

kubectl apply -k config/default/

This command uses kubectl with the -k flag to apply the kustomize configuration located in the config/default/ directory. The kustomize configuration includes the manager.yaml, role.yaml, role_binding.yaml, and service_account.yaml manifests.

After running these commands, the OldPodList operator will be deployed to your Kubernetes cluster. You can monitor the status of the operator's Deployment by running:

kubectl get deployments -n <namespace>

Replace <namespace> with the namespace where you deployed the operator. If you used the default kustomize configuration, the namespace will be oldpodlist-operator-system.

With the operator deployed, you can now create and manage OldPodList custom resources in your Kubernetes cluster.

Creating an Instance of the Custom Resource

In this section, we’ll demonstrate how to create an instance of the OldPodList custom resource. We'll provide an example YAML manifest, explain how to apply it using kubectl, and discuss how the operator handles the custom resource. We'll also include a Mermaid diagram to help visualize the flow of the operator's actions.

Example YAML Manifest

Create a file called oldpodlist-in-defaultns.yaml and add the following contents:

apiVersion: example.com/v1
kind: OldPodList
metadata:
  name: oldpodlist-in-defaultns
spec:
  namespace: default
  daysOld: 15

This manifest creates an instance of the OldPodList custom resource named oldpodlist-in-defaultns in the default namespace. It specifies that the operator should list all the pods in the default namespace that have not been recreated or updated for more than 15 days.

Applying the Manifest

To apply the manifest and create an instance of the OldPodList custom resource, run the following command:

kubectl apply -f oldpodlist-in-defaultns.yaml

Operator Handling

Once you’ve applied the manifest, the operator’s controller will detect the creation of the new OldPodList instance. The controller will then query the Kubernetes API to list all pods in the specified namespace and filter those that have not been recreated or updated for more than the specified number of days.

The controller will then update the status field of the OldPodList resource with the list of old pods, along with a timestamp indicating when the operation was performed.

This diagram depicts the sequence of actions taken by the operator when handling an OldPodList instance. The user creates the custom resource, which the Kubernetes API notifies the operator about. The operator then retrieves the list of pods from the Kubernetes API, filters the old pods, and updates the custom resource's status with the results.

Monitoring and Troubleshooting

In this section, we’ll cover how to monitor the logs of the custom controller running within the OldPodList operator. Monitoring logs can help you identify potential issues and understand the behavior of the custom controller. We'll also discuss some common problems that you may encounter and suggest ways to resolve them.

Monitoring Logs

The custom controller’s logs can be found in the operator’s container running as a Deployment in the Kubernetes cluster. To stream the logs, you need to find the pod name for the operator and use kubectl logs to view the logs. Follow these steps:

Find the operator’s pod: List the pods in the namespace where the operator is deployed. If you used the default kustomize configuration, the namespace will be oldpodlist-operator-system. Run the following command:

kubectl get pods -n oldpodlist-operator-system

Look for the pod with a name that starts with oldpodlist-operator-controller-manager.

2. Stream the logs: Replace <operator-pod> with the pod name from the previous step and run the following command:

kubectl logs -f <operator-pod> -n oldpodlist-operator-system -c manager

This command streams the logs from the manager container of the operator pod. You can view the logs to monitor the custom controller's actions and identify potential issues.

Common Issues and Solutions

Here are some common issues you may encounter while working with the custom OldPodList operator and suggestions on how to resolve them:

Failed to list pods or update the custom resource: If you see errors related to listing pods or updating the custom resource, it’s likely an issue with the RBAC permissions. Make sure the operator’s ClusterRole has the necessary permissions to list pods and update the OldPodList custom resources. Double-check the config/rbac/role.yaml file and ensure it includes the correct rules.
Operator not detecting custom resource events: If the custom controller is not reacting to the creation, update, or deletion of OldPodList instances, there might be an issue with the controller's informers or event handlers. Review the custom controller's code and make sure it's set up to watch OldPodList resources and handle their events correctly.
Custom resource status not updating: If the custom resource’s status is not updated with the list of old pods, the custom controller might not be processing the resource correctly. Examine the custom controller’s logic and make sure it’s filtering the pods based on the specified criteria and updating the OldPodList status accordingly.

By monitoring the custom controller’s logs and addressing common issues, you can ensure the smooth operation of the OldPodList operator in your Kubernetes cluster.

Cleaning Up

In this section, we’ll guide you through cleaning up the deployed resources and removing the OldPodList operator from your Kubernetes cluster. It's important to perform these steps if you no longer need the operator or want to redeploy it after making changes to the code.

Removing the Custom Resource Instances

First, delete any OldPodList custom resource instances that you've created. This ensures that the operator is not actively processing any resources when you remove it from the cluster. Run the following command:

kubectl delete oldpodlists.example.com --all

This command deletes all OldPodList instances in the current namespace. If you have created instances in other namespaces, switch to those namespaces and run the command again.

Removing the Operator

To remove the OldPodList operator, along with its associated RBAC rules and service account, run the following command from the root of your operator project:

kubectl delete -k config/default/

This command uses kubectl with the -k flag to delete the kustomize configuration located in the config/default/ directory. The kustomize configuration includes the manager.yaml, role.yaml, role_binding.yaml, and service_account.yaml manifests.

Removing the Custom Resource Definition (CRD)

Finally, delete the OldPodList CRD from your Kubernetes cluster by running:

kubectl delete -f config/crd/bases/example.com_oldpodlists.yaml

This command removes the OldPodList custom resource type from your cluster.

After completing these steps, the OldPodList operator, its associated resources, and custom resource instances will be removed from your Kubernetes cluster. You can now redeploy the operator if needed, or simply leave the cluster without the operator if you no longer require its functionality.

Conclusion

In this blog post, we’ve walked you through developing and deploying a Kubernetes custom resource using the Operator SDK. We covered the essentials of Kubernetes custom resources and operators, set up the development environment, and created a custom resource definition (CRD) to list old pods in a namespace. We then implemented the operator’s controller logic, built and deployed the operator to a Kubernetes cluster, and demonstrated how to create an instance of the custom resource.

By following this tutorial, you’ve gained valuable insights into building custom resources and operators for Kubernetes. This knowledge will enable you to create more advanced operators tailored to your specific needs, further enhancing the capabilities of your Kubernetes clusters.

I encourage you to try the tutorial, experiment with the custom resource and operator, and share your feedback. Your experiences and suggestions can help improve the tutorial and inspire others in the Kubernetes community to build even more powerful custom resources and operators. Happy coding!

References

Here are some references to help you dive deeper into Kubernetes custom resources, operators, and the Operator SDK:

Kubernetes Custom Resources: Official documentation on extending the Kubernetes API using custom resources. https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
Kubernetes Operators: An introduction to operators in Kubernetes and their benefits. https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
Operator SDK: The official GitHub repository of the Operator SDK, which provides tools and libraries to build, test, and package operators. https://github.com/operator-framework/operator-sdk
Operator Framework: Official documentation for the Operator Framework, a toolkit to manage Kubernetes native applications, called operators, in an effective, automated, and scalable way. https://operatorframework.io/
Kubebuilder: An SDK for building Kubernetes APIs using custom resource definitions (CRDs). It is an alternative to the Operator SDK that provides similar functionalities. https://book.kubebuilder.io/
CoreOS Operator Pattern: A blog post that introduces the operator pattern and explains its benefits. https://coreos.com/blog/introducing-operators.html
Writing Kubernetes Operators with Operator SDK: A blog post that provides a detailed introduction to writing Kubernetes operators using the Operator SDK. https://developers.redhat.com/blog/2020/08/28/writing-kubernetes-operators-with-the-operator-sdk/

These references will help you further understand the concepts discussed in this tutorial and expand your knowledge of Kubernetes custom resources and operators.