Testing Kubernetes Controllers with the E2E-Framework
Using the features of the Kubernetes-SIGs/e2e-framework to create end-to-end testing of Kubernetes controllers and operators
In a previous post, I introduced the E2E-Framework, a Kubernetes-SIGs project, for testing components running in a Kubernetes cluster. This post can be considered part two in which I will explore how to use the framework to test custom Kubernetes controllers running on a cluster.
End-to-End testing of a controller
If you are reading this, chances are you have written (or interested in writing) a Kubernetes controller/operator using either Kubernetes-SIGs/Kubebuilder or the Operator SDK. These are both frameworks that you can use to create your own Kubernetes controllers/operators.
This post assumes that you are familiar with the concept of Kubernetes controllers. It does not provide an in-depth coverage of controllers nor does it discuss how to create them.
The Kubebuilder project comes with a tutorial called Building CronJob which demonstrates how to create your own cron job controller (in case, you know, you are not happy with the one that comes with Kubernetes). This blog post shows you how to create end-to-end tests to exercise the components of that controller.
See the Github repository for complete source code listing of e2e-framework tests
Infrastructure setup
Running end-to-end tests of cluster components require several infrastructural steps that must be established before the controller is tested. As you will see below, the e2e-framework provide many constructs to facilitate these setup steps.
Define a Go TestMain
To get started, let’s define a test suite using a Go TestMain function to orchestrate the steps for the test. We will also create an e2e-framework env.Environment variable to define our setup steps:
var (
testEnv env.Environment
)
func TestMain(m *testing.M) {
...
}Setup Kind
Next, we will use the Environment.Setup method to create a new Kind cluster and install a namespace on the cluster:
var (
testEnv env.Environment
namespace = "cronjob"
)
func TestMain(m *testing.M) {
testEnv = env.New()
kindClusterName := "kind-test"
kindCluster := kind.NewCluster(kindClusterName)
testEnv.Setup(
envfuncs.CreateCluster(kindCluster, kindClusterName),
envfuncs.CreateNamespace(namespace),
...
)
}Install prometheus and cert-manager
Next, we will define a custom step function that is used to install prometheus and cert-manager. The step uses the e2e-framework’s wait package to wait for the cert manager deployment to be ready before continuing:
var (
...
certmgrVer = "v1.13.1"
certMgrUrl = fmt.Sprintf("https://github.com/jetstack/cert-manager/releases/download/%s/cert-manager.yaml", certmgrVer)
promVer = "v0.60.0"
promUrl = fmt.Sprintf("https://github.com/prometheus-operator/prometheus-operator/releases/download/%s/bundle.yaml", promVer)
)
func TestMain(m *testing.M) {
...
testEnv.Setup(
func(ctx context.Context, cfg *envconf.Config) (context.Context, error) {
// install prometheus
if p := utils.RunCommand(
fmt.Sprintf("kubectl apply -f %s --server-side", promUrl),
); p.Err() != nil {
return ctx, p.Err()
}
// install cert-manager
if p := utils.RunCommand(
fmt.Sprintf("kubectl apply -f %s", certMgrUrl),
); p.Err() != nil {
return ctx, p.Err()
}
// wait for certmgr deployment to be ready
client := cfg.Client()
if err := wait.For(
conditions.New(client.Resources()).
DeploymentAvailable("cert-manager-webhook", "cert-manager"),
wait.WithTimeout(5*time.Minute),
wait.WithInterval(10*time.Second),
); err != nil {
return ctx, err
}
return ctx, nil
},
)
}Install binary dependencies
The next function define steps to install binary dependencies (kustomize and controller-gen) needed to generate the required source and configuration files for the controller:
var (
...
kustomizeVer = "v5.1.1"
ctrlgenVer = "v0.13.0"
)
func TestMain(m *testing.M) {
...
testEnv.Setup(
...
func(ctx context.Context, cfg *envconf.Config) (context.Context, error) {
// install kubstomize binary
if p := utils.RunCommand(
fmt.Sprintf("go install sigs.k8s.io/kustomize/kustomize/v5@%s", kustomizeVer),
); p.Err() != nil {
return ctx, p.Err()
}
// install controller-gen binary
if p := utils.RunCommand(
fmt.Sprintf("go install sigs.k8s.io/controller-tools/cmd/controller-gen@%s", ctrlgenVer),
); p.Err() != nil {
return ctx, p.Err()
}
return ctx, nil
},
)
}Generate and deploy components
Using the binaries installed in the previous step, we can now define a step function to do the followings:
- Generate the configuration and source files for the controller
- Build and deploy the controller container image to Docker
- Load the built image into Kind
- Deploy the controller components to the local Kind cluster
While it sounds complicated, the e2e-framework makes it easy. Let’s see how that is done:
func TestMain(m *testing.M) {
...
testEnv.Setup(
...
func(ctx context.Context, cfg *envconf.Config) (context.Context, error) {
// gen manifest files
if p := utils.RunCommand(
`controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases`,
); p.Err() != nil {
return ctx, p.Err()
}
// gen api objects
if p := utils.RunCommand(
`controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."`,
); p.Err() != nil {
return ctx, p.Err()
}
// Build the docker image
if p := utils.RunCommand(
fmt.Sprintf("docker build -t %s .", dockerImage),
); p.Err() != nil {
return ctx, p.Err()
}
// Load the docker image into kind
if err := kindCluster.LoadImage(ctx, dockerImage); err != nil {
return ctx, err
}
// Deploy the controller components
if p := utils.RunCommand(
`bash -c "kustomize build config/default | kubectl apply --server-side -f -"`,
); p.Err() != nil {
return ctx, p.Err()
}
// wait for the controller deployment to be ready
client := cfg.Client()
if err := wait.For(
conditions.New(client.Resources()).
DeploymentAvailable("cronjob-controller-manager", "cronjob-system"),
wait.WithTimeout(3*time.Minute),
wait.WithInterval(10*time.Second),
); err != nil {
return ctx, err
}
return ctx, nil
},
)
}Teardown
Lastly, let’s use the e2e-framework Environment.Finish method to define some steps to tear down the infrastructure after all tests are completed.
testEnv.Finish(
// remove cluster components
func(ctx context.Context, cfg *envconf.Config) (context.Context, error) {
utils.RunCommand(fmt.Sprintf("kubectl delete -f %s", promUrl))
utils.RunCommand(fmt.Sprintf("kubectl delete -f %s", certMgrUrl))
utils.RunCommand(`bash -c "kustomize build config/default | kubectl delete -f -"`)
return ctx, nil
},
// delete namespace and destroy cluster
envfuncs.DeleteNamespace(namespace),
envfuncs.DestroyCluster(kindClusterName),
)Launch the test
The last thing to do in TestMain is to trigger the Go test functions in the package. This is done using the Run method:
func TestMain(m *testing.M) {
...
os.Exit(testEnv.Run(m))
}Controller unit tests
Now that we have setup an infrastructure with a Kubernetes cluster, we can use Go test functions to define unit tests for the controller.
But first, let’s define a CronJob struct literal that we will use in the tests:
var (
cronjobName = "cronjob-controller"
cronjob = &cronjobV1.CronJob{
TypeMeta: metaV1.TypeMeta{
APIVersion: "batch.tutorial.kubebuilder.io/v1",
Kind: "CronJob",
},
ObjectMeta: metaV1.ObjectMeta{
Name: cronjobName,
Namespace: namespace,
},
Spec: cronjobV1.CronJobSpec{
Schedule: "1 * * * *",
JobTemplate: batchV1.JobTemplateSpec{
Spec: batchV1.JobSpec{
Template: coreV1.PodTemplateSpec{
Spec: coreV1.PodSpec{
Containers: []coreV1.Container{{
Name: "test-container",
Image: "test-image",
}},
RestartPolicy: coreV1.RestartPolicyOnFailure,
},
},
},
},
},
}
)The test function
Remember, the e2e-framework uses the standard built-in Go test framework. So to express your test, you start by defining a new Go test function:
func TestCron(t *testing.T) {
}Test feature
Next we use the e2e-framework to define a test feature which allows you to express different setup and assessment steps for particular feature or functionality of your code:
func TestCron(t *testing.T) {
feature := features.New("Cronjob Controller")
}Next, using the feature, we will start by setting up a watcher to watch for job pods created by our controller and puts them on channel podCreationSig to be processed later in the test:
func TestCron(t *testing.T) {
podCreationSig := make(chan *coreV1.Pod)
feature := features.New("Cronjob Controller")
feature.Setup(func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
client := cfg.Client()
// setup watcher for creation of job v1.Pods by cronjob-controller
if err := client.Resources(namespace).
Watch(&coreV1.PodList{}).
WithAddFunc(func(obj interface{}) {
pod := obj.(*coreV1.Pod)
if strings.HasPrefix(pod.Name, "cronjob-controller") {
podCreationSig <- pod
}
}).Start(ctx); err != nil {
t.Fatal(err)
}
return ctx
})
...
}Assess CRD installation
Next, lets add an assessment to ensure the CronJob CRD is deployed on the cluster :
func TestCron(t *testing.T) {
...
feature.Assess("CRD installed", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
client := cfg.Client()
apiextensionsV1.AddToScheme(client.Resources().GetScheme())
name := "cronjobs.batch.tutorial.kubebuilder.io"
var crd apiextensionsV1.CustomResourceDefinition
if err := client.Resources().Get(ctx, name, "", &crd); err != nil {
t.Fatalf("CRD not found: %s", err)
}
if crd.Spec.Group != "batch.tutorial.kubebuilder.io" {
t.Fatalf("Cronjob CRD has unexpected group: %s", crd.Spec.Group)
}
return ctx
})
}Assess CronJob creation
In the next assessment, we will create a new instance of a CronJob in the cluster and use the wait package to wait for it to be created by the controller or fail the test if it’s not created within the specified time:
func TestCron(t *testing.T) {
...
feature.Assess("Cronjob creation", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
client := cfg.Client()
cronjobV1.AddToScheme(client.Resources(namespace).GetScheme())
if err := client.Resources().Create(ctx, cronjob); err != nil {
t.Fatalf("Failed to crete cronjob: %s", err)
}
// wait for resource to be created
if err := wait.For(
conditions.New(client.Resources()).
ResourceMatch(cronjob, func(object k8s.Object) bool {
return true
},
),
wait.WithTimeout(3*time.Minute),
wait.WithInterval(30*time.Second),
); err != nil {
t.Fatal(err)
}
return ctx
})
}Assess job pod creation
When the controller creates a new CronJob custom resource, it is expected to also create a pod that will run the scheduled job.
The next assessment uses the podCreationSig channel to block until it receives an instance of the created pod from the watcher (from earlier). If the pod is received within the specified time window, it is tested otherwise the test is failed:
func TestCron(t *testing.T) {
...
feature.Assess("Watcher received pod job", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
select {
case <-time.After(30 * time.Second):
t.Error("Timed out wating for job pod creation by cronjob contoller")
case pod := <-podCreationSig:
t.Log("Pod created by cronjob-controller")
refname := pod.GetOwnerReferences()[0].Name
if !strings.HasPrefix(refname, cronjobName) {
t.Fatalf("Job pod has unexpected owner ref: %#v", refname)
}
}
return ctx
})
}Test the feature
The last step, in our test function, is to use the e2e-framework Environment.Test method to test the defined feature:
func TestCron(t *testing.T) {
...
testEnv.Test(t, feature.Feature())
}Running the test
We can use the standard Go test tool to build and run the test in one command:
go test -v ./testBecause e2e-framework uses Go, running the test will automatically pull down all required package dependencies (no Make tasks required):
go test -v ./e2e-test
go: downloading sigs.k8s.io/e2e-framework v0.3.0
go: downloading k8s.io/api v0.28.3
go: downloading k8s.io/apimachinery v0.28.3
...You can also build a test binary first:
go test -c -o controller.test ./testAnd then run the test:
./controller.test -vConclusion
This blog post showcases some of the features of the Kubernetes-SIGs/e2e-framework that facilitate the end-to-end testing of Kubernetes controllers and operators. Using Kubebuilder’s CronJob controller as an example, this post show how to use the e2e-framework to automate the following:
- Download Kind and setup a cluster
- Install and configure prometheus and cert-manager
- Build and deploy the controller on the cluster
- Define test functions that exercise the controller and its custom resources
The features discussed here are only a small subset of the features supported in the e2e-framework project. Visit the project’s examples directory to explore other features and see why e2e-framework continues to gain traction.
