Kube-green: An operator to reduce CO2 footprint of your kubernetes clusters
Overview
As we all know that kubernetes is a popular container orchestration platform. Many Organisations have already adopted kubernetes platform to run their applications & products.
Typically in any organisation, we usually see people creating kubernetes clusters to do either development work or for testing or for running production workload etc. This is fine to run workloads but keep that in mind cost is directly proportional to the number of clusters that you create.
So question is -> is there a way to reduce the amount that we spend on clusters?
Ans is -> There are multiple best practices that need to be followed to achieve that . One of the best practice is to shutdown the workloads when you dont need them. We can achieve this using kube-green.
What is kube-green
kube-green is a simple k8s addon that automatically shuts down (some of) your resources when you don’t need them. If we shuts down the resources(for ex: pods) underlying computing resources will become free (or available) and then we can leverage the cluster autoscaler to reduce the number of nodes. So this is how we can save lot of money.
Use Case
- Do we need to run pods during weekends?
- Do we need to run pods at night?
If you want to implement any of the above use case then go for kube-green.
How It works?
kube-green stops the following resources.
- deployments
- cronjobs
Note: By default only deployment pods will be stopped. If you want cronjob pods need to stopped you need to explicitly enable it. We will discuss about this scenario later.
Creating Kubernetes cluster
We need kubernetes cluster to test kube-green functionality. You can use any of the kubernetes cluster for example AWS EKS or Google GKE or Azure AKS or minikube or kind etc etc..
I am going to use kind kubernetes cluster.
kind create cluster --name kube-green
Note: if you dont know what is kind & how to create kubernetes cluster using kind then refer to my other blog.
Now lets switch to new context using below command.
kubectl cluster-info --context kind-kube-green
You can verify cluster using any of the below commands.
kubectl get nodes
kubectl get componentstatus
Installing certmanager
We need to install cert manager because kube-green uses certmanager to issue certificates. Execute the below command to install the cert-manager.
kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml
wait until all the pods goes into running state.
kubectl get pods -n cert-manager
Installing kube-green
kube-green can be installed in multiple ways i.e either kubectl apply or kustomize or via operator.
In this demo I am going to use the simplest method i.e kubectl apply. Execute the below command.
kubectl apply -f https://github.com/kube-green/kube-green/releases/latest/download/kube-green.yaml
The above command creates bunch of resources which you can see from the above screenshot. New namespace “kube-green” gets created and controller pod will be deployed in it.
kubectl get pods -n kube-green
The above command also creates a custom resource definition named “sleepinfos.kube-green.com” which can be seen by executing the below command.
kubectl get crds | grep sleep
This CRD will be used to define when the resources will be shutdown and wakeup.
Testing
Before we tell kube-green to shutdown some of the resources. Lets deploy some sample applications.
kubectl create deployment frontend --image purushothamkdr453/frontend --replicas 3
kubectl create deployment backend --image purushothamkdr453/frontend --replicas 3
kubectl get pods
There are currently 6 pods running which are from 2 different deployments.
Scenario-1: Shutting down all the resources
Lets create sleepinfo crd with the below content. create a file named “sleep-info.yaml” with below content.
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
name: sleep-test
spec:
weekdays: "*"
sleepAt: "15:00"
wakeUpAt: "15:05"
timeZone: "Asia/Kolkata"
kubectl apply -f sleep-info.yaml
Following are the attributes specified in CRD.
weekdays — * indicates all days, for specific days you can mention the range 1–5(1 is Monday, 2 is Tuesday etc).
SleepAt -at what time resources should be shutdown
wakeUpAt — at what time resource should be wakeup
timeZone — timezone in which you want to run the sleepinfo logic. you can get the other timezone details here.
Adjust these details according to you.
at 15.00PM IST, pods are shutdown. Screenshot below for reference.
at 15.05PM IST, pods are wakeup. Screenshot below for reference.
lets delete the sleepinfo crd.
kubectl delete -f sleep-info.yaml
Scenario-2: Exclude specific resources from shutting down
Lets try to exclude specific resource(in this case backend deployment resource) from shutting down.
Create a file named “sleepinfo-exclude.yaml” with below content.
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
name: working-hours
spec:
weekdays: "*"
sleepAt: "16:17"
wakeUpAt: "16:22"
timeZone: "Asia/Kolkata"
excludeRef:
- apiVersion: "apps/v1"
kind: Deployment
name: backend
kubectl apply -f sleepinfo-exclude.yaml
kubectl get sleepinfo -A
at 16.17 IST, excluding backend all other resources(pods) are shutdown.
at 16.22 resources which are shutdown are back.
kubectl delete -f sleepinfo-exclude.yaml
Scenario-3: shutting down cron resources
In the previous scenarios we have seen how to shutdown deployment resources. Now lets look at how to suspend cronjobs.
Lets deploy a cronjob by executing below commands.
wget https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/application/job/cronjob.yaml
kubectl apply -f cronjob.yamlkubectl get cronjobs
Cron job creates pods under the good.
kubectl get pods | grep hello
create a file named “sleep-cron.yaml” with below content.
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
name: sleep-cronjobs
spec:
weekdays: "*"
sleepAt: "17:05"
wakeUpAt: "17:10"
timeZone: "Asia/Kolkata"
suspendCronJobs: true
suspendDeployments: false
kubectl apply -f sleep-cron.yaml
kubectl get sleepinfo sleep-cronjobs
we have included 2 new attributes in the above crd. let us take a look at them.
suspendCronJobs -> By default cron jobs will not be suspended i.e default value is false. we need to enable it explicitly by setting .
suspendDeployments -> By default deployments will be suspended i.e default value is true. Just incase if you want to disable it then set it to false.
at 17.05PM IST, cron job will be suspended. Screenshot below for reference.
kubectl get cronjobs
at 17.10PM IST cronjob will be restored. Screenshot below.
Now lets delete the sleepInfo resource.
kubectl delete sleepinfo sleep-cronjobs
Where is the State Stored
how does the kube-green controller knows what to restore & when to restore. Complete state information is stored in a secret where the “SleepInfo” crd has been created. For example
kubectl get secrets
References:
Summary:
In this blog post we have seen what is kube-green and how we can use kube-green to suspend deployments or cronjobs. Feel free to comment if you have any queries.