Kubernetes Admission Controllers: Enhance Security and Ensure Compliance

Image source : https://wallpaperaccess.com/kubernetes

Introduction

Kubernetes is a powerful container orchestration platform that allows developers and operators to manage containerized applications with ease. One key aspect of Kubernetes that contributes to its flexibility and extensibility is the use of admission controllers. In this article, we will explore Kubernetes admission controllers, how they work, and how you can use them to enhance the security and management of your Kubernetes clusters.

We’ll also dive into a practical example, where we will create a custom admission controller in Go that blocks users from deploying Persistent Volume Claims (PVCs) larger than 10GB. By the end of this article, you will have a solid understanding of Kubernetes admission controllers and how to build and deploy your own.

Prerequisites

To follow this guide and deploy the admission controller, you need to have the following:

• A working knowledge of Kubernetes and its concepts
• Familiarity with Golang
• Access to a Kubernetes cluster (v1.16 or later)
• kubectl installed and configured to work with your cluster
• Docker installed and configured

Before diving into the content of this article, it’s essential to note that some prior knowledge is necessary to fully understand and follow along with the concepts and examples provided. If you're new to these topics, I recommend first exploring introductory resources on Kubernetes, Go, and Docker before continuing with this article.

What are Kubernetes Admission Controllers?

Kubernetes admission controllers are plugins that govern and enforce how the cluster is used. They act as gatekeepers, intercepting requests to the Kubernetes API server and modifying or rejecting them based on specific rules or policies. There are two types of admission controllers:

1. Validating admission controllers: These controllers validate the request, ensuring it meets specific criteria before allowing it to pass through. They do not modify the request; they only approve or deny it based on the defined rules.
2. Mutating admission controllers: These controllers modify the request, usually by adding or updating specific fields, before allowing it to proceed.

Kubernetes has several built-in admission controllers that cover various aspects of cluster management. However, you can also create custom admission controllers tailored to your specific needs.

Why Use Admission Controllers?

Admission controllers offer several benefits, including:

1. Enforcing best practices and policies for cluster resources
2. Enhancing security by restricting access to specific resources
3. Ensuring compliance with internal or external regulations
4. Managing resources and quota limits

Building a Custom Admission Controller in Go

In this section, I’ll guide you through the process of creating a custom validating admission controller in Go that helps manage PVC sizes in a company that employs dynamic provisioning for developers. Our admission controller will ensure efficient resource allocation and prevent potential storage overallocation by rejecting any PVC creation or update request if the requested size is greater than 10GB. By implementing this admission controller, organizations can strike a balance between empowering developers to dynamically provision storage resources and maintaining control over storage limits to avoid unnecessary expenses and optimize resource utilization.

If you would like to jump directly to deploying the admission controller, you can find the completed application and manifest files at the Github repository here (https://github.com/ashwinphilipgeorge/pvc-admission-controller)

Step 1: Writing the Golang code

First, we’ll write the Go code for the admission controller into file called main.go. This code will define a webhook server that listens for incoming admission review requests and validates the PVCs based on the size constraint.

package main

import (
 "encoding/json"
 "fmt"
 "io/ioutil"
 "log"
 "net/http"

 admission "k8s.io/api/admission/v1"
 corev1 "k8s.io/api/core/v1"
 "k8s.io/apimachinery/pkg/api/resource"
 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/apimachinery/pkg/runtime"
 "k8s.io/apimachinery/pkg/runtime/serializer"
)

var (
 runtimeScheme = runtime.NewScheme()
 codecs        = serializer.NewCodecFactory(runtimeScheme)
 deserializer  = codecs.UniversalDeserializer()
)

func main() {
 mux := http.NewServeMux()
 mux.HandleFunc("/validate", handleValidate)
 log.Fatal(http.ListenAndServeTLS(":443", "/certs/tls.crt", "/certs/tls.key", mux))
}

func handleValidate(w http.ResponseWriter, r *http.Request) {
 var admissionReview admission.AdmissionReview

 body, err := readRequestBody(r)
 if err != nil {
  http.Error(w, fmt.Sprintf("failed to read request: %v", err), http.StatusBadRequest)
  return
 }

 if _, _, err := deserializer.Decode(body, nil, &admissionReview); err != nil {
  http.Error(w, fmt.Sprintf("failed to deserialize request: %v", err), http.StatusBadRequest)
  return
 }

 admissionResponse := validate(admissionReview.Request)
 admissionReview.Response = admissionResponse

 res, err := json.Marshal(admissionReview)
 if err != nil {
  http.Error(w, fmt.Sprintf("failed to marshal response: %v", err), http.StatusInternalServerError)
  return
 }

 w.WriteHeader(http.StatusOK)
 w.Write(res)
}

func validate(req *admission.AdmissionRequest) *admission.AdmissionResponse {
 var pvc corev1.PersistentVolumeClaim

 if err := json.Unmarshal(req.Object.Raw, &pvc); err != nil {
  return &admission.AdmissionResponse{
   UID:     req.UID,
   Allowed: false,
   Result: &metav1.Status{
    Message: fmt.Sprintf("failed to unmarshal PVC object: %v", err),
    Code:    http.StatusBadRequest,
   },
  }
 }

 size := pvc.Spec.Resources.Requests[corev1.ResourceStorage]
 maxSize := resource.MustParse("10Gi")

 if size.Cmp(maxSize) > 0 {
  return &admission.AdmissionResponse{
   UID:     req.UID,
   Allowed: false,
   Result: &metav1.Status{
    Message: "PVC size exceeds 10GB limit",
    Code:    http.StatusForbidden,
   },
  }
 }

 return &admission.AdmissionResponse{
  UID:     req.UID,
  Allowed: true,
 }
}

func readRequestBody(r *http.Request) ([]byte, error) {
 if r.Body == nil {
  return nil, fmt.Errorf("Request body is empty")
 }

 defer r.Body.Close()
 return ioutil.ReadAll(r.Body)
}

Step 2: Dockerization

Create a Dockerfile in the same directory as your main.go file with the following content:

FROM golang:1.19 AS build

WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o pvc-admission-controller

FROM ubuntu

COPY --from=build /src/pvc-admission-controller /app/pvc-admission-controller
RUN chmod +x /app/pvc-admission-controller

EXPOSE 443
CMD ["/app/pvc-admission-controller"]

Build the Docker image using the following command, replacing <your-dockerhub-username> with your Docker Hub username :

docker build -t <your-dockerhub-username>/pvc-admission-controller .

Push the built Docker image to Docker Hub using the following command:

docker push <your-dockerhub-username>/pvc-admission-controller

If you don’t already have a Docker Hub account, sign up for one at Docker Hub.

Step 3: Cert-Manager

Before deploying the custom validating admission controller, we need to install and set up cert-manager in the Kubernetes cluster. Cert-manager is a popular Kubernetes add-on that automates the management and issuance of TLS certificates. It streamlines the process of obtaining, renewing, and using certificates, making it easier to manage webhook server TLS configurations.

Follow the steps in the cert-manager documentation to deploy cert-manager into your cluster.

Step 4: Kubernetes Manifests

With our application containerized and cert-manager ready, we’ll now create the necessary Kubernetes manifests to deploy the admission controller in our cluster. In a newly created directory called manifests , add the following manifest files:

namespace.yaml: To create a dedicated namespace for the admission controller:

apiVersion: v1
kind: Namespace
metadata:
  name: pvc-admission-controller

deployment.yaml: Deploy the admission controller as a Kubernetes Deployment (replacing <your-dockerhub-username> with your Docker Hub username) :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pvc-admission-controller
  namespace: pvc-admission-controller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pvc-admission-controller
  template:
    metadata:
      labels:
        app: pvc-admission-controller
    spec:
      serviceAccountName: pvc-admission-controller
      containers:
        - name: pvc-admission-controller
          image: <your-dockerhub-username>/pvc-admission-controller:latest
          ports:
            - containerPort: 443
              protocol: TCP
          volumeMounts:
            - name: certs
              mountPath: /certs
      volumes:
        - name: certs
          secret:
            secretName: pvc-admission-controller-certs

service.yaml: Expose the admission controller as a Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: pvc-admission-controller
  namespace: pvc-admission-controller
spec:
  selector:
    app: pvc-admission-controller
  ports:
    - protocol: TCP
      port: 443
      targetPort: 443

certs.yaml: With cert-manager installed in the cluster, we can now create an Issuer and Certificate resource to obtain and manage the self-signed TLS certificates for our admission controller webhook server. The certificates will be stored in a Kubernetes Secret called pvc-admission-controller-certs.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: pvc-admission-controller-issuer
  namespace: pvc-admission-controller
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: pvc-admission-controller-certs
  namespace: pvc-admission-controller
spec:
  secretName: pvc-admission-controller-certs
  dnsNames:
    - pvc-admission-controller.pvc-admission-controller.svc
    - pvc-admission-controller.pvc-admission-controller.svc.cluster.local
  issuerRef:
    name: pvc-admission-controller-issuer
  duration: 262800h # 30 years
  renewBefore: 730h # ~1 months

validating-webhook.yaml: Register the custom admission controller as a ValidatingWebhookConfiguration:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pvc-admission-controller
  annotations:
    cert-manager.io/inject-ca-from: pvc-admission-controller/pvc-admission-controller-certs
webhooks:
  - name: pvc-admission-controller.pvc-admission-controller.svc.cluster.local
    rules:
      - apiGroups:
          - ""
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - persistentvolumeclaims
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions: ["v1"]
    clientConfig:
      service:
        name: pvc-admission-controller
        namespace: pvc-admission-controller
        path: "/validate"
      caBundle: Cg==

Time to finally deploy your very own custom admission controller! You can do this by running kubectl apply -f manifests. This will create all the above resources. To delete these resources, you can simply run kubectl delete -f manifests .

Testing the Admission Controller

With the custom admission controller deployed, it’s time to test it. Let’s create two PVCs: one within the allowed size limit and another that exceeds the limit.

Create a file named test-pvc.yaml with the following content:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: small-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: large-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

Now, apply the manifest file using kubectl:

kubectl apply -f test-pvc.yaml

You should see the following output:

persistentvolumeclaim/small-pvc created
Error from server (PVC size exceeds 10GB limit): error when creating "test-pvc.yaml": admission webhook "pvc-admission-controller.example.com" denied the request: PVC size exceeds 10GB limit

As expected, the admission controller allowed the creation of the smaller PVC (5Gi) but rejected the larger PVC (20Gi).

Debugging and Monitoring Admission Controllers

When working with admission controllers, it is essential to keep an eye on logs and metrics for debugging and monitoring purposes. You can access the logs of the custom admission controller using the following command:

kubectl logs -n pvc-admission-controller deploy/pvc-admission-controller

To monitor the admission controller’s performance and resource usage, you can use Kubernetes-native tools like Prometheus and Grafana.

Optimizations

The custom validating admission controller I’ve provided in this article is a basic example that demonstrates the process of implementing and deploying a simple admission controller to enforce PVC size restrictions. However, it’s important to note that there are various optimizations and improvements that can be made to enhance the functionality and security of the controller. Let’s explore some possible optimizations:

Dynamic Quotas

In my example, I hardcoded the maximum PVC size (10GB) directly in the admission controller code. This approach might not be ideal in a production environment, as it lacks flexibility and requires a code change and redeployment to modify the limit. A better approach would be to pass the maximum PVC size as a configuration parameter (e.g., via environment variables or ConfigMaps) to make it easier to adjust the limit without modifying the code.

Namespace-based Quotas

Another optimization to consider is implementing namespace-based quotas. Instead of applying a global limit on PVC size, you can enforce different size limits for PVCs in different namespaces. This approach allows for more granular control over resource allocation and can be useful in multi-tenant clusters, where different teams or applications have different storage requirements.

Advanced Security Practices

My example uses a self-signed Issuer to generate TLS certificates for the webhook server, which may not be the best choice for a production environment. In a production setting, consider using a trusted certificate authority (CA) to issue the certificates, or use an intermediate CA specifically dedicated to signing webhook server certificates.

Additionally, you may want to introduce more advanced security practices, such as implementing role-based access control (RBAC) for the admission controller, setting up network policies to restrict access to the webhook server, or employing Kubernetes security contexts to limit the privileges of the admission controller’s container.

In summary, while our example provides a solid starting point for understanding and deploying a custom validating admission controller, there are numerous optimizations and enhancements that can be made to adapt the controller to specific production environments, improve its flexibility, and enhance its security and performance.

Conclusion

In this article, we learned about Kubernetes admission controllers and their benefits in ensuring a secure and compliant cluster environment. We also walked through the process of creating a custom admission controller in Golang that restricts users from deploying PVCs larger than 10GB.

Admission controllers are a powerful mechanism to enforce policies and best practices in your Kubernetes cluster. By building custom admission controllers, you can tailor the rules to fit your organization’s specific needs and maintain a robust and compliant infrastructure.