Validating Admission Policies with Kubernetes (GA in 1.30)

Updated on May 3rd, 2023 — Update from Kubernetes 1.28 as beta to Kubernetes 1.30 as GA.

Updated on August 16th, 2023 — Update from Kubernetes 1.26 as alpha to Kubernetes 1.28 as beta.

Kubernetes 1.26 (in alpha), Kubernetes 1.28 (in beta) and Kubernetes 1.30 (in GA) introduced a new feature: Validating Admission Policies.

Validating admission policies use the Common Expression Language (CEL) to offer a declarative, in-process alternative to validating admission webhooks.

CEL was first introduced to Kubernetes for the Validation rules for CustomResourceDefinitions. This enhancement expands the use of CEL in Kubernetes to support a far wider range of admission use cases.

This enhancement avoids much of this complexity of admission webhooks by embedding CEL expressions into Kubernetes resources instead of calling out to a remote webhook binary.

If you want to learn more about this feature and where it’s coming from, I encourage you to watch Joe Betz’s sessions at KubeCon NA 2022 or KubeCon EU 2023. I found them very insightful.

In this blog article, let’s see in actions how we could leverage this new Validating Admission Policies feature. Here is what will be accomplished throughout this blog article:

• Create a Kubernetes cluster with the Validating Admission Policies feature
• Create a simple policy with max of 3 replicas for any Deployments
• Pass parameters to the policy
• Exclude namespaces from the policy
• Limitations, gaps and thoughts
• Conclusion

Note: while testing this feature by leveraging its associated blog and doc, it was also the opportunity for me to open my first PRs in the kubernetes/website repo to fix some frictions I faced: https://github.com/kubernetes/website/pull/38893 and https://github.com/kubernetes/website/pull/38908.

Create a Kubernetes cluster with the Validating Admission Policies feature

With GKE since May 1st in the Rapid channel, you can now create a cluster in 1.30:

gcloud container clusters create-auto gke-vap \
    --location northamerica-northeast1 \
    --release-channel rapid \
    --cluster-version 1.30.0-gke.1167000

With Kind since May 13th, you can now create a cluster in 1.30:

kind create cluster \
    --image kindest/node:v1.30.0

Once the cluster is provisioned, we can check that the Validating Admission Policies feature is availabe with the two associated resources ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding:

kubectl api-resources \
    | grep ValidatingAdmissionPolicy

validatingadmissionpolicies                             admissionregistration.k8s.io/v1     false        ValidatingAdmissionPolicy
validatingadmissionpolicybindings                       admissionregistration.k8s.io/v1     false        ValidatingAdmissionPolicyBinding

Before jumping in creating and testing the policies, let’s deploy a sample app in our cluster that we could leverage later in this blog:

kubectl create ns sample-app
kubectl create deployment sample-app \
    --image=nginx \
    --replicas 5 \
    -n sample-app

Create a simple policy with max of 3 replicas for any Deployments

Let’s do it, let’s deploy our first policy!

This policy is composed by one ValidatingAdmissionPolicy defining the validation with the CEL expression and one ValidatingAdmissionPolicyBinding binding the policy to the appropriate resources in the cluster:

cat << EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: max-replicas-deployments
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["apps"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["deployments"]
  validations:
    - expression: "object.spec.replicas <= 3"
EOF
cat << EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: max-replicas-deployments
spec:
  policyName: max-replicas-deployments
  validationActions:
    - Deny
EOF

Now, let’s try to deploy an app with 5 replicas:

kubectl create deployment nginx \
    --image=nginx \
    --replicas 5

We can see that our policy is enforced, great!

error: failed to create deployment: deployments.apps "nginx" is forbidden: ValidatingAdmissionPolicy 'max-replicas-deployments' with binding 'max-replicas-deployments' denied request: failed expression: object.spec.replicas <= 3

So that’s for new admission requests, but what about our existing app we previously deployed? Interestingly, there is nothing telling me that my existing resources are not compliant. Nonetheless, kubectl rollout restart deployments sample-app -n sample-app or kubectl scale deployment sample-app --replicas 6 -n sample-app for example will fail, like expected.

Pass parameters to the policy

With the policy we just created we hard-coded the number of replicas we allow, but what if you want to have this more customizable? Here comes a really interesting feature where you can pass parameters!

The paramKind field allows you to pass an existing CRD that you could create by yourself or you can easily leverage existing ones like ConfigMap or Secrets. Let’s update our policy with a ConfigMap to achieve this:

cat << EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: max-replicas-deployments
spec:
  failurePolicy: Fail
  paramKind:
    apiVersion: v1
    kind: ConfigMap
  matchConstraints:
    resourceRules:
    - apiGroups:   ["apps"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["deployments"]
  validations:
  - expression: "params != null"
    message: "params missing but required to bind to this policy"
  - expression: "has(params.data.maxReplicas)"
    message: "params.data.maxReplicas missing but required to bind to this policy"
  - expression: "object.spec.replicas <= int(params.data.maxReplicas)"
    messageExpression: "'object.spec.replicas must be no greater than ' + string(params.data.maxReplicas)"
    reason: Invalid
EOF
kubectl create ns policies-configs
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: max-replicas-deployments
  namespace: policies-configs
data:
  maxReplicas: "3"
EOF
cat << EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: max-replicas-deployments
spec:
  paramRef:
    name: max-replicas-deployments
    namespace: policies-configs
    parameterNotFoundAction: Deny
  policyName: max-replicas-deployments
  validationActions:
    - Deny
EOF

Note: because ConfigMap has to reside in a namespace, we are also creating a dedicated policies-configs namespace.

Now, let’s try to deploy an app with 5 replicas:

kubectl create deployment nginx \
    --image=nginx \
    --replicas 5

We can see that our policy is still enforced with a new message, great!

error: failed to create deployment: deployments.apps "nginx" is forbidden: ValidatingAdmissionPolicy 'max-replicas-deployments' with binding 'max-replicas-deployments' denied request: object.spec.replicas must be no greater than 3

Exclude namespaces from the policy

One interesting feature with policies is the ability to exclude namespaces per policy. Very helpful to avoid breaking clusters with policies on system namespaces for example.

Here, we will use a namespaceSelector on our ValidatingAdmissionPolicyBinding to exclude system namespaces as well as our own allow-listed namespace:

cat << EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: max-replicas-deployments
spec:
  paramRef:
    name: max-replicas-deployments
    namespace: policies-configs
    parameterNotFoundAction: Deny
  policyName: max-replicas-deployments
  validationActions:
    - Deny
  matchResources:
    namespaceSelector:
      matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: NotIn
        values:
        - kube-node-lease
        - kube-public
        - kube-system
        - allow-listed
EOF

Note: in order to have this namespaceSelector expression working, we are assuming that we are in a Kubernetes cluster version 1.22+ which automatically adds the kubernetes.io/metadata.name label on any Namespaces. Very convenient for our use case to exclude namespaces from a policy.

Now, let’s try to deploy an app with 5 replicas in the default namespace:

kubectl create deployment nginx \
    --image=nginx \
    --replicas 5

We can see that our policy is still enforced, great!

error: failed to create deployment: deployments.apps "nginx" is forbidden: ValidatingAdmissionPolicy 'max-replicas-deployments' with binding 'max-replicas-deployments' denied request: object.spec.replicas must be no greater than 3

On the other hand, we are able to deploy it in the allow-listed namespace:

kubectl create ns allow-listed
kubectl create deployment nginx \
    --image=nginx \
    --replicas 5 \
    -n allow-listed

namespace/allow-listed created
deployment.apps/nginx created

Sweet! And that’s it for the demos!

If you want to see another example, I gave enforcing a label on a Namespace a try in there: Only one label to improve your Kubernetes security posture, with the Pod Security Admission (PSA) — just do it!.

Limitations, gaps and thoughts

Based on my experience with Gatekeeper policies or Kyverno; and the quick tests that I have done so far with this Validation Admission Policies feature, here are some limitations, gaps and thoughts that I’m seeing:

• Failure policy Ignore — I don’t seem to understand yet what this failure policy Ignore as opposed to Fail does. I get the same behavior with both…
• Just for admission — It’s just for admission, not for evaluating existing resources already in a cluster.
• Client-side validation — Not able to evaluate the resources against policies outside of a cluster, like we can do with the gator or the kyverno CLIs. Update on May 30th 2023: the kyverno CLI can validate Validating Admission Policies, wow!
• Inline parameters — Inline parameters in ValidatingAdmissionPolicyBinding would be way more easier, today we need to create our own CRD or have resources like ConfigMap, Secrets, etc.
• Cluster-wide exempted namespaces — Repeating the namespaceSelector expression for all the ValidatingAdmissionPolicyBinding could generate more work and errors, exempting namespaces cluster-wide would be really great.
• Mutating — Mutating is not possible today.
• Container image signature verification — Advanced scenario like leveraging cosign with Kyverno, OPA Gatekeeper or Sigstore’s Policy Controller doesn’t exist.
• Referential constraints — Advanced scenario where I want a policy making sure that any Namespace has a NetworkPolicy or an AuthorizationPolicy, based on referential constraints with Gatekeeper could be really helpful. It doesn’t seem to be supported today.
• Workload resources — Policies on Pods are important but could be tricky with the workload resources generating them, think about Deployments, ReplicaSet, Jobs, Daemonsets, etc. I haven’t tested it yet.

Conclusion

We were able to create our own policy, pass a parameter to it and exclude some namespaces. Finally, some limitations and gaps in comparison to Admission Controllers, like OPA Gatekeeper or Kyverno, were discussed.

This feature now in GA in Kubernetes 1.30 is really promising, very easy to leverage and yet powerful. I really like the fact that it’s also out of the box in Kubernetes and that it’s a very light footprint in my cluster as opposed to have others CRDs/containers in my cluster like we have with Gatekeeper or Kyverno.

I think this image below taken from Joe Betz’s session at KubeCon NA 2022 is a good summary about the positioning of this feature versus the advanced scenarios covered by webhooks:

Curious to know when to use OPA Gatekeeper or Kyverno in comparison to this new upstream feature in Kubernetes? Good questions, here are 2 resources explaining how they could be complementery used:

• Validating Admission Policies in Kyverno — by Mariam Fahmy
• Securing 900 Kubernetes Clusters Without PSP — Mercedes-Benz’ Journey to ValidatingAdmissionPolicies — Tobias Giese & Tjark Rasche, Mercedes-Benz

I’m really looking forward to seeing the next iterations on this feature as it has already reach the GA state.

More resources

• Only one label to improve your Kubernetes security posture, with the Pod Security Admission (PSA) — just do it!
• Kubernetes Validating Admission Policies: A Practical Example | Kubernetes
• Validating Admission Policies in Kyverno | by Mariam Fahmy | Medium
• Kubernetes 1.30: Validating Admission Policy Is Generally Available | Kubernetes
• Securing 900 Kubernetes Clusters Without PSP — Mercedes-Benz’ Journey to ValidatingAdmissionPolicies — Tobias Giese & Tjark Rasche, Mercedes-Benz

Originally posted at mathieu-benoit.github.io.