Simplifying management of RBAC and authentication in Kubernetes

API Server is the interface for managing the cluster and its resources. There are different ways to access it. The API can be accessed by human users, such as administrators/developers, or by Kubernetes service accounts, which are special accounts that can be used by pods.

Basic concepts

In Kubernetes, a service account is a way to give an identity to the processes that run inside a pod. A service account corresponds to a ServiceAccount object in the Kubernetes API.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account # this is the name of the service account
  namespace: default

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: my-service-account # this specifies the service account to use
  containers:
  - name: my-container
    image: my-image

Kubernetes understands the concept of a user, but it does not manage users itself. There is no user API in Kubernetes, and users are expected to be managed by an external system.

Authorization methods:

• Client certificates: This method uses X.509 certificates to authenticate users. The certificates are issued by a trusted certificate authority (CA) and contain the user’s identity and group information. The API server verifies the certificates using the CA’s public key.
• Bearer tokens: The tokens are issued by an external identity provider, such as an OIDC provider and contain the user’s identity and group information. The API server verifies the tokens using the identity provider’s public key.
• Basic authentication: This method uses a username and password to authenticate users. Not recommended.
• and other described here

Client certificate method

Any user who has a valid certificate signed by the CA can authenticate to the cluster. Kubernetes uses the common name field in the certificate’s subject (e.g., “/CN=developer1”) to determine the username of the user. Then, it applies the role based access control (RBAC) rules to check if the user has the necessary permissions to perform a specific operation on a resource.

Group

Besides the username, certificates can also contain other information about the user, such as group membership. Kubernetes can use this information to further refine the authorization process, by applying RBAC rules based on the user’s group (organization) membership. For example, if a user has a certificate with the subject /CN=alice/O=developers, Kubernetes will recognize that the user belongs to the Organization (O=developers). Then, it will check if there are role bindings associated to this group. This way, Kubernetes can enforce more flexible access control policies.

List of steps to create a user certificate, add RBAC roles and bind it to a user:

• Generate a private key for the user (i.e., via openssl)
• Create a certificate signing request (CSR) with the user’s information, such as name and group
• Create a YAML file with the CertificateSigningRequest API object and apply it with kubectl
• Approve the CertificateSigningRequest using kubectl
• Retrieve the signed certificate from the CertificateSigningRequest to .crt file using kubectl
• Create a kubeconfig file with the signed certificate and the private key
• Create a YAML file with the Role or ClusterRole objects and the permissions for the user
• Create a YAML file with the RoleBinding or ClusterRoleBinding object and the bind user or group to a role
• Apply both YAML files to the cluster and test the user’s authorization

Exact commands for every step can be found here.

In order to do extra verification, we can execute:

kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d > client.crt
openssl x509 -in client.crt -noout -subject

The command shall print something like:

subject=CN = alice

For example, decoded certificate for k3s (default configuration):

subject=O = system:masters, CN = system:admin

Wow, it is quite a long procedure to follow, but it can be simplified as presented in the next sections.

Drawbacks of client certificates:

• Client certificates are hard to manage and scale, as they require manual generation, distribution, rotation for each user that needs cluster access.
• Client certificates might be prone to compromise, as they require users to store and handle private keys, which can be leaked or stolen if not done properly.

That creates another problem.

Usually we have some means of certificate revocation, such as CRL or OSCP. CRL (Certificate Revocation List) contains revoked certificates that are no longer trusted by the certificate authority (CA) that issued them. An alternative to CRLs is the certificate validation protocol known as Online Certificate Status Protocol (OCSP). OCSP allows clients to query the CA or a designated responder about the status of a specific certificate, and receive a response indicating whether the certificate is valid or revoked. OCSP has the primary benefit of requiring less network bandwidth and enabling near real-time status checks.

Kubernetes does not support revoking client certificates natively

There is no easy way to revoke user access to a cluster when it is no longer required by them (Kubernetes does not support CRL or OCSP natively). This violates the principle of least privilege, which states that users should only have the minimum level of access required to perform their tasks and nothing more.

One option is to remove the permissions from the user or group associated with the client by deleting or modifying the role binding or the role. However, these methods do not change the fact that the user or client application is presenting the valid certificate to the API server. That may have security risks as exposes a vulnerability for RBAC policies that are poorly written and can enable back doors.

There is an issue in Kubernetes GitHub about supporting the revocation of client certificates that are used to authenticate users to the Kubernetes API server. The issue was opened on Dec, 2015. The issue is still open and has not been resolved yet.

One of the solution to this problem is to use OIDC. Using it in Kubernetes allows you to secure your cluster by authenticating users from an OpenID Connect identity provider, which eliminates the need to create client certificates for each user.

You can use the same identity provider for multiple clusters, which simplifies the user experience and reduces the maintenance overhead. With OIDC, managing users, updating users and deleting them will be much easier.

Scenario where client certificate can be useful

One scenario where certificates can be useful is when there is an emergency situation that requires access to the Kubernetes API server, but the OpenID Connect (OIDC) provider is unavailable (or just not configured yet). For example, this can happen when the cluster is first bootstrapped or when there is a network outage. In such cases, certificates can provide a fallback mechanism to authenticate users to the API server.

Note: a good practice for securing Kubernetes cluster is to make the API server private, so that it is not exposed to the public internet and can only be accessed within a private network.

Good practices for EKS are available here.

OIDC in EKS

If we want to use an alternative to IAM for authenticating users to our EKS cluster, we can use an OIDC compatible identity provider. OIDC is a protocol that adds login and profile information to OAuth 2.0, which allows to use our organization’s existing procedures for managing user accounts. However, we can only use one OIDC identity provider per cluster, and its issuer URL must be publicly accessible for EKS to verify signing keys.

Associating an OIDC identity provider with EKS

We need to provide some configuration parameters that match our identity provider’s settings. These parameters are:

• Issuer URL: This is the URL of the OIDC issuer that allows the API server to discover public signing keys for verifying tokens. The URL must begin with https:// and should correspond to the iss claim in the provider’s OIDC ID tokens.
• Client ID: This is the identifier of the app client that is registered with the OIDC identity provider. The client ID should correspond to the aud claim in the provider’s OIDC ID tokens.
• Username claim: This is an optional parameter that specifies the name of the claim that contains a unique identifier for the user. By default, this is sub, but we can use another claim if we want. For example, we can use email as the value for this parameter. EKS maps this claim to the username attribute of the Kubernetes user.
• Groups claim: This is an optional parameter that specifies the name of the claim that contains an array of group names for the user. For example, we can use groups as the value for this parameter. EKS maps this claim to the groups attribute of the Kubernetes user.
• Username prefix: This is an optional parameter that specifies a prefix that is prepended to username claims to prevent conflicts with other authentication strategies. For example, if we use oidc: as the value for this parameter, a username alice@example.com will be mapped to oidc:alice@example.com in Kubernetes.
• Groups prefix: This is an optional parameter that specifies a prefix that is prepended to groups claims to prevent conflicts with other authentication strategies. For example, if we use oidc: as the value for this parameter, a group name developers will be mapped to oidc:developers in Kubernetes.

Groups claim is a crucial parameter! It determines the user’s group membership in the ID token. Groups are the key to assign different permissions and roles to different types of users using Kubernetes RBAC.

For example, we can create distinct groups for developers, testers, and admins, and grant them different levels of access to our cluster resources. To use groups claim with Cognito as our OIDC identity provider, we must create a group in our Cognito user pool and assign user to it. EKS can map the cognito:groupsclaim in the ID token to the groups attribute of the Kubernetes user. The group(s) that a user belongs to are included in ID token from Cognito in the cognito:groups claim.

For example, to create a group named developers using Terraform, we can use the following resource:

resource "aws_cognito_user_group" "developers" {
  name = "developers"
  user_pool_id = aws_cognito_user_pool.main.id
  role_arn = aws_iam_role.group_role.arn
}

To associate an OIDC identity provider with our cluster named my-cluster, we can use the following Terraform resource:

resource "aws_eks_identity_provider_config" "this" {
  cluster_name = "my-cluster"

  oidc {
    issuer_url    = "<issuer_url>"
    client_id     = "<audience>"
    username_claim = "email"
    groups_claim   = "cognito:groups"
   }
}

Or as an option, we can run the following command:

eksctl associate identityprovider -f identity-provider.yaml

An example indentity-provider.yaml is available here.

OIDC is a feature of the official Kubernetes and to use this functionality, we need to set the flags --oidc-issuer-url, --oidc-client-idor --oidc-groups-claimin the API server.

How to link group from claim to a Kubernetes Role

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

Config above defines a role, which is a way to set permissions within a namespace (default in this case). A role can be assigned to one or more subjects using a RoleBinding. In this case, role can read pods and watch for changes.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader-rolebinding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader
subjects:
- kind: Group 
  name: oidc:developers 
  apiGroup: rbac.authorization.k8s.io

Here, we assign a role (byroleRef) to a set of subjects within a namespace. A subject can be a:

• user,
• group, or
• service account.

The roleRef field specifies the role that the RoleBinding refers to, which is also named pod-reader. The subjects field specifies the subjects that the RoleBinding applies to, which is a group named oidc:developers.

Difference between OIDC provider URL and OIDC authentication in EKS

The OIDC provider URL and OIDC authentication are two different concepts in Amazon EKS cluster configuration.

The OIDC provider URL is the URL of the OIDC issuer associated with your cluster. It allows your cluster to use AWS IAM roles for service accounts. This way, you can provide fine-grained permissions to your pods without using AWS credentials.

The OIDC authentication is a feature that allows you to authenticate users to your cluster from an external OIDC identity provider, such as Cognito, or Google. This way, you can use your existing identity provider to manage access to your cluster resources.

Now that we have covered two authentication methods — client certificates and OIDC identity providers, let’s now address the RBAC management.

Tools for Managing RBAC and Users in Kubernetes

Managing RBAC and users in Kubernetes can be complex and tedious, especially when you have a large number of resources and users with different levels of access. Fortunately, there are some tools that can help with this task, by simplifying the creation, modification, deletion, and auditing of RBAC policies.

There are different types of tools for RBAC in Kubernetes, depending on your needs and use cases. Some types are:

Tools to display permissions: These tools help you to review and audit the roles and permissions of users, service accounts, or groups in your cluster. For example:

• rakkess: A tool to show the access rights of users and service accounts for different Kubernetes resources.
• RBAC Lookup: A tool to find roles attached to any user, service account, or group name in your Kubernetes cluster.
• rback: A tool to visualize the RBAC roles and bindings in a Kubernetes cluster.

Tools to scan and find vulnerable rules: These tools help you to detect and fix potential security problems in your RBAC policies. For example:

• kubeaudit: A tool to check your Kubernetes clusters for common security issues.
• kubernetes-rbac-audit: A tool to audit the RBAC policies in a Kubernetes cluster.
• Popeye: A tool that scans Kubernetes cluster and reports potential issues with deployed configurations. Useful for detecting unused RBAC.

Tools to generate or manage policies: These tools help you to create and update RBAC policies based on various sources and criteria. For example:

• audit2rbac: A tool to create RBAC policies from Kubernetes audit logs.
• Permission Manager by SIGHUP: It allows you to create users, assign roles and permissions, and export kubeconfig file via a user-friendly web UI

Usually tools may serve multiple purposes such as rbac-tool which simplifies querying and creation of RBAC policies.

audit2rbac in k3s

First we need to get audit.logfile to analyse. In k3s, you need to enable audit logging on the k3s server

Step 1: Create an audit policy file

An audit policy file defines the rules for what events should be recorded and what data they should include.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata

This is a simple policy that logs the metadata of all requests. You can customize the policy according to your needs. Save it to /var/lib/rancher/k3s/server/audit.yaml

Step 2: Create an audit log directory

You need to create a directory where the audit logs will be stored. You can create a directory named /var/lib/rancher/k3s/server/logs/ with the following command:

sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/logs

Step 3: Pass flags to the k3s server

You need to pass some flags to the k3s server to enable audit logging and specify the policy file and the log path.

k3s server \
--kube-apiserver-arg='audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log' \
--kube-apiserver-arg='audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml'

Step 4: Verify the audit log

You should be able to see the audit log in. You can use tail or lesscommands to view the log. For example:

tail -f /var/lib/rancher/k3s/server/logs/audit.log

Step 5: Generating RBAC manifests

After performing operations on the API server from <user> account, you can generate RBAC policy by usingaudit.log

audit2rbac -f /var/lib/rancher/k3s/server/logs/audit.log --user <user> > user-roles.yaml

File user-roles.yaml will contain objects Role/ClusterRole and RoleBinding /ClusterRoleBinding objects.

Conclusion

There are different ways to authenticate users to a Kubernetes cluster, such as client certificates and OIDC identity providers.

OIDC identity providers is a better way of managing users than using client certificates.

OIDC identity providers allow you to integrate your existing identity management system with your cluster, without having to create and manage client certificates for each user.

Managing RBAC in Kubernetes can be a challenging task, especially when you have a large number of users and resources. Fortunately, there are some tools that can help you with RBAC, such as audit2rbac or rbac-tool. These tools can help you generate RBAC policies from audit logs, audit existing RBAC policies for compliance and security or visualize RBAC policies in a graph.

Thanks.