Running Nested clusters in Kubernetes using vCluster (K3s in K3s in K8s)

Many organizations have adopted Kubernetes but they face many challenges. One of the main challenges is cost since running K8s in a cloud or on-prem can be so costly in wide teams with many applications, that is what we will be talking about in this blog.

What are Nested Clusters?

Nested means to fit one object inside another. In this blog, we will be having a virtual cluster inside another virtual cluster. Crazy?!! right, follow up to learn more.

Let's take a scenario to understand more about this, Imagine an organization with over 20 projects each having an environment for Development, Testing, and Production. These projects are assigned to different teams to manage them.

One of the main concerns of the Organization is how they will be able to manage each team and ensure Isolation while also making sure their infrastructure is secure.

Why Use Nested Clusters for Kubernetes?

Nested clusters are a powerful way to improve the security, performance, and scalability of Kubernetes workloads. By isolating different workloads, testing new features, and scaling workloads as needed, nested clusters can help organizations to achieve their business goals.

When to Use Nested Clusters

Nested clusters are a good choice for organizations that have a need to:

• Isolate different workloads: Nested clusters can be used to isolate different workloads from each other, such as development, staging, and production. This can help to improve security and performance.
• Test new features: Nested clusters can be used to test new features or applications without affecting the production cluster. This can help to reduce the risk of introducing bugs or breaking changes.
• Scale workloads: Nested clusters can be scaled up or down as needed, which can help to improve scalability.

The Importance of Nested Clusters

Nested clusters are becoming increasingly important as Kubernetes becomes more widely adopted. As organizations deploy more and more Kubernetes workloads, the need for a way to isolate those workloads and scale them as needed becomes more critical.

Nested clusters provide a way to do both of those things, making them an essential tool for any organization that is using Kubernetes.

The Benefits of Nested Clusters

There are a number of benefits to using nested clusters, including:

• Improved security: Nested clusters can help to improve security by isolating different workloads from each other. This can help to prevent unauthorized access to sensitive data.
• Improved performance: Nested clusters can help to improve performance by dedicating resources to each team. This can help to reduce the risk of resource contention.
• Improved scalability: Nested clusters can help to improve scalability by allowing you to scale up or down workloads as needed.

Before we get to the demo let's first understand what virtual clusters are:

Vclusters

A vcluster is a virtual Kubernetes cluster that runs inside an existing Kubernetes cluster, it can either be on any cloud provider or running on-premise. It is a lightweight (K3s) and scalable way to create isolated Kubernetes environments for development, testing, or staging.

The vcluster project was started in 2018 by Loft Labs. The first version of vcluster was released in 2019. Since then, vcluster has been used by a number of organizations, including Google, Red Hat, and IBM.

Why use vclusters:

• Isolation: vclusters are isolated from each other, so they cannot see or access each other’s resources. This helps to ensure that each vcluster has its own dedicated resources, which can help to prevent resource contention.
• Scalability: vclusters can be scaled up or down as needed, which makes them a good fit for workloads that have fluctuating resource requirements.
• Cost-efficiency: vclusters can help to reduce costs by allowing you to share the underlying Kubernetes cluster with multiple vclusters.
• Ease of use: vclusters can be easily created and managed using the vcluster CLI tool.

You can learn more about vClusters here

Demo Project

In this demo project we will be creating a virtual cluster for 3 teams team-A team-B and Team C, We will also be creating the Dev, Test, and Production environment for Team- A. Finally, we will deploy an application.

Prerequisites

1. Kubernetes Cluster
2. Helm installed

Installing vcluster

In order to run vcluster we need first to install the vcluster cli using this binary or find the right one for your OS here

curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster

Check the version of vcluster

vcluster --version

Check the namespaces in our main Cluster

kubectl get ns

Creating each team vcluster

To create a vcluster using the cli

vcluster create team-a

Vcluster created a namespace in the main cluster

Two pods are deployed in the namespace coredns and the vcluster

By default, vclusters run as a single pod (scheduled by a StatefulSet) that consists of 2 containers:

• Control Plane: This container contains API server, controller manager and a connection (or mount) of the data store. By default, vclusters use sqlite as data store and run the API server and controller manager of k3s, which is a certified Kubernetes distribution
• Syncer: It copies the pods that are created within the vcluster to the underlying host cluster. Then, the host cluster will actually schedule the pod and the vcluster will keep the vcluster pod and host cluster pod in sync.

Deploy a vcluster for Team B and Team C

vcluster create team-b
vcluster create team-c

You should have similar namespaces as this

List all the vcluster using the vcluster cli

vcluster list

How to view the Kubeconfig of a vcluster

The vcluster CLI tool will create a secret in the vcluster namespace that contains the kubeconfig

kubectl get secret -n vcluster-team-a 

How to view the Kubeconfig

kubectl get secret -n vcluster-team-a -o yaml

How to connect to the vcluster

You can use the Kubeconfig and share it to teams to connect to the vcluster and create applications or even create another vcluster, You can also connect to the vcluster using the vcluster cli

vcluster connect team-a

This will open another Terminal which is connected to the vcluster.

View the namespaces in the vcluster

kubectl get ns

As we can see there is nothing deployed in the vcluster yet, Next we will be deploying the dev cluster, test cluster, and prod cluster.

Creating different clusters for the environments

From the terminal connected with team-a cluster lets create the development cluster

vcluster create dev

Vcluster has detected that we are creating a vcluster inside another vcluster and prompted us to confirm.

Select Yes so that the vcluster cli can deploy the dev cluster in team-a cluster

Create the other environments using the vcluster cli

vcluster create test
vcluster create prod

View all the environments created

vcluster list

View the namespaces in team-a

kubectl get ns

Deploying podinfo application in team-a Dev enviroment

Before we deploy the application let's first connect to the dev environment

vcluster connect dev

Let's view the namespaces in the dev environment

Let's deploy the podinfo application

Let's check everything running in the dev environment

Podinfo was deployed through a deployment, which had one pod running and a service.

We can view the application via port-forwarding

View the podinfo application

Let's check what is running in Team-a Cluster.

First you need to disconnect from the dev cluster.

vcluster disconnect

Let's check all that is running in the dev namespace.

Here we can see that there are 3 pods running, the vcluster, coredns and the podinfo. There is no deployment because in vcluster only the lower resources are synced to the Main Cluster for scheduling.

Main Cluster

Let's connect back to the main cluster and check what is happening there.

Here we can also see all the pods that are running for the other vclusters and also we can see the Podinfo pod but not the deployment.

Conclusion

In this blog post, we explored the use of nested vclusters in Kubernetes. We created different teams with their own clusters, and then created different environments within those clusters. This demonstrated the high level of isolation that vclusters can provide for large teams. We also deployed the podinfo application in the dev environment for Team A and saw how the pod was synchronized from the dev cluster environment to the team-a cluster environment and finally to the main cluster.

To dive deeper into Loft Labs and its offerings, stay tuned for my upcoming blog post. If you have any inquiries or would like to connect, feel free to reach out via email at onai.rotich@gmail.com or through Twitter or LinkedIn. Together, let’s simplify Kubernetes management and make it more accessible.