K8S Cluster as a Service with vCluster: Part1

7 min readJan 8, 2024

Introduction

In this series, I’m excited to talk about “K8S Cluster as a Service”. But what exactly does that mean? Well, simply put, it’s all about creating a system or mechanism that allows us to easily set up multiple K8S clusters, just like we do with Cloud Providers. Let’s explore how we can make this process smoother and more accessible together!

In Part 1, I’ll be delving into the groundwork needed and the essential tools required to set up our system.

Ways to provision a K8S Clusters

As you embark on your K8S journey, you’ll quickly realize there are several ways to provision K8S clusters. Exploring these various methods will give you a broader understanding and help you choose the best approach for your needs.

The Kubernetes playground stands out as the easiest way to get your hands on a cluster. It’s completely free, no payments required! However, it comes with a catch: the cluster it provides is temporary and not suitable for long-term use. Plus, it might not support all the advanced features that Kubernetes offers. Still, it’s a fantastic starting point to explore and learn the basics!

Another approach that offers more flexibility and access to a wider range of Kubernetes features is leveraging tools like Kind, MiniKube, or K3S to set up local K8S clusters. This method allows you to provision clusters using your own system. It’s excellent for local applications, learning and testing, but it’s not recommended for production environments due to its limitations in scalability and reliability.

Absolutely! For heightened security and compliance with regulations, setting up K8S clusters on bare metal systems or VMs is a solid choice. It’s a production-ready approach, offering more control and customization. However, managing and maintaining these clusters can be quite complex and might become challenging in the long run, requiring substantial effort and expertise to keep everything running smoothly.

If you have the budget and prefer a hassle-free approach, Managed K8S Clusters from providers like GCP’s GKE, Azure’s AKS, or AWS’s EKS are fantastic options. They offer fully managed solutions where the provider takes care of cluster maintenance, ensuring they are production-ready and supporting all Kubernetes features. This allows you to focus on your applications while leaving the infrastructure management to the experts.

What is Virtual Cluster ?

Let’s explain what does “Virtual Cluster” mean?

Virtual clusters are fully functional Kubernetes clusters that operate on top of other Kubernetes clusters.
Virtual clusters utilize the worker nodes and networking infrastructure of the host cluster.
They possess their own control plane and schedule all workloads into a designated namespace within the host cluster.
Similar to virtual machines, virtual clusters partition a sole physical cluster into distinct, multiple entities.

Why we need Virtual Cluster?

There are numerous ways we can effectively utilize Virtual Clusters, which might not be feasible with real Clusters.

Kubernetes itself already provides namespaces for multiple environments, they are limited in terms of cluster-scoped resources and control-plane usage:

Cluster-Scoped Resources: Installing an operator in different versions at the same time is not possible within a single cluster (different CRDs). Like installing two different version of ArgoCD in the same cluster.
Shared Kubernetes control plane. This can become problematic when there’s a high volume of objects requiring management by the control plane.

Setting up a K8S Cluster involves a bunch of management and maintenance, making it tough to handle lots of clusters for different needs. Plus, it can get pretty pricey too!

Absolutely! Preview and test environments are like a playground for both Dev and SRE/DevOps teams. For Devs, having their own Virtual Cluster means they can test their services without even breathing on the production or staging setups. No need to fuss with setting up local clusters either!

As an SRE, I’m in Virtual Cluster heaven — I use it day in, day out for testing new tools or tinkering with fresh features like those in ArgoCD. It’s like having a sandbox where I can play around without worrying about breaking anything in the real world!

vCluster

vCluster is like a magic wand — it helps create virtual K8S Clusters right on top of your existing K8S Cluster. This tool can provision various types of K8S Cluster distributions, like K3S, Vanilla K8S, and K0S. It’s like having a cluster creator within your cluster!

When you provision a Virtual Cluster, it’s got all the key Kubernetes components: the API server, controller manager, and a storage backend (think etcd, sqlite, mysql, and so on). And if needed, you can even throw in a scheduler! It’s like having a mini-Kubernetes universe ready to go.

vCluster deploys a hypervisor component within the Virtual Cluster, emulating networking and worker nodes within its virtual environment.

Here, we’ll dive into some comparisons between different types of multi-tenancy or various use cases.

The beauty of Virtual Clusters lies in their isolation — they come with their own control planes and happily share the same nodes or cluster.
When it comes to setting up Virtual Clusters with vCluster, all you really need are basic permissions to deploy Pods, Services, and PVCs. You don’t have to fuss about cluster-level permissions. The user who kicks off the vCluster creation gets to be the all-powerful admin of that Virtual Cluster.
When it comes to costs, having K8S Clusters on top of an existing cluster tends to be a lot more budget-friendly compared to having separate clusters for each tenant or use case.
Since Virtual Clusters operate atop a main cluster, the resources from that host cluster can be shared among these virtual entities. It’s like having a communal pool where resources can be divvied up between Virtual Clusters. Plus, there’s even an option to share resources between different Virtual Clusters or with the main host cluster.
Absolutely, performance is a big deal! When you fire up a Pod in a Virtual Cluster, it’s akin to running it directly in the host cluster — no performance penalties like you’d see with Container-in-Container approaches. Think of it like the Pod in the Virtual Cluster being synced seamlessly to the host cluster, behaving just like any other normal pod. Smooth sailing without any performance hiccups!

I won’t delve into the specifics of how vCluster provisions Virtual Clusters or their operational details here. The vCluster Documentation offers a more comprehensive explanation. I recommend reading it before proceeding to Part 2 of this series.

Provision Virtual Cluster using vCluster

Sounds like an exciting part ahead! Walking through the steps of provisioning a Virtual Cluster using vCluster is going to be a fun ride. Let’s dive in and get those virtual gears turning!

First get that vCluster CLI up and running! Head over to the documentation and follow along with the steps to install it.

Now, let’s work some magic! Run this command to provision our very own “demo” Virtual Cluster right into the current namespace.

vcluster create demo

Once the provisioning works its magic, we’re in! We’ll seamlessly connect to the Virtual Cluster since vCluster takes care of configuring Kubectl. If we ever want to step back into the host cluster:

vcluster disconnect

We can get list of Virtual Cluster:

vcluster list

And we can connect to our Virtual Cluster:

vcluster connect demo

From host Cluster, a Virtual Cluster is just 2 Pods:

❯ kubectl get pods -n demo
NAME                                            READY   STATUS    RESTARTS   AGE
coredns-56d44fc4b4-jqzxs-x-kube-system-x-demo   1/1     Running   0          103s
demo-0                                          2/2     Running   0          2m

Now let’s create a Deployment in our Virtual Cluster:

kubectl create deployment nginx --image=nginx

❯ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
nginx-76d6c9b8c-bdrfx   1/1     Running   0          7s

Let’s check out the host cluster, as we mentioned earlier, where a Pod runs directly. It’s like getting a backstage pass to see the real action happening in the main cluster!

❯ kubectl get pods -n demo
NAME                                            READY   STATUS    RESTARTS   AGE
coredns-56d44fc4b4-jqzxs-x-kube-system-x-demo   1/1     Running   0          6m4s
demo-0                                          2/2     Running   0          6m21s
nginx-76d6c9b8c-bdrfx-x-default-x-demo          1/1     Running   0          92s

We can easily delete Virtual Clusters:

vcluster delete demo

Next Step

In the upcoming post, I’ll be shining the spotlight on our very own system — a mechanism that’s all about deploying Virtual Clusters. Get ready for a closer look at its fantastic features:

Automated Process
User requests for a Virtual Cluster and Admin would approve it
User can select tools that need to be installed on Virtual Cluster
Tools will be installed automatically
Ingress configurations to route traffic outside of Host Cluster to vCluster (Internal / Internet)
User should access cluster using Kubectl

Please feel free to ask me any questions or leave comments about my post. I’m seeking ways to improve my content. Thank you!

Resources

https://www.vcluster.com/