K3s on Raspberry Pi and ClusterHat

Learn how to deploy k3s to a Raspberry Pi cluster with ClusterHat and ClusterCTRL.

Hardware

• 1 x 8GB Raspberry Pi 4
• 4 x Raspberry Pi Zero 2 W
• 1 x ClusterHat v2.5
• 5 x SD Card

Custom OS Images

Because you are using a ClusterHat, you will want to use custom OS images. For this guide you will be using the 64-bit versions of each and the CNAT Lite version for the controller. Enable SSH for each of the images. The username does not have to be pi but the username needs to be the same, across each of the nodes, for this guide. As you burn each of the SD Cards, place each one into their respective device, to keep them organized. The P1 node is the slot that is on the same side as the ethernet controller. The nodes increase, in order, to P4, which is on the same side as the power and HDMI connectors. You can refer to an earlier post, for detailed steps using Raspberry Pi Imager to create the SD Cards.

First Boot

With each SD card inserted into their respective nodes, you can turn on the control node. The ClusterHat will provide power and networking to the additional nodes. Power for each of the nodes can be controlled independently. You can connect to the device over SSH via WIFI, or via ethernet, depending on what you enabled in the previous step.

Connect to the control node and power on P1:

pi@control-node:~ $ clusterctrl on p1

You will see the green LED light up and the node will begin booting up.

Power up the other nodes:

pi@control-node:~ $ clusterctrl on p2 && clusterctrl on P3 && clusterctrl on p4

Update the OS Images

Update each of the images, from the upstream repositories, to get any of the changes made since the OS image was released.

Start with the control node:

pi@control-node:~ $ sudo apt update && sudo apt dist-upgrade -y

Optionally, add the following lines to /etc/hosts, on the control node:

172.19.181.1    p1
172.19.181.2    p2
172.19.181.3    p3
172.19.181.4    p4

You can also use the IP address directly.

Update each of the additional nodes via ssh:

pi@control-node:~ $ ssh p1 'sudo apt update && sudo apt dist-upgrade -y'
pi@control-node:~ $ ssh p2 'sudo apt update && sudo apt dist-upgrade -y'
pi@control-node:~ $ ssh p3 'sudo apt update && sudo apt dist-upgrade -y'
pi@control-node:~ $ ssh p4 'sudo apt update && sudo apt dist-upgrade -y'

At the time of writing this article, an additional change is required for the base OS on each node. This change is needed in order to run containers, as the defaults for cgroups has changed recently.

Add the following configuration to /boot/cmdline.txt on each node and then reboot:

cgroup_memory=1 cgroup_enable=memory

Ready for k3s

The documentation for k3s is incredible. For this guide, you will use another tool called k3sup to deploy the k3s cluster.

First install k3sup:

pi@control-node:~ $ sudo curl -sLS https://get.k3sup.dev | sh

Install the control node of k3sup:

pi@control-node:~ $ k3sup install --ip 172.19.181.254 --user $(whoami)

That process will end with output similar to this:

...
Saving file to: /home/pi/kubeconfig

# Test your cluster with:
export KUBECONFIG=/home/pi/kubeconfig
kubectl config set-context default
kubectl get node -o wide
pi@control-node:~ $

After setting up your context and verifying your control node is ready, add the other nodes to the cluster.

pi@control-node:~ $ k3sup join --ip 172.19.181.1 --server-ip 172.19.181.254 --user $(whoami)
pi@control-node:~ $ k3sup join --ip 172.19.181.2 --server-ip 172.19.181.254 --user $(whoami)
pi@control-node:~ $ k3sup join --ip 172.19.181.3 --server-ip 172.19.181.254 --user $(whoami)
pi@control-node:~ $ k3sup join --ip 172.19.181.4 --server-ip 172.19.181.254 --user $(whoami)

You now have a kubernetes cluster, deployed to about $100 of hardware.

pi@control-node:~ $ kubectl get node
NAME STATUS ROLES AGE VERSION
control-node Ready control-plane,master 25m v1.24.3+k3s1
p1 Ready <none> 20m v1.24.3+k3s1
p2 Ready <none> 20m v1.24.3+k3s1
p3 Ready <none> 22m v1.24.3+k3s1
p4 Ready <none> 19m v1.24.3+k3s1
pi@control-node:~ $

Test deployment

Deploy an ARM64 native Spring Boot application to the cluster:

Create a file, spring-boot-native-pi.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-boot-native-pi
spec:
  selector:
    matchLabels:
      app: springbootnativepi
  replicas: 4
  template:
    metadata:
      labels:
        app: springbootnativepi
    spec:
      containers:
      - name: springbootnativepi
        image: dashaun/spring-pi-native:focal-arm64
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: springbootnativepi
  ports:
    - protocol: TCP
      port: 8080

Verify:

pi@control-node:~ $ kubectl get pods
NAME READY STATUS RESTARTS AGE
spring-boot-native-pi-59f7d7659c-hvlp8 1/1 Running 0 42m
spring-boot-native-pi-59f7d7659c-dp7nn 1/1 Running 0 42m
spring-boot-native-pi-59f7d7659c-sf899 1/1 Running 0 42m
spring-boot-native-pi-59f7d7659c-99vfq 1/1 Running 0 42m
pi@control-node:~ $

Summary

In this guide, you learned how to deploy k3s on top of the custom OS images provided by ClusterHat, with Raspberry Pi.