Intro
I have a Synology NAS DS920+ which performs excellently, and I love it. Over the years, I have added more and more applications that I self-host. I’m currently hosting:
Vaultwarden for password management
Gitea git repository for hosting my game projects that have larger file requirements
Obsidian LiveSync server
Jellyfin for movie/TV-show streaming
Jupyter Notebook
Immich for sharing photos with friends and family
Keycloak for identity and access management
Paperless-ngx for document management
As I added more and more applications, the memory footprint of these applications has started to increase to a point where I felt like I might want to offload some applications to a separate machine.
Since I plan on completing the Kubernetes Administrator Certification, I thought, why not run my own cluster where I can run these applications? This is why I thought a Raspberry Pi Kubernetes cluster would be an ideal fit.
The Planned Setup
The final setup will have the Synology NAS as the storage solution for the cluster. So all the data will still be stored there. Over time, I will move applications from the NAS to the rasberry pi cluster. The Synology NAS will then still be the entry point for internet traffic, but if the application runs on the cluster, it will forward that traffic to the cluster. The traffic between the NAS and the cluster is going to be HTTPS encrypted.
What You’ll Need
What we’re going to need:
3x Raspberry Pis
3x Raspberry Pi official power supplies
4 port unmanaged network switch depending on the size of the cluster
A power strip
4 short Ethernet cables
A Raspberry Pi cluster acrylic case or similar
Fans for the Raspberry Pis
SD cards like, for example, SanDisk Extreme PLUS 32GB, A1, U3, V30 or similar
Unfortunately we will need the power-supplies and won’t be able to use multi port usb chargers to power the rasberry pis. This is because the usb-chargers cannot deliver the power reliably enough. Think about it. Usb chargers are for charging phones not powering computers, so therefore we need to stick to the cumbersome power supplies.
Flashing the OS
I’m using the Raspberry Pi Imager found here: https://www.raspberrypi.com/software/
Download the application, and flash the os onto the sd cards one by one.
As OS, I use Raspberry Pi OS Lite (64-bit).
Set the hostname to rpi-node-#.local (or use whatever naming scheme you prefer). The hostname needs to be unique, so number the nodes accordingly.
Set the username, password, and enable SSH and pass your public key. This will simplify SSH access and make it more secure – win-win.
Once all SD cards are flashed, insert them into the RPIs and connect them all to power and internet.
Setting Up The Cluster
We’ll set up a K3s cluster with one master node and two workers using k3sup to simplify the process. You can use k3sup or copy-paste the commands manually.
K3s comes with a preconfigured load balancer, ServiceLB, but it exposes node addresses, leading to multiple external IPs, which isn’t ideal. Instead, we’ll use MetalLB for a single external IP, providing real load balancing across nodes.
K3s also includes Traefik, a reverse proxy and load balancer, but we won’t install it at first we’ll install it later and configure it separateley.
Why two load balancers? MetalLB (Layer 4) distributes traffic across IPs, while Traefik (Layer 7) routes traffic based on hostnames (it can do more but for now let’s assume this). Combining MetalLB and Traefik gives a powerful setup with one IP for requests, forwarding them to the appropriate services.
now install k3sup
To set up a K3s cluster is pretty simple, but to ssh into all the machines is still annoying so I created a script that will do all of it for us. The script can be found here:
https://gist.github.com/VincentSchmid/1075eb6d06f2b56bd8e9efac5871e492
You can run the script like so:
The $RPI_1_IP will be the master node and the other two will be the worker nodes. Also you can add more or less the 3 nodes if you like.
What the script does:
It will enable cgroup features on the kernel, required for running containerized applications
It will make sure that sudo can be run without a password. This is unfortunately needed for K3s to install.
It will set the hostnames of the Pis based on the ordering you set
It will reboot the Raspberry Pis
It will check if the operations were successful
It will install K3s server on the master node and then join the worker nodes to the cluster
After the script ran successfully, you should have a working cluster and the kubeconfig should be in the current working directory, which you can copy to ~/.kube/config/
try running:
Setup Load Balancing and a Dashboard
Next, we’re going to install MetalLB, Traefik, and Rancher, a cool Kubernetes management dashboard. You could also go for the Kubernetes dashboard: https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/
Below I’m going to explain in detail how to set up the following apps.
Cert Manager
Cert Manager helps you create certificates and rotate those certificates, but in this setup, we just need it for Rancher to work, and that could be a reason to simply use the Kubernetes dashboard instead of Rancher.
Here is how we install Cert Manager:
values.yaml:
enabled: true
replicaCount: 2
extraArgs:
– –dns01-recursive-nameservers=1.1.1.1:53,1.1.1.1:53
– –dns01-recursive-nameservers-only=true
podDnsConfig:
nameservers:
– “1.1.1.1″
– “9.9.9.9″
Installation steps:
helm install cert-manager jetstack/cert-manager –namespace cert-manager –create-namespace –version v1.15.2 -f values.yaml
MetalLB
For MetalLB to work, it needs some free external IPs. You need to set a free IP range from your local network under spec.addresses:
metallb-config.yaml:
kind: IPAddressPool
metadata:
name: pool
namespace: metallb-system
spec:
addresses:
– 192.168.1.242-192.168.1.250 # set this range to a free range on your network
—
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: l2-advertisement
namespace: metallb-system
spec:
ipAddressPools:
– pool
To make sure these addresses will stay free, you can shrink your local network on your router. I reserved the top 14 IPs on my network for MetalLB, but really, until now I only needed one.
Run the following commands to install MetalLB:
helm repo add metallb https://metallb.github.io/metallb
helm install
metallb metallb/metallb –namespace metallb-system
kubectl apply -f metallb-config.yml
Traefik
Now I’m going to configure Traefik in a way that it can create certificates itself for ingresses. This only works if you use a single Traefik replica, but it will simplify your ingress definitions down the line. You will just need to add an annotation to the ingress for it to work. This Traefik setup will automatically redirect to 443.
In the values.yaml, you will need to set the “metallb.universe.tf/loadBalancerIPs:” annotation to an IP that is within your free IP range you assigned to MetalLB before. This will be the IP Traefik will use as an external IP and which will listen to the requests.
values.yaml:
– “–global.sendanonymoususage=false”
– “–global.checknewversion=false”
additionalArguments:
– “–serversTransport.insecureSkipVerify=true”
– “–log.level=DEBUG”
– “–certificatesresolvers.default.acme.tlschallenge=true”
– “–certificatesresolvers.default.acme.email=your-email@example.com”
– “–certificatesresolvers.default.acme.storage=/data/acme.json”
– “–certificatesresolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory”
deployment:
enabled: true
replicas: 1
annotations: {}
podAnnotations: {}
additionalContainers: []
initContainers: []
ports:
web:
redirectTo:
port: websecure
websecure:
port: 443
tls:
enabled: true
certResolver: default
ingressRoute:
dashboard:
enabled: false
providers:
kubernetesCRD:
enabled: true
ingressClass: traefik-external
allowExternalNameServices: false
allowCrossNamespace: true
kubernetesIngress:
enabled: true
allowExternalNameServices: false
publishedService:
enabled: false
rbac:
enabled: true
service:
enabled: true
type: LoadBalancer
annotations:
metallb.universe.tf/loadBalancerIPs: 192.168.1.242 # one of the MetalLB address DHCP range assigned
resources:
requests:
cpu: “100m”
memory: “50Mi”
limits:
cpu: “300m”
memory: “150Mi”
persistence:
enabled: true
name: data
accessMode: ReadWriteOnce
size: 128Mi
path: /data
storageClass: “local-path”
Now run:
kubectl create namespace traefik
helm install –namespace=traefik traefik traefik/traefik -f values.yaml
Rancher
First, set a rancher password:
Now run:
kubectl create namespace cattle-system
helm install rancher rancher-latest/rancher –namespace cattle-system –set hostname=rancher.cluster.local –set bootstrapPassword=$RANCHER_PASSWORD
Now the Traefik load balancer will listen on the IP specified for requests with the Rancher hostname: rancher.cluster.local. For each service you expose with a hostname, you will need to add a DNS entry in your router or on your local machine’s host file. For example, rancher.cluster.local points to the Traefik load balancer IP. Now if you make the request rancher.cluster.local, it will upgrade the request to HTTPS, and Traefik will forward it to the service.
Log in to Rancher using the password you specified.
Congratulations, now you’ve got a nice starting point and can start deploying applications!
Next Steps
Now that you’ve got a cluster it’s time to think about storage. I’m using the smb csi driver to mount my nas to my containers using storage classes. I’ve also setup argocd and deploy my applications using gitops.
