Self-Hosted Blog Part 3 - OS & Kubernetes

In part 2, we got our hardware ordered and assembled. Now, we need to turn it into a functioning cluster. Rather than re-invent the wheel, I recommend you follow the Ubuntu tutorial for installing Ubuntu on Raspberry Pis.

A few things to note as you go through that tutorial. First, I strongly recommend using Ethernet rather than WiFi, and I didn’t put WiFi in the build of materials. Secondly, I recommend assigning Static IPs to make discovery easier. To do this, you’ll need to modify your netplan configuration. I modified /etc/netplan/50-cloud-init.yaml to look like:

network:
    ethernets:
        eth0:
            dhcp4: false
            optional: true
            addresses: [192.168.68.2/22]
            gateway4: 192.168.68.1
            nameservers:
                addresses: [192.168.68.9, 1.1.1.1, 1.0.0.1]
                search: ["typhoon.org"]
    version: 2

Note, I setup custom DNS settings for my LAN. Note, there are some DNS dependencies. All the hostnames on my network resolve properly. There are many ways to accomplish this. I created static entries in my Pihole DNS server. Once you have all four machines on the network and you can access them with SSH, we can move on to getting MicroK8S up and running.

Ubuntu tutorials once again are great, but I will walk you through setting up K8S myself as there were a few gotchas.

Before we get MicroK8S installed, there are a couple of things we want to change with kernel parameters that I found out the hard way. In Ubuntu for Raspberry Pi, the way to set Kernel parameters at bootup is to modify the contents of /boot/firmware/cmdline.txt. In other versions of Ubuntu, this involves some Grub commands. Took some research to find this out, as sysctl.conf settings were also not taking effect on reboot. In my environment, I prepended the following:

systemd.unified_cgroup_hierarchy=0 ipv6.disable=1 cgroup_enable=memory cgroup_memory=1

Let’s go through each:

  • cgroup_enable=memory cgroup_memory=1 - MicroK8S itself will warn you if these are not set
  • ipv6.disable=1 - Found out the hard way that IPv6 does not work will with MicroK8S.
  • systemd.unified_cgroup_hierarchy=0 - Necessary, temporarily for me, on Ubuntu 21.10 which has cgroup v2 as the default

Prepend these values to /boot/firmware/cmdline.txt. Next, we need to install some kernel modules specifically for Raspberry Pi, which was a hard fought lesson when I rebuilt the cluster. This is a minor Raspberry Pi specific caveat that was not easy to discover. Run:

sudo apt install linux-modules-extra-raspi

Reboot each node. Next, on each node, get MicroK8s installed:

sudo snap install microk8s --classic --channel=latest/stable
sudo snap alias microk8s.kubectl kubectl

Quality of life, you’re going to be using kubectl a lot, so aliasing microk8s.kubectl on the nodes to kubectl saves a bunch of typing. Now, one of the nodes will be the initial leader node and we want to add other nodes to that. On that node, run:

microk8s add-node

This will print out a set of instructions for you to run on another node to add it to the cluster. Node, you’ll need to run microk8s add-node a different time for each node you’re adding to the cluster. This takes a few minutes. After everything is running, run kubectl get nodes and it should look something like:

NAME    STATUS   ROLES    AGE     VERSION
nanny   Ready    <none>   6d16h   v1.21.9-3+a49cacd624c159
billy   Ready    <none>   6d16h   v1.21.9-3+a49cacd624c159
pan     Ready    <none>   6d16h   v1.21.9-3+a49cacd624c159
ian     Ready    <none>   6d16h   v1.21.9-3+a49cacd624c159

All my node names are goat themed. Now that we have all the nodes checking in, we need a few microk8s extensions. Two are critical for our needs: DNS and an ingress controller. Doing this after the nodes have checked in was important, because otherwise they will attempt to resolve DNS using local systemd rather than CoreDNS. From the first node, run:

microk8s enable dns
microk8s enable ingress

This should modify the configurations of all running nodes. If for some reason DNS is not working in your containers, check cat /var/snap/microk8s/current/args/kubelet and you should see these command line parameters at the bottom:

--cluster-domain=cluster.local
--cluster-dns=10.152.183.10

Now, we have all the OS and cluster infrastructure we need in order to host the workload. Next, we need to get a static site created and containerized to get it running in our MicroK8S cluster.

Posts in this Series