It has been too long since I’ve written a post, far short of my one post a week target. The pandemic is definitely an indirect factor but really an overly demanding job and responsibilities as a father. Porbably should post another time about it.

Goal and starting place

So I acquired a Raspberry Pi 4 with the intent of using it as a lower power cool k8s node to compliment the two AMD64 nodes I am currently running. As a matter of setting up the Pi I enabled SSH and added the following command line parameters to enable cgroups via /boot/cmdline.txt:

cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1

This brings the full kernel command line to the following:

console=serial0,115200 console=tty1 root=PARTUUID=f2d1950f-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1

The system is running in Raspbian ARM7 mode or 32-bit mode. Sounds fine for now but might look into moving to later ARM ABIs supported by the chip.

Installing Docker and Kubernetes

Baesd on the work from github.com/alexellis I should be able to install k8s for Raspbian via the following:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update -q
sudo apt-get install -qy kubeadm

So let’s give that a whirrl. Most of these commands a definitely inert, simply adding information. The install took a bit based on the throughput of the SD card I am using. In the future I might consider using my NAS as a faster storage device.

This definitely installed kubeadm correctly. Now to remember how to rename the device and have a node join the cluster.

Fixing the hostname

So there appear to be many people who do so via tradition editing of /etc/hostname. I know systemd provides a new mechanism and I would prefer to do use that. Based on the man page this should be straight forward: sudo hostnamectl hostname blueberry….and that failed. Turns out the Raspbian version I am running, Raspbian GNU/Linux 10 (buster), uses set-hostname like so: sudo hostnamectl set-hostname blueberry.

Joining the Cluster

I use kubeadm on my on-prem cluster and never remember how to join a node. Luckily ServerLab has an excellent succient blog post to do so! This effecively boils down to:

root@control-plane# kubeadm token create --print-join-command
stdout: kubeadm join control-plane:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>
root@blueberry#  kubeadm join control-plane:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Unfortunately this resulted in the following

KERNEL_VERSION: 5.4.51-v7l+
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: missing
	[WARNING SystemVerification]: missing optional cgroups: hugetlb
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
	[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
	[ERROR Swap]: running with swap on is not supported. Please disable swap
	[ERROR SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.4.51-v7l+/modules.dep.bin'\nmodprobe: FATAL: Module configs not found in directory /lib/modules/5.4.51-v7l+\n", err: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

So in order of warnings and errors:

  • hugetlb are not enabled in the kernel the device is running, so maybe I can enable this in the future?
  • bridge-nf-call-iptables does not exist appears like iptables is not setup. Looks like I might need to load br_netfilter. Unfortunately a simple modprobe did not work: 
    blueberry# modprobe br_netfilter
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.4.51-v7l+/modules.dep.bin'
modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.51-v7l+
  • I saw a recommendation to run sudo apt install --reinstall raspberrypi-bootloader raspberrypi-kernel then restart. This resolved that issue and confirmed the hostname changes stuck properly. From my understanding there is an upgraded kernel version avaiable: 5.10.17-v7l+.
  • ip_forward – I just went with classic editing /etc/sysctl.conf which I should bring under configuration management later. Then I ran sudo sysctl net.ipv4.ip_forward=1 to enable it this boot.
  • Swap is something I choose to keep enabled right now. I know it is against k8s best practices right now however there is not much memory available on the device. Preferably the node keeps going until I have a chnace to fix it and the kubelet does a decent job tracking memory. This is done by appending --ignore-preflight-errors Swap to the kubeadm command.

Docker missing?

I was hoping the CRI would install whatever it needed to run however I received the following error:

[preflight] Running pre-flight checks
[preflight] WARNING: Couldn't create the interface used for talking to the container runtime: docker is required for container runtime: exec: "docker": executable file not found in $PATH
	[WARNING Swap]: running with swap on is not supported. Please disable swap
	[WARNING SystemVerification]: missing optional cgroups: hugetlb

The command then hung. I am going to break down and installed via the offical Docker instructions.

Takes a while

I run Flannel. It took a while for the node to get the CNI installed and running. After that the other daemonset resources kicked off. Some daemonset resources are not booting but that is a problem for future Mark.