Raspberry Pi 4 K8S Node
• Mark Eschbach
It has been too long since I’ve written a post, far short of my one post a week target. The pandemic is definitely an indirect factor but really an overly demanding job and responsibilities as a father. Porbably should post another time about it.
Goal and starting place
So I acquired a Raspberry Pi 4 with the intent of using it as a lower power cool k8s node to compliment the two AMD64
nodes I am currently running. As a matter of setting up the Pi I enabled SSH and added the following command line
parameters to enable
cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1
This brings the full kernel command line to the following:
console=serial0,115200 console=tty1 root=PARTUUID=f2d1950f-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1
The system is running in Raspbian ARM7 mode or 32-bit mode. Sounds fine for now but might look into moving to later ARM ABIs supported by the chip.
Installing Docker and Kubernetes
Baesd on the work from github.com/alexellis I should be able to install k8s for Raspbian via the following:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt-get update -q sudo apt-get install -qy kubeadm
So let’s give that a whirrl. Most of these commands a definitely inert, simply adding information. The install took a bit based on the throughput of the SD card I am using. In the future I might consider using my NAS as a faster storage device.
This definitely installed kubeadm correctly. Now to remember how to rename the device and have a node join the cluster.
Fixing the hostname
So there appear to be many people who do so via tradition editing of
/etc/hostname. I know
systemd provides a new
mechanism and I would prefer to do use that. Based on the man page
this should be straight forward:
sudo hostnamectl hostname blueberry….and that failed. Turns out the Raspbian
version I am running,
Raspbian GNU/Linux 10 (buster), uses
set-hostname like so:
sudo hostnamectl set-hostname blueberry.
Joining the Cluster
kubeadm on my on-prem cluster and never remember how to join a node. Luckily ServerLab
has an excellent succient blog post to do so! This effecively boils down to:
root@control-plane# kubeadm token create --print-join-command stdout: kubeadm join control-plane:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash> root@blueberry# kubeadm join control-plane:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>
Unfortunately this resulted in the following
KERNEL_VERSION: 5.4.51-v7l+ OS: Linux CGROUPS_CPU: enabled CGROUPS_CPUACCT: enabled CGROUPS_CPUSET: enabled CGROUPS_DEVICES: enabled CGROUPS_FREEZER: enabled CGROUPS_MEMORY: enabled CGROUPS_PIDS: enabled CGROUPS_HUGETLB: missing [WARNING SystemVerification]: missing optional cgroups: hugetlb error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1 [ERROR Swap]: running with swap on is not supported. Please disable swap [ERROR SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.4.51-v7l+/modules.dep.bin'\nmodprobe: FATAL: Module configs not found in directory /lib/modules/5.4.51-v7l+\n", err: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher
So in order of warnings and errors:
hugetlbare not enabled in the kernel the device is running, so maybe I can enable this in the future?
bridge-nf-call-iptables does not existappears like
iptablesis not setup. Looks like I might need to load
br_netfilter. Unfortunately a simple
modprobedid not work:
blueberry# modprobe br_netfilter modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.4.51-v7l+/modules.dep.bin' modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/5.4.51-v7l+
- I saw a recommendation to run
sudo apt install --reinstall raspberrypi-bootloader raspberrypi-kernelthen restart. This resolved that issue and confirmed the hostname changes stuck properly. From my understanding there is an upgraded kernel version avaiable:
ip_forward– I just went with classic editing
/etc/sysctl.confwhich I should bring under configuration management later. Then I ran
sudo sysctl net.ipv4.ip_forward=1to enable it this boot.
Swapis something I choose to keep enabled right now. I know it is against k8s best practices right now however there is not much memory available on the device. Preferably the node keeps going until I have a chnace to fix it and the kubelet does a decent job tracking memory. This is done by appending
--ignore-preflight-errors Swapto the
I was hoping the CRI would install whatever it needed to run however I received the following error:
[preflight] Running pre-flight checks [preflight] WARNING: Couldn't create the interface used for talking to the container runtime: docker is required for container runtime: exec: "docker": executable file not found in $PATH [WARNING Swap]: running with swap on is not supported. Please disable swap [WARNING SystemVerification]: missing optional cgroups: hugetlb
The command then hung. I am going to break down and installed via the offical Docker instructions.
Takes a while
I run Flannel. It took a while for the node to get the CNI installed and running. After that the other
resources kicked off. Some
daemonset resources are not booting but that is a problem for future Mark.