Upgrading HomeLab from 1.23 to 1.24
• Mark Eschbach
1.24 was released about a month ago. Normally I am happy to be an early adopter however 1.24.0
has some system changes which also need to occur from my understanding, related to the removal of
from my observations on Twitter the k8s crew was pretty high stress about it when communicating. Understandable since
they probably catch a lot of flack :-( even though they are being very upfront about the change.
is a bridge for migrating Kubernetes from a direct dependency on Docker to using their Container Runtime Interface.
containerd via Docker under the hood. You
can configure the system to continue to do so. However, I am up for the challenge of streamlining towards
and something more interesting in the future!
Distraction: Kata Containers
Someone had the great idea to run containers under Intel’s HyperV setup using Qemu. Looks like
the project eventually rolled into Kata Containers. Looks like an open source
project which promises all the benefits of containers with all the benefits of running VMs. Something I should look
into in the future for my Intel nodes. This came from pulling on the thread for
containerd was the core of Docker, pulled out and donated as the open source core of running containers. The
community has centralized on this environment to run containers. containerd/containerd
does the best job at describing who they are and what they do. Particularly interesting is the operations documentation
pointing out the usage of
/var/lib/containerd which is important if you have a small root disk on some systems…like
one of my nodes.
Pausing to expand the control plane
Long story short: I was running a single node as the control plane. In order to keep the family acceptance factor of my systems high I need to expand the control plane so work loads can shift as I am getting things done.
me@Luft:> kubectl drain bronosaurs --ignore-daemonsets && kubectl delete bronosaurs root@arkhal# kubeadm init phase upload-certs --upload-certs I0622 10:04:46.486015 3938201 version.go:255] remote version is much newer: v1.24.2; falling back to: stable-1.23 [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07 root@bronosaurs# kubeadm reset root@bronosaurs# kubeadm join api-server.k8s.internal:6443 --token terv1c.ys1q6j7wsn01vkqq --control-plane --discovery-token-ca-cert-hash sha256:0e86e637b6428a4a5fd5c08edc78975a477225e3ec54e48db2b6243beecd0fe3 --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07 [omitted 'cause there is a lot] me@Luft:> kubectl drain blueberry --ignore-daemonsets && kubectl delete blueberry root@blueberry# kubeadm reset root@blueberry# kubeadm join api-server.k8s.internal:6443 --token terv1c.ys1q6j7wsn01vkqq --control-plane --discovery-token-ca-cert-hash sha256:0e86e637b6428a4a5fd5c08edc78975a477225e3ec54e48db2b6243beecd0fe3 --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07
This had a hiccup Flannel did not reuse the previous subnets. Required hand editing
/run/flannel/subnet.env to fix.
Raspberry Pi also hit the
unaligned atomic operation which claims to be fixed in 1.24…so I have to roll forward to
get the node to work.
Do the migration dance
Took the plunge on the Pi since it must be upgrade in order to be meet the requirements. Steps
> containerd config default | sudo tee /etc/containerd/config.toml > sudo systemctl restart containerd > kubectl edit node blueberry
You will need to modify the annotation
kubeadm.alpha.kubernetes.io/cri-socket to equal
# apt-mark unhold kubeadm kubectl kubelet # apt-get update && apt-get install -y kubeadm=1.24.2-00 kubectl=1.24.2-00 kubelet=1.24.2-00 # apt-mark hold kubeadm kubectl kubelet # kubeadm upgrade plan # kubeadm upgrade apply 1.24.2
Debugging and other notes:
crictlworks similar to the
/var/lib/kubelet/kubeadm-flags.envneed the following string added: