1.24 was released about a month ago. Normally I am happy to be an early adopter however 1.24.0 has some system changes which also need to occur from my understanding, related to the removal of dockershim. Honestly from my observations on Twitter the k8s crew was pretty high stress about it when communicating. Understandable since they probably catch a lot of flack :-( even though they are being very upfront about the change.

What is/was dockershim?

dockershim is a bridge for migrating Kubernetes from a direct dependency on Docker to using their Container Runtime Interface. dockershim drives containerd via Docker under the hood. You can configure the system to continue to do so. However, I am up for the challenge of streamlining towards containerd and something more interesting in the future!

Distraction: Kata Containers

Someone had the great idea to run containers under Intel’s HyperV setup using Qemu. Looks like the project eventually rolled into Kata Containers. Looks like an open source project which promises all the benefits of containers with all the benefits of running VMs. Something I should look into in the future for my Intel nodes. This came from pulling on the thread for frakti.

containerd?

containerd was the core of Docker, pulled out and donated as the open source core of running containers. The community has centralized on this environment to run containers. containerd/containerd does the best job at describing who they are and what they do. Particularly interesting is the operations documentation pointing out the usage of /var/lib/containerd which is important if you have a small root disk on some systems…like one of my nodes.

Pausing to expand the control plane

Long story short: I was running a single node as the control plane. In order to keep the family acceptance factor of my systems high I need to expand the control plane so work loads can shift as I am getting things done.

me@Luft:> kubectl drain bronosaurs --ignore-daemonsets && kubectl delete bronosaurs
root@arkhal# kubeadm init phase upload-certs --upload-certs
I0622 10:04:46.486015 3938201 version.go:255] remote version is much newer: v1.24.2; falling back to: stable-1.23
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07
root@bronosaurs# kubeadm reset
root@bronosaurs# kubeadm join api-server.k8s.internal:6443 --token terv1c.ys1q6j7wsn01vkqq  --control-plane --discovery-token-ca-cert-hash sha256:0e86e637b6428a4a5fd5c08edc78975a477225e3ec54e48db2b6243beecd0fe3  --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07
[omitted 'cause there is a lot]
me@Luft:> kubectl drain blueberry --ignore-daemonsets && kubectl delete blueberry
root@blueberry# kubeadm reset
root@blueberry# kubeadm join api-server.k8s.internal:6443 --token terv1c.ys1q6j7wsn01vkqq  --control-plane --discovery-token-ca-cert-hash sha256:0e86e637b6428a4a5fd5c08edc78975a477225e3ec54e48db2b6243beecd0fe3  --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07

This had a hiccup Flannel did not reuse the previous subnets. Required hand editing /run/flannel/subnet.env to fix. Raspberry Pi also hit the unaligned atomic operation which claims to be fixed in 1.24…so I have to roll forward to get the node to work.

Do the migration dance

Took the plunge on the Pi since it must be upgrade in order to be meet the requirements. Steps

> containerd config default | sudo tee /etc/containerd/config.toml
> sudo systemctl restart containerd
> kubectl edit node blueberry

You will need to modify the annotation kubeadm.alpha.kubernetes.io/cri-socket to equal unix:///run/containerd/containerd.sock.

# apt-mark unhold kubeadm kubectl kubelet
# apt-get update && apt-get install -y kubeadm=1.24.2-00 kubectl=1.24.2-00 kubelet=1.24.2-00
# apt-mark hold kubeadm kubectl kubelet
# kubeadm upgrade plan
# kubeadm upgrade apply 1.24.2

Debugging and other notes:

  • crictl works similar to the docker command.
  • Sometimes /var/lib/kubelet/kubeadm-flags.env need the following string added: --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock
References:
  • https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/_print/