In a kubernetes cluster (microk8s) the calico pod on the worker node is stuck in state "Init: 0/2"

General Discussions microk8s
1

In a kubernetes cluster the calico pod on the worker node is stuck in state “Init: 0/2” when added to the cluster.

On the control node, however, the calico node pod stays in running state.

The cluster runs on Microk8s v1.31/stable on top of Ubuntu Server 24.04 on both the control node and the worker node.

In the kubernetes dashboard i can see the following error in the failing calico pod:
(combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: failed to mount rootfs component &{overlay overlay [workdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/237259/work upperdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/237259/fs lowerdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs]}: invalid argument: unknown

Why can’t the calico pod be initialised on the worker node?

Can someone please suggest some possible reason for the problem?
Is something missing in the configuration?

2

Do you have CNI installed?

3

Thx for the question.
Calico is the CNI used in microk8s.
As you can see below the calico pod on the worker node is stuck in init state. The question is why.

admin@controlnode:~$ kubectl get pods -n kube-system -o wide
NAME                                         READY   STATUS     RESTARTS      AGE   IP             NODE          NOMINATED NODE   READINESS GATES
calico-kube-controllers-759cd8b574-phwdz     1/1     Running    0             17h   10.1.88.129    controlnode   <none>           <none>
calico-node-8tg5k                            0/1     Init:0/2   0             15h   10.42.0.101    workernode1   <none>           <none>
calico-node-gdzt6                            1/1     Running    0             15h   q.x.y.z        controlnode   <none>           <none>
coredns-7896dbf49-54hdl                      1/1     Running    0             17h   10.1.88.130    controlnode   <none>           <none>
dashboard-metrics-scraper-6b96ff7878-d9rkf   1/1     Running    0             17h   10.1.88.132    controlnode   <none>           <none>
hostpath-provisioner-5fbc49d86c-tzhbq        1/1     Running    1 (15h ago)   17h   10.1.88.134    controlnode   <none>           <none>
kubernetes-dashboard-7d869bcd96-cxrqj        1/1     Running    0             17h   10.1.88.133    controlnode   <none>           <none>
metrics-server-d6f74bb9f-9dwp7               1/1     Running    0             17h   10.1.88.131    controlnode   <none>           <none>
4

Hello, you can take a look at the following link CreatePodSandbox errors in RPI 4 cluster · Issue #1278 · canonical/microk8s · GitHub If it still doesn’t work, I suggest you switch to docker

5

Hi, sorry for the long delay, finally got time to check this out.
I looked at the link you suggested but it doesn’t look like the same problem.
Do you mean I should switch the CNI to docker?

6

dmesg on the worker node contains:
↔
[ 591.782426] overlayfs: upper fs missing required features.
[ 606.778133] overlayfs: upper fs does not support tmpfile.
[ 606.785279] overlayfs: upper fs does not support RENAME_WHITEOUT.
[ 606.785311] overlayfs: failed to set xattr on upper
[ 606.785319] overlayfs: …falling back to redirect_dir=nofollow.
[ 606.785325] overlayfs: …falling back to uuid=null.
[ 606.785332] overlayfs: …falling back to xino=off.
[ 606.785338] overlayfs: upper fs missing required features.
[ 617.781824] overlayfs: upper fs does not support tmpfile.
[ 617.789312] overlayfs: upper fs does not support RENAME_WHITEOUT.
[ 617.789347] overlayfs: failed to set xattr on upper
[ 617.789355] overlayfs: …falling back to redirect_dir=nofollow.
[ 617.789363] overlayfs: …falling back to uuid=null.
[ 617.789371] overlayfs: …falling back to xino=off.
[ 617.789378] overlayfs: upper fs missing required features.
[ 631.771703] overlayfs: upper fs does not support tmpfile.

The worker node boots of TFTP/NFS. Is it possible to configure the required features?