# Kubernetes cluster - Back-off restarting failed container in kube-system namespace **URL:** \ **Category:** General Discussions\ **Created:** [February 5, 2024, 3:13pm UTC](https://discuss.kubernetes.io/t/kubernetes-cluster-back-off-restarting-failed-container-in-kube-system-namespace/26971 "2024-02-05T15:13:31Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![czezz](https://avatars.discourse-cdn.com/v4/letter/c/54ee81/32.png) [@czezz](https://discuss.kubernetes.io/u/czezz)\ **Post date:** [February 5, 2024, 3:13pm UTC](https://discuss.kubernetes.io/t/kubernetes-cluster-back-off-restarting-failed-container-in-kube-system-namespace/26971/1 "2024-02-05T15:13:32Z")
Cluster information: ```auto Kubernetes version: v1.28.2 Cloud being used: Virtualbox Installation method: Kubernetes Cluster VirtualBox Host OS: Ubuntu 22.04.3 LTS CNI and version: calico CRI and version: containerd://1.7.2 ``` Cluster contains with 1 Master node and 2 Worker nodes. Once cluster is started for a moment (matter of 1-2 minutes since startup) looks good: ```auto lab@master:~$ kubectl -nkube-system get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-kube-controllers-7ddc4f45bc-4qx7l 1/1 Running 12 (2m11s ago) 13d 10.10.219.98 master calico-node-bqlnm 1/1 Running 3 (2m11s ago) 4d2h 192.168.1.164 master calico-node-mrd86 1/1 Running 105 (2d20h ago) 4d2h 192.168.1.165 worker01 calico-node-r6w9s 1/1 Running 110 (2d20h ago) 4d2h 192.168.1.166 worker02 coredns-5dd5756b68-njtpf 1/1 Running 11 (2m11s ago) 13d 10.10.219.100 master coredns-5dd5756b68-pxn8l 1/1 Running 10 (2m11s ago) 13d 10.10.219.99 master etcd-master 1/1 Running 67 (2m11s ago) 13d 192.168.1.164 master kube-apiserver-master 1/1 Running 43 (2m11s ago) 13d 192.168.1.164 master kube-controller-manager-master 1/1 Running 47 (2m11s ago) 13d 192.168.1.164 master kube-proxy-ffnzb 1/1 Running 122 (95s ago) 12d 192.168.1.165 worker01 kube-proxy-hf4mx 1/1 Running 108 (78s ago) 12d 192.168.1.166 worker02 kube-proxy-ql576 1/1 Running 15 (2m11s ago) 13d 192.168.1.164 master kube-scheduler-master 1/1 Running 46 (2m11s ago) 13d 192.168.1.164 master metrics-server-54cb77cffd-q292x 0/1 CrashLoopBackOff 68 (18s ago) 3d21h 10.10.30.94 worker02 ``` However, after some minutes later, pods in kube-system namespace start flapping/crashing. ```auto lab@master:~$ kubectl -nkube-system get po NAME READY STATUS RESTARTS AGE calico-kube-controllers-7ddc4f45bc-4qx7l 1/1 Running 12 (19m ago) 13d calico-node-bqlnm 0/1 Running 3 (19m ago) 4d2h calico-node-mrd86 0/1 CrashLoopBackOff 111 (2m28s ago) 4d2h calico-node-r6w9s 0/1 CrashLoopBackOff 116 (2m15s ago) 4d2h coredns-5dd5756b68-njtpf 1/1 Running 11 (19m ago) 13d coredns-5dd5756b68-pxn8l 1/1 Running 10 (19m ago) 13d etcd-master 1/1 Running 67 (19m ago) 13d kube-apiserver-master 1/1 Running 43 (19m ago) 13d kube-controller-manager-master 1/1 Running 47 (19m ago) 13d kube-proxy-ffnzb 0/1 CrashLoopBackOff 127 (42s ago) 12d kube-proxy-hf4mx 0/1 CrashLoopBackOff 113 (2m17s ago) 12d kube-proxy-ql576 1/1 Running 15 (19m ago) 13d kube-scheduler-master 1/1 Running 46 (19m ago) 13d metrics-server-54cb77cffd-q292x 0/1 CrashLoopBackOff 73 (64s ago) 3d22h ``` It is completely unclear to me what is wrong and by checking pods description I see repeating events: ```auto lab@master:~$ kubectl -nkube-system logs kube-proxy-ffnzb . . . Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Killing 2d20h (x50 over 3d1h) kubelet Stopping container kube-proxy Warning BackOff 2d20h (x1146 over 3d1h) kubelet Back-off restarting failed container kube-proxy in pod kube-proxy-ffnzb_kube-system(79f808ba-f450-4103-80a9-0e75af2e77cf) Normal Pulled 8m11s (x3 over 10m) kubelet Container image "registry.k8s.io/kube-proxy:v1.28.6" already present on machine Normal Created 8m10s (x3 over 10m) kubelet Created container kube-proxy Normal Started 8m10s (x3 over 10m) kubelet Started container kube-proxy Normal SandboxChanged 6m56s (x4 over 10m) kubelet Pod sandbox changed, it will be killed and re-created. Normal Killing 4m41s (x4 over 10m) kubelet Stopping container kube-proxy Warning BackOff 12s (x28 over 10m) kubelet Back-off restarting failed container kube-proxy in pod kube-proxy-ffnzb_kube-system(79f808ba-f450-4103-80a9-0e75af2e77cf) ``` **Note!** This situation does not prevent me to deploy some example deployments (nginx) - it seems to be running stable. Yet, I tried to add metrics-server and this one is crashing (possibly it is related to CrashLoopBackOff pods in kube-system namespace) Any ideas what might be wrong/where else to look to troubleshoot? ---
**Author:** ![fox-md](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/fox-md/32/10613_2.png) [@fox-md](https://discuss.kubernetes.io/u/fox-md)\ **Post date:** [February 5, 2024, 4:10pm UTC](https://discuss.kubernetes.io/t/kubernetes-cluster-back-off-restarting-failed-container-in-kube-system-namespace/26971/2 "2024-02-05T16:10:32Z")
Hi, Try to configure SystemdCgroup for containerd as per official guide: [Container Runtimes | Kubernetes](https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd) ---
**Author:** ![czezz](https://avatars.discourse-cdn.com/v4/letter/c/54ee81/32.png) [@czezz](https://discuss.kubernetes.io/u/czezz)\ **Post date:** [February 5, 2024, 7:51pm UTC](https://discuss.kubernetes.io/t/kubernetes-cluster-back-off-restarting-failed-container-in-kube-system-namespace/26971/3 "2024-02-05T19:51:19Z")
Hi @fox-md, Thank you for reply and the hint. **First** of all on master node, i was missing /etc/containerd/config.toml so I generated it as following: ```auto sudo containerd config default | sudo tee /etc/containerd/config.toml ``` **Secondly** , in /etc/containerd/config.toml I have changed **SystemdCgroup = false** to **SystemdCgroup = true** and then restarted containerd: systemctl restart containerd That put me in the following situation: ```auto lab@master:~$ kubectl -nkube-system get po The connection to the server master:6443 was refused - did you specify the right host or port? lab@master:~$ kubectl get nodes The connection to the server master:6443 was refused - did you specify the right host or port? ``` Then I have reverted the value back to **SystemdCgroup = false** **HOWEVER** , on nodes I keep this value as **SystemdCgroup = true** (otherwise calico and kube-proxy would keep getting into CrashLoopBackOff state). Since then, cluster seems to be runnign quite stable (includeing metrics-server). ```auto lab@master:~$ kubectl -nkube-system get po NAME READY STATUS RESTARTS AGE calico-kube-controllers-7ddc4f45bc-4qx7l 1/1 Running 7 (21m ago) 14d calico-node-c4rxp 1/1 Running 5 (48m ago) 54m calico-node-dhzr8 1/1 Running 6 (21m ago) 14d calico-node-wss6l 1/1 Running 6 71m coredns-5dd5756b68-njtpf 1/1 Running 6 (21m ago) 14d coredns-5dd5756b68-pxn8l 1/1 Running 5 (21m ago) 14d etcd-master 1/1 Running 61 (21m ago) 14d kube-apiserver-master 1/1 Running 37 (21m ago) 14d kube-controller-manager-master 1/1 Running 41 (21m ago) 14d kube-proxy-mgsdr 1/1 Running 5 (49m ago) 54m kube-proxy-ql576 1/1 Running 9 (21m ago) 14d kube-proxy-zl68t 1/1 Running 7 (16m ago) 71m kube-scheduler-master 1/1 Running 40 (21m ago) 14d metrics-server-98bc7f888-xtdxd 1/1 Running 6 (16m ago) 65m ``` I am not sure why this makes difference (Master and Workers are pretty much the same image). Anyway, thank you for the hint 🙂 ps. Also before all the above, I have disabled apparmor (but this might be irrelevant): ```auto sudo systemctl stop apparmor && sudo systemctl disable apparmor ```