Also seeing that the calico pod is going into CrashLoopBackOff
Cluster information:
Kubernetes version: v1.25.3
Cloud being used: (put bare-metal if not on a public cloud) AWS
Installation method: kubeadm
Host OS: Ubuntu 22.04.1 LTS
CNI and version: docker.io/calico/node:v3.24.4
CRI and version:
You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.
buntu@ip-10-0-30-11:~$ kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-f79f7749d-pvkh8 1/1 Running 0 2d4h 192.168.37.195 ip-10-0-30-11 <none> <none>
calico-node-nm2bt 0/1 Running 0 2d4h 10.0.30.11 ip-10-0-30-11 <none> <none>
calico-node-s6rkv 0/1 CrashLoopBackOff 553 (4m29s ago) 2d2h 10.0.27.233 ip-10-0-27-233 <none> <none>
coredns-565d847f94-6wt69 1/1 Running 0 2d4h 192.168.37.194 ip-10-0-30-11 <none> <none>
coredns-565d847f94-mh8zz 1/1 Running 0 2d4h 192.168.37.193 ip-10-0-30-11 <none> <none>
etcd-ip-10-0-30-11 1/1 Running 786 2d4h 10.0.30.11 ip-10-0-30-11 <none> <none>
kube-apiserver-ip-10-0-30-11 1/1 Running 755 2d4h 10.0.30.11 ip-10-0-30-11 <none> <none>
kube-controller-manager-ip-10-0-30-11 1/1 Running 823 2d4h 10.0.30.11 ip-10-0-30-11 <none> <none>
kube-proxy-r6xbf 1/1 Running 0 2d4h 10.0.30.11 ip-10-0-30-11 <none> <none>
kube-proxy-td68k 1/1 Running 470 (6m18s ago) 2d2h 10.0.27.233 ip-10-0-27-233 <none> <none>
kube-scheduler-ip-10-0-30-11 1/1 Running 852 2d4h 10.0.30.11 ip-10-0-30-11 <none> <none>```
NAME READY STATUS RESTARTS AGE
hello-node-7f48bfb94f-2qprl 1/1 Running 0 58s
my-nginx-77d5cb496b-2vk5w 0/1 CrashLoopBackOff 426 (79s ago) 44h
my-nginx-77d5cb496b-l56mx 1/1 Running 0 44h
test2-749b6d69bf-9669z 0/1 CrashLoopBackOff 6 (78s ago) 19m
testl-7f545b6db7-qh5pv 1/1 Running 5 (3m10s ago) 20m.
21m Normal Pulled pod/testl-7f545b6db7-qh5pv Successfully pulled image "httpd" in 107.625376ms
13m Warning BackOff pod/testl-7f545b6db7-qh5pv Back-off restarting failed container
14m Normal Pulled pod/testl-7f545b6db7-qh5pv Successfully pulled image "httpd" in 119.498692ms
22m Normal SuccessfulCreate replicaset/testl-7f545b6db7 Created pod: testl-7f545b6db7-qh5pv
22m Normal ScalingReplicaSet deployment/testl Scaled up replica set testl-7f545b6db7 to 1
ubuntu@ip-10-0-30-11:~$ kubectl describe pods -n kube-system calico-node-s6rkv
Name: calico-node-s6rkv
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Service Account: calico-node
Node: ip-10-0-27-233/10.0.27.233
Start Time: Mon, 07 Nov 2022 14:48:05 +0000
Labels: controller-revision-hash=95b799fb5
k8s-app=calico-node
pod-template-generation=1
Annotations: <none>
Status: Running
IP: 10.0.27.233
IPs:
IP: 10.0.27.233
Controlled By: DaemonSet/calico-node
Init Containers:
upgrade-ipam:
Container ID: containerd://9da5bb2a6874e65507bbc5785338b828d4b833c6831a94d1e176d4658d93cc16
Image: docker.io/calico/cni:v3.24.4
Image ID: docker.io/calico/cni@sha256:ac80dd5309268d8aeda72d80d42f6ea854cfe78b7323e4f31825df1bcb69e0b4
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/calico-ipam
-upgrade
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 09 Nov 2022 16:55:49 +0000
Finished: Wed, 09 Nov 2022 16:55:49 +0000
Ready: True
Restart Count: 15
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
/var/lib/cni/networks from host-local-net-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-58cnc (ro)
install-cni:
Container ID: containerd://4e159d51d3d062473e00a4a90074d8df660de9518c0aab4d781dec8708377b1e
Image: docker.io/calico/cni:v3.24.4
Image ID: docker.io/calico/cni@sha256:ac80dd5309268d8aeda72d80d42f6ea854cfe78b7323e4f31825df1bcb69e0b4
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/install
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 09 Nov 2022 16:55:50 +0000
Finished: Wed, 09 Nov 2022 16:55:51 +0000
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
CNI_CONF_NAME: 10-calico.conflist
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
SLEEP: false
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-58cnc (ro)
mount-bpffs:
Container ID: containerd://cf02fe2cbb56222e285f29266cf4e040da65e1472fb7fa251d9936bc07401613
Image: docker.io/calico/node:v3.24.4
Image ID: docker.io/calico/node@sha256:68dd802f1f6d1c9e564950d72757b275bb8755bced64faaaa004b2979f8f38bc
Port: <none>
Host Port: <none>
Command:
calico-node
-init
-best-effort
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 09 Nov 2022 16:55:51 +0000
Finished: Wed, 09 Nov 2022 16:55:51 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/nodeproc from nodeproc (ro)
/sys/fs from sys-fs (rw)
/var/run/calico from var-run-calico (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-58cnc (ro)
Containers:
calico-node:
Container ID: containerd://fe5420c8e88453d84013c687455571dae85e4521b89ae81d92ceda3aae24e4bd
Image: docker.io/calico/node:v3.24.4
Image ID: docker.io/calico/node@sha256:68dd802f1f6d1c9e564950d72757b275bb8755bced64faaaa004b2979f8f38bc
Port: <none>
Host Port: <none>
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 09 Nov 2022 16:55:46 +0000
Finished: Wed, 09 Nov 2022 16:55:48 +0000
Ready: False
Restart Count: 554
Requests:
cpu: 250m
Liveness: exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=10s period=10s #success=1 #failure=6
Readiness: exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=10s period=10s #success=1 #failure=3
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
DATASTORE_TYPE: kubernetes
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
CLUSTER_TYPE: k8s,bgp
IP: autodetect
CALICO_IPV4POOL_IPIP: Always
CALICO_IPV4POOL_VXLAN: Never
CALICO_IPV6POOL_VXLAN: Never
FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_VXLANMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_WIREGUARDMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_HEALTHENABLED: true
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/sys/fs/bpf from bpffs (rw)
/var/lib/calico from var-lib-calico (rw)
/var/log/calico/cni from cni-log-dir (ro)
/var/run/calico from var-run-calico (rw)
/var/run/nodeagent from policysync (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-58cnc (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
var-run-calico:
Type: HostPath (bare host directory volume)
Path: /var/run/calico
HostPathType:
var-lib-calico:
Type: HostPath (bare host directory volume)
Path: /var/lib/calico
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
sys-fs:
Type: HostPath (bare host directory volume)
Path: /sys/fs/
HostPathType: DirectoryOrCreate
bpffs:
Type: HostPath (bare host directory volume)
Path: /sys/fs/bpf
HostPathType: Directory
nodeproc:
Type: HostPath (bare host directory volume)
Path: /proc
HostPathType:
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
cni-log-dir:
Type: HostPath (bare host directory volume)
Path: /var/log/calico/cni
HostPathType:
host-local-net-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/cni/networks
HostPathType:
policysync:
Type: HostPath (bare host directory volume)
Path: /var/run/nodeagent
HostPathType: DirectoryOrCreate
kube-api-access-58cnc:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 37m (x11 over 72m) kubelet Container image "docker.io/calico/node:v3.24.4" already present on machine
Warning BackOff 114s (x329 over 70m) kubelet Back-off restarting failed contai
on my worker node I see: in journalctl -u kublet :
Nov 09 16:17:58 ip-10-0-27-233 kubelet[817]: E1109 16:17:58.974708 817 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"httpd\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=httpd pod=test2-749b6d69bf-9669z_default(e7a8d766-c758-4c21-9d43-c3e5fe8be1ba)\"" pod="default/test2-749b6d69bf-9669z" podUID=e7a8d766-c758-4c21-9d43-c3e5fe8be1ba
Nov 09 16:17:58 ip-10-0-27-233 kubelet[817]: I1109 16:17:58.974551 817 scope.go:115] "RemoveContainer" containerID="e323bf5fc53bf2e2b9feb5b1ce310476b5a40aa5d0404c00e87062a0d037e27f"
Nov 09 16:17:58 ip-10-0-27-233 kubelet[817]: E1109 16:17:58.973658 817 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"calico-node\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=calico-node pod=calico-node-s6rkv_kube-system(599471cc-b2a6-4c98-a72e-8c0028e7ab16)\"" pod="kube-system/calico-node-s6rkv" podUID=599471cc-b2a6-4c98-a72e-8c0028e7ab16
Nov 09 16:17:58 ip-10-0-27-233 kubelet[817]: I1109 16:17:58.972615 817 scope.go:115] "RemoveContainer" containerID="8216e8b6c0743802be2763a367616ebd6c650a1fbd03cf177b705624e7706484"
Nov 09 16:17:47 ip-10-0-27-233 kubelet[817]: E1109 16:17:47.966310 817 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"calico-node\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=calico-node pod=calico-node-s6rkv_kube-system(599471cc-b2a6-4c98-a72e-8c0028e7ab16)\"" pod="kube-system/calico-node-s6rkv" podUID=599471cc-b2a6-4c98-a72e-8c0028e7ab16
Nov 09 16:17:47 ip-10-0-27-233 kubelet[817]: I1109 16:17:47.965863 817 scope.go:115] "RemoveContainer" containerID="8216e8b6c0743802be2763a367616ebd6c650a1fbd03cf177b705624e7706484"
Nov 09 16:17:45 ip-10-0-27-233 kubelet[817]: E1109 16:17:45.938100 817 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"httpd\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=httpd pod=test2-749b6d69bf-9669z_default(e7a8d766-c758-4c21-9d43-c3e5fe8be1ba)\"" pod="default/test2-749b6d69bf-9669z" podUID=e7a8d766-c758-4c21-9d43-c3e5fe8be1ba
Nov 09 16:17:45 ip-10-0-27-233 kubelet[817]: E1109 16:17:45.938307 817 pod_workers.go:965] **"Error syncing pod,** skipping" err="failed to \"StartContainer\" for \"my-nginx\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=my-nginx pod=my-nginx-77d5cb496b-2vk5w_default(aa5cba4a-fb77-47ed-82f8-1caceb4295bb)\"" pod="default/my-nginx-77d5cb496b-2vk5w" podUID=aa5cba4a-fb77-47ed-82f8-1caceb4295bb
Nov 09 16:17:45 ip-10-0-27-233 kubelet[817]: I1109 16:17:45.937952 817 scope.go:115] "RemoveContainer" containerID="cdc688893bec66783740e219c1afc7b3ec55f5dc4c510c36e7778e4341d99f73"
Nov 09 16:17:45 ip-10-0-27-233 kubelet[817]: I1109 16:17:45.937612 817 scope.go:115] "RemoveContainer" containerID="e323bf5fc53bf2e2b9feb5b1ce310476b5a40aa5d0404c00e87062a0d037e27f"
ubuntu@ip-10-0-30-11:~$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-0-27-233 Ready <none> 2d2h v1.25.3 10.0.27.233 <none> Ubuntu 22.04.1 LTS 5.15.0-1022-aws containerd://1.5.9-0ubuntu3
ip-10-0-30-11 Ready control-plane 2d4h v1.25.3 10.0.30.11 <none> Ubuntu 22.04.1 LTS 5.15.0-1022-aws containerd://1.6.9
ubuntu@ip-10-0-30-11:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-node-7f48bfb94f-2qprl 1/1 Running 33 (6m41s ago) 3h2m 192.168.160.68 ip-10-0-27-233 <none> <none>
my-nginx-77d5cb496b-2vk5w 0/1 CrashLoopBackOff 461 (2m ago) 47h 192.168.160.126 ip-10-0-27-233 <none> <none>
my-nginx-77d5cb496b-l56mx 1/1 Running 0 47h 192.168.37.197 ip-10-0-30-11 <none> <none>
test2-749b6d69bf-9669z 0/1 CrashLoopBackOff 39 (72s ago) 3h20m 192.168.160.69 ip-10-0-27-233 <none> <none>
testl-7f545b6db7-qh5pv 1/1 Running 36 (2m21s ago) 3h21m 192.168.160.67 ip-10-0-27-233 <none> <none>
What am I missing?