I have some problems on given pods on given nodes (still creating the same pods on the same nodes even if I dleete them to reschedule them). They keep going to CrashLoopBackoff without any errors.
If I create othe pods on the same nodes, they start flawlessly.
Example1: I created a statefulset of proxysql in the mariadb namespace, it worked out on given nodes (1,4,5). If I create another same statefulset on mysql namespace, it fails (for example on node 1).
Exmaple 2: if I crate a phpMyAdmin deployment with pods on node 1, it fails.
Only error found: "Error syncing pod, skipping" err="failed to \"StartContainer\" in kubelet for the selected node 1
Example describe and logs:
kubectl describe -n mariadb pod/phpmyadmin-mariadb-7bb95f8df9-whdbt
Name: phpmyadmin-mariadb-7bb95f8df9-whdbt
Namespace: mariadb
Priority: 0
Service Account: default
Node: k8w1/10.0.50.54
Start Time: Fri, 06 Oct 2023 23:33:51 +0200
Labels: app.kubernetes.io/instance=phpmyadmin-mariadb
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=phpmyadmin
app.kubernetes.io/version=5.2.1
helm.sh/chart=phpmyadmin-12.1.2
pod-template-hash=7bb95f8df9
Annotations: cni.projectcalico.org/containerID: 0a142cf9c5eceacc5067b4865b3b01562147a32a8a31178e01a2a8046544fe7f
cni.projectcalico.org/podIP: 10.50.14.12/32
cni.projectcalico.org/podIPs: 10.50.14.12/32
k8s.v1.cni.cncf.io/network-status:
[{
"name": "k8s-pod-network",
"ips": [
"10.50.14.12"
],
"default": true,
"dns": {}
}]
Status: Running
IP: 10.50.14.12
IPs:
IP: 10.50.14.12
Controlled By: ReplicaSet/phpmyadmin-mariadb-7bb95f8df9
Containers:
phpmyadmin:
Container ID: containerd://695f383030622b01428f5f1cd4896e1ffe2f57ec8d829ec470bb6863d4d8028b
Image: docker.io/bitnami/phpmyadmin:5.2.1-debian-11-r103
Image ID: docker.io/bitnami/phpmyadmin@sha256:521ab8ff46e067aac9b5f6683f37dafc72b9d2e1778a542a1fe0bbdb26076e5f
Ports: 8080/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 135
Started: Fri, 06 Oct 2023 23:45:27 +0200
Finished: Fri, 06 Oct 2023 23:45:29 +0200
Ready: False
Restart Count: 7
Liveness: http-get http://:http/ delay=30s timeout=30s period=10s #success=1 #failure=6
Readiness: http-get http://:http/ delay=30s timeout=30s period=10s #success=1 #failure=6
Environment:
BITNAMI_DEBUG: false
DATABASE_PORT_NUMBER: 6033
DATABASE_HOST: proxysql.mariadb.svc.cluster.local
PHPMYADMIN_ALLOW_NO_PASSWORD: false
DATABASE_ENABLE_SSL: no
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-scwsq (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-scwsq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 14m default-scheduler Successfully assigned mariadb/phpmyadmin-mariadb-7bb95f8df9-whdbt to k8w1
Normal AddedInterface 14m multus Add eth0 [10.50.14.12/32] from k8s-pod-network
Normal Pulling 14m kubelet Pulling image "docker.io/bitnami/phpmyadmin:5.2.1-debian-11-r103"
Normal Pulled 13m kubelet Successfully pulled image "docker.io/bitnami/phpmyadmin:5.2.1-debian-11-r103" in 15.87s (15.87s including waiting)
Normal Created 12m (x4 over 13m) kubelet Created container phpmyadmin
Normal Started 12m (x4 over 13m) kubelet Started container phpmyadmin
Normal Pulled 11m (x4 over 13m) kubelet Container image "docker.io/bitnami/phpmyadmin:5.2.1-debian-11-r103" already present on machine
Warning BackOff 3m58s (x51 over 13m) kubelet Back-off restarting failed container phpmyadmin in pod phpmyadmin-mariadb-7bb95f8df9-whdbt_mariadb(3a13d06b-b1cd-473a-862f-3a8e4798526d)
kubectl logs -n mariadb pod/phpmyadmin-mariadb-7bb95f8df9-whdbt
phpmyadmin 21:45:27.65
phpmyadmin 21:45:27.65 Welcome to the Bitnami phpmyadmin container
phpmyadmin 21:45:27.66 Subscribe to project updates by watching https://github.com/bitnami/containers
phpmyadmin 21:45:27.66 Submit issues and feature requests at https://github.com/bitnami/containers/issues
phpmyadmin 21:45:27.66
phpmyadmin 21:45:27.67 INFO ==> ** Starting phpMyAdmin setup **
phpmyadmin 21:45:27.70 INFO ==> Generating sample certificates
Generating RSA private key, 4096 bit long modulus (2 primes)
........++++
....................................................................................................................++++
e is 65537 (0x010001)
Signature ok
subject=CN = example.com
Getting Private key
realpath: /bitnami/apache/conf: No such file or directory
phpmyadmin 21:45:28.67 INFO ==> Configuring Apache ServerTokens directive
phpmyadmin 21:45:28.70 INFO ==> Configuring PHP options
phpmyadmin 21:45:28.71 INFO ==> Setting PHP expose_php option
phpmyadmin 21:45:28.73 INFO ==> Setting PHP output_buffering option
phpmyadmin 21:45:28.79 WARN ==> The usage of 'PHPMYADMIN_ALLOW_NO_PASSWORD' is deprecated and will soon be removed. Use 'DATABASE_ALLOW_NO_PASSWORD' instead.
phpmyadmin 21:45:28.79 INFO ==> Configuring phpMyAdmin
phpmyadmin 21:45:28.80 INFO ==> Setting database host option
phpmyadmin 21:45:28.81 INFO ==> Setting database port number option
phpmyadmin 21:45:28.83 INFO ==> Setting AllowNoPassword option
phpmyadmin 21:45:28.85 INFO ==> Setting blowfish_secret option to a randomly generated value
phpmyadmin 21:45:28.88 INFO ==> Enabling web server application configuration for phpMyAdmin
phpmyadmin 21:45:29.16 INFO ==> ** phpMyAdmin setup finished! **
phpmyadmin 21:45:29.19 INFO ==> ** Starting Apache **
journalctl -u containerd | tail
Oct 06 23:45:27 k8w1 containerd[2241708]: time="2023-10-06T23:45:27.263763742+02:00" level=info msg="CreateContainer within sandbox \"0a142cf9c5eceacc5067b4865b3b01562147a32a8a31178e01a2a8046544fe7f\" for container &ContainerMetadata{Name:phpmyadmin,Attempt:7,}"
Oct 06 23:45:27 k8w1 containerd[2241708]: time="2023-10-06T23:45:27.333258968+02:00" level=info msg="CreateContainer within sandbox \"0a142cf9c5eceacc5067b4865b3b01562147a32a8a31178e01a2a8046544fe7f\" for &ContainerMetadata{Name:phpmyadmin,Attempt:7,} returns container id \"695f383030622b01428f5f1cd4896e1ffe2f57ec8d829ec470bb6863d4d8028b\""
Oct 06 23:45:27 k8w1 containerd[2241708]: time="2023-10-06T23:45:27.334291337+02:00" level=info msg="StartContainer for \"695f383030622b01428f5f1cd4896e1ffe2f57ec8d829ec470bb6863d4d8028b\""
Oct 06 23:45:27 k8w1 containerd[2241708]: time="2023-10-06T23:45:27.507634697+02:00" level=info msg="RemoveContainer for \"5a8add4b41b74b4d981d32d8bcfc96e57f0fa56defe3355b0057c32398de8e1d\""
Oct 06 23:45:27 k8w1 containerd[2241708]: time="2023-10-06T23:45:27.537822455+02:00" level=info msg="RemoveContainer for \"5a8add4b41b74b4d981d32d8bcfc96e57f0fa56defe3355b0057c32398de8e1d\" returns successfully"
Oct 06 23:45:27 k8w1 containerd[2241708]: time="2023-10-06T23:45:27.657239401+02:00" level=info msg="StartContainer for \"695f383030622b01428f5f1cd4896e1ffe2f57ec8d829ec470bb6863d4d8028b\" returns successfully"
Oct 06 23:45:29 k8w1 containerd[2241708]: time="2023-10-06T23:45:29.750466682+02:00" level=info msg="shim disconnected" id=695f383030622b01428f5f1cd4896e1ffe2f57ec8d829ec470bb6863d4d8028b
Oct 06 23:45:29 k8w1 containerd[2241708]: time="2023-10-06T23:45:29.750602831+02:00" level=warning msg="cleaning up after shim disconnected" id=695f383030622b01428f5f1cd4896e1ffe2f57ec8d829ec470bb6863d4d8028b namespace=k8s.io
Oct 06 23:45:29 k8w1 containerd[2241708]: time="2023-10-06T23:45:29.750627322+02:00" level=info msg="cleaning up dead shim"
Oct 06 23:45:29 k8w1 containerd[2241708]: time="2023-10-06T23:45:29.777531185+02:00" level=warning msg="cleanup warnings time=\"2023-10-06T23:45:29+02:00\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=2688457 runtime=io.containerd.runc.v2\n"
journalctl -u kubelet | tail
Oct 06 23:50:12 k8w1 kubelet[2241986]: I1006 23:50:12.259144 2241986 scope.go:117] "RemoveContainer" containerID="695f383030622b01428f5f1cd4896e1ffe2f57ec8d829ec470bb6863d4d8028b"
Oct 06 23:50:12 k8w1 kubelet[2241986]: E1006 23:50:12.260023 2241986 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"phpmyadmin\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=phpmyadmin pod=phpmyadmin-mariadb-7bb95f8df9-whdbt_mariadb(3a13d06b-b1cd-473a-862f-3a8e4798526d)\"" pod="mariadb/phpmyadmin-mariadb-7bb95f8df9-whdbt" podUID="3a13d06b-b1cd-473a-862f-3a8e4798526d"
Oct 06 23:50:24 k8w1 kubelet[2241986]: I1006 23:50:24.259530 2241986 scope.go:117] "RemoveContainer" containerID="695f383030622b01428f5f1cd4896e1ffe2f57ec8d829ec470bb6863d4d8028b"
Oct 06 23:50:24 k8w1 kubelet[2241986]: E1006 23:50:24.260286 2241986 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"phpmyadmin\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=phpmyadmin pod=phpmyadmin-mariadb-7bb95f8df9-whdbt_mariadb(3a13d06b-b1cd-473a-862f-3a8e4798526d)\"" pod="mariadb/phpmyadmin-mariadb-7bb95f8df9-whdbt" podUID="3a13d06b-b1cd-473a-862f-3a8e4798526d"
Oct 06 23:50:35 k8w1 kubelet[2241986]: I1006 23:50:35.259067 2241986 scope.go:117] "RemoveContainer" containerID="695f383030622b01428f5f1cd4896e1ffe2f57ec8d829ec470bb6863d4d8028b"
Oct 06 23:50:38 k8w1 kubelet[2241986]: I1006 23:50:38.345019 2241986 scope.go:117] "RemoveContainer" containerID="695f383030622b01428f5f1cd4896e1ffe2f57ec8d829ec470bb6863d4d8028b"
Oct 06 23:50:38 k8w1 kubelet[2241986]: I1006 23:50:38.345854 2241986 scope.go:117] "RemoveContainer" containerID="350dc473f25c20d2bc668baca162a18fd776ba32783065a1e3e020c903d6b514"
Oct 06 23:50:38 k8w1 kubelet[2241986]: E1006 23:50:38.346787 2241986 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"phpmyadmin\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=phpmyadmin pod=phpmyadmin-mariadb-7bb95f8df9-whdbt_mariadb(3a13d06b-b1cd-473a-862f-3a8e4798526d)\"" pod="mariadb/phpmyadmin-mariadb-7bb95f8df9-whdbt" podUID="3a13d06b-b1cd-473a-862f-3a8e4798526d"
Oct 06 23:50:41 k8w1 kubelet[2241986]: I1006 23:50:41.627312 2241986 scope.go:117] "RemoveContainer" containerID="350dc473f25c20d2bc668baca162a18fd776ba32783065a1e3e020c903d6b514"
Oct 06 23:50:41 k8w1 kubelet[2241986]: E1006 23:50:41.628006 2241986 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"phpmyadmin\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=phpmyadmin pod=phpmyadmin-mariadb-7bb95f8df9-whdbt_mariadb(3a13d06b-b1cd-473a-862f-3a8e4798526d)\"" pod="mariadb/phpmyadmin-mariadb-7bb95f8df9-whdbt" podUID="3a13d06b-b1cd-473a-862f-3a8e4798526d"
How can I solve it?
Cluster information:
Kubernetes version: 1.28.2
Cloud being used: (put bare-metal if not on a public cloud): bare-metal (vm)
Installation method: kubeadm
Host OS: ubuntu 22.04
CNI and version: Calico 3.26.1 + multus/wereabouts
CRI and version: containerd://1.6.24