Kafka Pods are not coming up!

General Discussions microk8s
1

Hi Team,

We have seen the existing Kafka pods went to CrashLoopBackOff mode, We tried to restart the pod, but the pods are not coming up now. We tried to run deployment file even tough not coming up and not showing the pods when we get pods. Please help us to get the pods up. Thanks

ordererorg-eehome kafka0-75945cfb48-m9w5j 0/1 CrashLoopBackOff 23 645d
ordererorg-eehome kafka1-78859c5b96-khkk2 0/1 CrashLoopBackOff 19 600d
ordererorg-eehome kafka2-6c4cf6cbc5-6dnwb 0/1 CrashLoopBackOff 23 645d
ordererorg-eehome kafka3-6b969bc586-7hxrw 0/1 CrashLoopBackOff 19 600d

Labelsalertname = KubePodCrashLooping
container = kafka3
instance = k8s-master
job = kubernetes-kube-state
namespace = ordererorg-eehome
pod = kafka3-6b969bc586-7hxrw
severity = criticalAnnotations
message = ordererorg-eehome/kafka3-6b969bc586-7hxrw (kafka3) is restarting 0.00 / second

2

Do you have any logs in your crashing pods?

3

Unfortunately no. because the issue was came because of changing path for the disk in cloud backend. The disk is allocated to Kafka mount previously. So when we see the mount permission it is changed to red only then we changed the permission to Read-Write mode and restarted pods, but the pods wont came back.

We tried to remove the deployments and recreated even though no luck. Then we have remove pv and pvc and created new pv and pvc but existing one went to termination and new one came to pending . Please find below.

[root@k8s-master ~]# kubectl get pv -n ordererorg-eehome
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
kafkas-ordererorg-eehome-pv 2Gi RWX Retain Terminating ordererorg-eehome/kafkas-ordererorg-eehome-pvc 2y113d
kafkas-ordererorg-eehome-pv1 2Gi RWX Retain Pending ordererorg-eehome/kafkas-ordererorg-eehome-pvc1 26m
zookeeper-ordererorg-eehome-pv 1Gi RWX Retain Bound ordererorg-eehome/zk-ordererorg-eehome-pvc 2y113d
zookeeper-ordererorg-eehome-pv1 1Gi RWX Retain Pending ordererorg-eehome/zk-ordererorg-eehome-pvc1 10m
[root@k8s-master ~]# kubectl get pvc -n ordererorg-eehome
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
kafkas-ordererorg-eehome-pvc Terminating kafkas-ordererorg-eehome-pv 2Gi RWX 2y113d
kafkas-ordererorg-eehome-pvc1 Pending

4

Hi Team,

We have resolved the issue. Now pods are come up.

  1. mount went to ready only mode – changed to read write
  2. we renewed kube certs long back we have restarted all api, scheduler, controller forgot to restart the master server due to which apis are failing, once we restrted the master pods issue resolved.
5

Hi, I am using Strimzi 0.18.0 version. After following installation steps, I notice that only Zookeeper pods are runnings. I don’t see any PODs related to … Describe the bug When using strimzi operator to try to cps tester start a Kafka infrastructure, kafka cluster pods don’t start.We have seen the existing Kafka pods went to CrashLoopBackOff mode, We tried to restart the pod, but the pods are not coming up now.

6

Points to be verified:

  1. Check the kube cert expiry
  2. Check kube api logs
  3. Try to restart master server