Hey everyone, I have run into an interesting issue which is me and my team were trying to setup bare-metal K8 cluster. We got one cluster perfectly running then we went for the next one. There We found the DNS is not being resolved inside the cluster. So I had went through almost all possible DNS debuggings but still could not get it up and running. But while googling found one article which suggested to restart the CoreDNS and we tried to do that. Interestingly new CoreDNS pods spun up and got running but never got into ready state. While we checked into logs we got this
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
And here is the kube-system pods listing while we try to restart coredns with the bellow command
kubectl -n kube-system rollout restart deployment coredns
kube-system coredns-78fcd69978-tk5fb 1/1 Running 0 2d12h
kube-system coredns-858997d697-lg9kr 0/1 Running 0 8h
kube-system coredns-858997d697-xgbbc 0/1 Running 0 8h
kube-system etcd-wsprdapp4.noar.dc 1/1 Running 0 2d12h
kube-system kube-apiserver-wsprdapp4.noar.dc 1/1 Running 0 2d12h
kube-system kube-controller-manager-wsprdapp4.noar.dc 1/1 Running 0 2d12h
kube-system kube-router-qxvnn 1/1 Running 0 2d12h
kube-system kube-router-zmzmx 1/1 Running 0 2d12h
kube-system kube-scheduler-wsprdapp4.noar.dc 1/1 Running 0 2d12h
Any help would be highly appreciated.
Cluster information:
Kubernetes version: 1.22.2
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: CentOS 7
CNI and version: cloudnativelabs/kube-router:latest
CRI and version: docker v20.10.8
You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.