CoreDNS CrashLoopBackOff

General Discussions
1

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: 1.15
Cloud being used: (put bare-metal if not on a public cloud)
Installation method: kubeadm and kubectl
Host OS: CentOS Linux release 7.6.1810 (Core)
CNI and version: kubernetes-cni-0.7.5-0.x86_64
CRI and version: N/A

CoreDNS is failing with CrashLoopBackOff error and more details below. appreciate any help.

kubectl logs -p coredns-5c98db65d4-gvq8n -n=kube-system

.:53

2019-08-01T18:28:29.302Z [INFO] CoreDNS-1.3.1

2019-08-01T18:28:29.302Z [INFO] linux/amd64, go1.11.4, 6b56a9c

CoreDNS-1.3.1

linux/amd64, go1.11.4, 6b56a9c

2019-08-01T18:28:29.302Z [INFO] plugin/reload: Running configuration MD5 = 5d5369fbc12f985709b924e721217843

2019-08-01T18:28:35.303Z [ERROR] plugin/errors: 2 6044359725485763008.6006489691758497056. HINFO: read udp 172.100.2.47:51870->10.40.52.2:53: i/o timeout

2019-08-01T18:28:38.303Z [ERROR] plugin/errors: 2 6044359725485763008.6006489691758497056. HINFO: read udp 172.100.2.47:37536->10.40.52.2:53: i/o timeout

2019-08-01T18:28:39.304Z [ERROR] plugin/errors: 2 6044359725485763008.6006489691758497056. HINFO: read udp 172.100.2.47:33555->10.40.52.2:53: i/o timeout

2019-08-01T18:28:40.304Z [ERROR] plugin/errors: 2 6044359725485763008.6006489691758497056. HINFO: read udp 172.100.2.47:46509->10.40.52.2:53: i/o timeout

2019-08-01T18:28:43.304Z [ERROR] plugin/errors: 2 6044359725485763008.6006489691758497056. HINFO: read udp 172.100.2.47:34647->10.40.52.2:53: i/o timeout

2019-08-01T18:28:46.305Z [ERROR] plugin/errors: 2 6044359725485763008.6006489691758497056. HINFO: read udp 172.100.2.47:58610->10.40.52.2:53: i/o timeout

2019-08-01T18:28:49.305Z [ERROR] plugin/errors: 2 6044359725485763008.6006489691758497056. HINFO: read udp 172.100.2.47:53868->10.40.52.2:53: i/o timeout

2019-08-01T18:28:52.305Z [ERROR] plugin/errors: 2 6044359725485763008.6006489691758497056. HINFO: read udp 172.100.2.47:37015->10.40.52.2:53: i/o timeout

E0801 18:28:54.302914 1 reflector.go:134] github.com/coredns/coredns/plugin/kubernetes/controller.go:322: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

E0801 18:28:54.302914 1 reflector.go:134] github.com/coredns/coredns/plugin/kubernetes/controller.go:322: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

thank you.

-Naren K

2

A common crash is caused if it can’t reach the API server. I don’t know the IP of the API server on your setup, but my guess is that it times out trying to connect.

Is that the IP of the API server? Can you see some logs to see why it times out?

Does this started to happen after some network configuration change? Can you snif traffic to see where it is lost?

3

Investigate the value at /etc/resolv.conf and more logs needed to arrive to decision.

You can also delete loops [kubectl edit cm coredns -n kube-system] and save it, but be careful, only do that if the logs are pointing to it.

4

Hi,
Can you share the steps if anything worked for this