Why coredns occurs this error: [ERROR] plugin/errors: 2 kube-dns.kube-system. AAAA: read udp 10.244.0.2:42616->8.8.4.4:53: i/o timeout v0.16.1

General Discussions
1

I install kubernetes v1.23.1 with flannel, the coredns version is 1.8.6.
when I execute ‘nslookup kube-dns.kube-system’ command in a pod, this error occured:

[ERROR] plugin/errors: 2 kube-dns.kube-system. AAAA: read udp 10.244.0.2:42616->8.8.4.4:53: i/o timeout
[ERROR] plugin/errors: 2 kube-dns.kube-system. A: read udp 10.244.0.2:54528->8.8.4.4:53: i/o timeout

My analysis as follows:
1 coredns pod is running
kube-system coredns-5dfff794cb-j6wkv 1/1 Running 0 174m 10.244.0.3 master
kube-system coredns-5dfff794cb-nbn4r 1/1 Running 0 174m 10.244.0.2 master

2 coredns service is running
kube-dns ClusterIP 10.1.0.10 53/UDP,53/TCP,9153/TCP 175m k8s-app=kube-dns

3 the pod ip assigned to the service
NAME ENDPOINTS AGE
kube-dns 10.244.0.2:53,10.244.0.3:53,10.244.0.2:53 + 3 more… 174m

4 pod can access the coredns ip:
PING 10.244.0.2 (10.244.0.2): 56 data bytes
64 bytes from 10.244.0.2: seq=0 ttl=62 time=0.762 ms
64 bytes from 10.244.0.2: seq=1 ttl=62 time=0.573 ms

5 pod can’t get kubernetes.default’s ip by service cluster ip:
/etc # nslookup kubernetes.default
Server: 10.1.0.10
Address: 10.1.0.10:53

;; connection timed out; no servers could be reached

6 pod can’t get kubernetes.default’s ip by coredns’s ip:
/etc # nslookup kubernetes.default 10.244.0.2
Server: 10.244.0.2
Address: 10.244.0.2:53

;; connection timed out; no servers could be reached

7 the all error logs of coredns service as follows:
[ERROR] plugin/errors: 2 kube-dns.kube-system. AAAA: read udp 10.244.0.2:42616->8.8.4.4:53: i/o timeout
[ERROR] plugin/errors: 2 kube-dns.kube-system. A: read udp 10.244.0.2:54528->8.8.4.4:53: i/o timeout

So, how can I further analysis and fix it, many thanks!

Cluster information:

Kubernetes version: 1.23.1
Cloud being used: my self vm
Installation method: yum and kubeadm init
Host OS: centos 7.4
CNI and version: docker 19.03.13
CRI and version: flannel 0.16.1

2

It’s a known bug of busybox 1.32, v1.28.3 working.

3

I’m also having this same issue, but it’s not working on that version of busybox either. It is also preventing me from running nginx pods, because it is trying to contact the coredns pod, but that pod can’t resolve the nameservers.