Kubernetes version: 1.20.0
Installation method: kubeadm
Host OS: BigCloud Enterprise Linux release 8.2.2107 (Core) （Like Centos）
CNI and version: flannel Quay
CRI and version: docker 19.03.5
Sorry, friends. I have encountered a problem regarding DNS.
I have set up a cluster and deployed our production application. However, there is an issue with connectivity between microservices. Upon investigation, I found that our services can not access the Eureka registry service using the internal cluster domain name.
The Eureka service is deployed on the nb-17-8 server. Pods deployed on the same server, nb-17-8, can access Eureka using DNS. For example, the three replicas of Eureka need to communicate with each other for synchronization (which works fine):
ndsp eureka-0 1/1 Running 1 71m 10.244.86.42 nb-17-8 <none> <none>
ndsp eureka-1 1/1 Running 1 71m 10.244.86.44 nb-17-8 <none> <none>
ndsp eureka-2 1/1 Running 1 71m 10.244.86.45 nb-17-8 <none> <none>
Within the Eureka pods on nb-17-8, they cannot access 10.96.0.1 or 10.96.0.10:
kubectl exec -it -n ndsp eureka-0 -- sh
/ # nc -v 10.96.0.1 443
10.96.0.1 (10.96.0.1:443) open
^Cpunt!
/ # nc -v 10.96.0.10 53
10.96.0.10 (10.96.0.10:53) open
^Cpunt!
On other hosts, the Service IPs are accessible:
[root@nb-17-57 ~]# nc -v 10.96.0.1 443
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 10.96.0.1:443.
^C
[root@nb-17-57 ~]# nc -v 10.96.0.10 53
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 10.96.0.10:53.
On the hosts, accessing the Eureka pod via domain name is not possible, but using the Pod IP works (from all hosts and pods):
[root@nb-17-57 ~]# curl -I 10.244.86.42:8761
HTTP/1.1 200
Content-Type: text/html;charset=UTF-8
Content-Language: en-US
Transfer-Encoding: chunked
Date: Wed, 29 May 2024 11:14:14 GMT
In the CoreDNS pod, there are the following error messages:
[ERROR] plugin/errors: 2 eureka-2.eureka. A: read udp 10.244.0.11:44838->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 eureka-2.eureka. A: read udp 10.244.0.11:51686->8.8.4.4:53: i/o timeout
[ERROR] plugin/errors: 2 eureka-2.eureka. A: read udp 10.244.0.11:54068->8.8.4.4:53: i/o timeout
All hosts have the following resolv.conf file content (since it’s a closed internal network, domain name resolution uses the hosts file and there is no internal DNS server):
cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
Does anyone have any good solutions?