Pod does not send any ARP request with self IP

General Discussions
1

in k8s v1.23.13 cluster, on certain node(hostname) pod cannot traceroute to kubernetes svc ip but ping works
meanwhile, on node traffic capture by nslookup kubernetes output like

...
09:47:34.625033 LLDP, length 221: <hostname>
09:47:36.285491 IP 172.18.88.13.58555 > 172.19.0.10.domain: 21926+ A? kubernetes.default.svc.cluster.local. (54)
09:47:41.286712 IP 172.18.88.13.58555 > 172.19.0.10.domain: 21926+ A? kubernetes.default.svc.cluster.local. (54)
09:47:41.318517 ARP, Request who-has 169.254.1.1 tell 172.18.88.13, length 28
09:47:41.318542 ARP, Reply 169.254.1.1 is-at ee:ee:ee:ee:ee:ee (oui Unknown), length 28
09:47:46.290649 IP 172.18.88.13.58555 > 172.19.0.10.domain: 21926+ A? kubernetes.default.svc.cluster.local. (54)
09:48:04.627400 LLDP, length 221: <hostname>
...

in contrast to normal node, found that an ARP request from pod self’s IP missing on abnormal one
could someone please provide some clue? thanks

Cluster information:

Kubernetes version: 1.23.13
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: Debian GNU/Linux 11 (bullseye), 5.10.158-2 (2022-12-13) x86_64 GNU/Linux
CNI and version: calico/cni:v3.23.1
CRI and version: containerd containerd.io 1.6.8 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6

2

Services are almost always implemented in some form of NAT, and ARP is not handled.

3

hi @thockin, in normal node, within pod, kubernetes nslookup traffic shows an ARP request from pod ip, looks like
…
16:34:53.586606 ARP, Request who-has 169.254.1.1 tell 172.18.219.45, length 28
16:34:53.586636 ARP, Reply 169.254.1.1 is-at ee:ee:ee:ee:ee:ee (oui Unknown), length 28
16:34:53.590602 ARP, Request who-has 172.18.219.45 tell host_name, length 28
16:34:53.590655 ARP, Reply 172.18.219.45 is-at 36:12:c0:d9:e9:43 (oui Unknown), length 28
16:34:59.513959 IP 172.18.219.45.59047 > 172.19.0.10.domain: 36941+ A? kubernetes.default.svc.cluster.local. (54)
16:34:59.514522 IP 172.19.0.10.domain > 172.18.219.45.59047: 36941*- 1/0/0 A 172.19.0.1 (106)
16:34:59.515077 IP 172.18.219.45.51632 > 172.19.0.10.domain: 57871+ AAAA? kubernetes.default.svc.cluster.local. (54)
16:34:59.515612 IP 172.19.0.10.domain > 172.18.219.45.51632: 57871*- 0/0/0 (54)
…