Kubernetes pods not resolving with service.namespace subdomains

General Discussions
, ,
1

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: v1.31.0
Cloud being used: bare-metal
Installation method: minikube
Host OS: Ubuntu 24.10 WSL on Windows 11 Enterprise
CNI and version: calicoVersion=“v3.28.1”
CRI and version: docker://27.2.0

I am running 2 pods busybox and nginx in the namespace called webcn
have exposed the nginx pod as a service

i am trying to understand working of name resolution in kubernetes within a namespace from busybox pod

vajra@ubuntu-minikube:~/kubernetes-learn$ : kubectl get pods busybox nginx  -n webcn
NAME      READY   STATUS    RESTARTS      AGE
busybox   1/1     Running   33 (8h ago)   56d
nginx     1/1     Running   33 (8h ago)   56d


vajra@ubuntu-minikube:~/kubernetes-learn$ : kubectl get svc nginx -n webcn
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
nginx   ClusterIP   10.108.64.229   <none>        80/TCP    54d

verifying name resolution from busybox pod running in the same namespace
#we can see the short name for service (nginx) and fqdn (nginx.webcn.svc.cluster.local) are resolving as expected also resolves without issues

vajra@ubuntu-minikube:~/kubernetes-learn$ : kubectl exec -it busybox -n webcn -- nslookup nginx
Server:         10.96.0.10
Address:        10.96.0.10:53

** server can't find nginx.svc.cluster.local: NXDOMAIN
** server can't find nginx.svc.cluster.local: NXDOMAIN
** server can't find nginx.cluster.local: NXDOMAIN

Name:   nginx.webcn.svc.cluster.local
Address: 10.108.64.229

** server can't find nginx.dns.podman: NXDOMAIN
** server can't find nginx.dns.podman: NXDOMAIN
** server can't find nginx.cluster.local: NXDOMAIN

command terminated with exit code 1
vajra@ubuntu-minikube:~/kubernetes-learn$ : kubectl exec -it busybox -n webcn -- nslookup nginx.webcn.svc.cluster.local
Server:         10.96.0.10
Address:        10.96.0.10:53


Name:   nginx.webcn.svc.cluster.local
Address: 10.108.64.229

#**but queries with service.namespace and service.namespace.svc are failing in my setup **
#which should also resolve correctly as per my understanding

vajra@ubuntu-minikube:~/kubernetes-learn$ : kubectl exec -it busybox -n webcn -- nslookup nginx.webcn
Server:         10.96.0.10
Address:        10.96.0.10:53

** server can't find nginx.webcn: NXDOMAIN

** server can't find nginx.webcn: NXDOMAIN

command terminated with exit code 1
vajra@ubuntu-minikube:~/kubernetes-learn$ : kubectl exec -it busybox -n webcn -- nslookup nginx.webcn.svc
Server:         10.96.0.10
Address:        10.96.0.10:53

** server can't find nginx.webcn.svc: NXDOMAIN

** server can't find nginx.webcn.svc: NXDOMAIN

command terminated with exit code 1

the DNS configuration on the busybox pod is configured with correct DNS service and proper search domains


vajra@ubuntu-minikube:~/kubernetes-learn$ : kubectl exec -it busybox -n webcn -- cat /etc/resolv.conf
nameserver 10.96.0.10
search webcn.svc.cluster.local svc.cluster.local cluster.local dns.podman
options ndots:5

vajra@ubuntu-minikube:~/kubernetes-learn$: kubectl get svc kube-dns -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   56d

Setup information

3 node minikube cluster

CNI : calico

vajra@ubuntu-minikube:~/kubernetes-learn$ : kubectl version
Client Version: v1.32.1
Kustomize Version: v5.5.0
Server Version: v1.31.0


vajra@ubuntu-minikube:~/kubernetes-learn$ : minikube version
minikube version: v1.34.0
commit: 210b148df93a80eb872ecbeb7e35281b3c582c61

vajra@ubuntu-minikube:~/kubernetes-learn$ : kubectl get nodes 
NAME           STATUS   ROLES           AGE   VERSION
minikube       Ready    control-plane   56d   v1.31.0
minikube-m02   Ready    <none>          56d   v1.31.0
minikube-m03   Ready    <none>          56d   v1.31.0

CoreDNS configuration

vajra@ubuntu-minikube:~/kubernetes-learn$ : kubectl get cm coredns  -n kube-system -o yaml
apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        hosts {
           10.255.255.254 host.minikube.internal
           fallthrough
        }
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2025-05-27T06:44:08Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "374"
  uid: 23598043-6063-439a-be6d-29b4ef14cac0

what am i missing ??

2

the DNS queries with [service].[namespace] and [service].[namespace].svc are failing in my setup
which should also resolve correctly as per my understanding.
need to understand why its failing

3

Updating your coreDNS with following should fix it

kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
-       fallthrough in-addr.arpa ip6.arpa
+       fallthrough in-addr.arpa ip6.arpa svc.cluster.local
        ttl 30
    }