Unable to resolve pod to pod (coredns)

General Discussions
1

Cluster information:

Kubernetes version: v1.18.6
Cloud being used: bare-metal
Installation method:
Host OS: Ubuntu20 LTS
CNI and version: Weave Net
CRI and version: docker://19.3.8

Hi,

NAME            STATUS   ROLES    AGE   VERSION
k8s-ams-node0   Ready    master   8d    v1.18.6
k8s-ams-node1   Ready    master   8d    v1.18.6
k8s-ams-node2   Ready    master   8d    v1.18.6
k8s-ams-node3   Ready    <none>   8d    v1.18.6

When i create a namespace eg. debug and deploy 2 containers in it they cannot resolve eachother :
I changed the default /etc/resolv.conf to match /run/systemd/resolve/resolv.conf so nothing matches localhost. (systemd-resolveconf)

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu1
  namespace: debug
  labels:
    app: ubuntu1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ubuntu1
  template:
    metadata:
      labels:
        app: ubuntu1
    spec:
      hostname: ubuntu1
      containers:
      - name: ubuntu1
        image: ubuntu:debug
        imagePullPolicy: Always

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu2
  namespace: debug
  labels:
    app: ubuntu2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ubuntu2
  template:
    metadata:
      labels:
        app: ubuntu2
    spec:
      hostname: ubuntu2
      containers:
      - name: ubuntu2
        image: ubuntu:debug
        imagePullPolicy: Always

Pods run on the same node.

kubectl get pods -n debug -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP          NODE            NOMINATED NODE   READINESS GATES
ubuntu1-84869f6f9d-csnp7   1/1     Running   0          77m   10.32.0.4   k8s-ams-node0   <none>           <none>
ubuntu2-788fffccd7-qh2rr   1/1     Running   0          77m   10.32.0.6   k8s-ams-node0   <none>           <none>

Pinging eachother works

root@ubuntu1:/# ping 10.32.0.4
PING 10.32.0.4 (10.32.0.4): 56 data bytes
64 bytes from 10.32.0.4: icmp_seq=0 ttl=64 time=0.072 ms

root@ubuntu1:/# ping 10.32.0.6
PING 10.32.0.6 (10.32.0.6): 56 data bytes
64 bytes from 10.32.0.6: icmp_seq=0 ttl=64 time=0.278 ms

root@ubuntu1:/# cat /etc/resolv.conf
nameserver 10.96.0.10
search debug.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

CoreDNS is working (as far as i can see)

kubectl get svc -n kube-system -o wide
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   8d    k8s-app=kube-dns

kubectl get pods -n kube-system
NAME                                    READY   STATUS    RESTARTS   AGE
coredns-66bff467f8-qz2t7                1/1     Running   0          88m
coredns-66bff467f8-x4l59                1/1     Running   0          88m

CoreDNS resolves the basics (internal/external)

root@ubuntu1:/# nslookup kubernetes.default
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

root@ubuntu1:/# nslookup google.com
Server:         10.96.0.10
Address:        10.96.0.10#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.20.110
Name:   google.com
Address: 2a00:1450:400e:80e::200e

But resolving eachother isn’t working with any of the search domains.

root@ubuntu1:/# nslookup ubuntu2
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find ubuntu2: NXDOMAIN

root@ubuntu1:/# nslookup ubuntu2.debug.svc.cluster.local
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find ubuntu2.debug.svc.cluster.local: NXDOMAIN

root@ubuntu1:/# nslookup ubuntu2.svc.cluster.local
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find ubuntu2.svc.cluster.local: NXDOMAIN

root@ubuntu1:/# nslookup ubuntu2.cluster.local
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find ubuntu2.cluster.local: NXDOMAIN

So i’m really puzzled.
Any info needed i can ofcourse provide.

2

In addition :

kubectl describe configmap coredns -n kube-system

Data
====
Corefile:
----
.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

Events:  <none>
3

Hi Martijn,

In your current setup you should be able to resolve by the IP hostname:

10-32-0-4.debug.pod.cluster.local

But in order to get a DNS under svc.cluster.local you need to put a service in front of the pod.

The example given here are all services:

I don’t think you can directly address a pod by the hostname because what happens if you have multiple replicas, they can’t all be ubuntu1.

Kind regards,
Stephen

4

That was simpler as i thought, the downside of working with UI’s to much.

Created a service, and its perfectly resolvable.

root@ubuntu1:/# nslookup ubuntu
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   ubuntu.debug.svc.cluster.local
Address: 10.110.51.149

Thanks for the quick reply.

1 Like