EC2 private DNS names do not resolve in the pods

General Discussions microk8s
1

Hello,

On an EC2 machine, a pod is accessing services outside of Microk8s but still in the same vpc (each host can ping each other by private DNS). However, if the URLs are from the private EC2 DNS they are not resolving from the pods (but are from the EC2 host):

From the Microk8s host, pinging another EC2 host:

ubuntu@ip-172-31-23-164:~$ ping ip-172-1-1-150.us-west-2.compute.internal
PING ip-172-1-1-150.us-west-2.compute.internal (172.1.1.150) 56(84) bytes of data.

From one pod, pinging another EC2 host:

Private domain:

root@server-zts87:/usr/share/server# ping ip-172-1-1-150.us-west-2.compute.internal
ping: unknown host

Public domain:

root@server-zts87:/usr/share/server# ping ec2-31-211-211-111.us-west-2.compute.amazonaws.com
PING 31-211-211-111.us-west-2.compute.amazonaws.com (31.211.211.111): 56 data bytes

Google:

root@server-zts87:/usr/share/server# ping google.com
PING google.com (216.58.194.174): 56 data bytes

Tried to add a stubDomains:

kubectl -n kube-system edit configmap/kube-dns
stubDomains: '{"us-west-2.compute.internal": ["127.0.0.53"]}'

microk8s dns addon is enabled but I did not see how we could tweak it to also include those. Is there a recommended way to do this?

Thank you,

2

I believe you need to update the upstreamNameservers in the kube-dns ConfigMap under the kube-system namespace. To do so first microk8s.enable dns and then microk8s.kubectl edit cm/kube-dns -n kube-system. Soon (in the 1.15 release) kube-dns will be replaced by coredns so the upstream nameservers will have to be updated with microk8s.kubectl edit cm/coredns -n kube-system.

3

I see thanks for the tips!

Still seeing the issue though, maybe the problem is with the IP of the internal DNS but doing a check on the actual node provides this.

cat /etc/resolv.conf 
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0
search us-west-2.compute.internal

Will give another try when 1.15 is out!

4

Found the issue!

studbDomains is the way to go but the DNS IP should come from:

systemd-resolve --status

Link 2 (ens5)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 172.31.0.2
          DNS Domain: us-west-2.compute.internal
5

How did you end up resolving this? I’m still struggling to get this to work. I’ve tried altering the dns server from google’s to internal ones to no avail.

6

So I think the DNS was switched to CoreDNS in a release and it broke again and I did not retry yet.

7

Ah, dang. Was hoping you’d be my saviour! It has indeed changed to CoreDNS. I even tried mapping the upstream directly to etc/resolv.conf to no avail.

Annoyingly, I’m trying to see the viability of converting a production server to microk8s, but this is quite a big blocker.

8

Can you show the coredns confgmap?

9

It’s just the vanilla microk8s coredns configmap: https://pastebin.com/FvkLHCbQ
I’ve edited it a few times to point directly to /etc/resolv.conf and to ec2’s internal dns, but nothing. At the moment, I’ve resorted to direct ip mapping and default microk8s coredns config.

10

Did you do it like this?

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        log . {
            class error
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
12

Yes, I did. It’s still the same issue, unfortunately. I do inspect the /etc/resolv.conf inside the pod, and it does display search eu-west-2.compute.internal, so I’m not sure if it’s directly an issue with the file?

I can ping the private ip directly inside the pod (172.x.x.x), but just the domain resolution is an issue.