EC2 private DNS names do not resolve in the pods

Hello,

On an EC2 machine, a pod is accessing services outside of Microk8s but still in the same vpc (each host can ping each other by private DNS). However, if the URLs are from the private EC2 DNS they are not resolving from the pods (but are from the EC2 host):

From the Microk8s host, pinging another EC2 host:

ubuntu@ip-172-31-23-164:~$ ping ip-172-1-1-150.us-west-2.compute.internal
PING ip-172-1-1-150.us-west-2.compute.internal (172.1.1.150) 56(84) bytes of data.

From one pod, pinging another EC2 host:

Private domain:

root@server-zts87:/usr/share/server# ping ip-172-1-1-150.us-west-2.compute.internal
ping: unknown host

Public domain:

root@server-zts87:/usr/share/server# ping ec2-31-211-211-111.us-west-2.compute.amazonaws.com
PING 31-211-211-111.us-west-2.compute.amazonaws.com (31.211.211.111): 56 data bytes

Google:

root@server-zts87:/usr/share/server# ping google.com
PING google.com (216.58.194.174): 56 data bytes

Tried to add a stubDomains:

kubectl -n kube-system edit configmap/kube-dns
stubDomains: '{"us-west-2.compute.internal": ["127.0.0.53"]}'

microk8s dns addon is enabled but I did not see how we could tweak it to also include those. Is there a recommended way to do this?

Thank you,

I believe you need to update the upstreamNameservers in the kube-dns ConfigMap under the kube-system namespace. To do so first microk8s.enable dns and then microk8s.kubectl edit cm/kube-dns -n kube-system. Soon (in the 1.15 release) kube-dns will be replaced by coredns so the upstream nameservers will have to be updated with microk8s.kubectl edit cm/coredns -n kube-system.

I see thanks for the tips!

Still seeing the issue though, maybe the problem is with the IP of the internal DNS but doing a check on the actual node provides this.

cat /etc/resolv.conf 
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0
search us-west-2.compute.internal

Will give another try when 1.15 is out!

Found the issue!

studbDomains is the way to go but the DNS IP should come from:

systemd-resolve --status

Link 2 (ens5)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 172.31.0.2
          DNS Domain: us-west-2.compute.internal