MicroK8s + Istio not forwarding out the '.' domains for query

General Discussions
1

Bit of a noob here.

I have installed istio using the istioctl command with the profile demo.
I have deployed an istio injected sleep pod.
I realised I cannot resolve public domains unless I append the . .

If I try to curl google.com, on my homelab DNS servers logs, I see these entries:

[INFO] 192.168.88.25:13263 - 53624 "AAAA IN google.com.my.lab.domain. udp 55 true 2048" NXDOMAIN qr,aa,rd 131 0.00007882s
[INFO] 192.168.88.25:55794 - 48121 "A IN google.com.my.lab.domain. udp 55 true 2048" NXDOMAIN qr,aa,rd 131 0.00016916s

If I try to resolve it by adding the . at the end of google.com, it works.

~ $ curl google.com
curl: (6) Could not resolve host: google.com

## note the .

~ $ curl google.com.
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://google.com/">here</A>.
</BODY></HTML>

Cluster information:

Kubernetes version: v1.27.13
Cloud being used: bare-metal
Installation method:
Host OS: Ubuntu 22.04.4 LTS

Coredns config:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        log . {
          class error
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward .  192.168.88.2
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors\n    health {\n      lameduck 5s\n    }\n    ready\n    log . {\n      class error\n    }\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward .  192.168.88.2\n    cache 30\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists","k8s-app":"kube-dns"},"name":"coredns","namespace":"kube-system"}}
  creationTimestamp: "2024-05-18T07:53:12Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
  resourceVersion: "2462704"
  uid: 4c7cb15d-e80d-4caf-9d55-efb86ca9e4e9

On a non istio sidecar injected pod, it works as intended.

root@httpd:/usr/local/apache2# curl google.com 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

Any ideas where should I start looking ?