External domain name resolution is not working inside pods

General Discussions
,
1

I have setup single node cluster on digital ocean droplet with ingress-nginx (hostNetwork: true). Installation method followed. My ingress service is working i am able to access Next.js using https on it. Internal Networking fine. i am able to curl one pod to another pod.

Issue :
My pod is crashing because pod is not able to resolve domain name of redis cloud database (not on same server. I am using redis service). When i changed domain name to IP then Reddis connection is working fine. My server is not able to call any API with domain name. Reddis credential is working fine on local system so there is no issue with credential. When i try to ping google.com from the pod then also i got error
How can i make pods to resolve all external domain name without affecting internal domain name working
Note :
I am using ingress-nginx with SSL certificate (https enabled)

Cluster information:

Kubernetes version:
Cloud being used: bare-metal
Installation method: Single Node
Host OS: Ubuntu 22.04.2 LTS
CNI and version: flannel:v0.24.0
CRI and version: containerd GitHub - containerd/containerd: An open and reliable container runtime v1.6.14 9ba4b250366a5ddde94bb7c9d1def331423aa323
Ingress : Ingress-nginx with hostNetwork true

Ping by domain name

kubectl exec -i -t dnsutils -- ping google.com
ping: unknown host google.com

Ping by IP address

kubectl exec -i -t dnsutils -- ping 142.251.214.142
PING 142.251.214.142 (142.251.214.142) 56(84) bytes of data.
64 bytes from 142.251.214.142: icmp_seq=1 ttl=56 time=204 ms
64 bytes from 142.251.214.142: icmp_seq=2 ttl=56 time=202 ms
64 bytes from 142.251.214.142: icmp_seq=3 ttl=56 time=203 ms
64 bytes from 142.251.214.142: icmp_seq=4 ttl=56 time=203 ms
64 bytes from 142.251.214.142: icmp_seq=5 ttl=56 time=203 ms
^C
--- 142.251.214.142 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms

kubectl exec -i -t dnsutils -- nslookup kubernetes.default
;; reply from unexpected source: 10.244.0.49#53, expected 10.96.0.10#53
;; reply from unexpected source: 10.244.0.49#53, expected 10.96.0.10#53

Host server
/etc/resolve.conf

nameserver 127.0.0.53
options edns0 trust-ad
search .

Host server
/run/systemd/resolve/stub-resolv.conf

nameserver 127.0.0.53
options edns0 trust-ad
search .

I have tried to add google DNS in .yml file but it didn’t work

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment
  namespace: nsp
spec:
  replicas: 1
  selector:
    matchLabels:
      serviceName: service-name
  template:
    metadata:
      labels:
        serviceName: service-name
    spec:
      containers:
        - name: container
          image: image-name
          ports:
            - containerPort: 5000
          env:
            - name: DOMAIN_TEST
              value: "https://playground.io"
          envFrom:
            - secretRef:
                name: playground-io-secrets
      dnsConfig:
        nameservers:
        - 10.96.0.10
        - 8.8.8.8
        - 8.8.4.4
        searches:
        - default.svc.cluster.local
        - svc.cluster.local
        - cluster.local
        - playground.io
      imagePullSecrets:
        - name: playground-io-pull-secret

kubectl -n kube-system edit configmap coredns

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-23T18:01:05Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "279"
  uid: 57898429-ca25-436a-a7da-afc0c4df25ec
2

kubectl exec -i -t dnsutils – nslookup kubernetes.default
Output was :

;; reply from unexpected source: 10.244.0.49#53, expected 10.96.0.10#53
;; reply from unexpected source: 10.244.0.49#53, expected 10.96.0.10#53

so i edited /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
And added --cluster-dns=10.244.0.49 to enviroment
Before edit

Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"

After Edit

Environment="--cluster-dns=10.244.0.49  KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"

Restarted following service

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Recreated deployments and it started working fine
For testing if configuration is working fine or not .i used this command.

kubectl exec -i -t dnsutils -- nslookup kubernetes.default
Server:         10.244.0.95
Address:        10.244.0.95#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

kubectl exec -i -t dnsutils -- nslookup google.com
Server:         10.244.0.95
Address:        10.244.0.95#53

Non-authoritative answer:
Name:   google.com
Address: 142.250.196.46

Removed this from deployment.yml file

dnsConfig:
        nameservers:
        - 10.96.0.10
        - 8.8.8.8
        - 8.8.4.4
        searches:
        - default.svc.cluster.local
        - svc.cluster.local
        - cluster.local
        - playground.io

i applied this configuration also but i dont think it affected anything :

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
data:
  upstreamNameservers: |-
    ["8.8.8.8", "8.8.4.4"]
1 Like
3

After more debugging i found that issue occurs when server restarts to resolve the conflict i need to run following command :

modprobe br_netfilter

found above command from github issue of kubernetes
so i create service on linux which will execute command whenever server get restarted

1 Like
4

@shivam99 it worked like a magic , Thank you soo much for sharing , searching this for hours but why and how to make it not happen after restart , like not using custom scripts any idea ?

5

@Ravi_Shankar
on ubuntu i did something like this by creating service execute systemmd_command.sh
modprobe_br_netfilter.sh

#!/bin/bash

/sbin/modprobe br_netfilter

systemmd_command.sh

#!/bin/bash

echo -e "Copying shell script"

cp ./modprobe_br_netfilter.sh /usr/local/bin/

echo -e "Setting up service to resolve issue on restart"

# Create a file by adding the following content

cat <<EOF | sudo tee /etc/systemd/system/modprobe_br_netfilter.service
[Unit]
Description=Load br_netfilter kernel module

[Service]
Type=oneshot
ExecStart=/usr/local/bin/modprobe_br_netfilter.sh

[Install]
WantedBy=multi-user.target
EOF

echo -e "Reloading daemon"

systemctl daemon-reload

echo -e "Enabling service"

sudo systemctl enable modprobe_br_netfilter.service

echo -e "Finished creating service"