CoreDNS not working

General Discussions
1

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: v.17.3
Cloud being used: (put bare-metal if not on a public cloud) No
Installation method: kubeadm
Host OS: RedHat 7.7
CNI and version: flannel
CRI and version:

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.
kube-system coredns-6955765f44-tblgj 0/1 Running 0 5d9h
kube-system coredns-6955765f44-x9fwc 0/1 Running 0 5d9h
kube-system etcd-podaac-testk8s 1/1 Running 1 5d9h
kube-system kube-apiserver-podaac-testk8s 1/1 Running 1 5d9h
kube-system kube-controller-manager-podaac-testk8s 1/1 Running 1 5d9h
kube-system kube-flannel-ds-amd64-8tfc5 1/1 Running 0 5d9h
kube-system kube-flannel-ds-amd64-dwf29 1/1 Running 1 5d9h
kube-system kube-flannel-ds-amd64-zvr82 1/1 Running 0 5d9h
kube-system kube-proxy-gz65p 1/1 Running 1 5d9h
kube-system kube-proxy-kg8xd 1/1 Running 0 5d9h
kube-system kube-proxy-xjlgp 1/1 Running 0 5d9h
kube-system kube-scheduler-podaac-testk8s 1/1 Running 1 5d9h

kubectl get service kube-dns --namespace=kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 5d9h

But no endpoint:
kubectl get ep kube-dns --namespace=kube-system
NAME ENDPOINTS AGE
kube-dns 5d9h

Error Log
E0220 02:30:26.768407 1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0220 02:30:26.768407 1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

After applied dnsutils.yaml:
manifests]# kubectl exec dnsutils cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local jpl.nasa.gov
options ndots:5

kubectl exec -ti dnsutils – nslookup kubernetes.default
;; connection timed out; no servers could be reached

command terminated with exit code 1

Thanks much in advance.

2

looking for iptables INPUT chain. Add new rule like tcp -dport 443 -j ACCEPT.

3

Thanks for your response.

Here is the output from iptables
iptables -L INPUT
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all – anywhere anywhere ctstate NEW /* kubernetes service portals /
KUBE-EXTERNAL-SERVICES all – anywhere anywhere ctstate NEW / kubernetes externally-visible service portals */

4

-A INPUT -m conntrack --ctstate NEW -m comment --comment “kubernetes service portals” -j KUBE-SERVICES
-A INPUT -m conntrack --ctstate NEW -m comment --comment “kubernetes externally-visible service portals” -j KUBE-EXTERNAL-SERVICES
it’s conntrack only. Not accept

5

problem on port 6443! add rule please:
iptables -A INPUT -p tcp -m tcp --dport 6443 -j ACCEPT

6

Thanks.
iptables -A INPUT -p tcp -m tcp --dport 6443 -j ACCEPT
iptables -L INPUT
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere
ACCEPT tcp – anywhere anywhere tcp dpt:sun-sr-https

7

Chain KUBE-SERVICES (2 references)
num target prot opt source destination
1 REJECT tcp – anywhere 10.96.0.10 /* kube-system/kube-dns:metrics has no endpoints / tcp dpt:9153 reject-with icmp-port-unreachable
2 REJECT udp – anywhere 10.96.0.10 / kube-system/kube-dns:dns has no endpoints / udp dpt:domain reject-with icmp-port-unreachable
3 REJECT tcp – anywhere 10.96.0.10 / kube-system/kube-dns:dns-tcp has no endpoints */ tcp dpt:domain reject-with icmp-port-unreachable

8

Here is the describe for coredns
kubectl describe pod coredns-6955765f44-tblgj --namespace=kube-system
Name: coredns-6955765f44-tblgj
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Labels: k8s-app=kube-dns
pod-template-hash=6955765f44
Annotations:
Status: Running
IP: 10.240.1.4
IPs:
IP: 10.240.1.4
Controlled By: ReplicaSet/coredns-6955765f44
Containers:
coredns:
Container ID: docker://0c447f94c1c5eb5c0576382c82bcb9110c3ee9d43801bb5c1dcbf144082a2863
Image: k8s.gcr.io/coredns:1.6.5
Image ID: docker-pullable://k8s.gcr.io/coredns@sha256:7ec975f167d815311a7136c32e70735f0d00b73781365df1befd46ed35bd4fe7
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Running
Started: Fri, 14 Feb 2020 17:09:42 +0000
Ready: False
Restart Count: 0
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
Mounts:
/etc/coredns from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from coredns-token-nxpg2 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
coredns-token-nxpg2:
Type: Secret (a volume populated by a Secret)
SecretName: coredns-token-nxpg2
Optional: false
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

Warning Unhealthy 3m24s (x53130 over 6d3h) kubelet, Readiness probe failed: HTTP probe failed with statuscode: 503

9

Changed back to use the previous cni, it works now.

10

i recommend you use calico. I don’t have any problem with them.

11

I am looking for coredns noive contributing references and little help getting started.
Want to contribute on add-on operators.

12

Good evening, good morning,
Sorry for digging up the topic, but I have the same problem, as a bonus, I have a calico pod that is not on the IP pool, try to do:

Readiness probe failed: Error initializing datastore: Get “https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default”: dial tcp 10.96.0.1:443: i/o timeout

Do you have any idea?

13

Hello, my problem is solved

14

How did you solve the problem?

I’m getting:

E0403 13:54:46.663175       1 reflector.go:140] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=49236": dial tcp 10.43.0.1:443: i/o timeout