Kubernetes offline installation, kube-dns has no endpoints. Need help to resolve the issue

General Discussions
,
1

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

MasterNode IP: 10.4.1.244

Kubernetes version: 1.18.4
Installation method: offline installation

Kubeadm command to create cluster:

kubeadm init --kubernetes-version=1.18.4 --pod-network-cidr=10.5.1.0/24 --service-cidr=10.6.0.0/16 --apiserver-advertise-address=10.4.1.244 --v=5

Host OS: Red Hat Enterprise Linux Server release 7.7 (Maipo)

CNI and version: calico v3.14
CRI and version: docker v20.10.14

Services present in cluster:

[root@k8smaster ~]# kubectl get svc -A
NAMESPACE     NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE

default       kubernetes   ClusterIP   10.6.0.1     <none>        443/TCP                  6d19h
kube-system   kube-dns     ClusterIP   10.6.0.10    <none>        53/UDP,53/TCP,9153/TCP   6d19h

Kube-dns service does not consist endpoints by default.

[root@k8smaster ~]# kubectl describe svc kube-dns -n kube-system
Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=KubeDNS
Annotations:       prometheus.io/port: 9153
                   prometheus.io/scrape: true
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.6.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         <none>
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         <none>
Port:              metrics  9153/TCP
TargetPort:        9153/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

Thus IPtables gets populated with REJECT entries.

[root@k8smaster ~]# iptables -L

Chain KUBE-SERVICES (3 references)
target     prot opt source               destination
REJECT     udp  --  anywhere             10.6.0.10            /* kube-system/kube-dns:dns has no endpoints */ udp dpt:domain reject-with icmp-port-unreachable
REJECT     tcp  --  anywhere             10.6.0.10            /* kube-system/kube-dns:dns-tcp has no endpoints */ tcp dpt:domain reject-with icmp-port-unreachable
REJECT     tcp  --  anywhere             10.6.0.10            /* kube-system/kube-dns:metrics has no endpoints */ tcp dpt:9153 reject-with icmp-port-unreachable

The Calico pods too are not running and falling to CrashLoopBackOff state:

[root@k8smaster ~]# kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-65f8bc95db-rx2hm   0/1     ContainerCreating   0          3d5h
kube-system   calico-node-gzh82                          0/1     CrashLoopBackOff    1169       6d
kube-system   coredns-66bff467f8-42ppc                   0/1     ContainerCreating   0          3d5h
kube-system   coredns-66bff467f8-cpkpv                   0/1     ContainerCreating   0          3d5h
kube-system   etcd-k8smaster                             1/1     Running             29         6d18h
kube-system   kube-apiserver-k8smaster                   1/1     Running             30         6d18h
kube-system   kube-controller-manager-k8smaster          1/1     Running             7          6d18h
kube-system   kube-proxy-wwj4g                           1/1     Running             7          6d18h
kube-system   kube-scheduler-k8smaster                   1/1     Running             7          6d18h

kubectl describe pods calico-node-gzh82 -n kube-system

Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Warning  Unhealthy  58m (x642 over 39h)     kubelet  (combined from similar events): Liveness probe failed: calico/node is not ready: Felix is not live: Get http://localhost:9099/liveness: dial tcp [::1]:9099: connect: connection refused
  Warning  Unhealthy  54m (x15 over 57m)      kubelet  Readiness probe errored: rpc error: code = Unknown desc = container not running (3c034204d666ba35cb9c1f35010655c613636524f6472b5d59822998c5c8a314)
  Normal   Pulled     46m (x525 over 41h)     kubelet  Container image "calico/node:v3.14.2" already present on machine
  Warning  Unhealthy  44m (x3 over 45m)       kubelet  Readiness probe errored: rpc error: code = Unknown desc = container not running (c5715a380668e501367b83a2da52cfc2f0b1f52cb2fa211320f09524dda62521)
  Normal   Killing    35m (x528 over 41h)     kubelet  Container calico-node failed liveness probe, will be restarted
  Warning  Unhealthy  19m (x5479 over 41h)    kubelet  Readiness probe failed: calico/node is not ready: BIRD is not ready: Failed to stat() nodename file: stat /var/lib/calico/nodename: no such file or directory
  Warning  BackOff    11m (x1408 over 41h)    kubelet  Back-off restarting failed container
  Warning  Unhealthy  7m26s (x1555 over 41h)  kubelet  Liveness probe failed: calico/node is not ready: bird/confd is not live: exit status 1
  Warning  Unhealthy  4m56s (x1413 over 41h)  kubelet  Liveness probe failed: calico/node is not ready: Felix is not live: Get http://localhost:9099/liveness: dial tcp [::1]:9099: connect: connection refused


[root@k8smaster ~]# kubectl logs calico-node-gzh82 -n kube-system
2022-08-25 04:54:04.083 [INFO][10] startup/startup.go 299: Early log level set to info
2022-08-25 04:54:04.084 [INFO][10] startup/startup.go 315: Using NODENAME environment for node name
2022-08-25 04:54:04.084 [INFO][10] startup/startup.go 327: Determined node name: k8smaster
2022-08-25 04:54:04.092 [INFO][10] startup/startup.go 359: Checking datastore connection
2022-08-25 04:54:04.094 [INFO][10] startup/startup.go 374: Hit error connecting to datastore - retry error=Get https://10.6.0.1:443/api/v1/nodes/foo: dial tcp 10.6.0.1:443: connect: network is unreachable
2022-08-25 04:54:05.098 [INFO][10] startup/startup.go 374: Hit error connecting to datastore - retry error=Get https://10.6.0.1:443/api/v1/nodes/foo: dial tcp 10.6.0.1:443: connect: network is unreachable
2022-08-25 04:54:06.099 [INFO][10] startup/startup.go 374: Hit error connecting to datastore - retry error=Get https://10.6.0.1:443/api/v1/nodes/foo: dial tcp 10.6.0.1:443: connect: network is unreachable
2022-08-25 04:54:07.110 [INFO][10] startup/startup.go 374: Hit error connecting to datastore - retry error=Get https://10.6.0.1:443/api/v1/nodes/foo: dial tcp 10.6.0.1:443: connect: network is unreachable

Contents of Kubeadm-config config map:

[root@k8smaster ~]# kubectl describe cm kubeadm-config -n kube-system
Name:         kubeadm-config
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Data
====
ClusterConfiguration:
----
apiServer:
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.18.4
networking:
  dnsDomain: cluster.local
  podSubnet: 10.5.1.0/24
  serviceSubnet: 10.6.0.0/16
scheduler: {}

ClusterStatus:
----
apiEndpoints:
  k8smaster:
    advertiseAddress: 10.4.1.244
    bindPort: 6443
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterStatus

Events:  <none>
2

can you send the output of:

kubectl describe deploy coredns -n kube-system

and

kubectl describe rs coredns-<replicaset> -n kube-system
3

@Theog75 Thankyou for response!

[root@k8smaster ~]# kubectl describe deploy coredns -n kube-system

Name:                   coredns
Namespace:              kube-system
CreationTimestamp:      Thu, 18 Aug 2022 16:05:30 +0530
Labels:                 k8s-app=kube-dns
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               k8s-app=kube-dns
Replicas:               2 desired | 2 updated | 2 total | 0 available | 2 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=kube-dns
  Service Account:  coredns
  Containers:
   coredns:
    Image:       k8s.gcr.io/coredns:1.6.7
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
  Volumes:
   config-volume:
    Type:               ConfigMap (a volume populated by a ConfigMap)
    Name:               coredns
    Optional:           false
  Priority Class Name:  system-cluster-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    False   ProgressDeadlineExceeded
OldReplicaSets:  <none>
NewReplicaSet:   coredns-66bff467f8 (2/2 replicas created)
Events:          <none>

[root@k8smaster ~]# kubectl describe rs coredns-66bff467f8 -n kube-system

Name:           coredns-66bff467f8
Namespace:      kube-system
Selector:       k8s-app=kube-dns,pod-template-hash=66bff467f8
Labels:         k8s-app=kube-dns
                pod-template-hash=66bff467f8
Annotations:    deployment.kubernetes.io/desired-replicas: 2
                deployment.kubernetes.io/max-replicas: 3
                deployment.kubernetes.io/revision: 1
Controlled By:  Deployment/coredns
Replicas:       2 current / 2 desired
Pods Status:    0 Running / 2 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           k8s-app=kube-dns
                    pod-template-hash=66bff467f8
  Service Account:  coredns
  Containers:
   coredns:
    Image:       k8s.gcr.io/coredns:1.6.7
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
  Volumes:
   config-volume:
    Type:               ConfigMap (a volume populated by a ConfigMap)
    Name:               coredns
    Optional:           false
  Priority Class Name:  system-cluster-critical
Events:                 <none>
4

One observation from system.
/etc/resolve.conf file is empty in the master node. Can that lead for kube-dns to have no endpoint??