# Kubernetes offline installation, kube-dns has no endpoints. Need help to resolve the issue **URL:** \ **Category:** General Discussions\ **Tags:** development, network\ **Created:** [August 25, 2022, 8:50am UTC](https://discuss.kubernetes.io/t/kubernetes-offline-installation-kube-dns-has-no-endpoints-need-help-to-resolve-the-issue/21081 "2022-08-25T08:50:07Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Gaug](https://avatars.discourse-cdn.com/v4/letter/g/c5a1d2/32.png) [@Gaug](https://discuss.kubernetes.io/u/Gaug)\ **Post date:** [August 25, 2022, 8:50am UTC](https://discuss.kubernetes.io/t/kubernetes-offline-installation-kube-dns-has-no-endpoints-need-help-to-resolve-the-issue/21081/1 "2022-08-25T08:50:07Z")
Asking for help? Comment out what you need so we can get more information to help you! ### Cluster information: MasterNode IP: 10.4.1.244 Kubernetes version: 1.18.4 Installation method: offline installation Kubeadm command to create cluster: ``` kubeadm init --kubernetes-version=1.18.4 --pod-network-cidr=10.5.1.0/24 --service-cidr=10.6.0.0/16 --apiserver-advertise-address=10.4.1.244 --v=5 ``` Host OS: Red Hat Enterprise Linux Server release 7.7 (Maipo) CNI and version: calico v3.14 CRI and version: docker v20.10.14 Services present in cluster: ``` [root@k8smaster ~]# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.6.0.1 443/TCP 6d19h kube-system kube-dns ClusterIP 10.6.0.10 53/UDP,53/TCP,9153/TCP 6d19h ``` Kube-dns service does not consist endpoints by default. ``` [root@k8smaster ~]# kubectl describe svc kube-dns -n kube-system Name: kube-dns Namespace: kube-system Labels: k8s-app=kube-dns kubernetes.io/cluster-service=true kubernetes.io/name=KubeDNS Annotations: prometheus.io/port: 9153 prometheus.io/scrape: true Selector: k8s-app=kube-dns Type: ClusterIP IP: 10.6.0.10 Port: dns 53/UDP TargetPort: 53/UDP Endpoints: Port: dns-tcp 53/TCP TargetPort: 53/TCP Endpoints: Port: metrics 9153/TCP TargetPort: 9153/TCP Endpoints: Session Affinity: None Events: ``` Thus IPtables gets populated with REJECT entries. ``` [root@k8smaster ~]# iptables -L Chain KUBE-SERVICES (3 references) target prot opt source destination REJECT udp -- anywhere 10.6.0.10 /* kube-system/kube-dns:dns has no endpoints */ udp dpt:domain reject-with icmp-port-unreachable REJECT tcp -- anywhere 10.6.0.10 /* kube-system/kube-dns:dns-tcp has no endpoints */ tcp dpt:domain reject-with icmp-port-unreachable REJECT tcp -- anywhere 10.6.0.10 /* kube-system/kube-dns:metrics has no endpoints */ tcp dpt:9153 reject-with icmp-port-unreachable ``` The Calico pods too are not running and falling to CrashLoopBackOff state: ``` [root@k8smaster ~]# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-65f8bc95db-rx2hm 0/1 ContainerCreating 0 3d5h kube-system calico-node-gzh82 0/1 CrashLoopBackOff 1169 6d kube-system coredns-66bff467f8-42ppc 0/1 ContainerCreating 0 3d5h kube-system coredns-66bff467f8-cpkpv 0/1 ContainerCreating 0 3d5h kube-system etcd-k8smaster 1/1 Running 29 6d18h kube-system kube-apiserver-k8smaster 1/1 Running 30 6d18h kube-system kube-controller-manager-k8smaster 1/1 Running 7 6d18h kube-system kube-proxy-wwj4g 1/1 Running 7 6d18h kube-system kube-scheduler-k8smaster 1/1 Running 7 6d18h kubectl describe pods calico-node-gzh82 -n kube-system Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 58m (x642 over 39h) kubelet (combined from similar events): Liveness probe failed: calico/node is not ready: Felix is not live: Get http://localhost:9099/liveness: dial tcp [::1]:9099: connect: connection refused Warning Unhealthy 54m (x15 over 57m) kubelet Readiness probe errored: rpc error: code = Unknown desc = container not running (3c034204d666ba35cb9c1f35010655c613636524f6472b5d59822998c5c8a314) Normal Pulled 46m (x525 over 41h) kubelet Container image "calico/node:v3.14.2" already present on machine Warning Unhealthy 44m (x3 over 45m) kubelet Readiness probe errored: rpc error: code = Unknown desc = container not running (c5715a380668e501367b83a2da52cfc2f0b1f52cb2fa211320f09524dda62521) Normal Killing 35m (x528 over 41h) kubelet Container calico-node failed liveness probe, will be restarted Warning Unhealthy 19m (x5479 over 41h) kubelet Readiness probe failed: calico/node is not ready: BIRD is not ready: Failed to stat() nodename file: stat /var/lib/calico/nodename: no such file or directory Warning BackOff 11m (x1408 over 41h) kubelet Back-off restarting failed container Warning Unhealthy 7m26s (x1555 over 41h) kubelet Liveness probe failed: calico/node is not ready: bird/confd is not live: exit status 1 Warning Unhealthy 4m56s (x1413 over 41h) kubelet Liveness probe failed: calico/node is not ready: Felix is not live: Get http://localhost:9099/liveness: dial tcp [::1]:9099: connect: connection refused [root@k8smaster ~]# kubectl logs calico-node-gzh82 -n kube-system 2022-08-25 04:54:04.083 [INFO][10] startup/startup.go 299: Early log level set to info 2022-08-25 04:54:04.084 [INFO][10] startup/startup.go 315: Using NODENAME environment for node name 2022-08-25 04:54:04.084 [INFO][10] startup/startup.go 327: Determined node name: k8smaster 2022-08-25 04:54:04.092 [INFO][10] startup/startup.go 359: Checking datastore connection 2022-08-25 04:54:04.094 [INFO][10] startup/startup.go 374: Hit error connecting to datastore - retry error=Get https://10.6.0.1:443/api/v1/nodes/foo: dial tcp 10.6.0.1:443: connect: network is unreachable 2022-08-25 04:54:05.098 [INFO][10] startup/startup.go 374: Hit error connecting to datastore - retry error=Get https://10.6.0.1:443/api/v1/nodes/foo: dial tcp 10.6.0.1:443: connect: network is unreachable 2022-08-25 04:54:06.099 [INFO][10] startup/startup.go 374: Hit error connecting to datastore - retry error=Get https://10.6.0.1:443/api/v1/nodes/foo: dial tcp 10.6.0.1:443: connect: network is unreachable 2022-08-25 04:54:07.110 [INFO][10] startup/startup.go 374: Hit error connecting to datastore - retry error=Get https://10.6.0.1:443/api/v1/nodes/foo: dial tcp 10.6.0.1:443: connect: network is unreachable ``` Contents of Kubeadm-config config map: ``` [root@k8smaster ~]# kubectl describe cm kubeadm-config -n kube-system Name: kubeadm-config Namespace: kube-system Labels: Annotations: Data ==== ClusterConfiguration: ---- apiServer: extraArgs: authorization-mode: Node,RBAC timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: v1.18.4 networking: dnsDomain: cluster.local podSubnet: 10.5.1.0/24 serviceSubnet: 10.6.0.0/16 scheduler: {} ClusterStatus: ---- apiEndpoints: k8smaster: advertiseAddress: 10.4.1.244 bindPort: 6443 apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterStatus Events: ``` ---
**Author:** ![Theog75](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/theog75/32/9241_2.png) [@Theog75](https://discuss.kubernetes.io/u/Theog75)\ **Post date:** [August 27, 2022, 10:56am UTC](https://discuss.kubernetes.io/t/kubernetes-offline-installation-kube-dns-has-no-endpoints-need-help-to-resolve-the-issue/21081/2 "2022-08-27T10:56:16Z")
can you send the output of: ```auto kubectl describe deploy coredns -n kube-system ``` and ```auto kubectl describe rs coredns- -n kube-system ``` ---
**Author:** ![Gaug](https://avatars.discourse-cdn.com/v4/letter/g/c5a1d2/32.png) [@Gaug](https://discuss.kubernetes.io/u/Gaug)\ **Post date:** [August 29, 2022, 4:47am UTC](https://discuss.kubernetes.io/t/kubernetes-offline-installation-kube-dns-has-no-endpoints-need-help-to-resolve-the-issue/21081/3 "2022-08-29T04:47:36Z")
@Theog75 Thankyou for response! [root@k8smaster ~]# kubectl describe deploy coredns -n kube-system ```auto Name: coredns Namespace: kube-system CreationTimestamp: Thu, 18 Aug 2022 16:05:30 +0530 Labels: k8s-app=kube-dns Annotations: deployment.kubernetes.io/revision: 1 Selector: k8s-app=kube-dns Replicas: 2 desired | 2 updated | 2 total | 0 available | 2 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 1 max unavailable, 25% max surge Pod Template: Labels: k8s-app=kube-dns Service Account: coredns Containers: coredns: Image: k8s.gcr.io/coredns:1.6.7 Ports: 53/UDP, 53/TCP, 9153/TCP Host Ports: 0/UDP, 0/TCP, 0/TCP Args: -conf /etc/coredns/Corefile Limits: memory: 170Mi Requests: cpu: 100m memory: 70Mi Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5 Readiness: http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: Mounts: /etc/coredns from config-volume (ro) Volumes: config-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: coredns Optional: false Priority Class Name: system-cluster-critical Conditions: Type Status Reason ---- ------ ------ Available False MinimumReplicasUnavailable Progressing False ProgressDeadlineExceeded OldReplicaSets: NewReplicaSet: coredns-66bff467f8 (2/2 replicas created) Events: ``` [root@k8smaster ~]# kubectl describe rs coredns-66bff467f8 -n kube-system ```auto Name: coredns-66bff467f8 Namespace: kube-system Selector: k8s-app=kube-dns,pod-template-hash=66bff467f8 Labels: k8s-app=kube-dns pod-template-hash=66bff467f8 Annotations: deployment.kubernetes.io/desired-replicas: 2 deployment.kubernetes.io/max-replicas: 3 deployment.kubernetes.io/revision: 1 Controlled By: Deployment/coredns Replicas: 2 current / 2 desired Pods Status: 0 Running / 2 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: k8s-app=kube-dns pod-template-hash=66bff467f8 Service Account: coredns Containers: coredns: Image: k8s.gcr.io/coredns:1.6.7 Ports: 53/UDP, 53/TCP, 9153/TCP Host Ports: 0/UDP, 0/TCP, 0/TCP Args: -conf /etc/coredns/Corefile Limits: memory: 170Mi Requests: cpu: 100m memory: 70Mi Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5 Readiness: http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: Mounts: /etc/coredns from config-volume (ro) Volumes: config-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: coredns Optional: false Priority Class Name: system-cluster-critical Events: ``` ---
**Author:** ![Gaug](https://avatars.discourse-cdn.com/v4/letter/g/c5a1d2/32.png) [@Gaug](https://discuss.kubernetes.io/u/Gaug)\ **Post date:** [September 6, 2022, 5:01am UTC](https://discuss.kubernetes.io/t/kubernetes-offline-installation-kube-dns-has-no-endpoints-need-help-to-resolve-the-issue/21081/4 "2022-09-06T05:01:19Z")
One observation from system. /etc/resolve.conf file is empty in the master node. Can that lead for kube-dns to have no endpoint??