Kube-apiserver is gone / The connection to the server master:6443 was refused - did you specify the right host or port?

czezz · January 17, 2024, 9:06pm

Cluster information:

Kubernetes version: 1.20.1
Cloud being used: Virtualbox
Installation method: Build your own Multi-node Kubernetes Cluster with Ubuntu(20) on VirtualBox - techbeatly
Host OS: Ubuntu 20.04.4 LTS
CNI and version: calico
CRI and version:

This is my virtualbox kubernetes lab which I didnt use for a few months.
I have started it today and any kubectl command ends up like this:

$ kubectl get nodes -o wide
The connection to the server master:6443 was refused - did you specify the right host or port?

$ kubectl get po --all-namespaces
The connection to the server master:6443 was refused - did you specify the right host or port?

Indeed, there is nothing listening on socket master:6443

$ ss -tl4np
State                           Recv-Q                          Send-Q                                                     Local Address:Port                                                      Peer Address:Port                          Process
LISTEN                          0                               4096                                                           127.0.0.1:34627                                                          0.0.0.0:*
LISTEN                          0                               4096                                                           127.0.0.1:10248                                                          0.0.0.0:*
LISTEN                          0                               4096                                                       192.168.1.190:2379                                                           0.0.0.0:*
LISTEN                          0                               4096                                                           127.0.0.1:2379                                                           0.0.0.0:*
LISTEN                          0                               4096                                                       192.168.1.190:2380                                                           0.0.0.0:*
LISTEN                          0                               4096                                                           127.0.0.1:2381                                                           0.0.0.0:*
LISTEN                          0                               4096                                                           127.0.0.1:33133                                                          0.0.0.0:*
LISTEN                          0                               4096                                                           127.0.0.1:10257                                                          0.0.0.0:*
LISTEN                          0                               4096                                                           127.0.0.1:10259                                                          0.0.0.0:*
LISTEN                          0                               4096                                                       127.0.0.53%lo:53                                                             0.0.0.0:*
LISTEN                          0                               128                                                              0.0.0.0:22                                                             0.0.0.0:*

ccd@master:~$ curl https://master:6443
curl: (7) Failed to connect to master port 6443: Connection refused

As what I can see on the other cluster, process responsible for service on port 6443 is kube-apiserver. There is no such binary.

Question:

How do I start kube-apiserver
Why all of the suddent kube-apiserver is gone. It was working quite good last time i was this lab.

Side note: kubelet service is running but in in error state.


systemctl status kubelet
? kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             +-10-kubeadm.conf
     Active: active (running) since Wed 2024-01-17 19:55:47 UTC; 44min ago
       Docs: https://kubernetes.io/docs/home/
   Main PID: 613 (kubelet)
      Tasks: 15 (limit: 2274)
     Memory: 128.5M
     CGroup: /system.slice/kubelet.service
             +-613 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.6

Jan 17 20:40:43 master kubelet[613]: E0117 20:40:43.451465     613 kubelet.go:2422] "Error getting node" err="node \"master\" not found"
Jan 17 20:40:43 master kubelet[613]: E0117 20:40:43.552175     613 kubelet.go:2422] "Error getting node" err="node \"master\" not found"
Jan 17 20:40:43 master kubelet[613]: E0117 20:40:43.653822     613 kubelet.go:2422] "Error getting node" err="node \"master\" not found"
Jan 17 20:40:43 master kubelet[613]: E0117 20:40:43.754428     613 kubelet.go:2422] "Error getting node" err="node \"master\" not found"
Jan 17 20:40:43 master kubelet[613]: E0117 20:40:43.855082     613 kubelet.go:2422] "Error getting node" err="node \"master\" not found"
Jan 17 20:40:43 master kubelet[613]: E0117 20:40:43.957374     613 kubelet.go:2422] "Error getting node" err="node \"master\" not found"
Jan 17 20:40:44 master kubelet[613]: E0117 20:40:44.058207     613 kubelet.go:2422] "Error getting node" err="node \"master\" not found"
Jan 17 20:40:44 master kubelet[613]: E0117 20:40:44.158993     613 kubelet.go:2422] "Error getting node" err="node \"master\" not found"
Jan 17 20:40:44 master kubelet[613]: E0117 20:40:44.259665     613 kubelet.go:2422] "Error getting node" err="node \"master\" not found"
Jan 17 20:40:44 master kubelet[613]: E0117 20:40:44.360617     613 kubelet.go:2422] "Error getting node" err="node \"master\" not found"

czezz · January 18, 2024, 5:19pm

[UPDATE]

It seems that in my lab kube-apiserver is running as docker container:


    root@master:~# docker ps
    CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS     NAMES
    0adc676e6077   595f327f224a           "kube-scheduler --au…"   10 minutes ago   Up 10 minutes             k8s_kube-scheduler_kube-scheduler-master_kube-system_172567758300a3c99b36d0d4efd9321a_7
    8e5200bd131f   25f8c7f3da61           "etcd --advertise-cl…"   10 minutes ago   Up 10 minutes             k8s_etcd_etcd-master_kube-system_dd8c7aeab8fa24c31e933da39dcafd96_7
    7ad4134212ce   df7b72818ad2           "kube-controller-man…"   10 minutes ago   Up 10 minutes             k8s_kube-controller-manager_kube-controller-manager-master_kube-system_b38aa758e725e1c490cee42d42ec8bff_7
    6baca0741b74   k8s.gcr.io/pause:3.6   "/pause"                 10 minutes ago   Up 10 minutes             k8s_POD_etcd-master_kube-system_dd8c7aeab8fa24c31e933da39dcafd96_7
    9a5572813b8a   k8s.gcr.io/pause:3.6   "/pause"                 10 minutes ago   Up 10 minutes             k8s_POD_kube-controller-manager-master_kube-system_b38aa758e725e1c490cee42d42ec8bff_7
    daca7dae837c   k8s.gcr.io/pause:3.6   "/pause"                 10 minutes ago   Up 10 minutes             k8s_POD_kube-scheduler-master_kube-system_172567758300a3c99b36d0d4efd9321a_7
    10871d3a6e66   k8s.gcr.io/pause:3.6   "/pause"                 10 minutes ago   Up 10 minutes             k8s_POD_kube-apiserver-master_kube-system_01ecc291decf2feb7dae990cc7eb8cb6_7

In COMMAND column it says “/pause”. I tried to unpause it but no luck

root@master:~# docker unpause 10871d3a6e66
Error response from daemon: Container 10871d3a6e66c897af6cfa33ea4c45668045ab15bb29beab938556937589e3ad is not paused

fox-md · January 18, 2024, 8:28pm

Hi,
Try checking /var/log/pods directory. You might find kubeapi server logs. Otherwise check kubelet logs.

czezz · January 23, 2024, 12:46pm

[WORKAROUND]
I could not find any solution for my problem.
As this is just my VirtualBox LAB system (and I do not mind the data/projects to lose), I have re-initiated it (cluster).

Reset master node:

sudo su -
kubeadm reset -f
rm -rf /etc/cni /etc/kubernetes /var/lib/dockershim /var/lib/etcd /var/lib/kubelet /var/run/kubernetes ~/.kube/*

iptables -F && iptables -X
iptables -t nat -F && iptables -t nat -X
iptables -t raw -F && iptables -t raw -X
iptables -t mangle -F && iptables -t mangle -X

systemctl restart docker

exit
rm -rf ~/.kube/*

Run kubeadm init


sudo kubeadm init --ignore-preflight-errors=NumCPU --control-plane-endpoint master:6443 --pod-network-cidr 10.10.0.0/16

Add certificates ($HOME/.kube/config)


mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Apply CNI (in my case calico)


kubectl apply -f calico.yaml

kubectl get nodes
NAME     STATUS   ROLES                  AGE   VERSION
master   Ready    control-plane,master   29m   v1.23.4

Rejoin Node/Worker:

Make sure to have master entry in /etc/hosts
Re-join the node:


sudo kubeadm reset
sudo kubeadm join master:6443 --token aaaaaa.aaaaaaaaaaaaaaaa \
        --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

mohitkamboj2799 · February 1, 2024, 11:48am

check container exist log using
docker logs container id

then see what is the error

Topic		Replies	Views
The connection to the server <host>:6443 was refused - did you specify the right host or port? General Discussions	80	553506	October 17, 2024
The connection to the server Master_IP:6443 was refused - did you specify the right host or port? General Discussions development , security , network , kubernetes-custom-resources	0	885	June 3, 2024
I keep having issues with my cluster General Discussions	0	379	May 30, 2024
The connection to the server 192.168.1.2:6443 was refused - did you specify the right host or port? General Discussions	7	41132	July 16, 2024
Kubernetes becomes unavailable; connection to server was refused General Discussions	1	1390	February 16, 2023

Kube-apiserver is gone / The connection to the server master:6443 was refused - did you specify the right host or port?

Cluster information:

Related Topics