Can't get Kubectl info from master node after joining a worker node

General Discussions
1

Cluster information:

kubectl version
Client Version: v1.29.14
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.14

Cloud being used: bare-metal
Installation method:
Host OS: AlmaLinux 8
CNI and version: Flannel ver: 0.26.4
CRI and version: cri-dockerd ver: 0.3.16

I have a master node and create my first worker node, before executing the command kubeadm join in the worker I could ping from the worker to the master and viceversa without troubles, now that I have executed the kubeadm join ... command I cannot ping between them anymore and I get this error:

[root@worker-1 ~]# kubectl get nodes -o wide
E0308 19:38:31.027307   59324 memcache.go:265] couldn't get current server API group list: Get "https://198.58.126.88:6443/api?timeout=32s": dial tcp 198.58.126.88:6443: connect: connection refused
E0308 19:38:32.051145   59324 memcache.go:265] couldn't get current server API group list: Get "https://198.58.126.88:6443/api?timeout=32s": dial tcp 198.58.126.88:6443: connect: connection refused
E0308 19:38:33.075350   59324 memcache.go:265] couldn't get current server API group list: Get "https://198.58.126.88:6443/api?timeout=32s": dial tcp 198.58.126.88:6443: connect: connection refused
E0308 19:38:34.099160   59324 memcache.go:265] couldn't get current server API group list: Get "https://198.58.126.88:6443/api?timeout=32s": dial tcp 198.58.126.88:6443: connect: connection refused
E0308 19:38:35.123011   59324 memcache.go:265] couldn't get current server API group list: Get "https://198.58.126.88:6443/api?timeout=32s": dial tcp 198.58.126.88:6443: connect: connection refused
The connection to the server 198.58.126.88:6443 was refused - did you specify the right host or port?

What might be causing this blocking on the worker node?
Thanks.

1 Like
2

If I run this:

[root@worker-1 ~]# iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X

The ping command starts to work:

[root@worker-1 ~]# ping 198.58.126.88
PING 198.58.126.88 (198.58.126.88) 56(84) bytes of data.
64 bytes from 198.58.126.88: icmp_seq=1 ttl=64 time=0.030 ms
64 bytes from 198.58.126.88: icmp_seq=2 ttl=64 time=0.025 ms

But after about one minute it gets blocked again:

[root@worker-1 ~]# ping 198.58.126.88
PING 198.58.126.88 (198.58.126.88) 56(84) bytes of data.
From 198.58.126.88 icmp_seq=1 Destination Port Unreachable
From 198.58.126.88 icmp_seq=2 Destination Port Unreachable
3

The blocked connection from the worker to the master starts to happen as soon as the kubelet service is running; if the kubelet service is stopped then I can ping back the master from the worker.

4

The final solution was to uninstall kubernetes and the cluster and install it back again. Now I can see both nodes Ready in the worker and master nodes.