Kubernetes Pods does not communicate at all

General Discussions
1

Hi,

after reinstalling my little kubernetes cluster I have the problem that all my pods come up but all pods have no network access. The pods are unable to reach each other via IP and are unable to reach coredns.

Services:

NAME                        TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)                                        AGE
postgres                    NodePort    192.168.139.4     <none>        5432:30535/TCP                                 135m
synaptic-enrol20-service    ClusterIP   192.168.189.20    <none>        9502/TCP                                       135m
synaptic-fss20-service      ClusterIP   192.168.199.89    <none>        9411/TCP,9401/TCP                              135m
synaptic-hashfs20-service   ClusterIP   192.168.129.47    <none>        9402/TCP,9405/TCP                              135m
synaptic-node20-service     NodePort    192.168.203.203   <none>        9201:30881/TCP,9210:31439/TCP,9202:30202/TCP   135m
synaptic-panel20-service    NodePort    192.168.149.174   <none>        8080:30121/TCP,9601:30610/TCP                  135m
synaptic-ris20-service      ClusterIP   192.168.205.28    <none>        9501/TCP,9210/TCP                              135m
synaptic-webdav20-service   ClusterIP   192.168.181.39    <none>        9521/TCP,8086/TCP                              135m

Endpoints:

NAME                        ENDPOINTS                                                  AGE
postgres                    192.168.2.124:5432                                         136m
synaptic-enrol20-service    192.168.2.122:9502                                         136m
synaptic-fss20-service      192.168.2.121:9401,192.168.2.121:9411                      136m
synaptic-hashfs20-service   192.168.2.120:9405,192.168.2.120:9402                      136m
synaptic-node20-service     192.168.2.117:9210,192.168.2.117:9201,192.168.2.117:9202   136m
synaptic-panel20-service    192.168.2.118:9601,192.168.2.118:8080                      136m
synaptic-ris20-service      192.168.2.119:9501,192.168.2.119:9210                      136m
synaptic-webdav20-service   192.168.2.123:8086,192.168.2.123:9521                      136m

Pods:

NAMESPACE     NAME                                   READY   STATUS             RESTARTS         AGE    IP               NODE             NOMINATED NODE   READINESS GATES
default       dnsutils                               1/1     Running            0                59m    192.168.2.125    synaptic-k8s-2   <none>           <none>
fss           fss-enrol20-59b5b789c7-fc5vr           1/1     Running            0                141m   192.168.2.122    synaptic-k8s-2   <none>           <none>
fss           fss-fss20-598f665494-8k5pn             1/1     Running            0                141m   192.168.2.121    synaptic-k8s-2   <none>           <none>
fss           fss-hashfs20-766476b-9s9fc             1/1     Running            0                141m   192.168.2.120    synaptic-k8s-2   <none>           <none>
fss           fss-node20-5b7b498dd5-gmr8t            1/1     Running            0                141m   192.168.2.117    synaptic-k8s-2   <none>           <none>
fss           fss-panel20-779d694d57-wj6jq           0/1     CrashLoopBackOff   26 (4m20s ago)   141m   192.168.2.118    synaptic-k8s-2   <none>           <none>
fss           fss-postgres-5fc5c4cb56-hhkcg          1/1     Running            0                141m   192.168.2.124    synaptic-k8s-2   <none>           <none>
fss           fss-ris20-74749bcd66-kr8lv             1/1     Running            0                141m   192.168.2.119    synaptic-k8s-2   <none>           <none>
fss           fss-webdav20-7f869b9484-q6v2r          1/1     Running            0                141m   192.168.2.123    synaptic-k8s-2   <none>           <none>
kube-system   coredns-5dd5756b68-dwjxf               1/1     Running            1 (42m ago)      149m   192.168.0.168    synaptic-k8s     <none>           <none>
kube-system   coredns-5dd5756b68-rrjp4               1/1     Running            1 (42m ago)      149m   192.168.0.167    synaptic-k8s     <none>           <none>
kube-system   etcd-synaptic-k8s                      1/1     Running            4 (42m ago)      149m   192.168.38.197   synaptic-k8s     <none>           <none>
kube-system   kube-apiserver-synaptic-k8s            1/1     Running            1 (42m ago)      149m   192.168.38.197   synaptic-k8s     <none>           <none>
kube-system   kube-controller-manager-synaptic-k8s   1/1     Running            1 (42m ago)      149m   192.168.38.197   synaptic-k8s     <none>           <none>
kube-system   kube-proxy-2qp9s                       1/1     Running            0                143m   192.168.38.91    synaptic-k8s-2   <none>           <none>
kube-system   kube-proxy-6rqf9                       1/1     Running            1 (42m ago)      149m   192.168.38.197   synaptic-k8s     <none>           <none>
kube-system   kube-scheduler-synaptic-k8s            1/1     Running            2 (42m ago)      149m   192.168.38.197   synaptic-k8s     <none>           <none>

NSlookup:

root@dnsutils:/$ nslookup kubernetes.default     
;; connection timed out; no servers could be reached

Ping:
never returns

root@dnsutils:/# ping 192.168.2.122
PING 192.168.2.122 (192.168.2.122) 56(84) bytes of data.

Any hints for diagnosing this ?

Thanks in advance.

Cluster information:

Kubernetes version:1.28.2-00
Cloud being used: bare-metal
Installation method: debian installer apt
Host OS: debian buster
CNI and version: 1.2.0-00
CRI and version: containerd.io 1.6.24-1

Setup:
kubectl get services kube-dns --namespace=kube-system

2

Just for the records I solved this by resetting all nodes. Rebooting and restarting them. This not a good solution because it does not explain what happened. I think that something with kube-router failed and killed all the network traffic.