Asking for help? Comment out what you need so we can get more information to help you!
Hi,
I have a strange error on my K8S cluster. it has been running for months without a problem and suddenly since yesterday I can not access my pods anymore. I even reinstalled the cluster with clean Linux images and updated my K8s version, but the problem persists.
I followed the steps on Debug Services | Kubernetes and the problem is that if a pod runs on port 80 I can connect to it, from inside the cluster as outside.
root@calibre-comics-6d7c4bd4cc-62ndf:/# curl 10.233.19.152
Hostname: whoami-78466cdd67-v99zq
IP: 127.0.0.1
IP: 10.233.94.4
RemoteAddr: 10.233.94.3:51520
GET / HTTP/1.1
Host: 10.233.19.152
User-Agent: curl/7.58.0
Accept: /
But if i try to connect to the service port of a pod not running on port 80, i get a connection refused.
root@calibre-comics-6d7c4bd4cc-62ndf:/# curl 10.233.27.194:8096
curl: (7) Failed to connect to 10.233.27.194 port 8096: Connection refused
root@calibre-comics-6d7c4bd4cc-62ndf:/# curl 10.233.120.5:8096
curl: (7) Failed to connect to 10.233.120.5 port 8096: Connection refused
root@rockpro64-1:~# kubectl get pods -n emby -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
emby-84594b7d77-kl4dn 1/1 Running 0 29m 10.233.120.5 rockpro64-5
root@rockpro64-1:~# kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
calibre-comics calibre-comics LoadBalancer 10.233.20.236 192.168.4.101 80:30162/TCP 22m
default kubernetes ClusterIP 10.233.0.1 443/TCP 41m
default whoami LoadBalancer 10.233.19.152 192.168.4.103 80:31210/TCP 17m
emby emby LoadBalancer 10.233.27.194 192.168.4.102 80:32634/TCP 19m
kube-system coredns ClusterIP 10.233.0.3 53/UDP,53/TCP,9153/TCP 32m
kube-system dashboard-metrics-scraper ClusterIP 10.233.40.217 8000/TCP 32m
kube-system kubernetes-dashboard ClusterIP 10.233.54.67 443/TCP 32m
The images are images that worked yesterday perfect and suddenly it stopped and they all have it, so i don’t think it is an application error. I would think it is for firewall problem or so but i really don’t know. Can you help ?
Thank you
Cluster information:
Kubernetes version: v1.18.10
Cloud being used: bare-metal
Installation method: kubespray
Host OS: Linux rockpro64-1 4.4.190-1233-rockchip-ayufan-gd3f1be0ed310 #1 SMP Wed Aug 28 08:59:34 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
CNI and version:
CRI and version:
You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.