NodePort Connection Refused

General Discussions
1

Hi everyone!

I have a stuck when using NodePort for allow External call to my Services (I Could not use LoadBalancer Type because Vsphere does not support this one).

svc:

image
image1409×630 22.7 KB

Trying on nodes:

root@cluster-agent01:/home/spadmin# netstat -lnpt | grep 30901
tcp        0      0 0.0.0.0:30901           0.0.0.0:*               LISTEN      28276/kube-proxy

telnet:

telnet 127.0.0.1 30901
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

Could you please help explain me what is my missing and how to solved it.

Thanks,
Edgar

2

logs on kube-proxy

I1105 02:56:50.028893   28276 service.go:273] Service prometheus-operator/prometheus-operator-prometheus-thanos updated: 1 ports
I1105 02:56:50.029407   28276 service.go:394] Updating existing service port "prometheus-operator/prometheus-operator-prometheus-thanos:grpc" at 10.43.240.74:10901/TCP
I1105 02:56:50.029472   28276 proxier.go:845] Syncing iptables rules
I1105 02:56:50.065117   28276 proxier.go:1680] Opened local port "nodePort for prometheus-operator/prometheus-operator-prometheus-thanos:grpc" (:30901/tcp)
I1105 02:56:50.070991   28276 proxier.go:809] syncProxyRules took 42.048763ms
3

I checked on iptables also but could not see any define for port 30901 :frowning:

root@cluster-agent01:/var/log/kube-audit\# iptables-save | grep KUBE-SVC | grep -v grep
:KUBE-SVC-34KHMJXDM3NH5E7C - [0:0]
:KUBE-SVC-BNKSIWLLZGVIY7PW - [0:0]
:KUBE-SVC-DTHJ27J4JTIRFMMR - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-GAIKHQAFXHCEIZQX - [0:0]
:KUBE-SVC-JD5MR3NA4I4DYORP - [0:0]
:KUBE-SVC-JTFAIQOSQRKTQWS3 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-PHKTDTNNKLKS6FJV - [0:0]
:KUBE-SVC-QMWWTXBG7KFJQKLO - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A KUBE-SERVICES -d 10.43.66.86/32 -p tcp -m comment --comment "ingress-nginx/default-http-backend cluster IP" -m tcp --dport 80 -j KUBE-SVC-JTFAIQOSQRKTQWS3
-A KUBE-SERVICES -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.43.156.15/32 -p tcp -m comment --comment "kube-system/metrics-server cluster IP" -m tcp --dport 443 -j KUBE-SVC-QMWWTXBG7KFJQKLO
-A KUBE-SERVICES -d 10.43.59.185/32 -p tcp -m comment --comment "pal-ceph-csi-rbd/pal-ceph-csi-rbd-provisioner-http-metrics:http-metrics cluster IP" -m tcp --dport 9299 -j KUBE-SVC-PHKTDTNNKLKS6FJV
-A KUBE-SERVICES -d 10.43.83.244/32 -p tcp -m comment --comment "prometheus-operator/prometheus-operator-kube-state-metrics:http cluster IP" -m tcp --dport 8080 -j KUBE-SVC-DTHJ27J4JTIRFMMR
-A KUBE-SERVICES -d 10.43.131.203/32 -p tcp -m comment --comment "prometheus-operator/prometheus-operator-node-exporter:metrics cluster IP" -m tcp --dport 9100 -j KUBE-SVC-BNKSIWLLZGVIY7PW
-A KUBE-SERVICES -d 10.43.193.44/32 -p tcp -m comment --comment "prometheus-operator/prometheus-operator-operator:http cluster IP" -m tcp --dport 8080 -j KUBE-SVC-34KHMJXDM3NH5E7C
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES -d 10.43.28.51/32 -p tcp -m comment --comment "pal-ceph-csi-rbd/pal-ceph-csi-rbd-nodeplugin-http-metrics:http-metrics cluster IP" -m tcp --dport 9299 -j KUBE-SVC-GAIKHQAFXHCEIZQX
-A KUBE-SVC-34KHMJXDM3NH5E7C -m comment --comment "prometheus-operator/prometheus-operator-operator:http" -j KUBE-SEP-MXD6VFWHCJUVSSEG
-A KUBE-SVC-BNKSIWLLZGVIY7PW -m comment --comment "prometheus-operator/prometheus-operator-node-exporter:metrics" -j KUBE-SEP-NXVPYAJKCVLNP7DZ
-A KUBE-SVC-DTHJ27J4JTIRFMMR -m comment --comment "prometheus-operator/prometheus-operator-kube-state-metrics:http" -j KUBE-SEP-3Z3O3CL5FEW4HZF2
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-SPW6OHKWQYLI5PJR
-A KUBE-SVC-GAIKHQAFXHCEIZQX -m comment --comment "pal-ceph-csi-rbd/pal-ceph-csi-rbd-nodeplugin-http-metrics:http-metrics" -j KUBE-SEP-LLFUWNFND4V47TQN
-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-SEP-2RAD5WKGWNVHY3DH
-A KUBE-SVC-JTFAIQOSQRKTQWS3 -m comment --comment "ingress-nginx/default-http-backend" -j KUBE-SEP-OKYHZQ2GXTI2XSII
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-2NIRRBB72GWCF3ZP
-A KUBE-SVC-PHKTDTNNKLKS6FJV -m comment --comment "pal-ceph-csi-rbd/pal-ceph-csi-rbd-provisioner-http-metrics:http-metrics" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-2VWHI5OCAM2AB6GB
-A KUBE-SVC-PHKTDTNNKLKS6FJV -m comment --comment "pal-ceph-csi-rbd/pal-ceph-csi-rbd-provisioner-http-metrics:http-metrics" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-62FB3SVBJP4C4RR3
-A KUBE-SVC-PHKTDTNNKLKS6FJV -m comment --comment "pal-ceph-csi-rbd/pal-ceph-csi-rbd-provisioner-http-metrics:http-metrics" -j KUBE-SEP-3X22LO7R2PIRIHDJ
-A KUBE-SVC-QMWWTXBG7KFJQKLO -m comment --comment "kube-system/metrics-server" -j KUBE-SEP-2AHCZLASALRZCLS7
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-4SSQDR46DMIXOOUO