openSUSE: NodePort Service not reachable externally

General Discussions
1

Cluster information:

Kubernetes version: 1.22.2
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: openSUSE Leap 15.3
CNI and version: flanell

Hi everybody,

for playing around with and learning K8s, on three VMs on my own Server I installed openSUSE 15.3 and built a K8s Cluster with help of this StackOverflow Article: How to install kubernetes in Suse Linux enterprize server 15 virtual machines? - Stack Overflow with this as a result:

kubix01:~ # k get nodes -o wide
NAME      STATUS   ROLES                  AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION         CONTAINER-RUNTIME
kubix01   Ready    control-plane,master   25h   v1.22.2   192.168.42.51   <none>        openSUSE Leap 15.3   5.3.18-59.27-default   docker://20.10.6-ce
kubix02   Ready    <none>                 25h   v1.22.2   192.168.42.52   <none>        openSUSE Leap 15.3   5.3.18-59.27-default   docker://20.10.6-ce
kubix03   Ready    <none>                 25h   v1.22.2   192.168.42.53   <none>        openSUSE Leap 15.3   5.3.18-59.27-default   docker://20.10.6-ce

For testing things out I made a new 3 Replica Deployment for a traefik/whoami Image with this yaml:


apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  labels:
    app: whoami
spec:
  replicas: 3
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - name: whoami
        image: traefik/whoami
        ports:
        - containerPort: 80

This results in three Pods spread over the 2 worker nodes as expected:


kubix01:~/k8s/whoami # k get pods -o wide
NAME                      READY   STATUS    RESTARTS      AGE   IP           NODE      NOMINATED NODE   READINESS GATES
whoami-8557b59f65-2qkvq   1/1     Running   2 (24h ago)   25h   10.244.2.7   kubix03   <none>           <none>
whoami-8557b59f65-4wnmd   1/1     Running   2 (24h ago)   25h   10.244.1.6   kubix02   <none>           <none>
whoami-8557b59f65-xhx5x   1/1     Running   2 (24h ago)   25h   10.244.1.7   kubix02   <none>           <none>

After that I created a NodePort Service for making things available to the world with this yaml:

apiVersion: v1
kind: Service
metadata:
  name: whoami
spec:
  type: NodePort
  selector:
    app: whoami
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
      nodePort: 30080

This is the result:

kubix01:~/k8s/whoami # k get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          25h
whoami       NodePort    10.105.214.86   <none>        8080:30080/TCP   25h


kubix01:~/k8s/whoami # k describe svc whoami
Name:                     whoami
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=whoami
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.105.214.86
IPs:                      10.105.214.86
Port:                     <unset>  8080/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30080/TCP
Endpoints:                10.244.1.6:80,10.244.1.7:80,10.244.2.7:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

So everything looks fine and I tested things out with curl:

  1. curl on one Cluster Node to PodIP:PodPort
kubix01:~/k8s/whoami # curl 10.244.1.6
Hostname: whoami-8557b59f65-4wnmd
IP: 127.0.0.1
IP: 10.244.1.6
RemoteAddr: 10.244.0.0:50380
GET / HTTP/1.1
Host: 10.244.1.6
User-Agent: curl/7.66.0
Accept: */*

kubix01:~/k8s/whoami # curl 10.244.1.7
Hostname: whoami-8557b59f65-xhx5x
IP: 127.0.0.1
IP: 10.244.1.7
RemoteAddr: 10.244.0.0:36062
GET / HTTP/1.1
Host: 10.244.1.7
User-Agent: curl/7.66.0
Accept: */*

kubix01:~/k8s/whoami # curl 10.244.2.7
Hostname: whoami-8557b59f65-2qkvq
IP: 127.0.0.1
IP: 10.244.2.7
RemoteAddr: 10.244.0.0:43924
GET / HTTP/1.1
Host: 10.244.2.7
User-Agent: curl/7.66.0
Accept: */*

==> Everything works as expected

  1. curl on Cluster Node to services ClusterIP:ClusterPort:

kubix01:~/k8s/whoami # curl 10.105.214.86:8080
Hostname: whoami-8557b59f65-xhx5x
IP: 127.0.0.1
IP: 10.244.1.7
RemoteAddr: 10.244.0.0:1106
GET / HTTP/1.1
Host: 10.105.214.86:8080
User-Agent: curl/7.66.0
Accept: */*

kubix01:~/k8s/whoami # curl 10.105.214.86:8080
Hostname: whoami-8557b59f65-4wnmd
IP: 127.0.0.1
IP: 10.244.1.6
RemoteAddr: 10.244.0.0:9707
GET / HTTP/1.1
Host: 10.105.214.86:8080
User-Agent: curl/7.66.0
Accept: */*

kubix01:~/k8s/whoami # curl 10.105.214.86:8080
Hostname: whoami-8557b59f65-2qkvq
IP: 127.0.0.1
IP: 10.244.2.7
RemoteAddr: 10.244.0.0:25577
GET / HTTP/1.1
Host: 10.105.214.86:8080
User-Agent: curl/7.66.0
Accept: */*

==> Everything fine, Traffic is LoadBalanced to the different pods.

  1. curl on Cluster Node to NodeIP:NodePort
kubix01:~/k8s/whoami # curl 192.168.42.51:30080
Hostname: whoami-8557b59f65-2qkvq
IP: 127.0.0.1
IP: 10.244.2.7
RemoteAddr: 10.244.0.0:5463
GET / HTTP/1.1
Host: 192.168.42.51:30080
User-Agent: curl/7.66.0
Accept: */*

kubix01:~/k8s/whoami # curl 192.168.42.52:30080
^C [NoAnswer]
kubix01:~/k8s/whoami # curl 192.168.42.53:30080
^C [NoAnswer]

==> NodePort Service is only working at the same Node, no answer from the other nodes

  1. curl from another Network Host to NodeIP:NodePort
user@otherhost:~$ curl 192.168.42.51:30080
^C [NoAnswer]
user@otherhost:~$ curl 192.168.42.52:30080
^C [NoAnswer]
user@otherhost:~$ curl 192.168.42.53:30080
^C [NoAnswer]

==> Service is not reachable from the outside at all, no answer on all nodes

Has anybody an idea what is going wrong here?

Thx in advance
T0mcat