Ingress Controller's External IP cannot be reached

General Discussions
1

Cluster information:

Kubernetes version: v1.17.2+k3s1
Cloud being used: bare-metal
Installation method: k3s install script
Host OS: Arch Liux ARM
CNI and version: Flannel (default for k3s… I don’t know how to check its version?)
CRI and version: Containerd (same as above)

Hi all, very new to Kubernetes, trying to set up a cluster on some Raspberry Pis using k3s. I have been following parts of this guide, and I have the main parts set up:

~
❯ kubectl get pods -n kube-system -o wide
NAME                                        READY   STATUS    RESTARTS   AGE   IP             NODE           NOMINATED NODE   READINESS GATES
metallb-speaker-knd9z                       1/1     Running   5          12d   192.168.0.20   alarm-master   <none>           <none>
cert-manager-549fd9dcb4-j49fp               1/1     Running   3          21h   10.42.0.63     alarm-master   <none>           <none>
coredns-d798c9dd-42mvn                      1/1     Running   4          12d   10.42.0.65     alarm-master   <none>           <none>
local-path-provisioner-58fb86bdfd-fkbdn     1/1     Running   118        22h   10.42.2.149    alarm-other    <none>           <none>
metallb-speaker-p8qgw                       1/1     Running   5          37d   192.168.0.43   alarm-other    <none>           <none>
cert-manager-webhook-6d57dbf4f-2h5cv        1/1     Running   1          21h   10.42.2.147    alarm-other    <none>           <none>
nginx-ingress-controller-74c5f87877-5qcfr   1/1     Running   1          21h   10.42.2.148    alarm-other    <none>           <none>
metallb-speaker-xbz2b                       1/1     Running   5          37d   192.168.0.69   alarm-j        <none>           <none>
metrics-server-6d684c7b5-zvgbh              1/1     Running   5          31d   10.42.1.88     alarm-j        <none>           <none>
cert-manager-cainjector-79f4496665-nbx9c    1/1     Running   113        21h   10.42.1.87     alarm-j        <none>           <none>
metallb-controller-75bf779d4f-j7tn5         1/1     Running   5          37d   10.42.1.86     alarm-j        <none>           <none>

~
❯ kubectl get pods -n nextcloud -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
db-77b9dbd5c7-ll6qh         1/1     Running   2          23h   10.42.0.62   alarm-master   <none>           <none>
nextcloud-5cfc497cb-bzh4t   1/1     Running   2          20h   10.42.0.64   alarm-master   <none>           <none>

~
❯ kubectl get svc -n kube-system -o wide
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE   SELECTOR
kube-dns                   ClusterIP      10.43.0.10     <none>          53/UDP,53/TCP,9153/TCP       37d   k8s-app=kube-dns
metrics-server             ClusterIP      10.43.37.132   <none>          443/TCP                      37d   k8s-app=metrics-server
nginx-ingress-controller   LoadBalancer   10.43.150.78   192.168.0.240   80:32088/TCP,443:32221/TCP   21h   app=nginx-ingress,component=controller,release=nginx-ingress
cert-manager-webhook       ClusterIP      10.43.45.100   <none>          443/TCP                      21h   app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook
cert-manager               ClusterIP      10.43.90.39    <none>          9402/TCP                     21h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager

~
❯ kubectl get svc -n nextcloud -o wide
NAME        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE   SELECTOR
db          ClusterIP   10.43.255.219   <none>        3306/TCP   23h   app=db
nextcloud   ClusterIP   10.43.253.99    <none>        8080/TCP   20h   app=nextcloud

The issue I’m having is that even though nginx-ingress-controller says its external IP is 192.168.0.240, I cannot access this IP from outside the cluster e.g. my PC.

Running curl 192.168.0.240 on my PC gives me:

curl: (7) Failed to connect to 192.168.0.240 port 80: No route to host

I don’t know how to debug this; all I know is that it (probably?) isn’t iptables related since I don’t have iptables running.

Any ideas of how I should fix this? Thanks!

1 Like
2

Assuming that you are able to access the ext.IP within the cluster nodes, it looks to be an issue with no routes to the IP range you used within the cluster for the metalLB. You may need to think about adding static routes from the machine ( which is outside the cluster) to the ext.IPs.

3

From a cluster node:

[sseneca@alarm-master ~]$ curl https://192.168.0.240
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
[sseneca@alarm-master ~]$ curl http://192.168.0.240 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.17.8</center>
</body>
</html>

So yeah, I can reach it from within.

So the Pi’s local IP address is 192.168.0.20, and I would add a route from there to 192.168.0.240? Would that be done with iptables? Would it be easier to use a different IP range within the cluster for MetalLB?

Thanks

4

I am also having this issue.

NAME                                             READY   STATUS    RESTARTS   AGE    IP             NODE             NOMINATED NODE   READINESS GATES
coredns-d798c9dd-lsdnp                           1/1     Running   5          37h    10.42.0.25     c271-k3s-ocrh    <none>           <none>
local-path-provisioner-58fb86bdfd-bcpl7          1/1     Running   5          37h    10.42.0.22     c271-k3s-ocrh    <none>           <none>
metrics-server-6d684c7b5-v9tmh                   1/1     Running   5          37h    10.42.0.24     c271-k3s-ocrh    <none>           <none>
metallb-speaker-4kbmw                            1/1     Running   0          4m7s   192.168.0.14   c271-k3s-agent   <none>           <none>
metallb-controller-75bf779d4f-nb47l              1/1     Running   0          4m7s   10.42.1.45     c271-k3s-agent   <none>           <none>
metallb-speaker-776p9                            1/1     Running   0          4m7s   192.168.0.13   c271-k3s-ocrh    <none>           <none>
nginx-ingress-default-backend-5b967cf596-554bq   1/1     Running   0          98s    10.42.1.46     c271-k3s-agent   <none>           <none>
nginx-ingress-controller-674675d5b6-blndp        1/1     Running   0          98s    10.42.1.47     c271-k3s-agent   <none>           <none>

❯ kubectl get services  -n kube-system -l app=nginx-ingress -o wide
NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE    SELECTOR
nginx-ingress-default-backend   ClusterIP      10.43.170.195   <none>         80/TCP                       112s   app=nginx-ingress,component=default-backend,release=nginx-ingress
nginx-ingress-controller        LoadBalancer   10.43.220.166   192.168.0.21   80:31735/TCP,443:31566/TCP   111s   app=nginx-ingress,component=controller,release=nginx-ingress

I can curl nginx-ingress-controller pod with IP 10.42.1.47 but not 192.168.0.21 from 192.168.0.13(Master) or 192.168.0.14(Worker)

@sseneca what’s you solution on this ?

5

No solution yet. In the reddit thread on this topic, a user suggests the issue is the networking setup, and a possible workaround is by buying a switch. Since I’m already planning on buying a switch, I’ll probably get one now and see if that fixes the issue, but I’m waiting to see if I should buy a “Layer 2” switch or a “Layer 3” switch.

6

Any luck with this? Have the same problem :confused:

7

The routes must be added and advertised to anyone no your network (configured at your router). Your home network is not aware of the IP range being used by metallb.

If you’d like an alternative - Restrict your DHCP range to only part of the range and then allocate the rest to your externalIP pool e.g.

Home network: 192.168.0.1/24
DHCP Range: 192.168.0.2-192.168.0.100
ExternalIP Range: 192.168.0.101-192.168.0.200

8

Hello, Thank you for the response!

I did exclude the IP range given to metallb from my router. (192.168.0.30-192.168.0.35)
Current setup is :
Home network: 192.168.0.1/24
DHCP Range: 192.168.0.100-192.168.0.250
Excluded Range: 192.168.0.2-192.168.0.99

Ingress (LoaBalancer type) is getting IP from metallb (192.168.0.30). Shot after setup I’m able to connect to nginx via browser, of course getting nginx 400, but after a while I’m losing connection and I cant connect anymore with arp table for given IP as incomplete :confused: no freaking clue why it is happening.
So maybe those routes are the way but have no clue how to setup this… (router model TL-WR841N)

edit: in my browser with https://localhost:37533/ getting nginx ingress page (getting via lens with port forwarding)

SOLUTION (for me at least): sudo ip link set wlan0 promisc on