[Learning k8s] ingress not working with HTTP 502 / 113: Host is unreachable

General Discussions
1

Cluster information:

Kubernetes version: 1.28.5
Cloud being used: bare metal / Hyper-V on my PC
Installation method: manual
Host OS: Windows 10 Hyper-V, Alma Linux 8.9 VMs
CNI and version: Calico v3.26.1
CRI and version: cri-o 1.26.4

1 control node, 2 worker nodes

So I am just learning k8s, and the lessons I am using have me set up Ingress at this point. Have done other stuff fine like spinning up deployments, Services, using storage (host path and NFS with StorageClass/PV/PVC), etc

This is actually the second time I have done this part, after trying and getting stuck at this point, I removed all the Ingress etc resources and started clean again. Got the same results…

Steps:

helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace

(this goes fine, no errors)

Ingress info:

[19:18 krc@alma8-controller ~]$ kc get all -n ingress-nginx
NAME                                            READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-controller-76df688779-mmxmk   1/1     Running   0          39s

NAME                                         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             LoadBalancer   10.105.68.42     <pending>     80:31040/TCP,443:32281/TCP   39s
service/ingress-nginx-controller-admission   ClusterIP      10.100.195.216   <none>        443/TCP                      39s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   1/1     1            1           39s

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-76df688779   1         1         1       39s
[19:18 krc@alma8-controller ~]$

Create deployment and service for ingress to connect to:

[19:19 krc@alma8-controller ~]$ kc create deploy nginxsvc --image=nginx --port=80
deployment.apps/nginxsvc created
[19:19 krc@alma8-controller ~]$
[19:19 krc@alma8-controller ~]$ kc get deployments
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
nfs-subdir-external-provisioner   1/1     1            1           47h
nginxsvc                          1/1     1            1           19s
webshop                           3/3     3            3           24h
[19:20 krc@alma8-controller ~]$
[19:21 krc@alma8-controller ~]$ kc expose deployment nginxsvc
service/nginxsvc exposed
[19:21 krc@alma8-controller ~]$
[19:21 krc@alma8-controller ~]$ kc get all --selector app=nginxsvc
NAME                            READY   STATUS    RESTARTS   AGE
pod/nginxsvc-5f8b7d4f4d-w97j8   1/1     Running   0          97s

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/nginxsvc   ClusterIP   10.106.138.205   <none>        80/TCP    22s

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginxsvc   1/1     1            1           97s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/nginxsvc-5f8b7d4f4d   1         1         1       97s
[19:21 krc@alma8-controller ~]$

Create ingress rule, have to delete validator hook again (hit this the first time and tried a lot of stuff before just using this workaround of deleting the hook… would like to figure out the real issue at some point… but maybe it is related to current issue? IDK)

[19:21 krc@alma8-controller ~]$
[19:22 krc@alma8-controller ~]$ kubectl create ingress nginxsvc --class=nginx --rule=nginxsvc.info/*=nginxsvc:80
error: failed to create ingress: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": context deadline exceeded
[19:22 krc@alma8-controller ~]$
[19:22 krc@alma8-controller ~]$ kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
validatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-admission" deleted
[19:22 krc@alma8-controller ~]$
[19:22 krc@alma8-controller ~]$ kubectl create ingress nginxsvc --class=nginx --rule=nginxsvc.info/*=nginxsvc:80
ingress.networking.k8s.io/nginxsvc created
[19:23 krc@alma8-controller ~]$

Create a forward… I don’t totally get this, but it seems to be a temporary thing, so put it in bg with &

[19:25 krc@alma8-controller ~]$
[19:25 krc@alma8-controller ~]$ kubectl port-forward -n ingress-nginx svc/ingress-nginx-controller 8080:80 &
[1] 142629
[19:25 krc@alma8-controller ~]$ Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

[19:25 krc@alma8-controller ~]$

(already have 127.0.0.1 nginxsvc.info in /etc/hosts)

Test connection

[19:27 krc@alma8-controller ~]$ curl nginxsvc.info:8080
Handling connection for 8080
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
[19:28 krc@alma8-controller ~]$

Key error in logs is:

"GET / HTTP/1.1" 502 150 "-" "curl/7.61.1" 82 3.078 [default-nginxsvc-80] [] 10.17.184.163:80, 10.17.184.163:80, 10.17.184.163:80 0, 0, 0 1.031, 1.024, 1.024 502, 502, 502 ca443635e154e19f0d833223f85fd02b
2024/01/20 00:28:01 [error] 61#61: *2492 connect() failed (113: Host is unreachable) while connecting to upstream, client: ::1, server: nginxsvc.info, request: "GET / HTTP/1.1", upstream: "http://10.17.184.163:80/", host: "nginxsvc.info:8080"

kc logs -n ingress-nginx ingress-nginx-controller-76df688779-mmxmk
full log: ingress-nginx logs error · GitHub

Pod it is supposed to connect to looks ok?

[19:33 krc@alma8-controller ~]$ kc get pod nginxsvc-5f8b7d4f4d-w97j8 -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP              NODE          NOMINATED NODE   READINESS GATES
nginxsvc-5f8b7d4f4d-w97j8   1/1     Running   0          13m   10.17.184.163   alma8-node2   <none>           <none>
[19:33 krc@alma8-controller ~]$

Anyway, I have searched and searched, and tried as many troubleshooting steps as I can figure out and I can’t solve it.

Not sure how to dig further into the guts of the Ingress system. Doesn’t help that even if you exec -it into a pod you can’t do much on most images, missing too many tools (I get why they are like that and it is right, but annoying). Maybe have to figure out how to use those debug sidecar containers…

Any help much appreciated!

2

Spun up a “multitool” pod, confirmed from in there I can curl to the ClusterIP for nginxsvc, returns ok and I can see it in the logs of the nginxsvc pod as well.

So it is something in the Ingress itself as far as I can see…

Edit: Pretty sure my calico install is like half-busted. Some stuff works, but other things just don’t. Just can’t connect to some ClusterIPs for example. Routes on one node are not the same as the other three. etc.

3

Hi, I’m pretty new to kubernetes itself so I’m not too sure it if will help, but in this cases I try to check every part of the network flow:

  1. From inside the pod does a curl to localhost answer correctly?
    What is the result of:
    kc exec pod/nginxsvc-5f8b7d4f4d-w97j8 -- curl http://localhost:8080
    and
    kc exec pod/nginxsvc-5f8b7d4f4d-w97j8 -- curl http://localhost:80

  2. if you spin up a second pod and try to curl the service does it work correctly? I assume you tried already with:
    kc exec pod/nginxsvc-5f8b7d4f4d-w97j8 -- curl http://10.106.138.205:80
    or
    kc exec pod/nginxsvc-5f8b7d4f4d-w97j8 -- curl http://nginxsvc:80

  3. In addition, I don’t know why you would forward from localhost since your ingress controller already has a working IP address assigned, in order to test it fully I would insert in your /etc/hosts the line 10.105.68.42 nginxsv.info and then check the connecction with:
    curl nginxsvc.info directly from your local machine (where the /etc/hosts is)

Expected workflow (without port forwarding):
local machine (nginxsvc.info) → /etc/hosts → IP of ingress controller → IP of service → IP of POD → nginx in POD

Try to check each component starting from the last (point 1 - localhost from POD) and see where it breaks, from there we can debug further into the problem.

Hope it helps :slight_smile:
Leo