NGINX Load Balancer in Front of ingress Controller - Not Working

General Discussions
1

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version:v1.18.6
Cloud being used: (put bare-metal if not on a public cloud) - pure bare-metal
Installation method: kubeadm
Host OS: CentOS 7
CNI and version: Docker version 19.03.12
CRI and version: Calico v3.14

I am relatively new to Kubernetes, i have just deployed HA K8s cluster on-premises. We are using NGINX ingress controller to access Apps outside of cluster and verify that i can access apps via dns name defined in ingress rules.

But due to some project requirements, i need to deploy nginx as load balancer . Issue comes when NGINX Load balancer is in front of INGRESS Controller. When i try to access apps via dns name ( dns name points to the ip of nginx load balancer ) is not routing the traffic to worker nodes.

Following are the entries for worker nodes in nginx config file (nginx load balancer) :

upstream backend {
   server 192.168.1.41;
   server 192.168.1.42;
}

# This server accepts all traffic to port 80 and passes it to the upstream.

server {
   listen 80;

   location / {
      proxy_pass http://backend;
   }
}

Following are details for NGINX ingress controller

[kadmin@k8s-master ~]$ kubectl get  all -n nginx-ingress
NAME                      READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-5kmf8   1/1     Running   1          23h
pod/nginx-ingress-kzp76   1/1     Running   1          23h

NAME                    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/nginx-ingress   NodePort   10.108.39.187   <none>        80:32606/TCP,443:31978/TCP   23h

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginx-ingress   2         2         2       2            2           <none>          23h
[kadmin@k8s-master ~]$

Note : I can’t go for MetalLB as it doesn’t support SSL termination. Apart from this i have noticed one strange issue if i don’t have specify the host parameter in ingress rules then ingress is not working. It only works when i specify host entries in ingress yaml file.

I have refer beneath official documentation to install ingress controller as daemonset.

https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/

Can anybody suggest and give some clues to resolve above said issues ??

If any further details are required, please let me know.

Thanks in advance

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

2

In the nginx lb server, configure the backend server as

192.168.1.41:3206
192.168.1.42:3206

if you plan to use 443, then

192.168.1.41:31978
192.168.1.42:31978

Let me know if it helps

3

Hey Ranjith,

Thanks for the reply and suggestion. I make the suggested changes in nginx.conf on my nginx-lb

upstream backend {
   server 192.168.1.41:32606;
   server 192.168.1.42:32606;
}
# This server accepts all traffic to port 80 and passes it to the upstream.
server {
   listen 80;
   location / {
      proxy_pass http://backend;
   }
}

Now when i try to access my web page via load balancer ip , i am getting bad gateway error on the browser and i am also getting following errors in error log file on nginx-lb

   [root@lb1 ~]# tailf /var/log/nginx/error.log
    2020/08/11 14:28:27 [crit] 1478#0: *1 connect() to 192.168.1.41:32606 failed (13: Permission denied) while connecting to upstream, client: 192.168.1.3, server: , request: "GET / HTTP/1.1", upstream: "http://192.168.1.41:32606/", host: "lb-http.local"
    2020/08/11 14:28:28 [error] 1478#0: *1 no live upstreams while connecting to upstream, client: 192.168.1.3, server: , request: "GET / HTTP/1.1", upstream: "http://backend/", host: "lb-http.local"
    2020/08/11 14:28:47 [crit] 1478#0: *1 connect() to 192.168.1.42:32606 failed (13: Permission denied) while connecting to upstream, client: 192.168.1.3, server: , request: "GET / HTTP/1.1", upstream: "http://192.168.1.42:32606/", host: "lb-http.local"
    2020/08/11 14:28:47 [crit] 1478#0: *1 connect() to 192.168.1.41:32606 failed (13: Permission denied) while connecting to upstream, client: 192.168.1.3, server: , request: "GET / HTTP/1.1", upstream: "http://192.168.1.41:32606/", host: "lb-http.local"
    2020/08/11 14:30:48 [crit] 1478#0: *8 connect() to 192.168.1.42:32606 failed (13: Permission denied) while connecting to upstream, client: 192.168.1.3, server: , request: "GET / HTTP/1.1", upstream: "http://192.168.1.42:32606/", host: "192.168.1.151"
    2020/08/11 14:30:48 [crit] 1478#0: *8 connect() to 192.168.1.41:32606 failed (13: Permission denied) while connecting to upstream, client: 192.168.1.3, server: , request: "GET / HTTP/1.1", upstream: "http://192.168.1.41:32606/", host: "192.168.1.151"
    2020/08/11 14:30:49 [error] 1478#0: *8 no live upstreams while connecting to upstream, client: 192.168.1.3, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://backend/favicon.ico", host: "192.168.1.151", referrer: "http://192.168.1.151/"
    2020/08/11 14:30:58 [error] 1478#0: *12 no live upstreams while connecting to upstream, client: 192.168.1.3, server: , request: "GET / HTTP/1.1", upstream: "http://backend/", host: "lb-http.local"
    2020/08/11 14:31:10 [crit] 1478#0: *12 connect() to 192.168.1.42:32606 failed (13: Permission denied) while connecting to upstream, client: 192.168.1.3, server: , request: "GET / HTTP/1.1", upstream: "http://192.168.1.42:32606/", host: "lb-http.local"
    2020/08/11 14:31:10 [crit] 1478#0: *12 connect() to 192.168.1.41:32606 failed (13: Permission denied) while connecting to upstream, client: 192.168.1.3, server: , request: "GET / HTTP/1.1", upstream: "http://192.168.1.41:32606/", host: "lb-http.local"
4

Permission denied error was because of SELinux, I have disabled it now and error messages are gone but still nginx-lb is not working.

I am still unable to access web page using nginx-lb. Any suggestions and clues ?

[kadmin@k8s-master ~]$ kubectl get  all -n nginx-ingress
NAME                      READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-5kmf8   1/1     Running   1          24h
pod/nginx-ingress-kzp76   1/1     Running   1          24h

NAME                    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/nginx-ingress   NodePort   10.108.39.187   <none>        80:32606/TCP,443:31978/TCP   24h

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginx-ingress   2         2         2       2            2           <none>          24h
[kadmin@k8s-master ~]$


[root@lb1 ~]# cat /etc/nginx/nginx.conf
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

#    server {
#        listen       80 default_server;
#        listen       [::]:80 default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

upstream backend {
   server 192.168.1.41:32606;
   server 192.168.1.42:32606;
}

# This server accepts all traffic to port 80 and passes it to the upstream.

server {
   listen 80;

   location / {
      proxy_pass http://backend;
   }
}

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2 default_server;
#        listen       [::]:443 ssl http2 default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers HIGH:!aNULL:!MD5;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

}
5

How are you testing the connection through the load balancer? Are you using curl? Are you pointing the DNS name to the load balancer when testing?

Kind regards,
Stephen

6

i am pointing the DNS name to the load balancer IP via hosts file of my system.

7

The LB log you provided shows the host In the last column.

Seems to shift between lb-http.local and what I am guessing is the LB IP address.

You need to connect with the actual DNS address of an ingress resource otherwise the HTTP headers will not contain the correct HTTP host header and the ingress controller will not be able to route the request to the correct service/pod.

Kind regards,
Stephen

8

Can you try this

curl --header “host: X.example.com” nginx-LB-IP

host: “x.example.com” should be your app name

9

I am using the same dns name which is mentioned in ingress rules and i have added the same dns name in my hosts file. But still i am getting “404 Not Found - nginx/1.19.1”

$ ping lb-http.local

Pinging lb-http.local [192.168.1.151] with 32 bytes of data:
Reply from 192.168.1.151: bytes=32 time<1ms TTL=64
Reply from 192.168.1.151: bytes=32 time<1ms TTL=64

$ curl http://lb-http.local/
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.19.1</center>
</body>
</html>
             

[kadmin@k8s-master ~]$ kubectl create -f test-ingress.yaml
ingress.networking.k8s.io/simple-fanout-example created
[kadmin@k8s-master ~]$
[kadmin@k8s-master ~]$ cat test-ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: simple-fanout-example
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: lb-http.local
    http:
      paths:
      - path:
        backend:
          serviceName: nginx-lb
          servicePort: 80
[kadmin@k8s-master ~]$
[kadmin@k8s-master ~]$ kubectl describe ingress simple-fanout-example
Name:             simple-fanout-example
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host           Path  Backends
  ----           ----  --------
  lb-http.local
                    nginx-lb:80 (172.16.140.8:80)
Annotations:     nginx.ingress.kubernetes.io/rewrite-target: /
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  35s   nginx-ingress-controller  Configuration for default/simple-fanout-example was added or updated
  Normal  AddedOrUpdated  34s   nginx-ingress-controller  Configuration for default/simple-fanout-example was added or updated
[kadmin@k8s-master ~]$
10 
$ curl --header "host: lb-http.local" http://192.168.1.151
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.19.1</center>
</body>
</html>

Here “192.168.1.151” is the load balancer ip

11

What is the Ingres pointing to? Is it an nginx web server?

Nginx is taking over the world :joy:

12

What is nginx-lb port 80 defined in your ingress resource?
There is a good chance the 404 in your last message came from that pod.

I would recommend removing the rewrite-target annotation just to keep things as simple as possible for now.

Is the nginx-lb hosting particular page that you expect to see in the curl output?

Kind regards,
Stephen

13

Is the nginx-lb hosting particular page that you expect to see in the curl output? — Yes, i am expecting default home page of nginx , if i points the dns name to one of the worker node’s ip then it is start displaying the nginx home page. so i can say pod / service both are working fine

I would recommend removing the rewrite-target annotation just to keep things as simple as possible for now. – I have removed the rewrite-target annotation in my ingress rules but still no success

[kadmin@k8s-master ~]$ kubectl describe ingress simple-fanout-example
Name:             simple-fanout-example
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host           Path  Backends
  ----           ----  --------
  lb-http.local
                    nginx-lb:80 (172.16.140.8:80)
Annotations:     <none>
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  27m   nginx-ingress-controller  Configuration for default/simple-fanout-example was added or updated
  Normal  AddedOrUpdated  27m   nginx-ingress-controller  Configuration for default/simple-fanout-example was added or updated
[kadmin@k8s-master ~]$

What is nginx-lb port 80 defined in your ingress resource? – It points to nginx pod

[kadmin@k8s-master ~]$ kubectl describe svc nginx-lb
Name:                     nginx-lb
Namespace:                default
Labels:                   run=nginx-lb
Annotations:              <none>
Selector:                 run=nginx-lb
Type:                     NodePort
IP:                       10.97.102.181
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  32167/TCP
Endpoints:                172.16.140.8:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
[kadmin@k8s-master ~]$

Note : I have tested the nginx load balancer in minikube setup, it is working fine there, i can access nginx page via dns name that points to nginx load balancer ip.

14

Could you make the following changes to nginx.conf

listen 80;

location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://backend;
}

1 Like
15

other than lb-http.local which is your ingress, do you have any other ingress resources configured for any other app ?

curl --header "host: lb-http.local" http://192.168.1.151
16

Hey Ranjith,

Thank you very much !!!

Your suggestions did the tick and resolved the issue.

i can access app via nginx load balancer ip

 $ curl --header "host: lb-http.local" 192.168.1.51
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
17

Thanks Ranjith, Your contribution to the K8s community is tremendous.