How to use ingress-nginx with secure backend?

General Discussions
1

Hi, I am using ingress-nginx for making kibana accessible from outside the cluster.
I have the task to completely encrypt the application, so I think I need the following chain

Browser -> TLS -> ingress-nginx -> TLS -> service (kibana).
Kibana will also use tls encryption to connect elasticsearch, which is working fine so far.

I am at this point now:

  • using insecure ingress-nginx (http) with an insecure backend (http) -> OK
  • using secure ingress-nginx(https) with an insecure backend (http) -> OK
  • using secureingress-nginx (https) with secure backend (https) -> FAILS

here is my ingress configuration:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  labels:
    cluster: test
    app: kibana
    product: elasticStack
  annotations:
  #  nginx.ingress.kubernetes.io/rewrite-target: /test-kibana
  # enable to use secure backend
    nginx.ingress.kubernetes.io/secure-backends: "true"
spec:
  tls:
  - hosts:
    - elastic.ingress.kubernetes.internal.dtpublic.de
    secretName: elastic.ingress.kubernetes-tls
  rules:
  - host: elastic.ingress.kubernetes
    http:
      paths:
      - path: /test-kibana
        backend:
          serviceName: test-kibana
          servicePort: 5601

When I call the ingress url in browser with https, I get 502 bad gateway.

Log of ingress pods are showing this:

2019/07/30 08:16:17 [error] 44#44: *59001 upstream prematurely closed connection while reading response header from upstream, client: 172.22.19.152, server: elastic.ingress.kubernetes.internal.dtpublic.de, request: "GET /test-kibana/ HTTP/2.0", upstream: "http://192.168.198.160:5601/test-kibana/", host: "elastic.ingress.kubernetes.internal.dtpublic.de"

As you can see, it is forwarded to http, not https.

In kibana’s log I see following error:

{"type":"error","@timestamp":"2019-07-30T08:16:17Z","tags":["connection","client","error"],"pid":1,"level":"error","error":{"message":"139688227514240:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:242:\n","name":"Error","stack":"Error: 139688227514240:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:242:\n"},"message":"139688227514240:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:242:\n"}

That is the identical error message as if i call the service via curl with http:
kubectl exec -it test-es-master-0 -- curl -k -L http://test-kibana:5601/test-kibana
instead of https:
kubectl exec -it test-es-master-0 -- curl -k -L https://test-kibana:5601/test-kibana

https from inside a pod against the service is working like charm.

So what do I need to change to use a secure backend via tls?

Thanks, Andreas

2

found the solution. I added a second annotation:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

That made the trick

1 Like
3

Great, thanks for sharing :slight_smile:

4

ingress controller - kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml

ingress nodeprot - kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/baremetal/service-nodeport.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: simple-fanout-example
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: service1
          servicePort: 4200
      - path: /bar
        backend:
          serviceName: service2
          servicePort: 8080
5

Can you tell how is it different from force-ssl-redirect annotation?

6

helm upgrade --install ingress-nginx ingress-nginx
–repo Welcome - NGINX Ingress Controller
–namespace ingress-nginx --create-namespace

7

There are 3 components: the user, which connects to nginx, which itself connects to the backend (sometimes called “upstream”).

force-ssl-redirect instructs nginx that when a user connects to it in HTTP, it will send a redirect to the same URL with HTTPS. This is used to prevent clients connecting to nginx over HTTP, which is not encrypted.

backend-protocol: "HTTPS" instruct nginx to connect to the backend using HTTPS, instead of plain HTTP. It’s invisible to the user.