Ingress-Nginx SSL Alert Number 70

General Discussions
1

I am using ingress-nginx (v1.9.5) on AKS and am attempting to allow it to connect to an HTTPS upstream service. The upstream service is configured with the port and TLS and the certificate works fine if I ssh into the ingress pod and run a curl -v command.However, if I try to connect from outside the cluster using the ingress, it fails with a 502 error and logs the following:

[error] 592#592: *39534 SSL_do_handshake() failed (SSL: error:0A00042E:SSL routines::tlsv1 alert protocol version:SSL alert number 70) while SSL handshaking to upstream, client: x.x.x.x, server: example.com, request: "POST /example HTTP/1.1", upstream: "[https://x.x.y.y:443/example](https://x.x.y.y/example)", host: "example.com"

I ran a pcap on the upstream and found that nginx is attempting to make a connection to the upstream via TLSv1.0, which obviously isn’t ideal. Is there something I’m missing in the configuration that would either force 1.3 or at least stop it from using 1.0?

Here’s my ingress definition:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - example.com
      secretName: example-tls
  rules:
    - host: example.com
      http:
        paths:
          - path: /example
            pathType: ImplementationSpecific
            backend:
              service:
                name: upstream-secure
                port:
                  name: https

Cluster information:

Kubernetes version: 1.27.7
Cloud being used: AKS
Installation method: Helm
Host OS: Ubuntu

2

Ah, the ol’ SSL handshake saga, with a twist of TLS version mismatch, no less! It’s like your Ingress is trying to throw it back to the '90s, insisting on using TLSv1.0. Not cool, especially when everyone’s moved on to the newer, shinier versions. Let’s see if we can give it a nudge in the right direction.

So, you’ve already done a great job ensuring that the backend communication is encrypted by setting nginx.ingress.kubernetes.io/backend-protocol: HTTPS in your Ingress annotations. However, it seems like Ingress-Nginx is still attempting to communicate using an outdated TLS version. We definitely want to avoid that, for security and compatibility reasons.

3

What you might want to explore is specifying the TLS version and ciphers that Nginx should use when communicating with your upstream services. This isn’t directly configurable through the Ingress resource but can be set globally for your Ingress controller or overridden on a per-Ingress basis using annotations.

Here’s a little plan to steer this ship back on course:

  1. Global Configuration: If you have access and the change is applicable cluster-wide, consider updating the Ingress-Nginx controller’s configmap to enforce stronger TLS versions for upstream communication. You’d be looking to set the ssl_protocols directive to something more modern, like TLSv1.2 or TLSv1.3.
  2. Per-Ingress Override: If a global change is too broad or you need to apply this on a case-by-case basis, you can use annotations on your individual Ingress resources to specify the TLS version. The catch is, Ingress-Nginx doesn’t directly expose an annotation for setting ssl_protocols for upstream communication. However, you can use the nginx.ingress.kubernetes.io/configuration-snippet annotation to inject custom Nginx configuration. It’s a bit of a hack and should be used judiciously, but it can get the job done.Here’s an illustrative snippet on how you might set it up:

yamlCopy code

nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_ssl_protocols TLSv1.2 TLSv1.3;
  proxy_ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';

Be mindful that this snippet goes directly into the Nginx configuration, so ensure the syntax is correct to avoid any unexpected issues.
3. Debugging and Verification: After applying the changes, keep an eye on the Ingress-Nginx logs and use tools like curl to verify that the TLS handshake is succeeding with the desired version of TLS. Since you’ve already demonstrated prowess with curl and packet captures, you’re all set on this front.

Remember, with great power comes great responsibility. Tweaking TLS settings can have broad implications, so test thoroughly in a dev or staging environment before rolling this out in production. And, as always, keep that Ingress-Nginx documentation handy for any deep dives or troubleshooting.