Ngnix ingress controller returns 502 bad gateway for grafana/prometheus

Nginx ingress controller route works fine for an application through port 443
And grafana/prometheus also works fine through external IP. Have downloaded the yaml for grafana & prometheus from charts repo.

But the ingress route fails for grafana and prometheus app stating 502 bad gateway though the service port also fine. Deployed everything in same namespace.

What else could be the reason?

Thanks,
Akshya

Please share the yaml

That’s likely because the nginx ingress cannot find the targets for grafana and prometheus – Can you double check the ingress rules were created in the prom/grafana namespace and that the serviceName and servicePort target the prom/grafana services?

Prometheus, grafana and the application all created in the same namespace, ingress route also pointing to the same namespace. Service name , service port also fine. This is my yaml file

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: [ingress name]
namespace: [namespace where all apps deployed]
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/backend-protocol: “HTTPS”
spec:
tls:

  • hosts:
    • [hostname]
      secretName: [secret]
      rules:
  • host: [hostname]
    http:
    paths:
    • backend:
      serviceName: [Working fine service]
      servicePort: 443
      path: /(.*)
    • backend:
      serviceName: grafana
      servicePort: 443
      path: /grafana(/|$)(.*)
    • backend:
      serviceName: prometheus-server
      servicePort: 443
      path: /prometheus(/|$)(.*)

Regards,
Akshya

This is the yaml file
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: [ingress name]
namespace: [namespace where all apps deployed]
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/backend-protocol: “HTTPS”
spec:
tls:

  • hosts:
    • [hostname]
      secretName: [secret]
      rules:
  • host: [hostname]
    http:
    paths:
    • backend:
      serviceName: [Working fine service]
      servicePort: 443
      path: /(.*)
    • backend:
      serviceName: grafana
      servicePort: 443
      path: /grafana(/|$)(.*)
    • backend:
      serviceName: prometheus-server
      servicePort: 443
      path: /prometheus(/|$)(.*)

Akshay, don’t see any issue as such. Do you see anything in the logs?

You can rollback and restart the nodes, most likely your issue should get fixed. If it still doesn’t then I’d replicate it at my end, will fix it then.