Ingress fails with All backend services are in Unhealthy state, but port forwarding from service works

General Discussions
1

Kubernetes version: Latest on GKE
Cloud being used: GKE
Installation method: using GKE
Host OS: unbuntu-containerd
CNI and version:
CRI and version:

I am trying to run prometheus to monitor my kubernetes cluster. I deployed prometheus and created a service to open port 9090 on the prometheus container.
Below is my service.yml:

kind: Service
apiVersion: v1
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector:
    app: prometheus
  type: NodePort
  ports:
  - name: webui
    port: 9090
    targetPort: 9090

In my deployment.yml, I have given the same name (webui) for port 9090 and I have only one replica in the pod.
I am creating ingress through the UI in GCP. Below is the yaml of the generated ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/backends: '{"k8s-be-31152--25f699f93b6f004b":"UNHEALTHY"}'
    ingress.kubernetes.io/forwarding-rule: k8s2-fr-tivb1lq8-monitoring-prometheus-9xkdyx19
    ingress.kubernetes.io/target-proxy: k8s2-tp-tivb1lq8-monitoring-prometheus-9xkdyx19
    ingress.kubernetes.io/url-map: k8s2-um-tivb1lq8-monitoring-prometheus-9xkdyx19
  creationTimestamp: "2020-11-25T00:29:11Z"
  finalizers:
  - networking.gke.io/ingress-finalizer-V2
  generation: 1
  name: prometheus
  namespace: monitoring
  resourceVersion: "4667031"
  selfLink: /apis/extensions/v1beta1/namespaces/monitoring/ingresses/prometheus
  uid: 4ca410f0-8af1-4b6f-8b34-83dca28be6c3
spec:
  backend:
    serviceName: prometheus
    servicePort: 9090
status:
  loadBalancer:
    ingress:
    - ip: 34.120.20.188

The issue is that Ingress creation fails with the error “All backend services are in unhealthy state”. But I tried the port forward option in service and I am able to access my pod with port 9090.

Has anyone faced this type of issue before? Any suggestion to fix this would be very helpful. Please let me know if any more details are needed (I am a new user of Kubernetes and still learning).

Thanks,
Srinidhi