GKE getting 404 issue

I am creating a GKE cluster and I am trying to setup an ingress, this is a simple ingress that I am starting with:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tf-poc-ingress
spec:
  rules:
    - http:
        paths:
        - path: /foo
          pathType: Exact
          backend:
            service:
              name: foo
              port:
                number: 3000
        - path: /*
          pathType: ImplementationSpecific
          backend:
            service:
              name: bar
              port:
                number: 3001

When I am accessing /foo it is saying { statusCode :404, message : Cannot GET /foo , error : Not Found }

but when I am accessing / it is returning the “bar” service which is correct, I tried switching “foo” and “bar” and whatever service it is from the path / is getting called correctly, is there something that I am missing here?

here are the service and other info:

Name:                   bar
Namespace:              default
CreationTimestamp:      Thu, 21 Mar 2024 02:29:32 +0800
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=bar
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=bar
  Containers:
   bar:
    Image:      cultofdev/ssv2authimage
    Port:       <none>
    Host Port:  <none>
    Environment:
      PORT:  3000
    Mounts:  <none>
  Volumes:   <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   bar-86d8977877 (2/2 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  4m12s  deployment-controller  Scaled up replica set bar-86d8977877 to 2

Service:

Name:                     bar
Namespace:                default
Labels:                   <none>
Annotations:              cloud.google.com/neg: {"ingress":true}
                          cloud.google.com/neg-status:
                            {"network_endpoint_groups":{"3000":"k8s1-af5a8b57-default-bar-3000-0dc0c226"},"zones":["asia-southeast1-a","asia-southeast1-b"]}
Selector:                 app=bar
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.124.65.196
IPs:                      10.124.65.196
Port:                     <unset>  3000/TCP
TargetPort:               3000/TCP
NodePort:                 <unset>  30000/TCP
Endpoints:                10.8.0.15:3000,10.8.1.9:3000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                Age   From            Message
  ----     ------                ----  ----            -------
  Warning  ProcessServiceFailed  12m   neg-controller  error processing service "default/bar": failed to ensure svc neg cr default/k8s1-af5a8b57-default-bar-3000-0dc0c226/3000 for existing port: servicenetworkendpointgroups.networking.gke.io "k8s1-af5a8b57-default-bar-3000-0dc0c226" already exists
  Normal   Create                12m   neg-controller  Created NEG "k8s1-af5a8b57-default-bar-3000-0dc0c226" for default/bar-k8s1-af5a8b57-default-bar-3000-0dc0c226-/3000-3000-GCE_VM_IP_PORT-L7 in "asia-southeast1-a".
  Normal   Create                12m   neg-controller  Created NEG "k8s1-af5a8b57-default-bar-3000-0dc0c226" for default/bar-k8s1-af5a8b57-default-bar-3000-0dc0c226-/3000-3000-GCE_VM_IP_PORT-L7 in "asia-southeast1-b".
  Normal   Attach                12m   neg-controller  Attach 1 network endpoint(s) (NEG "k8s1-af5a8b57-default-bar-3000-0dc0c226" in zone "asia-southeast1-a")
  Normal   Attach                12m   neg-controller  Attach 1 network endpoint(s) (NEG "k8s1-af5a8b57-default-bar-3000-0dc0c226" in zone "asia-southeast1-b")

Ingress:

Name:             tf-poc-ingress
Labels:           <none>
Namespace:        default
Address:          35.241.29.37
Ingress Class:    <none>
Default backend:  foo:3001 (10.8.0.14:3001,10.8.1.8:3001)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /foo   foo:3001 (10.8.0.14:3001,10.8.1.8:3001)
              /bar   bar:3000 (10.8.0.15:3000,10.8.1.9:3000)
Annotations:  ingress.kubernetes.io/backends: {"k8s1-af5a8b57-default-bar-3000-0dc0c226":"HEALTHY","k8s1-af5a8b57-default-foo-3001-bf4282ec":"HEALTHY"}
              ingress.kubernetes.io/forwarding-rule: k8s2-fr-481qr3j9-default-tf-poc-ingress-dx3599zy
              ingress.kubernetes.io/target-proxy: k8s2-tp-481qr3j9-default-tf-poc-ingress-dx3599zy
              ingress.kubernetes.io/url-map: k8s2-um-481qr3j9-default-tf-poc-ingress-dx3599zy
Events:
  Type     Reason     Age                   From                     Message
  ----     ------     ----                  ----                     -------
  Warning  Translate  6m27s (x17 over 11m)  loadbalancer-controller  Translation failed: invalid ingress spec: could not find port "&ServiceBackendPort{Name:,Number:3003,}" in service "default/foo"
  Normal   Sync       3m12s                 loadbalancer-controller  UrlMap "k8s2-um-481qr3j9-default-tf-poc-ingress-dx3599zy" created
  Normal   Sync       3m8s                  loadbalancer-controller  TargetProxy "k8s2-tp-481qr3j9-default-tf-poc-ingress-dx3599zy" created
  Normal   Sync       2m49s                 loadbalancer-controller  ForwardingRule "k8s2-fr-481qr3j9-default-tf-poc-ingress-dx3599zy" created
  Normal   Sync       2m48s (x5 over 11m)   loadbalancer-controller  Scheduled for sync
  Normal   IPChanged  2m48s                 loadbalancer-controller  IP is now 35.241.29.37

Logs of deployment:

{
insertId: “r3zmu5v4lzjjd5fw”
labels: {3}
logName: “projects/tf-poc-19586/logs/stdout”
receiveTimestamp: “2024-03-18T17:43:26.128523948Z”
resource: {2}
severity: “INFO”
textPayload: “[32m[Nest] 19  - [39m03/18/2024, 5:43:24 PM [32m    LOG[39m [38;5;3m[NestApplication] [39m[32mNest application successfully started[39m[38;5;3m +4ms[39m”
timestamp: “2024-03-18T17:43:24.827447369Z”
}

Cluster information:

Kubernetes version: 1.27
Cloud being used: GKE
Installation method: managed service
Host OS: GKE container optimized
CNI and version: GKE default
CRI and version: GKE default