Ingress giving 503 error

General Discussions
1

Cluster information:

Kubernetes version:
Cloud being used: bare-metal
Installation method: helm
Host OS: CentOS 7.9
CNI and version: Calico
CRI and version: docker://20.10.13

I’m trying to expose an application via a service and ingress, and have the following yaml:

apiVersion: v1
kind: Service
metadata:
  name: api-gateway 
  namespace: namespace-here 
spec:
  ports:
  - name: http 
    port: 6764 
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/component: api-gateway 
    app.kubernetes.io/instance: instance-here 
    app.kubernetes.io/part-of: app-here 
  type: NodePort

A describe of the service is as follows:

Name:                     api-gateway
Namespace:                namespace-here 
Labels:                   <none>
Annotations:              field.cattle.io/publicEndpoints:
                            [{"addresses":["ip-address-here"],"port":32458,"protocol":"TCP","serviceName":"namespace-here:api-gateway","allNodes":true}]
Selector:                 app.kubernetes.io/component=api-gateway,app.kubernetes.io/instance=instance-here,app.kubernetes.io/part-of=app-here
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       x.y.z.190
IPs:                      x.y.z.190
Port:                     http  6764/TCP
TargetPort:               http/TCP
NodePort:                 http  32458/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

So, I realize there should be endpoints listed, but I’m not sure why they’re not, as the fields in the selector match those of the pods, so that I do see the expected pod when I run the command here:

% k get pods -l app.kubernetes.io/component=api-gateway,app.kubernetes.io/instance=instance-here,app.kubernetes.io/part-of=app-here
NAME                              READY   STATUS    RESTARTS   AGE
api-gateway-5b788dc6d6-hk5h9      1/1     Running   0          92m

I can also use k9s to forward the port for both this pod and this service to localhost:6764, and access the expected webpage.

So, while the page at Debug Services | Kubernetes says I should have endpoints listed (i.e. not None), I’m not so sure this is valid. Given this, it would seem to me that the pod and service are working OK.

I have the following ingress yaml that references the above service, along with the describe output of the ingress implemented with that yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: namespace-here 
spec:
  ingressClassName: nginx
  rules:
  - host: fqdn-of-hostname-here 
    http:
      paths:
      - backend:
          service:
            name: api-gateway 
            port:
              number: 6764 
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - fqdn-of-hostname-here


%  k describe ing -n namespace-here
Name:             api-ingress
Labels:           <none>
Namespace:        namespace-here
Address:          ip-address-removed 
Ingress Class:    nginx
Default backend:  <default>
TLS:
  SNI routes fqdn-of-ingress-here
Rules:
  Host                  Path  Backends
  ----                  ----  --------
  fqdn-of-ingress-here 
                        /   api-gateway:6764 (<none>)
Annotations:            field.cattle.io/publicEndpoints:
                          [{"addresses":["ip-address-removed"],"port":443,"protocol":"HTTPS","serviceName":"namespace-here:api-gateway","ingressName":"namespace-here:api-ingress...
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    2m51s (x3 over 2m56s)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    2m51s (x3 over 2m56s)  nginx-ingress-controller  Scheduled for sync

There are no network policies defined in this namespace.

From what I can tell, everything is set up properly. Yet, I’m still seeing a HTTP 503 error when I try to access https://fqdn-of-ingress-here. Any thoughts as to why?

2

Resolved, had the port (should have been 80) and targetPort (should have been 6764) switched.