Getting 404 when accessing the Kubernetes Dashboard via the LoadBalancer

Hi!

I am getting a HTTP error 404 when I try to access the Kubernetes Dashboard via the LoadBalancer. I does work when I access it via the cluster IP.

Here are my configurations.

 > kubectl get nodes,all,ing,ingressclass,pv -A
NAME                       STATUS   ROLES                  AGE   VERSION
node/k8s-master.example.com   Ready    control-plane,master   17d   v1.20.2
node/k8s-slave1.example.com   Ready    <none>                 17d   v1.20.2
node/k8s-slave2.example.com   Ready    <none>                 17d   v1.20.2

NAMESPACE        NAME                                                 READY   STATUS    RESTARTS   AGE
default          pod/k8s-dash-kubernetes-dashboard-869d64d489-hkc9l   1/1     Running   0          17d
default          pod/k8s-ingress-nginx-controller-76d87b876b-94kl6    1/1     Running   0          17d
default          pod/nginx-errors-5465c9bfd7-dxvfq                    1/1     Running   0          46h
kube-system      pod/cilium-dvtw2                                     1/1     Running   0          17d
kube-system      pod/cilium-l95ct                                     1/1     Running   0          17d
kube-system      pod/cilium-operator-696dc48d8d-t6kb4                 1/1     Running   0          17d
kube-system      pod/cilium-x46zw                                     1/1     Running   0          17d
kube-system      pod/coredns-74ff55c5b-6txht                          1/1     Running   0          17d
kube-system      pod/coredns-74ff55c5b-fhqxr                          1/1     Running   0          17d
kube-system      pod/etcd-k8s-master.example.com                         1/1     Running   0          17d
kube-system      pod/kube-apiserver-k8s-master.example.com               1/1     Running   0          17d
kube-system      pod/kube-controller-manager-k8s-master.example.com      1/1     Running   0          17d
kube-system      pod/kube-proxy-f4525                                 1/1     Running   0          17d
kube-system      pod/kube-proxy-g6njq                                 1/1     Running   0          17d
kube-system      pod/kube-proxy-k776w                                 1/1     Running   0          17d
kube-system      pod/kube-scheduler-k8s-master.example.com               1/1     Running   0          17d
metallb-system   pod/controller-65db86ddc6-j927b                      1/1     Running   0          17d
metallb-system   pod/speaker-6pq65                                    1/1     Running   0          17d
metallb-system   pod/speaker-fdmrv                                    1/1     Running   0          17d
metallb-system   pod/speaker-mmv9j                                    1/1     Running   0          17d

NAMESPACE     NAME                                             TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
default       service/k8s-dash-kubernetes-dashboard            ClusterIP      10.108.249.46    <none>         443/TCP                      17d
default       service/k8s-ingress-nginx-controller             LoadBalancer   10.98.229.138    10.44.37.201   80:31188/TCP,443:30459/TCP   17d
default       service/k8s-ingress-nginx-controller-admission   ClusterIP      10.101.206.3     <none>         443/TCP                      17d
default       service/kubernetes                               ClusterIP      10.96.0.1        <none>         443/TCP                      17d
default       service/nginx-errors                             ClusterIP      10.102.207.223   <none>         80/TCP                       46h
kube-system   service/kube-dns                                 ClusterIP      10.96.0.10       <none>         53/UDP,53/TCP,9153/TCP       17d

NAMESPACE        NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system      daemonset.apps/cilium       3         3         3       3            3           <none>                   17d
kube-system      daemonset.apps/kube-proxy   3         3         3       3            3           kubernetes.io/os=linux   17d
metallb-system   daemonset.apps/speaker      3         3         3       3            3           kubernetes.io/os=linux   17d

NAMESPACE        NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
default          deployment.apps/k8s-dash-kubernetes-dashboard   1/1     1            1           17d
default          deployment.apps/k8s-ingress-nginx-controller    1/1     1            1           17d
default          deployment.apps/nginx-errors                    1/1     1            1           46h
kube-system      deployment.apps/cilium-operator                 1/1     1            1           17d
kube-system      deployment.apps/coredns                         2/2     2            2           17d
metallb-system   deployment.apps/controller                      1/1     1            1           17d

NAMESPACE        NAME                                                       DESIRED   CURRENT   READY   AGE
default          replicaset.apps/k8s-dash-kubernetes-dashboard-869d64d489   1         1         1       17d
default          replicaset.apps/k8s-ingress-nginx-controller-76d87b876b    1         1         1       17d
default          replicaset.apps/nginx-errors-5465c9bfd7                    1         1         1       46h
kube-system      replicaset.apps/cilium-operator-696dc48d8d                 1         1         1       17d
kube-system      replicaset.apps/coredns-74ff55c5b                          2         2         2       17d
metallb-system   replicaset.apps/controller-65db86ddc6                      1         1         1       17d

NAMESPACE   NAME                                              CLASS               HOSTS                    ADDRESS        PORTS     AGE
default     ingress.networking.k8s.io/k8s-ingress-dashboard   k8s-ingress-class   kubernetes.example.com   10.44.37.201   80, 443   46h

NAMESPACE   NAME                                               CONTROLLER             PARAMETERS                            AGE
            ingressclass.networking.k8s.io/k8s-ingress-class   k8s.io/ingress-nginx   IngressParameters/k8s-ingress-class   157m

NAMESPACE   NAME                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
            persistentvolume/dev-nfs   1Gi        RWX            Retain           Available                                   17d
 > kubectl describe service k8s-ingress-nginx-controller
Name:                     k8s-ingress-nginx-controller
Namespace:                default
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=k8s-ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/version=0.43.0
                          helm.sh/chart=ingress-nginx-3.22.0
Annotations:              meta.helm.sh/release-name: k8s-ingress-nginx
                          meta.helm.sh/release-namespace: default
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=k8s-ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Families:              <none>
IP:                       10.98.229.138
IPs:                      10.98.229.138
LoadBalancer Ingress:     10.44.37.201
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31188/TCP
Endpoints:                10.0.0.75:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30459/TCP
Endpoints:                10.0.0.75:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
 > kubectl describe service k8s-dash-kubernetes-dashboard 
Name:              k8s-dash-kubernetes-dashboard
Namespace:         default
Labels:            app.kubernetes.io/component=kubernetes-dashboard
                   app.kubernetes.io/instance=k8s-dash
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=kubernetes-dashboard
                   app.kubernetes.io/version=2.1.0
                   helm.sh/chart=kubernetes-dashboard-4.0.0
                   kubernetes.io/cluster-service=true
Annotations:       meta.helm.sh/release-name: k8s-dash
                   meta.helm.sh/release-namespace: default
Selector:          app.kubernetes.io/component=kubernetes-dashboard,app.kubernetes.io/instance=k8s-dash,app.kubernetes.io/name=kubernetes-dashboard
Type:              ClusterIP
IP Families:       <none>
IP:                10.108.249.46
IPs:               10.108.249.46
Port:              https  443/TCP
TargetPort:        https/TCP
Endpoints:         10.0.0.2:8443
Session Affinity:  None
Events:            <none>
 > kubectl describe ing k8s-ingress-dashboard
Name:             k8s-ingress-dashboard
Namespace:        default
Address:          10.44.37.201
Default backend:  nginx-errors:80 (10.0.0.157:8080)
TLS:
  k8s-secret-dash terminates kubernetes.example.com
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  kubernetes.example.com  
                          /dashboard   k8s-dash-kubernetes-dashboard:443 (10.0.0.2:8443)
Annotations:              nginx.ingress.kubernetes.io/rewrite-target: /
Events:                   <none>
 > kubectl describe ingressclass k8s-ingress-class
Name:         k8s-ingress-class
Labels:       <none>
Annotations:  ingressclass.kubernetes.io/is-default-class: True
Controller:   k8s.io/ingress-nginx
Parameters:
  Kind:  IngressParameters
  Name:  k8s-ingress-class
Events:  <none>

Cluster information:

Kubernetes version: 1.20.2
Cloud being used: bare-metal
Installation method: (not sure what to put here)
Host OS: Ubuntu 20.04.1
CNI and version: Cilium v1.9.4
CRI and version: Docker v20.10.2

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

Somehow it didn’t work for me as well using ingress.
Create a service of type LoadBalancer and access directly.
http:///dashboard.html

@esudheere ,

Thank you for testing my environment and confirming that I am not crazy. :slight_smile:

Please recommend an Ingress controller that would work better than the NGINX Ingress controller. I would like to get it working that way because I am planning to have many applications behind it in the future.

Also, please forgive me if I am defining things wrong. I am in the learning phase. :slight_smile:

Regards,
J