Microk8s 127.0.0.1:16443 was refused after rebooting

General Discussions microk8s
1

I install microk8s 1.28/stable on Ubuntu 22.04 LTS. It works OK.
After rebooting, when I run kubectl, sometime it works fine, and sometime it complains “The connection to the server 127.0.0.1:16443 was refused”. But I can access 127.0.0.1:16443 on FireFox.
Here are several outputs:

I can get node information, but sometime refused:

administer@gdsl1:~$ microk8s kubectl get node
NAME    STATUS   ROLES    AGE   VERSION
gdsl1   Ready    <none>   46m   v1.28.13

administer@gdsl1:~$ microk8s kubectl get node
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?

administer@gdsl1:~$ microk8s kubectl get node
NAME    STATUS   ROLES    AGE   VERSION
gdsl1   Ready    <none>   46m   v1.28.13

There is no error on the events:

administer@gdsl1:~$ microk8s kubectl get events -A
No resources found

administer@gdsl1:~$ microk8s kubectl get events -A
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?

Sometime I can see the cluster-info, sometime I cannot:

administer@gdsl1:~$ microk8s kubectl cluster-info
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?

administer@gdsl1:~$ microk8s kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:16443
CoreDNS is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Sorry I can not send the whole cluster-info dump file. But here are some errors from the output of “microk8s kubectl cluster-info dump”:

==== START logs for container calico-kube-controllers of pod kube-system/calico-kube-controllers-77bd7c5b-fb66f ====
Request log error: an error on the server ("unknown") has prevented the request from succeeding (get pods calico-kube-controllers-77bd7c5b-fb66f)
==== END logs for container calico-kube-controllers of pod kube-system/calico-kube-controllers-77bd7c5b-fb66f ====
==== START logs for container upgrade-ipam of pod kube-system/calico-node-8p4wj ====
Request log error: an error on the server ("unknown") has prevented the request from succeeding (get pods calico-node-8p4wj)
==== END logs for container upgrade-ipam of pod kube-system/calico-node-8p4wj ====
==== START logs for container install-cni of pod kube-system/calico-node-8p4wj ====
Request log error: an error on the server ("unknown") has prevented the request from succeeding (get pods calico-node-8p4wj)
==== END logs for container install-cni of pod kube-system/calico-node-8p4wj ====
==== START logs for container calico-node of pod kube-system/calico-node-8p4wj ====
Request log error: an error on the server ("unknown") has prevented the request from succeeding (get pods calico-node-8p4wj)
==== END logs for container calico-node of pod kube-system/calico-node-8p4wj ====
==== START logs for container coredns of pod kube-system/coredns-864597b5fd-m27bn ====
Request log error: an error on the server ("unknown") has prevented the request from succeeding (get pods coredns-864597b5fd-m27bn)
==== END logs for container coredns of pod kube-system/coredns-864597b5fd-m27bn ====
==== START logs for container dashboard-metrics-scraper of pod kube-system/dashboard-metrics-scraper-5657497c4c-jq4j2 ====
Request log error: an error on the server ("unknown") has prevented the request from succeeding (get pods dashboard-metrics-scraper-5657497c4c-jq4j2)
==== END logs for container dashboard-metrics-scraper of pod kube-system/dashboard-metrics-scraper-5657497c4c-jq4j2 ====
==== START logs for container hostpath-provisioner of pod kube-system/hostpath-provisioner-7df77bc496-s7cl8 ====
Request log error: an error on the server ("unknown") has prevented the request from succeeding (get pods hostpath-provisioner-7df77bc496-s7cl8)
==== END logs for container hostpath-provisioner of pod kube-system/hostpath-provisioner-7df77bc496-s7cl8 ====
==== START logs for container kubernetes-dashboard of pod kube-system/kubernetes-dashboard-54b48fbf9-lsxtk ====
Request log error: an error on the server ("unknown") has prevented the request from succeeding (get pods kubernetes-dashboard-54b48fbf9-lsxtk)
==== END logs for container kubernetes-dashboard of pod kube-system/kubernetes-dashboard-54b48fbf9-lsxtk ====
==== START logs for container metrics-server of pod kube-system/metrics-server-848968bdcd-6vhx4 ====
Request log error: an error on the server ("unknown") has prevented the request from succeeding (get pods metrics-server-848968bdcd-6vhx4)
==== END logs for container metrics-server of pod kube-system/metrics-server-848968bdcd-6vhx4 ====

Any idea how to solve it?

Thanks.

2

I also try to run dashboard-proxy but failed:

administer@gdsl1:~$ microk8s dashboard-proxy
Checking if Dashboard is running.
Infer repository core for addon dashboard
Waiting for Dashboard to come up.
Trying to get token from microk8s-dashboard-token
Waiting for secret token (attempt 0)
Dashboard will be available at https://127.0.0.1:10443
Use the following token to login:
eyJhbGciOiJSUzI1NiIsImtpZCI6InhDM0FmN0I2azQwd3loVHdLLXAzaG5yQlh1M0d5cnNxdHRRLTNZNV9WelEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJtaWNyb2s4cy1kYXNoYm9hcmQtdG9rZW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImIzZTQ4NDU0LThmMmUtNDNmNi1hODFkLTRiZTdmMzQwNDkyMSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpkZWZhdWx0In0.Vp64LcBTtKvSGbxm4KvUehZdR-oruSgZYIBnA_0DGM9WCkTVx-gW5VJT1cqrtJq6__P6LzjlE2LqoPPEzfwQvB-OvF9q0e3SfSsomGylMESN-F2gGEIzmo3djU94DH_QrvkojtzfpHIFOjiyXuhSkiscI9c5Ys2yuyMTTBGXRH6-zprB0PnB4Er5p-m4cqL6A6MhlGVkmT5ti7-sSbquNvEZR_ZzgbZs6NS2BkOIc_wIBWUIJp0fMXmTYefvAehlHHzyVwjRQf_4lms_1Ca3EJWHuAlt5AyEIMXfdxpJseHDXnSYBGKV3g7DXYlfGL6VxWNBbM8ljNVPiph53S1kAw
error: error upgrading connection: error dialing backend: dial tcp 138.253.112.21:10250: connect: connection refused
Traceback (most recent call last):
  File "/snap/microk8s/7207/scripts/wrappers/dashboard_proxy.py", line 111, in <module>
    dashboard_proxy()
  File "/snap/microk8s/7207/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/snap/microk8s/7207/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/snap/microk8s/7207/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/microk8s/7207/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/snap/microk8s/7207/scripts/wrappers/dashboard_proxy.py", line 105, in dashboard_proxy
    check_output(command)
  File "/snap/microk8s/7207/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/snap/microk8s/7207/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/snap/microk8s/7207/microk8s-kubectl.wrapper', 'port-forward', '-n', 'kube-system', 'service/kubernetes-dashboard', '10443:443', '--address', '0.0.0.0']' returned non-zero exit status 1.