Not ready after running: microk8s status --wait-ready

General Discussions microk8s
1

Hi all,

We’re starting microk8s on a ci pipeline to provide kube services etc like this:

sudo snap install microk8s --classic
sudo microk8s status --wait-ready
sudo microk8s enable dns registry storage
sudo microk8s status --wait-ready

Still getting problems with those services (dns,registry,storage) not being ready after that, e.g:

Error: Get "https://10.1.0.4:16443/api/v1/namespaces/default/persistentvolumeclaims/local-mongo-pv-claim": dial tcp 10.1.0.4:16443: connect: connection refused

error when creating "/root/snap/microk8s/1769/tmp/temp.coredns.yaml": Post "https://127.0.0.1:16443/apis/rbac.authorization.k8s.io/v1/clusterroles?fieldManager=kubectl-client-side-apply": read tcp 127.0.0.1:48292->127.0.0.1:16443: read: connection reset by peer
error when retrieving current configuration of:
Resource: "rbac.authorization.k8s.io/v1, Resource=clusterrolebindings", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding"
Name: "coredns", Namespace: ""
from server for: "/root/snap/microk8s/1769/tmp/temp.coredns.yaml": Get "https://127.0.0.1:16443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/coredns": dial tcp 127.0.0.1:16443: connect: connection refused
Failed to enable dns

Registry not being ready was solved by pinging it until it responds - it does take a good few minutes - machines on ci not that powerful however.

What’s the correct way to do this with microk8s?

2

What’s the machine specs?

3

It’s running on a Standard_DS2_v2 (https://docs.microsoft.com/en-us/azure/virtual-machines/dv2-dsv2-series#dsv2-series). Basically 7gb ram, 2 vCPU cores, 14gb SSD.

4

Yeah looking at kubectl after doing microk8s status --wait-ready (https://gist.github.com/dave-yotta/a7ee033476b158dae6d4fbfd0da14a88) can see that the hostpath provisioner, registry and coredns pods are all still ContainerCreating or Pending.

I’ll try adding some kubectl wait --for-condition based on existing pods. Sounds like microk8s should be doing this in that command already?

5

Ok I’ve tried that (https://gist.github.com/dave-yotta/de1005cd21ae027cf4c2ced7e86bca42). All I get now is:

waiting for pod pod/calico-node-lmz2v to be ready
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?

(https://gist.github.com/dave-yotta/6db9e7c4a28f88c1e1c0cb97b92286a1)

6

It looks like the apiserver keeps restarting. Do you mind filing a github issue? And if possible upload the inspect tarball?

7

Yeah I’ve actually just done that: https://github.com/ubuntu/microk8s/issues/1710

I’ve not figured out out a way to get the tarball from the cloud CI yet, but I’ll keep trying :slight_smile:

1 Like