Calico-node-xxxx "Hit error connecting to datastore" in a newly setup worker node

Is this something I have to worry about?

To give some context, I’m creating a simple Microk8s cluster with one controller and one worker node.

I have two separate virtual machines that belong to the same private network of

  • The controller machine at will be referred to as kube-controller

  • The worker machine at will be referred to as kube-worker-test-1

I went ahead and installed Microk8s on both machines, tried a simple nginx deployment on kube-controller and made sure it’s working as intended. Now, I want to create a multi-node cluster so I followed the documentation.

kube-controller $ microk8s add-node

kube-worker $ microk8s join --worker

kube-controller $ kubectl get nodes

All is well so far. However, I could see that a pod automatically created by kube-worker-test-1 keeps crashing with:

Hit error connecting to datastore - retry error=Get “”: dial tcp ││ 3: i/o timeout

Is this an error or a bad configuration from my side? I’ve seen a similar thread but I don’t know if it’s the same thing. I’ve also seen a possible solution on GitHub about changing the pod-network-cidr but I don’t seem to understand it.

I’m still new to Kubernetes and MicroK8s so any input on the matter is very much appreciated.

I got the <kubernetes-server-cluster-ip> using kubectl get service kubernetes -o wide.

From kube-controller (, I tried doing a curl and it responded successfully!

curl https://<kubernetes-server-cluster-ip>:443/api/v1/nodes/foo -k

From kube-worker-test-1 (, I also tried doing a curl and here is the output:

curl https://<kubernetes-server-cluster-ip>:443/api/v1/nodes/foo -k # Did not respond
curl -k # Responded successfully!

Now I guess I’m looking for a way to pass as the datastore IP to the worker on

I found a /var/snap/microk8s/current/args/cni-network/calico-kubeconfig file which contains this config

- name: local
    server: https://[]:443

I tried changing the server key to for both the controller and the worker but, then restarted microk8s but still no luck. The logs still show that is being used.

I haven’t used microkis but from doc

The list of API servers is stored in /var/snap/microk8s/current/args/traefik/provider.yaml . Please, review this list right after the join operation completes to verify all control plane endpoints are there. If your API servers are behind a load balancer you will need to replace the detected endpoints in this file with the entry your provided by the load balancer.

Seems that do some magic viewing your interfaces… If k8s try to do magic maybe it’s will do fail too…

My advice from de distance :slight_smile: try to do from the beginning changing this file before to add the first node

Hope it’s help

No luck unfortunately. Although I managed to narrow down the problem I think.

You put in /var/snap/microk8s/current/args/traefik/provider.yaml and view the change in /var/snap/microk8s/current/args/cni-network/calico-kubeconfig?? and no comunication?

I’ve seen that /var/snap/microk8s/current/args/traefik/provider.yaml was configured successfully by microk8s. I also tried overriding and removing then re-adding the node but it still won’t work.

One interesting thing I noticed is that /var/snap/microk8s/current/args/cni-network/calico-kubeconfig always gets overridden when I restart the cluster.

# Kubeconfig file for Calico CNI plugin.
apiVersion: v1
kind: Config
- name: local
    server: https://[]:443 # => I tried setting it to but it always changes back to this value. 
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJ...

I’ve solved this here!