Error getting ClusterInformation - Forbidden [solved]

General Discussions microk8s
1

I am trying to create a pod, but getting the following error: (pod describe)

Warning  FailedCreatePodSandBox  118s (x1607 over 6h)  kubelet  (combined from similar events):
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox 
"885133771bfde866b301d5ecc41ae578c24480cf7e9c7806e12c5f576d32a5b7": error getting ClusterInformation: 
Get https://[10.152.183.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: Forbidden

is this a rolebinding issue?
I could get the info via proxy:

curl --noproxy "*" http://localhost:8001/apis/crd.projectcalico.org/v1/clusterinformations/default
{"apiVersion":"crd.projectcalico.org/v1","kind":"ClusterInformation","metadata":{"annotations":{"projectcalico.org/metadata":"{\"uid\":\"68f0d789-38e2-4dec-9838-841711f476f3\",\"creationTimestamp\":\"2021-02-15T13:38:53Z\"}"},"creationTimestamp":"2021-02-15T13:38:53Z","generation":1,"managedFields":[{"apiVersion":"crd.projectcalico.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:projectcalico.org/metadata":{}}},"f:spec":{".":{},"f:calicoVersion":{},"f:clusterGUID":{},"f:clusterType":{},"f:datastoreReady":{}}},"manager":"Go-http-client","operation":"Update","time":"2021-02-15T13:38:53Z"}],"name":"default","resourceVersion":"2633357","selfLink":"/apis/crd.projectcalico.org/v1/clusterinformations/default","uid":"9cf8750c-0a66-4250-9b80-61022c47673c"},"spec":{"calicoVersion":"v3.13.2","clusterGUID":"9bcc29dc31264db899cbea927e6ddfcf","clusterType":"k8s,bgp,kubeadm,kdd","datastoreReady":true}}
2

Hi @laplasz, how does your microk8s.kubectl get all -A look like? What distribution are you on? How can I reproduce this localy?

3 
~$ microk8s.kubectl get all -A
NAMESPACE     NAME                                          READY   STATUS              RESTARTS   AGE
kube-system   pod/calico-kube-controllers-847c8c99d-qtpfn   0/1     ContainerCreating   0          21h
kube-system   pod/coredns-86f78bb79c-5tbzl                  0/1     ContainerCreating   0          20h
kube-system   pod/calico-node-4xnqg                         1/1     Running             1          21h
default       pod/pilot-0                                   0/2     ContainerCreating   0          12h

NAMESPACE     NAME                                TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
default       service/kubernetes                  ClusterIP      10.152.183.1     <none>        443/TCP                         21h
default       service/eric-tcc-titansim-service   LoadBalancer   10.152.183.142   <pending>     4000:32415/TCP,7103:30725/TCP   21h
kube-system   service/kube-dns                    ClusterIP      10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP          20h
default       service/pilot-service               LoadBalancer   10.152.183.100   <pending>     4000:30173/TCP,7103:31524/TCP   12h

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   21h

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   0/1     1            0           21h
kube-system   deployment.apps/coredns                   0/1     1            0           20h

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-847c8c99d   1         1         0       21h
kube-system   replicaset.apps/coredns-86f78bb79c                  1         1         0       20h

calico-kube-controllers got stuck in containerCreating with the same error

~$ microk8s.ctr version  
Client:
  Version:  v1.3.7
  Revision: 8fba4e9a7d01810a393d5d25a3621dc101981175

Server:
  Version:  v1.3.7
  Revision: 8fba4e9a7d01810a393d5d25a3621dc101981175
  UUID: abfacce8-45d8-4348-8699-2b7b5892268b

it was a fresh install using snap…

4

is it possible that microk8s using the system proxy? how can I disable proxy settings for microk8s?
yes, that was it
i had to update /etc/environment with NO_PROXY for microk8s IPs