Calico-kube-controllers stuck on ContainerCreating

General Discussions
1

Asking for help? Comment out what you need so we can get more information to help you!

Hi
What could be the reason for the following? 1st focusing on calico-kube-controllers-56fcbf9d6b-l8vc7 0/1 ContainerCreating

labuser@kub-master:~/work/calico$ kubectl get pods -A -o wide
NAMESPACE     NAME                                       READY   STATUS              RESTARTS        AGE   IP             NODE         NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-56fcbf9d6b-l8vc7   0/1     ContainerCreating   0               43m   <none>         kub-master   <none>           <none>
kube-system   calico-node-7nddr                          0/1     CrashLoopBackOff    15 (2m3s ago)   43m   10.5.183.151   kub-master   <none>           <none>
kube-system   coredns-64897985d-bblws                    0/1     ContainerCreating   0               44m   <none>         kub-master   <none>           <none>
kube-system   coredns-64897985d-zlsp4                    0/1     ContainerCreating   0               44m   <none>         kub-master   <none>           <none>
kube-system   etcd-kub-master                            1/1     Running             34              44m   10.5.183.151   kub-master   <none>           <none>
kube-system   kube-apiserver-kub-master                  1/1     Running             2               44m   10.5.183.151   kub-master   <none>           <none>
kube-system   kube-controller-manager-kub-master         1/1     Running             2               44m   10.5.183.151   kub-master   <none>           <none>
kube-system   kube-scheduler-kub-master                  1/1     Running             10              44m   10.5.183.151   kub-master   <none>           <none>
labuser@kub-master:~/work/calico$ kubectl describe pod calico-kube-controllers-56fcbf9d6b-l8vc7 -n kube-system

Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Warning  FailedScheduling        45m                  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Normal   Scheduled               44m                  default-scheduler  Successfully assigned kube-system/calico-kube-controllers-56fcbf9d6b-l8vc7 to kub-master
  Warning  FailedCreatePodSandBox  43m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "1d1497626db83fededd5e586dd9e1948af1be89c99d738f40840a29afda52ffc" network for pod "calico-kube-controllers-56fcbf9d6b-l8vc7": networkPlugin cni failed to set up pod "calico-kube-controllers-56fcbf9d6b-l8vc7_kube-system" network: error getting ClusterInformation: Get "https://[10.96.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp 10.96.0.1:443: i/o timeout, failed to clean up sandbox container "1d1497626db83fededd5e586dd9e1948af1be89c99d738f40840a29afda52ffc" network for pod "calico-kube-controllers-56fcbf9d6b-l8vc7": networkPlugin cni failed to teardown pod "calico-kube-controllers-56fcbf9d6b-l8vc7_kube-system" network: error getting ClusterInformation: Get "https://[10.96.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp 10.96.0.1:443: i/o timeout]
  Normal   SandboxChanged          3m6s (x69 over 43m)  kubelet            Pod sandbox changed, it will be killed and re-created.

Cluster information:

Kubernetes version:

kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:52:18Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

Cloud being used: bare-metal
Installation method: git clones and apt-get
Host OS: 20.04.3 LTS (Focal Fossa)
CNI and version: calico

cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.1",

CRI and version:

labuser@kub-master:~/work/calico$ docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:36 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
2

The clusterInformation problem I solved with this:

sudo vim.tiny /var/snap/microk8s/current/args/kubelet
Add this to bottom: --node-ip=

sudo vim.tiny /var/snap/microk8s/current/args/kube-apiserver
Add this to bottom: --advertise-address=