Calico-kube-controllers stuck on ContainerCreating

Asking for help? Comment out what you need so we can get more information to help you!

Hi
What could be the reason for the following? 1st focusing on calico-kube-controllers-56fcbf9d6b-l8vc7 0/1 ContainerCreating

labuser@kub-master:~/work/calico$ kubectl get pods -A -o wide
NAMESPACE     NAME                                       READY   STATUS              RESTARTS        AGE   IP             NODE         NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-56fcbf9d6b-l8vc7   0/1     ContainerCreating   0               43m   <none>         kub-master   <none>           <none>
kube-system   calico-node-7nddr                          0/1     CrashLoopBackOff    15 (2m3s ago)   43m   10.5.183.151   kub-master   <none>           <none>
kube-system   coredns-64897985d-bblws                    0/1     ContainerCreating   0               44m   <none>         kub-master   <none>           <none>
kube-system   coredns-64897985d-zlsp4                    0/1     ContainerCreating   0               44m   <none>         kub-master   <none>           <none>
kube-system   etcd-kub-master                            1/1     Running             34              44m   10.5.183.151   kub-master   <none>           <none>
kube-system   kube-apiserver-kub-master                  1/1     Running             2               44m   10.5.183.151   kub-master   <none>           <none>
kube-system   kube-controller-manager-kub-master         1/1     Running             2               44m   10.5.183.151   kub-master   <none>           <none>
kube-system   kube-scheduler-kub-master                  1/1     Running             10              44m   10.5.183.151   kub-master   <none>           <none>
labuser@kub-master:~/work/calico$ kubectl describe pod calico-kube-controllers-56fcbf9d6b-l8vc7 -n kube-system

Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Warning  FailedScheduling        45m                  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Normal   Scheduled               44m                  default-scheduler  Successfully assigned kube-system/calico-kube-controllers-56fcbf9d6b-l8vc7 to kub-master
  Warning  FailedCreatePodSandBox  43m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "1d1497626db83fededd5e586dd9e1948af1be89c99d738f40840a29afda52ffc" network for pod "calico-kube-controllers-56fcbf9d6b-l8vc7": networkPlugin cni failed to set up pod "calico-kube-controllers-56fcbf9d6b-l8vc7_kube-system" network: error getting ClusterInformation: Get "https://[10.96.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp 10.96.0.1:443: i/o timeout, failed to clean up sandbox container "1d1497626db83fededd5e586dd9e1948af1be89c99d738f40840a29afda52ffc" network for pod "calico-kube-controllers-56fcbf9d6b-l8vc7": networkPlugin cni failed to teardown pod "calico-kube-controllers-56fcbf9d6b-l8vc7_kube-system" network: error getting ClusterInformation: Get "https://[10.96.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp 10.96.0.1:443: i/o timeout]
  Normal   SandboxChanged          3m6s (x69 over 43m)  kubelet            Pod sandbox changed, it will be killed and re-created.

Cluster information:

Kubernetes version:

kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:52:18Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

Cloud being used: bare-metal
Installation method: git clones and apt-get
Host OS: 20.04.3 LTS (Focal Fossa)
CNI and version: calico

cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.1",

CRI and version:
labuser@kub-master:~/work/calico$ docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:36 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0