Coredns stuck in "ContainerCreating"

I try use kube-vip to create k8s cluster
after create CNI ,coredns staus still in ““ContainerCreating””

[root@k8s-1 ~]# kubectl get pods -A
NAMESPACE         NAME                                        READY   STATUS              RESTARTS       AGE
kube-system       coredns-64897985d-q65n4                     0/1     ContainerCreating   0              35m
kube-system       coredns-64897985d-q6xqp                     0/1     ContainerCreating   0              35m
kube-system       etcd-k8s-1.localdomain                      1/1     Running             11             35m
kube-system       etcd-k8s-2.localdomain                      1/1     Running             0              34m
kube-system       kube-apiserver-k8s-1.localdomain            1/1     Running             11             35m
kube-system       kube-apiserver-k8s-2.localdomain            1/1     Running             0              34m
kube-system       kube-controller-manager-k8s-1.localdomain   1/1     Running             5 (34m ago)    35m
kube-system       kube-controller-manager-k8s-2.localdomain   1/1     Running             0              34m
kube-system       kube-proxy-l2b9m                            1/1     Running             0              35m
kube-system       kube-proxy-mwmhw                            1/1     Running             0              34m
kube-system       kube-scheduler-k8s-1.localdomain            1/1     Running             13 (34m ago)   35m
kube-system       kube-scheduler-k8s-2.localdomain            1/1     Running             0              34m
kube-system       kube-vip-k8s-1.localdomain                  1/1     Running             2 (34m ago)    35m
tigera-operator   tigera-operator-59fc55759-2k5hb             1/1     Running             0              33m

and coredns got certificate error

[root@k8s-1 ~]# kubectl describe pods coredns-64897985d-q65n4  -n kube-system
Name:                 coredns-64897985d-q65n4
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 k8s-1.localdomain/192.168.5.141
Start Time:           Thu, 17 Feb 2022 00:06:20 +0800
Labels:               k8s-app=kube-dns
                      pod-template-hash=64897985d
Annotations:          <none>
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        ReplicaSet/coredns-64897985d
Containers:
  coredns:
    Container ID:
    Image:         k8s.gcr.io/coredns/coredns:v1.8.6
    Image ID:
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cmqd7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  kube-api-access-cmqd7:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               36m                  default-scheduler  Successfully assigned kube-system/coredns-64897985d-q65n4 to k8s-1.localdomain
  Warning  FailedCreatePodSandBox  36m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6c6d4460ff7ebe99bdae732730e8ec8f9ee4988a66fe12f44a191346fbb62e59": error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  Warning  FailedCreatePodSandBox  35m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b1b20eab1496bdd1e74e7b50099b0bac8c7f23c4b12a48f9dc30d2bab032f594": error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  Warning  FailedCreatePodSandBox  35m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f436d1cfd0a4311df69c8396275609838ca63d777177a902936b9c91ce31251a": error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  Warning  FailedCreatePodSandBox  35m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d5c1933a107bbf86d7d6a61cf772b66ea420f7f9a13eea3e4e22c3827e7655d9": error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  Warning  FailedCreatePodSandBox  35m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "512ce9fac5213d45e8aafa23877fd1ba3189cbc33352dd3f24e31ae921093a67": error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  Warning  FailedCreatePodSandBox  34m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a5f3b4a25c47b293fbcff1eed269d0fc7d6058496d772940944f26d5dc962640": error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  Warning  FailedCreatePodSandBox  34m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "aa45414d8e089bd4bae2df454a95eec8b455780599a5efd6e9f775c6864c6335": error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  Warning  FailedCreatePodSandBox  34m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5a88f317279a56fbf22e57b94b535fe9d3bf55ce4c0daa9bddacb14812604f3a": error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  Warning  FailedCreatePodSandBox  34m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a55ab49bac077d486526347473667796756b05e91292be23e00593e31ed9997a": error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  Warning  FailedCreatePodSandBox  54s (x155 over 34m)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "79eab646d8b61b6a0639953d1d23ddb2ae2ae5e745555ee91d145593a3984b09": error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

if I don’t use kube-vip , create single control-plane node k8s
it’s can work
how do I resolve this problem

Cluster information:

Kubernetes version:1.23.3
Cloud being used: bare-metal
Host OS: rocky linux 8.5
CNI and version: calico v1.25.0
CRI and version: containerd 1.4.12

i had similar problem because i changed CP hostname after creating the cluster. I re-ran
‘kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml’ and it fixed the problem.

1 Like

Thanks that fixed my problem.