Hi! I’m trying to set up a one-machine cluster.
Cluster information:
Kubernetes version: 1.20.1-00
Cloud being used: bare-metal
Installation method: apt-get (deb https://apt.kubernetes.io/ kubernetes-xenial main)
Host OS: Ubuntu 18.04
CNI and version: don’t know
CRI and version: cri-o 1.0.0~rc92.3
What happens
I run: sudo kubeadm init --pod-network-cidr=10.100.0.0/12 --cri-socket /var/run/crio/crio.sock
It hangs then timeouts:
[init] Using Kubernetes version: v1.20.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [servername kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.19]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [servername localhost] and IPs [192.168.1.19 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [servername localhost] and IPs [192.168.1.19 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all Kubernetes containers running in cri-o/containerd using crictl:
- 'crictl --runtime-endpoint /var/run/crio/crio.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint /var/run/crio/crio.sock logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
kubelet seems to be running fine:
systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Wed 2021-01-06 12:11:26 CET; 6min ago
Docs: https://kubernetes.io/docs/home/
Main PID: 3476 (kubelet)
Tasks: 34 (limit: 19660)
CGroup: /system.slice/kubelet.service
└─3476 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib
However, its log in journalctl shows lots of errors:
-- Logs begin at Mon 2020-07-27 13:37:42 CEST, end at Wed 2021-01-06 12:18:48 CET. --
Jan 06 12:17:26 servername kubelet[3476]: E0106 12:17:26.524601 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:26 servername kubelet[3476]: E0106 12:17:26.624701 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:26 servername kubelet[3476]: E0106 12:17:26.724842 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:26 servername kubelet[3476]: E0106 12:17:26.824945 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:26 servername kubelet[3476]: E0106 12:17:26.925071 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.025373 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.125526 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.225785 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.325984 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.426104 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.526261 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.626374 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.726505 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.826773 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.926858 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.938566 3476 kubelet.go:2160] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/cni/net.d/. Has your network provider started?
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.027013 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.127353 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.227469 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.305230 3476 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://192.168.1.19:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 192.168.1.19:6443: connect: connection refused
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.327724 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.350584 3476 controller.go:144] failed to ensure lease exists, will retry in 7s, error: Get "https://192.168.1.19:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/servername?timeout=10s": dial tcp 192.168.1.19:6443: connect: connection refused
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.427823 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.528084 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.628301 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.728457 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.828607 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.928835 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.029072 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.129359 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.229471 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.329544 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.429728 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.529828 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.580205 3476 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"servername.1657a029b7bb4562", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"servername", UID:"servername", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node servername status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"servername"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbff58499325b5d62, ext:6662808557, loc:(*time.Location)(0x70c9020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbff58499325b5d62, ext:6662808557, loc:(*time.Location)(0x70c9020)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Post "https://192.168.1.19:6443/api/v1/namespaces/default/events": dial tcp 192.168.1.19:6443: connect: connection refused'(may retry after sleeping)
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.629959 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.730058 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.830296 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.930569 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.030776 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.131064 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: I0106 12:17:30.212036 3476 kubelet_node_status.go:71] Attempting to register node servername
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.212343 3476 kubelet_node_status.go:93] Unable to register node "servername" with API server: Post "https://192.168.1.19:6443/api/v1/nodes": dial tcp 192.168.1.19:6443: connect: connection refused
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.231211 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.331600 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.431806 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.531973 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.632219 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.732383 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.832462 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.932735 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.032976 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.133211 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.233260 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.333439 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.433526 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.533751 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.633836 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.734118 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.827486 3476 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = cri-o configured with systemd cgroup manager, but did not receive slice as parent: /kubepods/burstable/pode8c72dd7a7db732e7138a115e6d914fa
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.827550 3476 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "etcd-servername_kube-system(e8c72dd7a7db732e7138a115e6d914fa)" failed: rpc error: code = Unknown desc = cri-o configured with systemd cgroup manager, but did not receive slice as parent: /kubepods/burstable/pode8c72dd7a7db732e7138a115e6d914fa
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.827569 3476 kuberuntime_manager.go:755] createPodSandbox for pod "etcd-servername_kube-system(e8c72dd7a7db732e7138a115e6d914fa)" failed: rpc error: code = Unknown desc = cri-o configured with systemd cgroup manager, but did not receive slice as parent: /kubepods/burstable/pode8c72dd7a7db732e7138a115e6d914fa
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.827632 3476 pod_workers.go:191] Error syncing pod e8c72dd7a7db732e7138a115e6d914fa ("etcd-servername_kube-system(e8c72dd7a7db732e7138a115e6d914fa)"), skipping: failed to "CreatePodSandbox" for "etcd-servername_kube-system(e8c72dd7a7db732e7138a115e6d914fa)" with CreatePodSandboxError: "CreatePodSandbox for pod \"etcd-servername_kube-system(e8c72dd7a7db732e7138a115e6d914fa)\" failed: rpc error: code = Unknown desc = cri-o configured with systemd cgroup manager, but did not receive slice as parent: /kubepods/burstable/pode8c72dd7a7db732e7138a115e6d914fa"
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.834552 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.934673 3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:32 servername kubelet[3476]: E0106 12:17:32.034759 3476 kubelet.go:2240] node "servername" not found
This goes on for more than 1000 lines.
Can somebody see what’s wrong?