[beginner] kubeadm init fails with timeout

Hi! I’m trying to set up a one-machine cluster.

Cluster information:

Kubernetes version: 1.20.1-00
Cloud being used: bare-metal
Installation method: apt-get (deb https://apt.kubernetes.io/ kubernetes-xenial main)
Host OS: Ubuntu 18.04
CNI and version: don’t know
CRI and version: cri-o 1.0.0~rc92.3

What happens

I run: sudo kubeadm init --pod-network-cidr=10.100.0.0/12 --cri-socket /var/run/crio/crio.sock

It hangs then timeouts:

[init] Using Kubernetes version: v1.20.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [servername kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.19]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [servername localhost] and IPs [192.168.1.19 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [servername localhost] and IPs [192.168.1.19 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.

Here is one example how you may list all Kubernetes containers running in cri-o/containerd using crictl:
- 'crictl --runtime-endpoint /var/run/crio/crio.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint /var/run/crio/crio.sock logs CONTAINERID'

error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

kubelet seems to be running fine:

systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
        └─10-kubeadm.conf
Active: active (running) since Wed 2021-01-06 12:11:26 CET; 6min ago
Docs: https://kubernetes.io/docs/home/
Main PID: 3476 (kubelet)
Tasks: 34 (limit: 19660)
CGroup: /system.slice/kubelet.service
        └─3476 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib

However, its log in journalctl shows lots of errors:

-- Logs begin at Mon 2020-07-27 13:37:42 CEST, end at Wed 2021-01-06 12:18:48 CET. --
Jan 06 12:17:26 servername kubelet[3476]: E0106 12:17:26.524601    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:26 servername kubelet[3476]: E0106 12:17:26.624701    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:26 servername kubelet[3476]: E0106 12:17:26.724842    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:26 servername kubelet[3476]: E0106 12:17:26.824945    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:26 servername kubelet[3476]: E0106 12:17:26.925071    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.025373    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.125526    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.225785    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.325984    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.426104    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.526261    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.626374    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.726505    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.826773    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.926858    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:27 servername kubelet[3476]: E0106 12:17:27.938566    3476 kubelet.go:2160] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/cni/net.d/. Has your network provider started?
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.027013    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.127353    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.227469    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.305230    3476 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://192.168.1.19:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 192.168.1.19:6443: connect: connection refused
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.327724    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.350584    3476 controller.go:144] failed to ensure lease exists, will retry in 7s, error: Get "https://192.168.1.19:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/servername?timeout=10s": dial tcp 192.168.1.19:6443: connect: connection refused
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.427823    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.528084    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.628301    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.728457    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.828607    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:28 servername kubelet[3476]: E0106 12:17:28.928835    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.029072    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.129359    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.229471    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.329544    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.429728    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.529828    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.580205    3476 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"servername.1657a029b7bb4562", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"servername", UID:"servername", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node servername status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"servername"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbff58499325b5d62, ext:6662808557, loc:(*time.Location)(0x70c9020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbff58499325b5d62, ext:6662808557, loc:(*time.Location)(0x70c9020)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Post "https://192.168.1.19:6443/api/v1/namespaces/default/events": dial tcp 192.168.1.19:6443: connect: connection refused'(may retry after sleeping)
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.629959    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.730058    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.830296    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:29 servername kubelet[3476]: E0106 12:17:29.930569    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.030776    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.131064    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: I0106 12:17:30.212036    3476 kubelet_node_status.go:71] Attempting to register node servername
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.212343    3476 kubelet_node_status.go:93] Unable to register node "servername" with API server: Post "https://192.168.1.19:6443/api/v1/nodes": dial tcp 192.168.1.19:6443: connect: connection refused
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.231211    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.331600    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.431806    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.531973    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.632219    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.732383    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.832462    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:30 servername kubelet[3476]: E0106 12:17:30.932735    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.032976    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.133211    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.233260    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.333439    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.433526    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.533751    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.633836    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.734118    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.827486    3476 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = cri-o configured with systemd cgroup manager, but did not receive slice as parent: /kubepods/burstable/pode8c72dd7a7db732e7138a115e6d914fa
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.827550    3476 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "etcd-servername_kube-system(e8c72dd7a7db732e7138a115e6d914fa)" failed: rpc error: code = Unknown desc = cri-o configured with systemd cgroup manager, but did not receive slice as parent: /kubepods/burstable/pode8c72dd7a7db732e7138a115e6d914fa
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.827569    3476 kuberuntime_manager.go:755] createPodSandbox for pod "etcd-servername_kube-system(e8c72dd7a7db732e7138a115e6d914fa)" failed: rpc error: code = Unknown desc = cri-o configured with systemd cgroup manager, but did not receive slice as parent: /kubepods/burstable/pode8c72dd7a7db732e7138a115e6d914fa
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.827632    3476 pod_workers.go:191] Error syncing pod e8c72dd7a7db732e7138a115e6d914fa ("etcd-servername_kube-system(e8c72dd7a7db732e7138a115e6d914fa)"), skipping: failed to "CreatePodSandbox" for "etcd-servername_kube-system(e8c72dd7a7db732e7138a115e6d914fa)" with CreatePodSandboxError: "CreatePodSandbox for pod \"etcd-servername_kube-system(e8c72dd7a7db732e7138a115e6d914fa)\" failed: rpc error: code = Unknown desc = cri-o configured with systemd cgroup manager, but did not receive slice as parent: /kubepods/burstable/pode8c72dd7a7db732e7138a115e6d914fa"
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.834552    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:31 servername kubelet[3476]: E0106 12:17:31.934673    3476 kubelet.go:2240] node "servername" not found
Jan 06 12:17:32 servername kubelet[3476]: E0106 12:17:32.034759    3476 kubelet.go:2240] node "servername" not found

This goes on for more than 1000 lines.

Can somebody see what’s wrong?