When I run kubeadm join to add node to k8s cluster, there is an error

Hello,

When I run kubeadm join to add node to k8s cluster, there is an error as follow:

[preflight] running pre-flight checks
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr] or no builtin kernel ipvs support: map[nf_conntrack_ipv4:{} ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{}]
you can solve this problem with following methods:

  1. Run 'modprobe – ’ to load missing kernel modules;
  2. Provide the missing builtin kernel ipvs support

[discovery] Trying to connect to API Server “10.112.130.81:6443”
[discovery] Created cluster-info discovery client, requesting info from “https://10.112.130.81:6443
[discovery] Requesting info from “https://10.112.130.81:6443” again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server “10.112.130.81:6443”
[discovery] Successfully established connection with API Server “10.112.130.81:6443”
[kubelet] Downloading configuration for the kubelet from the “kubelet-config-1.12” ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap…

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- ‘systemctl status kubelet’
- ‘journalctl -xeu kubelet’
timed out waiting for the condition

Does anyone know why?

the output messages with command systemctl status kubelet :

● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since 二 2021-06-22 15:44:13 CST; 5min ago
Docs: Kubernetes Documentation | Kubernetes
Main PID: 3320 (kubelet)
Tasks: 59
Memory: 46.0M
CPU: 9.630s
CGroup: /system.slice/kubelet.service
└─3320 /usr/bin/kubelet
6月 22 15:47:55 lab640-S2600CWR kubelet[3320]: I0622 15:47:55.868919 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:48:05 lab640-S2600CWR kubelet[3320]: I0622 15:48:05.913652 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:48:15 lab640-S2600CWR kubelet[3320]: I0622 15:48:15.961051 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:48:26 lab640-S2600CWR kubelet[3320]: I0622 15:48:26.008733 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:48:36 lab640-S2600CWR kubelet[3320]: I0622 15:48:36.050610 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll

the output messages with command journalctl -xeu kubelet :

6月 22 15:47:15 lab640-S2600CWR kubelet[3320]: I0622 15:47:15.675684 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:47:25 lab640-S2600CWR kubelet[3320]: I0622 15:47:25.719316 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:47:35 lab640-S2600CWR kubelet[3320]: I0622 15:47:35.767315 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:47:45 lab640-S2600CWR kubelet[3320]: I0622 15:47:45.813969 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:47:55 lab640-S2600CWR kubelet[3320]: I0622 15:47:55.868919 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:48:05 lab640-S2600CWR kubelet[3320]: I0622 15:48:05.913652 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:48:15 lab640-S2600CWR kubelet[3320]: I0622 15:48:15.961051 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:48:26 lab640-S2600CWR kubelet[3320]: I0622 15:48:26.008733 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:48:36 lab640-S2600CWR kubelet[3320]: I0622 15:48:36.050610 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:48:46 lab640-S2600CWR kubelet[3320]: I0622 15:48:46.099551 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:48:56 lab640-S2600CWR kubelet[3320]: I0622 15:48:56.149845 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:49:06 lab640-S2600CWR kubelet[3320]: I0622 15:49:06.203749 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:49:16 lab640-S2600CWR kubelet[3320]: I0622 15:49:16.253241 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:49:26 lab640-S2600CWR kubelet[3320]: I0622 15:49:26.302561 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:49:36 lab640-S2600CWR kubelet[3320]: I0622 15:49:36.351625 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll
6月 22 15:49:46 lab640-S2600CWR kubelet[3320]: I0622 15:49:46.397393 3320 kubelet_node_status.go:277] Setting node annotation to enable volume controll

Cluster information:

Kubernetes version: 1.12.9
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: ubuntu16.04
CNI and version:
CRI and version:

1.12 has been out of support for some time (2 years+), and I’d strongly recommend updating. With that said, IIRC that error was related to some left over bad config. You can try kubeadm reset -f (and then restart) to try joining the node again.