HA Cluster initialization issues

Raphy · November 28, 2023, 5:07pm

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version:
Cloud being used: (put bare-metal if not on a public cloud)
Installation method:
Host OS: Ubuntu 22.04
CNI and version:
CRI and version:

During the initialization of the very first control-plane-nodes (3 control-plane-nodes + 3 worker-nodes) I’m getting these errors :

root@k8s-eu-1-control-plane-node-1:~# sudo kubeadm init --control-plane-endpoint k82-eu-1-load-balancer-dns-1:53 --upload-certs --v=8 --ignore-preflight-errors=Port-6443
I1128 17:42:29.255247   31193 initconfiguration.go:117] detected and using CRI socket: unix:///var/run/containerd/containerd.sock
I1128 17:42:29.256565   31193 interface.go:432] Looking for default routes with IPv4 addresses
I1128 17:42:29.256574   31193 interface.go:437] Default route transits interface "eth0"
I1128 17:42:29.256762   31193 interface.go:209] Interface eth0 is up
I1128 17:42:29.256983   31193 interface.go:257] Interface "eth0" has 2 addresses :[aa.aaa.aaa.aa/19 10.0.0.30/32].
I1128 17:42:29.257001   31193 interface.go:224] Checking addr  aa.aaa.aaa.aa/19.
I1128 17:42:29.257012   31193 interface.go:231] IP found aa.aaa.aaa.aa
I1128 17:42:29.257029   31193 interface.go:263] Found valid IPv4 address aa.aaa.aaa.aa for interface "eth0".
I1128 17:42:29.257036   31193 interface.go:443] Found active IP aa.aaa.aaa.aa 
I1128 17:42:29.257163   31193 kubelet.go:196] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
I1128 17:42:29.267107   31193 version.go:187] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt
[init] Using Kubernetes version: v1.28.4
[preflight] Running pre-flight checks
I1128 17:42:29.559049   31193 checks.go:563] validating Kubernetes and kubeadm version
I1128 17:42:29.559082   31193 checks.go:168] validating if the firewall is enabled and active
I1128 17:42:29.568760   31193 checks.go:203] validating availability of port 6443
    [WARNING Port-6443]: Port 6443 is in use
I1128 17:42:29.569105   31193 checks.go:203] validating availability of port 10259
I1128 17:42:29.569133   31193 checks.go:203] validating availability of port 10257
I1128 17:42:29.569152   31193 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml
I1128 17:42:29.569164   31193 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml
I1128 17:42:29.569171   31193 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml
I1128 17:42:29.569179   31193 checks.go:280] validating the existence of file /etc/kubernetes/manifests/etcd.yaml
I1128 17:42:29.569186   31193 checks.go:430] validating if the connectivity type is via proxy or direct
I1128 17:42:29.569212   31193 checks.go:469] validating http connectivity to first IP address in the CIDR
I1128 17:42:29.569229   31193 checks.go:469] validating http connectivity to first IP address in the CIDR
I1128 17:42:29.569235   31193 checks.go:104] validating the container runtime
I1128 17:42:29.627509   31193 checks.go:639] validating whether swap is enabled or not
I1128 17:42:29.628210   31193 checks.go:370] validating the presence of executable crictl
I1128 17:42:29.628302   31193 checks.go:370] validating the presence of executable conntrack
I1128 17:42:29.628330   31193 checks.go:370] validating the presence of executable ip
I1128 17:42:29.628358   31193 checks.go:370] validating the presence of executable iptables
I1128 17:42:29.628393   31193 checks.go:370] validating the presence of executable mount
I1128 17:42:29.628433   31193 checks.go:370] validating the presence of executable nsenter
I1128 17:42:29.628453   31193 checks.go:370] validating the presence of executable ebtables
I1128 17:42:29.628482   31193 checks.go:370] validating the presence of executable ethtool
I1128 17:42:29.628514   31193 checks.go:370] validating the presence of executable socat
I1128 17:42:29.628539   31193 checks.go:370] validating the presence of executable tc
I1128 17:42:29.628557   31193 checks.go:370] validating the presence of executable touch
I1128 17:42:29.628579   31193 checks.go:516] running all checks
I1128 17:42:29.645127   31193 checks.go:401] checking whether the given node name is valid and reachable using net.LookupHost
I1128 17:42:29.645200   31193 checks.go:605] validating kubelet version
I1128 17:42:29.719616   31193 checks.go:130] validating if the "kubelet" service is enabled and active
I1128 17:42:29.732450   31193 checks.go:203] validating availability of port 10250
I1128 17:42:29.732572   31193 checks.go:329] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1128 17:42:29.732629   31193 checks.go:329] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1128 17:42:29.732650   31193 checks.go:203] validating availability of port 2379
I1128 17:42:29.732675   31193 checks.go:203] validating availability of port 2380
I1128 17:42:29.732694   31193 checks.go:243] validating the existence and emptiness of directory /var/lib/etcd
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I1128 17:42:29.732832   31193 checks.go:828] using image pull policy: IfNotPresent
I1128 17:42:29.797031   31193 checks.go:846] image exists: registry.k8s.io/kube-apiserver:v1.28.4
I1128 17:42:29.841337   31193 checks.go:846] image exists: registry.k8s.io/kube-controller-manager:v1.28.4
I1128 17:42:29.886504   31193 checks.go:846] image exists: registry.k8s.io/kube-scheduler:v1.28.4
I1128 17:42:29.927376   31193 checks.go:846] image exists: registry.k8s.io/kube-proxy:v1.28.4
W1128 17:42:29.971812   31193 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
I1128 17:42:30.009450   31193 checks.go:846] image exists: registry.k8s.io/pause:3.9
I1128 17:42:30.043123   31193 checks.go:846] image exists: registry.k8s.io/etcd:3.5.9-0
I1128 17:42:30.072814   31193 checks.go:846] image exists: registry.k8s.io/coredns/coredns:v1.10.1
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I1128 17:42:30.072992   31193 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I1128 17:42:30.335319   31193 certs.go:519] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k82-eu-1-load-balancer-dns-1 k8s-eu-1-control-plane-node-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 38.242.249.60]
[certs] Generating "apiserver-kubelet-client" certificate and key
I1128 17:42:30.661803   31193 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I1128 17:42:30.777524   31193 certs.go:519] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I1128 17:42:31.015301   31193 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I1128 17:42:31.154876   31193 certs.go:519] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-eu-1-control-plane-node-1 localhost] and IPs [38.242.249.60 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-eu-1-control-plane-node-1 localhost] and IPs [38.242.249.60 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I1128 17:42:31.674814   31193 certs.go:78] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I1128 17:42:31.927500   31193 kubeconfig.go:103] creating kubeconfig file for admin.conf
W1128 17:42:31.927953   31193 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
I1128 17:42:32.224910   31193 kubeconfig.go:103] creating kubeconfig file for kubelet.conf
W1128 17:42:32.225431   31193 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I1128 17:42:32.304014   31193 kubeconfig.go:103] creating kubeconfig file for controller-manager.conf
W1128 17:42:32.304371   31193 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I1128 17:42:32.491972   31193 kubeconfig.go:103] creating kubeconfig file for scheduler.conf
W1128 17:42:32.492400   31193 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I1128 17:42:32.674599   31193 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I1128 17:42:32.674664   31193 manifests.go:102] [control-plane] getting StaticPodSpecs
I1128 17:42:32.675000   31193 certs.go:519] validating certificate period for CA certificate
I1128 17:42:32.675079   31193 manifests.go:128] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I1128 17:42:32.675093   31193 manifests.go:128] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver"
I1128 17:42:32.675107   31193 manifests.go:128] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I1128 17:42:32.675113   31193 manifests.go:128] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver"
I1128 17:42:32.675121   31193 manifests.go:128] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver"
I1128 17:42:32.676061   31193 manifests.go:157] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I1128 17:42:32.676086   31193 manifests.go:102] [control-plane] getting StaticPodSpecs
I1128 17:42:32.676281   31193 manifests.go:128] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I1128 17:42:32.676296   31193 manifests.go:128] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager"
I1128 17:42:32.676302   31193 manifests.go:128] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I1128 17:42:32.676310   31193 manifests.go:128] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I1128 17:42:32.676321   31193 manifests.go:128] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I1128 17:42:32.676330   31193 manifests.go:128] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager"
I1128 17:42:32.676339   31193 manifests.go:128] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager"
I1128 17:42:32.677157   31193 manifests.go:157] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I1128 17:42:32.677177   31193 manifests.go:102] [control-plane] getting StaticPodSpecs
I1128 17:42:32.677340   31193 manifests.go:128] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I1128 17:42:32.677808   31193 manifests.go:157] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
I1128 17:42:32.677840   31193 kubelet.go:67] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
I1128 17:42:32.917314   31193 waitcontrolplane.go:83] [wait-control-plane] Waiting for the API server to be healthy
I1128 17:42:32.918088   31193 loader.go:395] Config loaded from file:  /etc/kubernetes/admin.conf
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I1128 17:42:32.930405   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:42:32.930428   31193 round_trippers.go:469] Request Headers:
I1128 17:42:32.930465   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:42:32.930480   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:42:42.939995   31193 round_trippers.go:574] Response Status:  in 10009 milliseconds
I1128 17:42:42.940042   31193 round_trippers.go:577] Response Headers:
I1128 17:42:43.441239   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:42:43.441283   31193 round_trippers.go:469] Request Headers:
I1128 17:42:43.441293   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:42:43.441301   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:42:53.441613   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:42:53.441661   31193 round_trippers.go:577] Response Headers:
I1128 17:42:53.940625   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:42:53.940670   31193 round_trippers.go:469] Request Headers:
I1128 17:42:53.940680   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:42:53.940687   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:43:03.941173   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:43:03.941252   31193 round_trippers.go:577] Response Headers:
I1128 17:43:04.441176   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:43:04.441214   31193 round_trippers.go:469] Request Headers:
I1128 17:43:04.441224   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:43:04.441230   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
[kubelet-check] Initial timeout of 40s passed.
I1128 17:43:14.442189   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:43:14.442229   31193 round_trippers.go:577] Response Headers:
I1128 17:43:14.941039   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:43:14.941104   31193 round_trippers.go:469] Request Headers:
I1128 17:43:14.941120   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:43:14.941140   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:43:24.941173   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:43:24.941223   31193 round_trippers.go:577] Response Headers:
I1128 17:43:24.941507   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:43:24.941528   31193 round_trippers.go:469] Request Headers:
I1128 17:43:24.941539   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:43:24.941548   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:43:34.942362   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:43:34.942404   31193 round_trippers.go:577] Response Headers:
I1128 17:43:35.440585   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:43:35.440632   31193 round_trippers.go:469] Request Headers:
I1128 17:43:35.440642   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:43:35.440648   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:43:45.441230   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:43:45.441291   31193 round_trippers.go:577] Response Headers:
I1128 17:43:45.441580   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:43:45.441606   31193 round_trippers.go:469] Request Headers:
I1128 17:43:45.441640   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:43:45.441650   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:43:55.441767   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:43:55.441817   31193 round_trippers.go:577] Response Headers:
I1128 17:43:55.940682   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:43:55.940714   31193 round_trippers.go:469] Request Headers:
I1128 17:43:55.940728   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:43:55.940741   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:44:05.940729   31193 round_trippers.go:574] Response Status:  in 9999 milliseconds
I1128 17:44:05.940767   31193 round_trippers.go:577] Response Headers:
I1128 17:44:06.440840   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:44:06.440913   31193 round_trippers.go:469] Request Headers:
I1128 17:44:06.440926   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:44:06.440944   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:44:16.441595   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:44:16.441630   31193 round_trippers.go:577] Response Headers:
I1128 17:44:16.940457   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:44:16.940492   31193 round_trippers.go:469] Request Headers:
I1128 17:44:16.940502   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:44:16.940508   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:44:26.941069   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:44:26.941125   31193 round_trippers.go:577] Response Headers:
I1128 17:44:27.441003   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:44:27.441044   31193 round_trippers.go:469] Request Headers:
I1128 17:44:27.441056   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:44:27.441067   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:44:37.441221   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:44:37.441293   31193 round_trippers.go:577] Response Headers:
I1128 17:44:37.441517   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:44:37.441529   31193 round_trippers.go:469] Request Headers:
I1128 17:44:37.441537   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:44:37.441543   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:44:47.442368   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:44:47.442425   31193 round_trippers.go:577] Response Headers:
I1128 17:44:47.941439   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:44:47.941515   31193 round_trippers.go:469] Request Headers:
I1128 17:44:47.941533   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:44:47.941549   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:44:57.941446   31193 round_trippers.go:574] Response Status:  in 9999 milliseconds
I1128 17:44:57.941502   31193 round_trippers.go:577] Response Headers:
I1128 17:44:58.440692   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:44:58.440758   31193 round_trippers.go:469] Request Headers:
I1128 17:44:58.440773   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:44:58.440782   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:45:08.441721   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:45:08.441776   31193 round_trippers.go:577] Response Headers:
I1128 17:45:08.949674   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:45:08.949723   31193 round_trippers.go:469] Request Headers:
I1128 17:45:08.949736   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:45:08.949746   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:45:18.949946   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:45:18.949992   31193 round_trippers.go:577] Response Headers:
I1128 17:45:19.440915   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:45:19.440948   31193 round_trippers.go:469] Request Headers:
I1128 17:45:19.440957   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:45:19.440963   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:45:29.441182   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:45:29.441220   31193 round_trippers.go:577] Response Headers:
I1128 17:45:29.940652   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:45:29.940696   31193 round_trippers.go:469] Request Headers:
I1128 17:45:29.940707   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:45:29.940716   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:45:39.941785   31193 round_trippers.go:574] Response Status:  in 10001 milliseconds
I1128 17:45:39.941833   31193 round_trippers.go:577] Response Headers:
I1128 17:45:40.440701   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:45:40.440730   31193 round_trippers.go:469] Request Headers:
I1128 17:45:40.440739   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:45:40.440746   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:45:50.441335   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:45:50.441394   31193 round_trippers.go:577] Response Headers:
I1128 17:45:50.941305   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:45:50.941372   31193 round_trippers.go:469] Request Headers:
I1128 17:45:50.941385   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:45:50.941395   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:46:00.941280   31193 round_trippers.go:574] Response Status:  in 9999 milliseconds
I1128 17:46:00.941348   31193 round_trippers.go:577] Response Headers:
I1128 17:46:00.941681   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:46:00.941693   31193 round_trippers.go:469] Request Headers:
I1128 17:46:00.941700   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:46:00.941707   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:46:10.942209   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:46:10.942258   31193 round_trippers.go:577] Response Headers:
I1128 17:46:11.441262   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:46:11.441303   31193 round_trippers.go:469] Request Headers:
I1128 17:46:11.441315   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:46:11.441325   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:46:21.441610   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:46:21.441649   31193 round_trippers.go:577] Response Headers:
I1128 17:46:21.940554   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:46:21.940588   31193 round_trippers.go:469] Request Headers:
I1128 17:46:21.940597   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:46:21.940604   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:46:31.941224   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:46:31.941272   31193 round_trippers.go:577] Response Headers:
I1128 17:46:32.441186   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:46:32.441237   31193 round_trippers.go:469] Request Headers:
I1128 17:46:32.441245   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:46:32.441252   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:46:42.441651   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:46:42.441699   31193 round_trippers.go:577] Response Headers:
I1128 17:46:42.940653   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:46:42.940688   31193 round_trippers.go:469] Request Headers:
I1128 17:46:42.940699   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:46:42.940706   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:46:52.940879   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:46:52.940925   31193 round_trippers.go:577] Response Headers:
I1128 17:46:52.941047   31193 round_trippers.go:463] GET https://k82-eu-1-load-balancer-dns-1:53/healthz?timeout=10s
I1128 17:46:52.941061   31193 round_trippers.go:469] Request Headers:
I1128 17:46:52.941068   31193 round_trippers.go:473]     Accept: application/json, */*
I1128 17:46:52.941078   31193 round_trippers.go:473]     User-Agent: kubeadm/v1.28.2 (linux/amd64) kubernetes/89a4ea3
I1128 17:47:02.941262   31193 round_trippers.go:574] Response Status:  in 10000 milliseconds
I1128 17:47:02.941311   31193 round_trippers.go:577] Response Headers:

Unfortunately, an error has occurred:
    timed out waiting for the condition

This error is likely caused by:
    - The kubelet is not running
    - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
    - 'systemctl status kubelet'
    - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
    - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
    Once you have found the failing container, you can inspect its logs with:
    - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID'
couldn't initialize a Kubernetes cluster
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase
    cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:108
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
    cmd/kubeadm/app/cmd/phases/workflow/runner.go:259
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
    cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
    cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
    cmd/kubeadm/app/cmd/init.go:111
github.com/spf13/cobra.(*Command).execute
    vendor/github.com/spf13/cobra/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
    vendor/github.com/spf13/cobra/command.go:1068
github.com/spf13/cobra.(*Command).Execute
    vendor/github.com/spf13/cobra/command.go:992
k8s.io/kubernetes/cmd/kubeadm/app.Run
    cmd/kubeadm/app/kubeadm.go:50
main.main
    cmd/kubeadm/kubeadm.go:25
runtime.main
    /usr/local/go/src/runtime/proc.go:250
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1598
error execution phase wait-control-plane
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
    cmd/kubeadm/app/cmd/phases/workflow/runner.go:260
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
    cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
    cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
    cmd/kubeadm/app/cmd/init.go:111
github.com/spf13/cobra.(*Command).execute
    vendor/github.com/spf13/cobra/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
    vendor/github.com/spf13/cobra/command.go:1068
github.com/spf13/cobra.(*Command).Execute
    vendor/github.com/spf13/cobra/command.go:992
k8s.io/kubernetes/cmd/kubeadm/app.Run
    cmd/kubeadm/app/kubeadm.go:50
main.main
    cmd/kubeadm/kubeadm.go:25
runtime.main
    /usr/local/go/src/runtime/proc.go:250
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1598

Output of journalctl -xeu kubelet : KubeletJournalCtlOutput.txt - Google Drive

root@k8s-eu-1-control-plane-node-1:~# crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause
579c2b9e5d17a       7fe0e6f37db33       41 seconds ago      Exited              kube-apiserver            50                  7d52f351045d2       kube-apiserver-k8s-eu-1-control-plane-node-1
9db9a2fe179e3       e3db313c6dbc0       16 minutes ago      Running             kube-scheduler            25                  d55a5e9d9be56       kube-scheduler-k8s-eu-1-control-plane-node-1
d3887c919854f       d058aa5ab969c       16 minutes ago      Running             kube-controller-manager   18                  e61c1eb6a8700       kube-controller-manager-k8s-eu-1-control-plane-node-1
root@k8s-eu-1-control-plane-node-1:~# 
root@k8s-eu-1-control-plane-node-1:~# crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs 579c2b9e5d17a
I1128 16:58:28.080267       1 options.go:220] external host was not specified, using 38.242.249.60
I1128 16:58:28.081342       1 server.go:148] Version: v1.28.4
I1128 16:58:28.081365       1 server.go:150] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
E1128 16:58:28.081652       1 run.go:74] "command failed" err="failed to create listener: failed to listen on 0.0.0.0:6443: listen tcp 0.0.0.0:6443: bind: address already in use"

root@k8s-eu-1-control-plane-node-1:~# ps xa | grep 6443
  33348 pts/0    R+     0:00 grep --color=auto 6443

If I do kubeadm init --pod-network-cidr=192.168.0.0/16 the initialization process goes fine

Based on what is described here: https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#keepalived-configuration

I defined:

/etc/haproxy/haproxy.cfg :

# https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#haproxy-configuration

# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log /dev/log local0
    log /dev/log local1 notice
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          20s
    timeout server          20s
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserver
    bind *:6445
    mode tcp
    option tcplog
    default_backend apiserverbackend

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
# https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#bootstrap-the-cluster

backend apiserverbackend
    #option httpchk GET /healthz
    option httpchk GET /livez
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
        server k82-eu-1-load-balancer-dns-1 ppp.pp.ppp.pp:53
        server k82-eu-1-load-balancer-dns-2 yyy.yy.yyy.yy:53

/etc/keepalived/keepalived.conf :

# https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#keepalived-configuration
# https://www.server-world.info/en/note?os=Ubuntu_22.04&p=keepalived&f=1

! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
    enable_script_security
}
vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    authentication {
        auth_type PASS
        auth_pass 42
    }
    virtual_ipaddress {
        10.0.0.30
    }
    track_script {
        check_apiserver
    }
}

/etc/keepalived/check_apiserver.sh :

# https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#keepalived-configuration
# https://www.server-world.info/en/note?os=Ubuntu_22.04&p=keepalived&f=1

#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

APISERVER_DEST_PORT=6445
APISERVER_VIP=10.0.0.30

curl --silent --max-time 2 --insecure https://localhost:${APISERVER_DEST_PORT}/ -o /dev/null || errorExit "Error GET https://localhost:${APISERVER_DEST_PORT}/"
if ip addr | grep -q ${APISERVER_VIP}; then
    curl --silent --max-time 2 --insecure https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/ -o /dev/null || errorExit "Error GET https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/"
fi

/etc/kubernetes/manifests/haproxy.yaml :

# https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#option-2-run-the-services-as-static-pods

apiVersion: v1
kind: Pod
metadata:
  name: haproxy
  namespace: kube-system
spec:
  containers:
  - image: haproxy:2.1.4
    name: haproxy
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: localhost
        path: /healthz
        port: 6445
        scheme: HTTPS
    volumeMounts:
    - mountPath: /usr/local/etc/haproxy/haproxy.cfg
      name: haproxyconf
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/haproxy/haproxy.cfg
      type: FileOrCreate
    name: haproxyconf
status: {}

/etc/kubernetes/manifests/keepalived.yaml :

# https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#option-2-run-the-services-as-static-pods

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  name: keepalived
  namespace: kube-system
spec:
  containers:
  - image: osixia/keepalived:2.0.17
    name: keepalived
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - NET_BROADCAST
        - NET_RAW
    volumeMounts:
    - mountPath: /usr/local/etc/keepalived/keepalived.conf
      name: config
    - mountPath: /etc/keepalived/check_apiserver.sh
      name: check
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/keepalived/keepalived.conf
    name: config
  - hostPath:
      path: /etc/keepalived/check_apiserver.sh
    name: check
status: {}

What am I doing wrong? How to make the initialization process work?

fox-md · November 28, 2023, 6:07pm

Hi,
Try checking with netstat command which process is holding port 6443 on the k8s-eu-1-control-plane-node-1 node.

Topic		Replies	Views
Kubeadm init gets stuck with "Error writing Crisocket information for the control-plane node" General Discussions	2	5808	December 4, 2019
Kubeadm init error on ubuntu 20.04 General Discussions	2	3311	January 10, 2024
Kubernets not working General Discussions	2	4822	October 15, 2019
Kubernetes installation issue with kubeadm init General Discussions deployment	3	1057	January 29, 2024
Kubernets installation erro General Discussions network	1	279	April 24, 2023

HA Cluster initialization issues

Cluster information:

Related Topics