# Problems with creating a high availability cluster using Kubeadm

**URL:** https://discuss.kubernetes.io/t/problems-with-creating-a-high-availability-cluster-using-kubeadm/22457
**Category:** General Discussions
**Tags:** development, minikube
**Created:** [December 28, 2022, 1:26am UTC](https://discuss.kubernetes.io/t/problems-with-creating-a-high-availability-cluster-using-kubeadm/22457 "2022-12-28T01:26:58Z")
**Posts on this page:** 1
**Page:** 1

<div class="post-metadata">

### Author: ![ailunbolinkenasi](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/ailunbolinkenasi/32/11462_2.png) [@ailunbolinkenasi](https://discuss.kubernetes.io/u/ailunbolinkenasi)
#### Post date: [December 28, 2022, 1:26am UTC](https://discuss.kubernetes.io/t/problems-with-creating-a-high-availability-cluster-using-kubeadm/22457/1 "2022-12-28T01:26:58Z")

</div>

### What happened?

The many times I have tried to create a highly available cluster using `kubeadm init --config kubeadm-config.yaml` --upload-certs, I have encountered the following problems

## This is the bug report I presented

1. Looks like there’s no link to [api.k8s.verbos.com](http://api.k8s.verbos.com)
2. I use Keepalived+Haproxy to load api-server

```shell
[root@containerd-master1 ~]# kubeadm init --config /root/kubeadm-config.yaml --v=5

```

```shell
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I1227 09:37:29.495368 39237 certs.go:111] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I1227 09:37:29.625377 39237 certs.go:519] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [api.k8s.verbos.com containerd-master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.10.0.1 10.1.6.48 10.1.6.24 10.1.6.45]
[certs] Generating "apiserver-kubelet-client" certificate and key
I1227 09:37:29.840751 39237 certs.go:111] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I1227 09:37:29.949095 39237 certs.go:519] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
I1227 09:37:30.204471 39237 certs.go:77] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I1227 09:37:30.434682 39237 kubeconfig.go:103] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I1227 09:37:30.622955 39237 kubeconfig.go:103] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I1227 09:37:30.764826 39237 kubeconfig.go:103] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I1227 09:37:31.043569 39237 kubeconfig.go:103] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I1227 09:37:31.231569 39237 kubelet.go:65] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I1227 09:37:31.322959 39237 manifests.go:99] [control-plane] getting StaticPodSpecs
I1227 09:37:31.323250 39237 certs.go:519] validating certificate period for CA certificate
I1227 09:37:31.323316 39237 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I1227 09:37:31.323323 39237 manifests.go:125] [control-plane] adding volume "etc-pki" for component "kube-apiserver"
I1227 09:37:31.323327 39237 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I1227 09:37:31.329137 39237 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I1227 09:37:31.329155 39237 manifests.go:99] [control-plane] getting StaticPodSpecs
I1227 09:37:31.329359 39237 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I1227 09:37:31.329371 39237 manifests.go:125] [control-plane] adding volume "etc-pki" for component "kube-controller-manager"
I1227 09:37:31.329377 39237 manifests.go:125] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I1227 09:37:31.329381 39237 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I1227 09:37:31.329386 39237 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I1227 09:37:31.329927 39237 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I1227 09:37:31.329947 39237 manifests.go:99] [control-plane] getting StaticPodSpecs
I1227 09:37:31.330141 39237 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I1227 09:37:31.330573 39237 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
I1227 09:37:31.330587 39237 etcd.go:103] [etcd] External etcd mode. Skipping the creation of a manifest for local etcd
I1227 09:37:31.330594 39237 waitcontrolplane.go:89] [wait-control-plane] Waiting for the API server to be healthy
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I1227 09:37:31.332356 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 1 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:32.333672 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 2 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:33.334389 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 3 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:34.335979 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 4 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:35.336694 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 5 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:36.337340 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 6 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:37.338720 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 7 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:38.339495 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 8 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:39.340194 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 9 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:40.340904 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 10 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:41.843152 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 1 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:42.843783 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 2 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:43.844609 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 3 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:44.845551 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 4 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:45.846356 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 5 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:46.846946 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 6 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:47.848392 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 7 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:48.849161 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 8 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:49.850777 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 9 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:50.851537 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 10 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
[apiclient] All control plane components are healthy after 21.018490 seconds
I1227 09:37:52.350344 39237 uploadconfig.go:110] [upload-config] Uploading the kubeadm ClusterConfiguration to a ConfigMap
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I1227 09:37:52.373290 39237 uploadconfig.go:124] [upload-config] Uploading the kubelet component config to a ConfigMap
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
I1227 09:37:52.386582 39237 uploadconfig.go:129] [upload-config] Preserving the CRISocket information for the control-plane node
I1227 09:37:52.386595 39237 patchnode.go:31] [patchnode] Uploading the CRI Socket information "/run/containerd/containerd.sock" to the Node API object "containerd-master1" as an annotation
[kubelet-check] Initial timeout of 40s passed.
Error writing Crisocket information for the control-plane node
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runUploadKubeletConfig
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init/uploadconfig.go:131
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:153
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:852
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:960
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:897
k8s.io/kubernetes/cmd/kubeadm/app.Run
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
        _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:225
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1371
error execution phase upload-config/kubelet
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:153
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:852
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:960
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:897
k8s.io/kubernetes/cmd/kubeadm/app.Run
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
        _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:225
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1371

```

## This is my Hosts file

```shell
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.1.6.48 containerd-master1
10.1.6.24 containerd-master2
10.1.6.45 containerd-master3
10.1.6.215 api.k8s.verbos.com

```

## This kubeadm.conf

```yaml
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.22.10
controlPlaneEndpoint: "api.k8s.verbos.com:6443"
apiServer:
  certSANs:
  - 10.1.6.48
  - 10.1.6.24
  - 10.1.6.45
etcd:
  external:
    endpoints:
      - https://10.1.6.46:2379 # 适当地更改 ETCD_0_IP
      - https://10.1.6.43:2379 # 适当地更改 ETCD_1_IP
      - https://10.1.6.47:2379 # 适当地更改 ETCD_2_IP
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
    keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
imageRepository: registry.aliyuncs.com/google_containers
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.10.0.0/16

```

## This is etcd status

- The three Etcd nodes are in the Running state

```shell
[root@containerd-work1 pki]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
5bcd8330b1fad 0048118155842 5 hours ago Running etcd 0 6ff6ad495f06b etcd-containerd-work1

```

## Check VIP communication status

```shell
[root@containerd-master1 ~]# nc -v api.k8s.verbos.com 6443
Connection to api.k8s.verbos.com (10.1.6.215) 6443 port [tcp/sun-sr-https] succeeded!

```

## This is Haproxy config

```conf
global
    log /dev/log local0 warning
    chroot /var/lib/haproxy
    pidfile /var/run/haproxy.pid
    maxconn 4000
    user haproxy
    group haproxy
    daemon

   stats socket /var/lib/haproxy/stats

defaults
  log global
  option httplog
  option dontlognull
        timeout connect 5000
        timeout client 50000
        timeout server 50000

frontend kube-apiserver
  bind *:6443
  mode tcp
  option tcplog
  default_backend kube-apiserver

backend kube-apiserver
    mode tcp
    option tcplog
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server kube-apiserver-1 10.1.6.48:6443 check # Replace the IP address with your own.
    server kube-apiserver-2 10.1.6.24:6443 check # Replace the IP address with your own.
    server kube-apiserver-3 10.1.6.45:6443 check # Replace the IP address with your own.

```

## This is Keepalived config

```conf
global_defs {
  notification_email {
  }
  router_id LVS_DEVEL
  vrrp_skip_check_adv_addr
  vrrp_garp_interval 0
  vrrp_gna_interval 0
}

vrrp_script chk_haproxy {
  script "killall -0 haproxy"
  interval 2
  weight 2
}

vrrp_instance haproxy-vip {
  state BACKUP
  priority 100
  interface ens192 # Network card
  virtual_router_id 60
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  unicast_src_ip 10.1.6.213 # The IP address of this machine
  unicast_peer {
    10.1.6.214 # The IP address of peer machines
  }

  virtual_ipaddress {
    10.1.6.215/24 # The VIP address
  }

  track_script {
    chk_haproxy
  }
}

```

### What did you expect to happen?

## What causes this problem?

### How can we reproduce it (as minimally and precisely as possible)?

## I don’t know what the problem is

### Anything else we need to know?

_No response_

### Kubernetes version

## This is Kubectl Version

```shell
[root@containerd-master1 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.10", GitCommit:"eae22ba6238096f5dec1ceb62766e97783f0ba2f", GitTreeState:"clean", BuildDate:"2022-05-24T12:56:35Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

```

### Cloud provider

> ****
>

### OS version

## This is Os Version

```shell
[root@containerd-master1 ~]# cat /etc/redhat-release 
Rocky Linux release 8.6 (Green Obsidian)

```

### Install tools

> ****
>

### Container runtime (CRI) and version (if applicable)

## This is Runtime Version

```shell
[root@containerd-master1 ~]# crictl -v
crictl version v1.25.0
[root@containerd-master1 ~]# containerd -v
containerd containerd.io 1.6.14 9ba4b250366a5ddde94bb7c9d1def331423aa323

```

### Related plugins (CNI, CSI, …) and versions (if applicable)

> ****
>