What happened?
The many times I have tried to create a highly available cluster using kubeadm init --config kubeadm-config.yaml --upload-certs, I have encountered the following problems
This is the bug report I presented
- Looks like there’s no link to api.k8s.verbos.com
- I use Keepalived+Haproxy to load api-server
[root@containerd-master1 ~]# kubeadm init --config /root/kubeadm-config.yaml --v=5
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I1227 09:37:29.495368 39237 certs.go:111] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I1227 09:37:29.625377 39237 certs.go:519] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [api.k8s.verbos.com containerd-master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.10.0.1 10.1.6.48 10.1.6.24 10.1.6.45]
[certs] Generating "apiserver-kubelet-client" certificate and key
I1227 09:37:29.840751 39237 certs.go:111] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I1227 09:37:29.949095 39237 certs.go:519] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
I1227 09:37:30.204471 39237 certs.go:77] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I1227 09:37:30.434682 39237 kubeconfig.go:103] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I1227 09:37:30.622955 39237 kubeconfig.go:103] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I1227 09:37:30.764826 39237 kubeconfig.go:103] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I1227 09:37:31.043569 39237 kubeconfig.go:103] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I1227 09:37:31.231569 39237 kubelet.go:65] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I1227 09:37:31.322959 39237 manifests.go:99] [control-plane] getting StaticPodSpecs
I1227 09:37:31.323250 39237 certs.go:519] validating certificate period for CA certificate
I1227 09:37:31.323316 39237 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I1227 09:37:31.323323 39237 manifests.go:125] [control-plane] adding volume "etc-pki" for component "kube-apiserver"
I1227 09:37:31.323327 39237 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I1227 09:37:31.329137 39237 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I1227 09:37:31.329155 39237 manifests.go:99] [control-plane] getting StaticPodSpecs
I1227 09:37:31.329359 39237 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I1227 09:37:31.329371 39237 manifests.go:125] [control-plane] adding volume "etc-pki" for component "kube-controller-manager"
I1227 09:37:31.329377 39237 manifests.go:125] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I1227 09:37:31.329381 39237 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I1227 09:37:31.329386 39237 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I1227 09:37:31.329927 39237 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I1227 09:37:31.329947 39237 manifests.go:99] [control-plane] getting StaticPodSpecs
I1227 09:37:31.330141 39237 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I1227 09:37:31.330573 39237 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
I1227 09:37:31.330587 39237 etcd.go:103] [etcd] External etcd mode. Skipping the creation of a manifest for local etcd
I1227 09:37:31.330594 39237 waitcontrolplane.go:89] [wait-control-plane] Waiting for the API server to be healthy
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I1227 09:37:31.332356 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 1 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:32.333672 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 2 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:33.334389 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 3 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:34.335979 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 4 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:35.336694 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 5 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:36.337340 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 6 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:37.338720 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 7 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:38.339495 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 8 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:39.340194 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 9 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:40.340904 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 10 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:41.843152 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 1 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:42.843783 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 2 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:43.844609 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 3 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:44.845551 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 4 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:45.846356 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 5 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:46.846946 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 6 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:47.848392 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 7 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:48.849161 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 8 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:49.850777 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 9 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
I1227 09:37:50.851537 39237 with_retry.go:171] Got a Retry-After 1s response for attempt 10 to https://api.k8s.verbos.com:6443/healthz?timeout=10s
[apiclient] All control plane components are healthy after 21.018490 seconds
I1227 09:37:52.350344 39237 uploadconfig.go:110] [upload-config] Uploading the kubeadm ClusterConfiguration to a ConfigMap
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I1227 09:37:52.373290 39237 uploadconfig.go:124] [upload-config] Uploading the kubelet component config to a ConfigMap
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
I1227 09:37:52.386582 39237 uploadconfig.go:129] [upload-config] Preserving the CRISocket information for the control-plane node
I1227 09:37:52.386595 39237 patchnode.go:31] [patchnode] Uploading the CRI Socket information "/run/containerd/containerd.sock" to the Node API object "containerd-master1" as an annotation
[kubelet-check] Initial timeout of 40s passed.
Error writing Crisocket information for the control-plane node
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runUploadKubeletConfig
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init/uploadconfig.go:131
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:153
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:852
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:960
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:897
k8s.io/kubernetes/cmd/kubeadm/app.Run
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
error execution phase upload-config/kubelet
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:153
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:852
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:960
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:897
k8s.io/kubernetes/cmd/kubeadm/app.Run
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
This is my Hosts file
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.1.6.48 containerd-master1
10.1.6.24 containerd-master2
10.1.6.45 containerd-master3
10.1.6.215 api.k8s.verbos.com
This kubeadm.conf
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.22.10
controlPlaneEndpoint: "api.k8s.verbos.com:6443"
apiServer:
certSANs:
- 10.1.6.48
- 10.1.6.24
- 10.1.6.45
etcd:
external:
endpoints:
- https://10.1.6.46:2379 # 适当地更改 ETCD_0_IP
- https://10.1.6.43:2379 # 适当地更改 ETCD_1_IP
- https://10.1.6.47:2379 # 适当地更改 ETCD_2_IP
caFile: /etc/kubernetes/pki/etcd/ca.crt
certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
imageRepository: registry.aliyuncs.com/google_containers
networking:
podSubnet: 10.244.0.0/16
serviceSubnet: 10.10.0.0/16
This is etcd status
- The three Etcd nodes are in the Running state
[root@containerd-work1 pki]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
5bcd8330b1fad 0048118155842 5 hours ago Running etcd 0 6ff6ad495f06b etcd-containerd-work1
Check VIP communication status
[root@containerd-master1 ~]# nc -v api.k8s.verbos.com 6443
Connection to api.k8s.verbos.com (10.1.6.215) 6443 port [tcp/sun-sr-https] succeeded!
This is Haproxy config
global
log /dev/log local0 warning
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
log global
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
frontend kube-apiserver
bind *:6443
mode tcp
option tcplog
default_backend kube-apiserver
backend kube-apiserver
mode tcp
option tcplog
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server kube-apiserver-1 10.1.6.48:6443 check # Replace the IP address with your own.
server kube-apiserver-2 10.1.6.24:6443 check # Replace the IP address with your own.
server kube-apiserver-3 10.1.6.45:6443 check # Replace the IP address with your own.
This is Keepalived config
global_defs {
notification_email {
}
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 2
weight 2
}
vrrp_instance haproxy-vip {
state BACKUP
priority 100
interface ens192 # Network card
virtual_router_id 60
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
unicast_src_ip 10.1.6.213 # The IP address of this machine
unicast_peer {
10.1.6.214 # The IP address of peer machines
}
virtual_ipaddress {
10.1.6.215/24 # The VIP address
}
track_script {
chk_haproxy
}
}
What did you expect to happen?
What causes this problem?
How can we reproduce it (as minimally and precisely as possible)?
I don’t know what the problem is
Anything else we need to know?
No response
Kubernetes version
This is Kubectl Version
[root@containerd-master1 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.10", GitCommit:"eae22ba6238096f5dec1ceb62766e97783f0ba2f", GitTreeState:"clean", BuildDate:"2022-05-24T12:56:35Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Cloud provider
OS version
This is Os Version
[root@containerd-master1 ~]# cat /etc/redhat-release
Rocky Linux release 8.6 (Green Obsidian)
Install tools
Container runtime (CRI) and version (if applicable)
This is Runtime Version
[root@containerd-master1 ~]# crictl -v
crictl version v1.25.0
[root@containerd-master1 ~]# containerd -v
containerd containerd.io 1.6.14 9ba4b250366a5ddde94bb7c9d1def331423aa323