The Kube-apiserver service in the high availability Kubernetes cluster keeps restarting

A high availability cluster with three Kube-Apiservers was deployed using binary packages. Today, the Kube-Apiserver-SRV1 nodes suddenly kept restarting, leaving the Kube-Apiserver-SRv2 and Kube-Apiserver-SRv3 nodes running normally. The log of the Kubapiserver-SRv1 node that keeps restarting is as follows:

Log file created at: 2022/09/24 15:07:08
Running on machine: ayunw-core-kube-master-srv1
Binary: Built with gc go1.15 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg


I0924 15:07:08.928157 2543051 flag/flags.go:59] FLAG: --add-dir-header="true"
I0924 15:07:08.928305 2543051 flag/flags.go:59] FLAG: --address="127.0.0.1"
I0924 15:07:08.928311 2543051 flag/flags.go:59] FLAG: --admission-control="[]"
I0924 15:07:08.928316 2543051 flag/flags.go:59] FLAG: --admission-control-config-file=""
I0924 15:07:08.928320 2543051 flag/flags.go:59] FLAG: --advertise-address="<nil>"
I0924 15:07:08.928322 2543051 flag/flags.go:59] FLAG: --allow-privileged="true"
I0924 15:07:08.928328 2543051 flag/flags.go:59] FLAG: --alsologtostderr="false"
I0924 15:07:08.928331 2543051 flag/flags.go:59] FLAG: --anonymous-auth="false"
I0924 15:07:08.928333 2543051 flag/flags.go:59] FLAG: --api-audiences="[]"
I0924 15:07:08.928338 2543051 flag/flags.go:59] FLAG: --apiserver-count="3"
I0924 15:07:08.928341 2543051 flag/flags.go:59] FLAG: --audit-log-batch-buffer-size="10000"
I0924 15:07:08.928344 2543051 flag/flags.go:59] FLAG: --audit-log-batch-max-size="1"
I0924 15:07:08.928346 2543051 flag/flags.go:59] FLAG: --audit-log-batch-max-wait="0s"
I0924 15:07:08.928350 2543051 flag/flags.go:59] FLAG: --audit-log-batch-throttle-burst="0"
I0924 15:07:08.928352 2543051 flag/flags.go:59] FLAG: --audit-log-batch-throttle-enable="false"
I0924 15:07:08.928354 2543051 flag/flags.go:59] FLAG: --audit-log-batch-throttle-qps="0"
I0924 15:07:08.928359 2543051 flag/flags.go:59] FLAG: --audit-log-format="json"
I0924 15:07:08.928361 2543051 flag/flags.go:59] FLAG: --audit-log-maxage="30"
I0924 15:07:08.928364 2543051 flag/flags.go:59] FLAG: --audit-log-maxbackup="3"
I0924 15:07:08.928367 2543051 flag/flags.go:59] FLAG: --audit-log-maxsize="200"
I0924 15:07:08.928370 2543051 flag/flags.go:59] FLAG: --audit-log-mode="blocking"
I0924 15:07:08.928373 2543051 flag/flags.go:59] FLAG: --audit-log-path="/data/log/kubernetes/audit/kube-apiserver/kube-apiserver-audit.log"
I0924 15:07:08.928377 2543051 flag/flags.go:59] FLAG: --audit-log-truncate-enabled="false"
I0924 15:07:08.928380 2543051 flag/flags.go:59] FLAG: --audit-log-truncate-max-batch-size="10485760"
I0924 15:07:08.928386 2543051 flag/flags.go:59] FLAG: --audit-log-truncate-max-event-size="102400"
I0924 15:07:08.928389 2543051 flag/flags.go:59] FLAG: --audit-log-version="audit.k8s.io/v1"
I0924 15:07:08.928392 2543051 flag/flags.go:59] FLAG: --audit-policy-file=""
I0924 15:07:08.928394 2543051 flag/flags.go:59] FLAG: --audit-webhook-batch-buffer-size="10000"
I0924 15:07:08.928397 2543051 flag/flags.go:59] FLAG: --audit-webhook-batch-initial-backoff="10s"
I0924 15:07:08.928400 2543051 flag/flags.go:59] FLAG: --audit-webhook-batch-max-size="400"
I0924 15:07:08.928403 2543051 flag/flags.go:59] FLAG: --audit-webhook-batch-max-wait="30s"
I0924 15:07:08.928405 2543051 flag/flags.go:59] FLAG: --audit-webhook-batch-throttle-burst="15"
I0924 15:07:08.928407 2543051 flag/flags.go:59] FLAG: --audit-webhook-batch-throttle-enable="true"
I0924 15:07:08.928409 2543051 flag/flags.go:59] FLAG: --audit-webhook-batch-throttle-qps="10"
I0924 15:07:08.928412 2543051 flag/flags.go:59] FLAG: --audit-webhook-config-file=""
I0924 15:07:08.928414 2543051 flag/flags.go:59] FLAG: --audit-webhook-initial-backoff="10s"
I0924 15:07:08.928416 2543051 flag/flags.go:59] FLAG: --audit-webhook-mode="batch"
I0924 15:07:08.928417 2543051 flag/flags.go:59] FLAG: --audit-webhook-truncate-enabled="false"
I0924 15:07:08.928419 2543051 flag/flags.go:59] FLAG: --audit-webhook-truncate-max-batch-size="10485760"
I0924 15:07:08.928421 2543051 flag/flags.go:59] FLAG: --audit-webhook-truncate-max-event-size="102400"
I0924 15:07:08.928423 2543051 flag/flags.go:59] FLAG: --audit-webhook-version="audit.k8s.io/v1"
I0924 15:07:08.928425 2543051 flag/flags.go:59] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
I0924 15:07:08.928427 2543051 flag/flags.go:59] FLAG: --authentication-token-webhook-config-file=""
I0924 15:07:08.928428 2543051 flag/flags.go:59] FLAG: --authentication-token-webhook-version="v1beta1"
I0924 15:07:08.928430 2543051 flag/flags.go:59] FLAG: --authorization-mode="[RBAC,Node]"
I0924 15:07:08.928433 2543051 flag/flags.go:59] FLAG: --authorization-policy-file=""
I0924 15:07:08.928434 2543051 flag/flags.go:59] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
I0924 15:07:08.928436 2543051 flag/flags.go:59] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
I0924 15:07:08.928438 2543051 flag/flags.go:59] FLAG: --authorization-webhook-config-file=""
I0924 15:07:08.928440 2543051 flag/flags.go:59] FLAG: --authorization-webhook-version="v1beta1"
I0924 15:07:08.928442 2543051 flag/flags.go:59] FLAG: --bind-address="0.0.0.0"
I0924 15:07:08.928444 2543051 flag/flags.go:59] FLAG: --cert-dir="/var/run/kubernetes"
I0924 15:07:08.928446 2543051 flag/flags.go:59] FLAG: --client-ca-file="/data/certs/CA/AYUNW-SKY-Kubernetes-CA.pem"
I0924 15:07:08.928448 2543051 flag/flags.go:59] FLAG: --cloud-config=""
I0924 15:07:08.928450 2543051 flag/flags.go:59] FLAG: --cloud-provider=""
I0924 15:07:08.928451 2543051 flag/flags.go:59] FLAG: --cloud-provider-gce-l7lb-src-cidrs="130.211.0.0/22,35.191.0.0/16"
I0924 15:07:08.928455 2543051 flag/flags.go:59] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
I0924 15:07:08.928459 2543051 flag/flags.go:59] FLAG: --contention-profiling="false"
I0924 15:07:08.928460 2543051 flag/flags.go:59] FLAG: --cors-allowed-origins="[]"
I0924 15:07:08.928463 2543051 flag/flags.go:59] FLAG: --default-not-ready-toleration-seconds="300"
I0924 15:07:08.928466 2543051 flag/flags.go:59] FLAG: --default-unreachable-toleration-seconds="300"
I0924 15:07:08.928468 2543051 flag/flags.go:59] FLAG: --default-watch-cache-size="100"
I0924 15:07:08.928470 2543051 flag/flags.go:59] FLAG: --delete-collection-workers="1"
I0924 15:07:08.928472 2543051 flag/flags.go:59] FLAG: --deserialization-cache-size="0"
I0924 15:07:08.928473 2543051 flag/flags.go:59] FLAG: --disable-admission-plugins="[]"
I0924 15:07:08.928475 2543051 flag/flags.go:59] FLAG: --egress-selector-config-file=""
I0924 15:07:08.928477 2543051 flag/flags.go:59] FLAG: --enable-admission-plugins="[NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,DefaultIngressClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,PodNodeSelector,PodSecurityPolicy]"
I0924 15:07:08.928493 2543051 flag/flags.go:59] FLAG: --enable-aggregator-routing="true"
I0924 15:07:08.928496 2543051 flag/flags.go:59] FLAG: --enable-bootstrap-token-auth="true"
I0924 15:07:08.928497 2543051 flag/flags.go:59] FLAG: --enable-garbage-collector="true"
I0924 15:07:08.928499 2543051 flag/flags.go:59] FLAG: --enable-logs-handler="true"
I0924 15:07:08.928500 2543051 flag/flags.go:59] FLAG: --enable-priority-and-fairness="true"
I0924 15:07:08.928502 2543051 flag/flags.go:59] FLAG: --enable-swagger-ui="false"
I0924 15:07:08.928505 2543051 flag/flags.go:59] FLAG: --encryption-provider-config=""
I0924 15:07:08.928507 2543051 flag/flags.go:59] FLAG: --endpoint-reconciler-type="lease"
I0924 15:07:08.928509 2543051 flag/flags.go:59] FLAG: --etcd-cafile="/data/certs/CA/AYUNW-SKY-Kubernetes-CA.pem"
I0924 15:07:08.928511 2543051 flag/flags.go:59] FLAG: --etcd-certfile="/data/certs/etcd/etcd.pem"
I0924 15:07:08.928513 2543051 flag/flags.go:59] FLAG: --etcd-compaction-interval="5m0s"
I0924 15:07:08.928515 2543051 flag/flags.go:59] FLAG: --etcd-count-metric-poll-period="1m0s"
I0924 15:07:08.928517 2543051 flag/flags.go:59] FLAG: --etcd-db-metric-poll-interval="30s"
I0924 15:07:08.928519 2543051 flag/flags.go:59] FLAG: --etcd-keyfile="/data/certs/etcd/etcd-key.pem"
I0924 15:07:08.928521 2543051 flag/flags.go:59] FLAG: --etcd-prefix="/registry"
I0924 15:07:08.928523 2543051 flag/flags.go:59] FLAG: --etcd-servers="[https://etcd1.core.ayunw.cn:2379,https://etcd2.core.ayunw.cn:2379,https://etcd3.core.ayunw.cn:2379]"
I0924 15:07:08.928529 2543051 flag/flags.go:59] FLAG: --etcd-servers-overrides="[]"
I0924 15:07:08.928532 2543051 flag/flags.go:59] FLAG: --event-ttl="168h0m0s"
I0924 15:07:08.928534 2543051 flag/flags.go:59] FLAG: --experimental-encryption-provider-config=""
I0924 15:07:08.928536 2543051 flag/flags.go:59] FLAG: --external-hostname="kube-apiserver.core.ayunw.cn"
I0924 15:07:08.928538 2543051 flag/flags.go:59] FLAG: --feature-gates=""
I0924 15:07:08.928542 2543051 flag/flags.go:59] FLAG: --goaway-chance="0.001"
I0924 15:07:08.928546 2543051 flag/flags.go:59] FLAG: --help="false"
I0924 15:07:08.928548 2543051 flag/flags.go:59] FLAG: --http2-max-streams-per-connection="0"
I0924 15:07:08.928550 2543051 flag/flags.go:59] FLAG: --insecure-bind-address="127.0.0.1"
I0924 15:07:08.928552 2543051 flag/flags.go:59] FLAG: --insecure-port="8080"
I0924 15:07:08.928553 2543051 flag/flags.go:59] FLAG: --kubelet-certificate-authority="/data/certs/CA/AYUNW-SKY-Kubernetes-CA.pem"
I0924 15:07:08.928556 2543051 flag/flags.go:59] FLAG: --kubelet-client-certificate="/data/certs/kubernetes/kubelet-client/kubelet-client.pem"
I0924 15:07:08.928558 2543051 flag/flags.go:59] FLAG: --kubelet-client-key="/data/certs/kubernetes/kubelet-client/kubelet-client-key.pem"
I0924 15:07:08.928561 2543051 flag/flags.go:59] FLAG: --kubelet-https="true"
I0924 15:07:08.928563 2543051 flag/flags.go:59] FLAG: --kubelet-port="10250"
I0924 15:07:08.928566 2543051 flag/flags.go:59] FLAG: --kubelet-preferred-address-types="[InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname]"
I0924 15:07:08.928569 2543051 flag/flags.go:59] FLAG: --kubelet-read-only-port="10255"
I0924 15:07:08.928571 2543051 flag/flags.go:59] FLAG: --kubelet-timeout="5s"
I0924 15:07:08.928573 2543051 flag/flags.go:59] FLAG: --kubernetes-service-node-port="0"
I0924 15:07:08.928575 2543051 flag/flags.go:59] FLAG: --livez-grace-period="0s"
I0924 15:07:08.928577 2543051 flag/flags.go:59] FLAG: --log-backtrace-at=":0"
I0924 15:07:08.928581 2543051 flag/flags.go:59] FLAG: --log-dir="/data/log/kubernetes/kube-apiserver"
I0924 15:07:08.928584 2543051 flag/flags.go:59] FLAG: --log-file="/data/log/kubernetes/kube-apiserver/kube-apiserver.log"
I0924 15:07:08.928586 2543051 flag/flags.go:59] FLAG: --log-file-max-size="1800"
I0924 15:07:08.928588 2543051 flag/flags.go:59] FLAG: --log-flush-frequency="5s"
I0924 15:07:08.928590 2543051 flag/flags.go:59] FLAG: --logging-format="text"
I0924 15:07:08.928592 2543051 flag/flags.go:59] FLAG: --logtostderr="false"
I0924 15:07:08.928593 2543051 flag/flags.go:59] FLAG: --master-service-namespace="default"
I0924 15:07:08.928597 2543051 flag/flags.go:59] FLAG: --max-connection-bytes-per-sec="0"
I0924 15:07:08.928599 2543051 flag/flags.go:59] FLAG: --max-mutating-requests-inflight="2000"
I0924 15:07:08.928600 2543051 flag/flags.go:59] FLAG: --max-requests-inflight="4000"
I0924 15:07:08.928602 2543051 flag/flags.go:59] FLAG: --min-request-timeout="1800"
I0924 15:07:08.928604 2543051 flag/flags.go:59] FLAG: --oidc-ca-file=""
I0924 15:07:08.928606 2543051 flag/flags.go:59] FLAG: --oidc-client-id=""
I0924 15:07:08.928607 2543051 flag/flags.go:59] FLAG: --oidc-groups-claim=""
I0924 15:07:08.928609 2543051 flag/flags.go:59] FLAG: --oidc-groups-prefix=""
I0924 15:07:08.928610 2543051 flag/flags.go:59] FLAG: --oidc-issuer-url=""
I0924 15:07:08.928612 2543051 flag/flags.go:59] FLAG: --oidc-required-claim=""
I0924 15:07:08.928616 2543051 flag/flags.go:59] FLAG: --oidc-signing-algs="[RS256]"
I0924 15:07:08.928619 2543051 flag/flags.go:59] FLAG: --oidc-username-claim="sub"
I0924 15:07:08.928621 2543051 flag/flags.go:59] FLAG: --oidc-username-prefix=""
I0924 15:07:08.928623 2543051 flag/flags.go:59] FLAG: --permit-port-sharing="false"
I0924 15:07:08.928624 2543051 flag/flags.go:59] FLAG: --port="8080"
I0924 15:07:08.928626 2543051 flag/flags.go:59] FLAG: --profiling="true"
I0924 15:07:08.928628 2543051 flag/flags.go:59] FLAG: --proxy-client-cert-file="/data/certs/kubernetes/proxy-client/proxy-client.pem"
I0924 15:07:08.928630 2543051 flag/flags.go:59] FLAG: --proxy-client-key-file="/data/certs/kubernetes/proxy-client/proxy-client-key.pem"
I0924 15:07:08.928633 2543051 flag/flags.go:59] FLAG: --request-timeout="1m0s"
I0924 15:07:08.928634 2543051 flag/flags.go:59] FLAG: --requestheader-allowed-names="[proxy-client]"
I0924 15:07:08.928637 2543051 flag/flags.go:59] FLAG: --requestheader-client-ca-file="/data/certs/CA-EXT/AYUNW-SKY-Kubernetes-EXT-CA.pem"
I0924 15:07:08.928639 2543051 flag/flags.go:59] FLAG: --requestheader-extra-headers-prefix="[X-Remote-Extra-]"
I0924 15:07:08.928650 2543051 flag/flags.go:59] FLAG: --requestheader-group-headers="[X-Remote-Group]"
I0924 15:07:08.928653 2543051 flag/flags.go:59] FLAG: --requestheader-username-headers="[X-Remote-User]"
I0924 15:07:08.928656 2543051 flag/flags.go:59] FLAG: --runtime-config="api/all=true"
I0924 15:07:08.928661 2543051 flag/flags.go:59] FLAG: --secure-port="443"
I0924 15:07:08.928663 2543051 flag/flags.go:59] FLAG: --service-account-api-audiences="[]"
I0924 15:07:08.928664 2543051 flag/flags.go:59] FLAG: --service-account-extend-token-expiration="false"
I0924 15:07:08.928666 2543051 flag/flags.go:59] FLAG: --service-account-issuer=""
I0924 15:07:08.928668 2543051 flag/flags.go:59] FLAG: --service-account-jwks-uri=""
I0924 15:07:08.928670 2543051 flag/flags.go:59] FLAG: --service-account-key-file="[/data/certs/kubernetes/service-account/service-account.pem]"
I0924 15:07:08.928675 2543051 flag/flags.go:59] FLAG: --service-account-lookup="true"
I0924 15:07:08.928677 2543051 flag/flags.go:59] FLAG: --service-account-max-token-expiration="0s"
I0924 15:07:08.928679 2543051 flag/flags.go:59] FLAG: --service-account-signing-key-file=""
I0924 15:07:08.928680 2543051 flag/flags.go:59] FLAG: --service-cluster-ip-range="10.10.0.0/16"
I0924 15:07:08.928682 2543051 flag/flags.go:59] FLAG: --service-node-port-range="20000-60000"
I0924 15:07:08.928704 2543051 flag/flags.go:59] FLAG: --show-hidden-metrics-for-version=""
I0924 15:07:08.928706 2543051 flag/flags.go:59] FLAG: --shutdown-delay-duration="0s"
I0924 15:07:08.928708 2543051 flag/flags.go:59] FLAG: --skip-headers="false"
I0924 15:07:08.928711 2543051 flag/flags.go:59] FLAG: --skip-log-headers="false"
I0924 15:07:08.928713 2543051 flag/flags.go:59] FLAG: --ssh-keyfile=""
I0924 15:07:08.928714 2543051 flag/flags.go:59] FLAG: --ssh-user=""
I0924 15:07:08.928716 2543051 flag/flags.go:59] FLAG: --stderrthreshold="2"
I0924 15:07:08.928718 2543051 flag/flags.go:59] FLAG: --storage-backend=""
I0924 15:07:08.928719 2543051 flag/flags.go:59] FLAG: --storage-media-type="application/vnd.kubernetes.protobuf"
I0924 15:07:08.928722 2543051 flag/flags.go:59] FLAG: --target-ram-mb="0"
I0924 15:07:08.928723 2543051 flag/flags.go:59] FLAG: --tls-cert-file="/data/certs/kubernetes/kube-apiserver/kube-apiserver.pem"
I0924 15:07:08.928726 2543051 flag/flags.go:59] FLAG: --tls-cipher-suites="[]"
I0924 15:07:08.928728 2543051 flag/flags.go:59] FLAG: --tls-min-version="VersionTLS12"
I0924 15:07:08.928730 2543051 flag/flags.go:59] FLAG: --tls-private-key-file="/data/certs/kubernetes/kube-apiserver/kube-apiserver-key.pem"
I0924 15:07:08.928732 2543051 flag/flags.go:59] FLAG: --tls-sni-cert-key="[]"
I0924 15:07:08.928735 2543051 flag/flags.go:59] FLAG: --token-auth-file=""
I0924 15:07:08.928737 2543051 flag/flags.go:59] FLAG: --v="8"
I0924 15:07:08.928739 2543051 flag/flags.go:59] FLAG: --version="false"
I0924 15:07:08.928742 2543051 flag/flags.go:59] FLAG: --vmodule=""
I0924 15:07:08.928744 2543051 flag/flags.go:59] FLAG: --watch-cache="true"
I0924 15:07:08.928746 2543051 flag/flags.go:59] FLAG: --watch-cache-sizes="[]"
I0924 15:07:08.929138 2543051 net/interface.go:400] Looking for default routes with IPv4 addresses
I0924 15:07:08.929142 2543051 net/interface.go:405] Default route transits interface "eth0"
I0924 15:07:08.929312 2543051 net/interface.go:208] Interface eth0 is up
I0924 15:07:08.929962 2543051 net/interface.go:256] Interface "eth0" has 2 addresses :[172.16.40.5/24 fe80::f828:ff:fe03:3cc8/64].
I0924 15:07:08.929985 2543051 net/interface.go:223] Checking addr  172.16.40.5/24.
I0924 15:07:08.929990 2543051 net/interface.go:230] IP found 172.16.40.5
I0924 15:07:08.929995 2543051 net/interface.go:262] Found valid IPv4 address 172.16.40.5 for interface "eth0".
I0924 15:07:08.929998 2543051 net/interface.go:411] Found active IP 172.16.40.5
I0924 15:07:08.930014 2543051 master/services.go:51] Setting service IP to "10.10.0.1" (read-write).
I0924 15:07:08.930312 2543051 app/server.go:163] Version: v1.19.0
I0924 15:07:08.936488 2543051 dynamiccertificates/dynamic_serving_content.go:111] Loaded a new cert/key pair for "serving-cert::/data/certs/kubernetes/kube-apiserver/kube-apiserver.pem::/data/certs/kubernetes/kube-apiserver/kube-apiserver-key.pem"
I0924 15:07:09.393148 2543051 dynamiccertificates/dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/data/certs/CA/AYUNW-SKY-Kubernetes-CA.pem"
I0924 15:07:09.393264 2543051 dynamiccertificates/dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "request-header::/data/certs/CA-EXT/AYUNW-SKY-Kubernetes-EXT-CA.pem"
I0924 15:07:09.393660 2543051 options/audit.go:296] No audit policy file provided, no events will be recorded for log backend
I0924 15:07:09.394734 2543051 admission/plugins.go:158] Loaded 13 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,PodSecurityPolicy,PodNodeSelector,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0924 15:07:09.394740 2543051 admission/plugins.go:161] Loaded 12 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurityPolicy,PodNodeSelector,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0924 15:07:09.394799 2543051 master/services.go:51] Setting service IP to "10.10.0.1" (read-write).
I0924 15:07:09.394936 2543051 dynamiccertificates/dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/data/certs/CA/AYUNW-SKY-Kubernetes-CA.pem"
I0924 15:07:09.395031 2543051 dynamiccertificates/dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "request-header::/data/certs/CA-EXT/AYUNW-SKY-Kubernetes-EXT-CA.pem"
I0924 15:07:09.395728 2543051 admission/plugins.go:158] Loaded 13 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,PodSecurityPolicy,PodNodeSelector,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0924 15:07:09.395733 2543051 admission/plugins.go:161] Loaded 12 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurityPolicy,PodNodeSelector,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0924 15:07:09.395955 2543051 server/config.go:638] Not requested to run hook priority-and-fairness-config-consumer
I0924 15:07:09.402993 2543051 clientv3/client.go:360] parsed scheme: "endpoint"
I0924 15:07:09.403025 2543051 endpoint/endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://etcd1.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd2.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd3.core.ayunw.cn:2379  <nil> 0 <nil>}]
I0924 15:07:09.497680 2543051 clientv3/client.go:360] parsed scheme: "endpoint"
I0924 15:07:09.497722 2543051 endpoint/endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://etcd1.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd2.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd3.core.ayunw.cn:2379  <nil> 0 <nil>}]
I0924 15:07:09.566146 2543051 factory/etcd3.go:271] Start monitoring storage db size metric for endpoint https://etcd1.core.ayunw.cn:2379 with polling interval 30s
I0924 15:07:09.566181 2543051 factory/etcd3.go:271] Start monitoring storage db size metric for endpoint https://etcd2.core.ayunw.cn:2379 with polling interval 30s
I0924 15:07:09.566186 2543051 factory/etcd3.go:271] Start monitoring storage db size metric for endpoint https://etcd3.core.ayunw.cn:2379 with polling interval 30s
I0924 15:07:09.566199 2543051 registry/storage_factory.go:53] Storage caching is enabled for *apiextensions.CustomResourceDefinition
I0924 15:07:09.566358 2543051 clientv3/client.go:360] parsed scheme: "passthrough"
I0924 15:07:09.566382 2543051 clientv3/client.go:360] parsed scheme: "passthrough"
I0924 15:07:09.566383 2543051 clientv3/client.go:360] parsed scheme: "passthrough"
I0924 15:07:09.566409 2543051 passthrough/passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://etcd3.core.ayunw.cn:2379  <nil> 0 <nil>}] <nil> <nil>}
I0924 15:07:09.566418 2543051 grpc/clientconn.go:948] ClientConn switching balancer to "pick_first"
I0924 15:07:09.566424 2543051 passthrough/passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://etcd2.core.ayunw.cn:2379  <nil> 0 <nil>}] <nil> <nil>}
I0924 15:07:09.566431 2543051 passthrough/passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://etcd1.core.ayunw.cn:2379  <nil> 0 <nil>}] <nil> <nil>}
I0924 15:07:09.566432 2543051 grpc/clientconn.go:948] ClientConn switching balancer to "pick_first"
I0924 15:07:09.566437 2543051 grpc/clientconn.go:948] ClientConn switching balancer to "pick_first"
I0924 15:07:09.566513 2543051 grpc/balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000d28a50, {CONNECTING <nil>}
I0924 15:07:09.566578 2543051 grpc/balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000f7a230, {CONNECTING <nil>}
I0924 15:07:09.566615 2543051 grpc/balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc001212170, {CONNECTING <nil>}
I0924 15:07:09.567044 2543051 registry/store.go:1378] Monitoring customresourcedefinitions.apiextensions.k8s.io count at <storage-prefix>//apiextensions.k8s.io/customresourcedefinitions
I0924 15:07:09.567148 2543051 cache/reflector.go:243] Listing and watching *apiextensions.CustomResourceDefinition from storage/cacher.go:/apiextensions.k8s.io/customresourcedefinitions
I0924 15:07:09.575621 2543051 clientv3/client.go:360] parsed scheme: "endpoint"
I0924 15:07:09.575649 2543051 endpoint/endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://etcd1.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd2.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd3.core.ayunw.cn:2379  <nil> 0 <nil>}]
I0924 15:07:09.575768 2543051 cacher/cacher.go:402] cacher (*apiextensions.CustomResourceDefinition): initialized
I0924 15:07:09.575774 2543051 cacher/watch_cache.go:521] Replace watchCache (rev: 1722071173)
I0924 15:07:09.642027 2543051 grpc/balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000d28a50, {READY <nil>}
I0924 15:07:09.643413 2543051 transport/controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
I0924 15:07:09.656421 2543051 registry/storage_factory.go:53] Storage caching is enabled for *apiextensions.CustomResourceDefinition
I0924 15:07:09.656521 2543051 registry/store.go:1378] Monitoring customresourcedefinitions.apiextensions.k8s.io count at <storage-prefix>//apiextensions.k8s.io/customresourcedefinitions
I0924 15:07:09.656614 2543051 cache/reflector.go:243] Listing and watching *apiextensions.CustomResourceDefinition from storage/cacher.go:/apiextensions.k8s.io/customresourcedefinitions
I0924 15:07:09.660983 2543051 cacher/cacher.go:402] cacher (*apiextensions.CustomResourceDefinition): initialized
I0924 15:07:09.660990 2543051 cacher/watch_cache.go:521] Replace watchCache (rev: 1722071173)
I0924 15:07:09.680006 2543051 master/services.go:51] Setting service IP to "10.10.0.1" (read-write).
I0924 15:07:09.680023 2543051 master/master.go:271] Using reconciler: lease
I0924 15:07:09.680096 2543051 storage/storage_factory.go:285] storing apiServerIPInfo in v1, reading as __internal from storagebackend.Config{Type:"", Prefix:"/registry", Transport:storagebackend.TransportConfig{ServerList:[]string{"https://etcd1.core.ayunw.cn:2379", "https://etcd2.core.ayunw.cn:2379", "https://etcd3.core.ayunw.cn:2379"}, KeyFile:"/data/certs/etcd/etcd-key.pem", CertFile:"/data/certs/etcd/etcd.pem", TrustedCAFile:"/data/certs/CA/AYUNW-SKY-Kubernetes-CA.pem", EgressLookup:(egressselector.Lookup)(nil)}, Paging:true, Codec:runtime.Codec(nil), EncodeVersioner:runtime.GroupVersioner(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000, DBMetricPollInterval:30000000000}
I0924 15:07:09.685823 2543051 clientv3/client.go:360] parsed scheme: "endpoint"
I0924 15:07:09.685846 2543051 endpoint/endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://etcd1.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd2.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd3.core.ayunw.cn:2379  <nil> 0 <nil>}]
I0924 15:07:09.757727 2543051 server/config.go:638] Not requested to run hook priority-and-fairness-config-consumer
I0924 15:07:09.758957 2543051 storage/storage_factory.go:285] storing podtemplates in v1, reading as __internal from storagebackend.Config{Type:"", Prefix:"/registry", Transport:storagebackend.TransportConfig{ServerList:[]string{"https://etcd1.core.ayunw.cn:2379", "https://etcd2.core.ayunw.cn:2379", "https://etcd3.core.ayunw.cn:2379"}, KeyFile:"/data/certs/etcd/etcd-key.pem", CertFile:"/data/certs/etcd/etcd.pem", TrustedCAFile:"/data/certs/CA/AYUNW-SKY-Kubernetes-CA.pem", EgressLookup:(egressselector.Lookup)(nil)}, Paging:true, Codec:runtime.Codec(nil), EncodeVersioner:runtime.GroupVersioner(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000, DBMetricPollInterval:30000000000}
I0924 15:07:09.764624 2543051 clientv3/client.go:360] parsed scheme: "endpoint"
I0924 15:07:09.764657 2543051 endpoint/endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://etcd1.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd2.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd3.core.ayunw.cn:2379  <nil> 0 <nil>}]
I0924 15:07:09.835832 2543051 registry/storage_factory.go:53] Storage caching is enabled for *core.PodTemplate
I0924 15:07:09.835993 2543051 registry/store.go:1378] Monitoring podtemplates count at <storage-prefix>//podtemplates
I0924 15:07:09.836062 2543051 cache/reflector.go:243] Listing and watching *core.PodTemplate from storage/cacher.go:/podtemplates
I0924 15:07:09.836061 2543051 storage/storage_factory.go:285] storing events in v1, reading as __internal from storagebackend.Config{Type:"", Prefix:"/registry", Transport:storagebackend.TransportConfig{ServerList:[]string{"https://etcd1.core.ayunw.cn:2379", "https://etcd2.core.ayunw.cn:2379", "https://etcd3.core.ayunw.cn:2379"}, KeyFile:"/data/certs/etcd/etcd-key.pem", CertFile:"/data/certs/etcd/etcd.pem", TrustedCAFile:"/data/certs/CA/AYUNW-SKY-Kubernetes-CA.pem", EgressLookup:(egressselector.Lookup)(nil)}, Paging:true, Codec:runtime.Codec(nil), EncodeVersioner:runtime.GroupVersioner(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000, DBMetricPollInterval:30000000000}
I0924 15:07:09.839178 2543051 cacher/cacher.go:402] cacher (*core.PodTemplate): initialized
I0924 15:07:09.839188 2543051 cacher/watch_cache.go:521] Replace watchCache (rev: 1722071174)
I0924 15:07:09.845150 2543051 clientv3/client.go:360] parsed scheme: "endpoint"
I0924 15:07:09.845179 2543051 endpoint/endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://etcd1.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd2.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd3.core.ayunw.cn:2379  <nil> 0 <nil>}]
I0924 15:07:10.400208 2543051 clientv3/client.go:360] parsed scheme: "endpoint"
I0924 15:07:10.400237 2543051 endpoint/endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://etcd1.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd2.core.ayunw.cn:2379  <nil> 0 <nil>} {https://etcd3.core.ayunw.cn:2379  <nil> 0 <nil>}]

Any one know what’s wrong with it,and could please help me to resolve this problem? Thanks.