kube-apiserver pod generated many Error proxying data from client to backend … errors.
Cluster information:
Kubernetes version: v1.23.13
Cloud being used: (put bare-metal if not on a public cloud)
Installation method: kubeadm (rpm packages)
Cluster: Single Node
Host OS: CentOS Linux release 7.9.2009
Host Kernel: 5.4.213-1.el7.elrepo.x86_64
Host IP: 10.130.200.205
CNI and version: flannel v0.19.2
CRI and version: docker://20.10.19
apiserver log
I1017 01:05:33.608669 1 server.go:565] external host was not specified, using 10.130.200.205
I1017 01:05:33.609261 1 server.go:172] Version: v1.23.13
I1017 01:05:33.982518 1 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I1017 01:05:33.984270 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I1017 01:05:33.984296 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I1017 01:05:33.986001 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I1017 01:05:33.986027 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W1017 01:05:34.057617 1 genericapiserver.go:538] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
I1017 01:05:34.058591 1 instance.go:274] Using reconciler: lease
W1017 01:05:35.169545 1 genericapiserver.go:538] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
W1017 01:05:35.171601 1 genericapiserver.go:538] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
W1017 01:05:35.187945 1 genericapiserver.go:538] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
W1017 01:05:35.189944 1 genericapiserver.go:538] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
W1017 01:05:35.196564 1 genericapiserver.go:538] Skipping API networking.k8s.io/v1beta1 because it has no resources.
W1017 01:05:35.199896 1 genericapiserver.go:538] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W1017 01:05:35.207910 1 genericapiserver.go:538] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
W1017 01:05:35.207927 1 genericapiserver.go:538] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W1017 01:05:35.210005 1 genericapiserver.go:538] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
W1017 01:05:35.210020 1 genericapiserver.go:538] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W1017 01:05:35.226177 1 genericapiserver.go:538] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W1017 01:05:35.233750 1 genericapiserver.go:538] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W1017 01:05:35.240259 1 genericapiserver.go:538] Skipping API apps/v1beta2 because it has no resources.
W1017 01:05:35.240273 1 genericapiserver.go:538] Skipping API apps/v1beta1 because it has no resources.
W1017 01:05:35.243104 1 genericapiserver.go:538] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
I1017 01:05:35.248204 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I1017 01:05:35.248220 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W1017 01:05:35.291353 1 genericapiserver.go:538] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
I1017 01:05:36.471826 1 dynamic_cafile_content.go:156] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I1017 01:05:36.471901 1 dynamic_cafile_content.go:156] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I1017 01:05:36.472054 1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key"
I1017 01:05:36.472307 1 secure_serving.go:266] Serving securely on [::]:6443
I1017 01:05:36.472363 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I1017 01:05:36.472578 1 dynamic_serving_content.go:131] "Starting controller" name="aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key"
I1017 01:05:36.472831 1 controller.go:83] Starting OpenAPI AggregationController
I1017 01:05:36.472885 1 apiservice_controller.go:97] Starting APIServiceRegistrationController
I1017 01:05:36.472905 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I1017 01:05:36.472942 1 customresource_discovery_controller.go:209] Starting DiscoveryController
I1017 01:05:36.473054 1 autoregister_controller.go:141] Starting autoregister controller
I1017 01:05:36.473076 1 cache.go:32] Waiting for caches to sync for autoregister controller
I1017 01:05:36.473110 1 available_controller.go:491] Starting AvailableConditionController
I1017 01:05:36.473145 1 controller.go:85] Starting OpenAPI controller
I1017 01:05:36.473153 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I1017 01:05:36.473220 1 naming_controller.go:291] Starting NamingConditionController
I1017 01:05:36.473262 1 establishing_controller.go:76] Starting EstablishingController
I1017 01:05:36.473322 1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I1017 01:05:36.473371 1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I1017 01:05:36.473406 1 crd_finalizer.go:266] Starting CRDFinalizer
I1017 01:05:36.473440 1 crdregistration_controller.go:111] Starting crd-autoregister controller
I1017 01:05:36.473454 1 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
I1017 01:05:36.473624 1 apf_controller.go:317] Starting API Priority and Fairness config controller
I1017 01:05:36.474174 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I1017 01:05:36.474194 1 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
I1017 01:05:36.474250 1 dynamic_cafile_content.go:156] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I1017 01:05:36.482705 1 dynamic_cafile_content.go:156] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I1017 01:05:36.498657 1 controller.go:611] quota admission added evaluator for: namespaces
I1017 01:05:36.549183 1 controller.go:611] quota admission added evaluator for: leases.coordination.k8s.io
I1017 01:05:36.573223 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I1017 01:05:36.573319 1 cache.go:39] Caches are synced for AvailableConditionController controller
I1017 01:05:36.573427 1 cache.go:39] Caches are synced for autoregister controller
I1017 01:05:36.573504 1 shared_informer.go:247] Caches are synced for crd-autoregister
I1017 01:05:36.573727 1 apf_controller.go:322] Running API Priority and Fairness config worker
I1017 01:05:36.574315 1 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller
I1017 01:05:36.582640 1 shared_informer.go:247] Caches are synced for node_authorizer
I1017 01:05:37.471791 1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I1017 01:05:37.482004 1 storage_scheduling.go:93] created PriorityClass system-node-critical with value 2000001000
I1017 01:05:37.486843 1 storage_scheduling.go:93] created PriorityClass system-cluster-critical with value 2000000000
I1017 01:05:37.486878 1 storage_scheduling.go:109] all system priority classes are created successfully or already exist.
I1017 01:05:38.029329 1 controller.go:611] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I1017 01:05:38.070225 1 controller.go:611] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
I1017 01:05:38.217412 1 alloc.go:329] "allocated clusterIPs" service="default/kubernetes" clusterIPs=map[IPv4:10.96.0.1]
W1017 01:05:38.225424 1 lease.go:233] Resetting endpoints for master service "kubernetes" to [10.130.200.205]
I1017 01:05:38.226880 1 controller.go:611] quota admission added evaluator for: endpoints
I1017 01:05:38.233187 1 controller.go:611] quota admission added evaluator for: endpointslices.discovery.k8s.io
I1017 01:05:39.812878 1 controller.go:611] quota admission added evaluator for: serviceaccounts
I1017 01:05:39.822320 1 controller.go:611] quota admission added evaluator for: deployments.apps
I1017 01:05:39.832609 1 alloc.go:329] "allocated clusterIPs" service="kube-system/kube-dns" clusterIPs=map[IPv4:10.96.0.10]
I1017 01:05:39.849736 1 controller.go:611] quota admission added evaluator for: daemonsets.apps
I1017 01:05:54.216643 1 controller.go:611] quota admission added evaluator for: controllerrevisions.apps
I1017 01:05:54.516866 1 controller.go:611] quota admission added evaluator for: replicasets.apps
I1017 01:05:56.098875 1 controller.go:611] quota admission added evaluator for: events.events.k8s.io
I1017 01:07:45.734538 1 alloc.go:329] "allocated clusterIPs" service="myapp/elasticsearch" clusterIPs=map[IPv4:10.103.12.186]
I1017 01:07:45.746103 1 alloc.go:329] "allocated clusterIPs" service="myapp/ha" clusterIPs=map[IPv4:10.110.249.78]
I1017 01:07:45.760346 1 alloc.go:329] "allocated clusterIPs" service="myapp/kibana" clusterIPs=map[IPv4:10.100.222.74]
I1017 01:07:45.779413 1 alloc.go:329] "allocated clusterIPs" service="myapp/myapp-master" clusterIPs=map[IPv4:10.110.54.97]
I1017 01:07:45.793457 1 alloc.go:329] "allocated clusterIPs" service="myapp/myapp-worker" clusterIPs=map[IPv4:10.106.61.187]
I1017 01:07:45.804772 1 alloc.go:329] "allocated clusterIPs" service="myapp/oval" clusterIPs=map[IPv4:10.108.194.212]
I1017 01:07:45.847196 1 controller.go:611] quota admission added evaluator for: statefulsets.apps
I1017 01:11:10.865629 1 alloc.go:329] "allocated clusterIPs" service="myapp/kafka" clusterIPs=map[IPv4:10.101.230.79]
I1017 01:11:11.190279 1 alloc.go:329] "allocated clusterIPs" service="myapp/mysql" clusterIPs=map[IPv4:10.100.50.72]
W1017 01:51:58.811545 1 watcher.go:229] watch chan error: etcdserver: mvcc: required revision has been compacted
E1017 01:52:22.065573 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:38278->10.130.200.205:10250: write: connection reset by peer
E1017 01:53:39.794894 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:60472->10.130.200.205:10250: write: broken pipe
E1017 01:53:51.422880 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:44174->10.130.200.205:10250: write: broken pipe
E1017 01:57:49.678945 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:54396->10.130.200.205:10250: write: broken pipe
E1017 01:58:10.975848 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:51654->10.130.200.205:10250: write: broken pipe
E1017 01:58:28.105041 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:48272->10.130.200.205:10250: write: broken pipe
E1017 01:58:41.975565 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:53162->10.130.200.205:10250: write: broken pipe
E1017 02:00:22.495011 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:52692->10.130.200.205:10250: write: broken pipe
E1017 02:00:37.948687 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:35772->10.130.200.205:10250: write: broken pipe
E1017 02:00:51.987581 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:40034->10.130.200.205:10250: write: broken pipe
E1017 02:01:45.992842 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:44068->10.130.200.205:10250: write: broken pipe
E1017 02:02:20.034440 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:46380->10.130.200.205:10250: write: broken pipe
E1017 02:04:07.801489 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:49626->10.130.200.205:10250: write: broken pipe
E1017 02:04:54.219208 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:60664->10.130.200.205:10250: write: connection reset by peer
E1017 02:06:37.813674 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:38396->10.130.200.205:10250: write: broken pipe
E1017 02:06:42.081533 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:38430->10.130.200.205:10250: write: connection reset by peer
E1017 02:06:58.041909 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:60592->10.130.200.205:10250: write: broken pipe
E1017 02:09:14.355440 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:60314->10.130.200.205:10250: write: broken pipe
E1017 02:09:22.656321 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:60430->10.130.200.205:10250: write: broken pipe
E1017 02:12:33.144473 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:33114->10.130.200.205:10250: write: broken pipe
E1017 02:13:19.464825 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:53560->10.130.200.205:10250: write: broken pipe
E1017 02:13:19.678108 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:53570->10.130.200.205:10250: write: broken pipe
E1017 02:13:37.155001 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:39412->10.130.200.205:10250: write: broken pipe
E1017 02:16:40.391769 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:45078->10.130.200.205:10250: write: broken pipe
E1017 02:17:04.931651 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:35614->10.130.200.205:10250: write: broken pipe
E1017 02:17:13.795787 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:35736->10.130.200.205:10250: write: connection reset by peer
E1017 02:17:56.079445 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:56366->10.130.200.205:10250: write: broken pipe
E1017 02:18:30.867526 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:40480->10.130.200.205:10250: write: broken pipe
E1017 02:21:11.854568 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:54568->10.130.200.205:10250: write: broken pipe
E1017 02:24:36.783575 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:59900->10.130.200.205:10250: write: broken pipe
E1017 02:24:41.014751 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:59954->10.130.200.205:10250: write: broken pipe
I1017 02:25:56.539034 1 trace.go:205] Trace[491245470]: "Get" url:/api/v1/namespaces/myapp/pods/myapp-master-0/log,user-agent:kubectl/v1.23.13 (linux/amd64) kubernetes/592eca0,audit-id:d5472521-ef7e-4b26-be61-2b7dd9f5094e,client:10.130.200.205,accept:application/json, */*,protocol:HTTP/2.0 (17-Oct-2022 02:25:15.120) (total time: 41418ms):
Trace[491245470]: ---"Transformed response object" 41416ms (02:25:56.538)
Trace[491245470]: [41.418658493s] [41.418658493s] END
E1017 02:26:41.241254 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:34284->10.130.200.205:10250: write: broken pipe
E1017 02:27:23.029938 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:58552->10.130.200.205:10250: write: broken pipe
E1017 02:27:48.936201 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:50668->10.130.200.205:10250: write: broken pipe
E1017 02:28:05.965957 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:33182->10.130.200.205:10250: write: broken pipe
E1017 02:28:23.453316 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:42956->10.130.200.205:10250: write: broken pipe
E1017 02:29:44.278637 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:36442->10.130.200.205:10250: write: broken pipe
E1017 02:29:48.566533 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:36474->10.130.200.205:10250: write: broken pipe
E1017 02:45:51.054759 1 watch.go:248] unable to encode watch object *v1.WatchEvent: http2: stream closed (&streaming.encoder{writer:responsewriter.outerWithCloseNotifyAndFlush{UserProvidedDecorator:(*metrics.ResponseWriterDelegator)(0xc0061416e0), InnerCloseNotifierFlusher:(*http2.responseWriter)(0xc001dde600)}, encoder:(*versioning.codec)(0xc009719d60), buf:(*bytes.Buffer)(0xc006141920)})
E1017 02:55:50.348489 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:47486->10.130.200.205:10250: write: broken pipe
E1017 02:55:54.407939 1 upgradeaware.go:436] Error proxying data from client to backend: read tcp 10.130.200.205:6443->10.130.200.205:48240: read: connection reset by peer
W1017 02:57:40.504773 1 watcher.go:229] watch chan error: etcdserver: mvcc: required revision has been compacted
E1017 03:14:27.905015 1 upgradeaware.go:450] Error proxying data from backend to client: write tcp 10.130.200.205:6443->10.130.200.205:33760: write: connection reset by peer
E1017 03:52:21.255603 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:59980->10.130.200.205:10250: write: broken pipe
E1017 03:52:29.683343 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:35080->10.130.200.205:10250: write: broken pipe
E1017 03:52:51.563900 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:57052->10.130.200.205:10250: write: connection reset by peer
E1017 03:53:38.914392 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:41912->10.130.200.205:10250: write: broken pipe
E1017 03:53:42.985257 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:41962->10.130.200.205:10250: write: broken pipe
E1017 03:53:55.424352 1 upgradeaware.go:450] Error proxying data from backend to client: write tcp 10.130.200.205:6443->10.130.200.205:50674: write: connection reset by peer
E1017 03:59:13.572641 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:41992->10.130.200.205:10250: write: broken pipe
E1017 03:59:14.007922 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:51390->10.130.200.205:10250: write: connection reset by peer
W1017 03:59:24.264581 1 watcher.go:229] watch chan error: etcdserver: mvcc: required revision has been compacted
E1017 03:59:39.846154 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:39698->10.130.200.205:10250: write: broken pipe
E1017 04:00:57.470342 1 upgradeaware.go:436] Error proxying data from client to backend: read tcp 10.130.200.205:6443->10.130.200.205:35630: read: connection reset by peer
E1017 04:01:51.384204 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:41242->10.130.200.205:10250: write: connection reset by peer
E1017 04:02:35.458179 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:47452->10.130.200.205:10250: write: broken pipe
E1017 04:03:06.121292 1 upgradeaware.go:436] Error proxying data from client to backend: read tcp 10.130.200.205:6443->10.130.200.205:38974: read: connection reset by peer
E1017 04:03:28.643006 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:44174->10.130.200.205:10250: write: broken pipe
E1017 04:03:45.862882 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:48746->10.130.200.205:10250: write: broken pipe
E1017 04:05:51.960653 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:57822->10.130.200.205:10250: write: broken pipe
E1017 04:09:02.241893 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:32958->10.130.200.205:10250: write: broken pipe
E1017 04:10:49.954186 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:40704->10.130.200.205:10250: write: broken pipe
W1017 04:13:15.271189 1 watcher.go:229] watch chan error: etcdserver: mvcc: required revision has been compacted
E1017 04:14:48.221833 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:40910->10.130.200.205:10250: write: broken pipe
E1017 04:15:52.166991 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:46080->10.130.200.205:10250: write: broken pipe
E1017 04:18:18.268785 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:58908->10.130.200.205:10250: write: broken pipe
E1017 04:19:57.393108 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:36626->10.130.200.205:10250: write: broken pipe
E1017 04:20:10.070902 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:45430->10.130.200.205:10250: write: broken pipe
E1017 04:21:01.329448 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:58506->10.130.200.205:10250: write: connection reset by peer
W1017 04:21:57.737892 1 watcher.go:229] watch chan error: etcdserver: mvcc: required revision has been compacted
E1017 04:22:27.321393 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:49074->10.130.200.205:10250: write: connection reset by peer
E1017 04:22:44.098129 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:41918->10.130.200.205:10250: write: connection reset by peer
E1017 04:24:23.365503 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:44608->10.130.200.205:10250: write: connection reset by peer
E1017 04:25:05.034497 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:33616->10.130.200.205:10250: write: broken pipe
E1017 04:25:40.062803 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:52666->10.130.200.205:10250: write: broken pipe
E1017 04:27:01.349967 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:55254->10.130.200.205:10250: write: broken pipe
E1017 04:27:17.616462 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:49348->10.130.200.205:10250: write: broken pipe
E1017 04:29:04.950264 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:40408->10.130.200.205:10250: write: broken pipe
E1017 04:29:43.151323 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:53860->10.130.200.205:10250: write: broken pipe
E1017 04:29:51.852935 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:58756->10.130.200.205:10250: write: broken pipe
E1017 04:30:04.240785 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:52076->10.130.200.205:10250: write: broken pipe
E1017 04:30:38.457206 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:46120->10.130.200.205:10250: write: broken pipe
E1017 04:32:16.454709 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:53444->10.130.200.205:10250: write: broken pipe
E1017 04:32:51.082389 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:48048->10.130.200.205:10250: write: broken pipe
E1017 04:33:12.906467 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:49022->10.130.200.205:10250: write: broken pipe
E1017 04:33:42.720727 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:50984->10.130.200.205:10250: write: broken pipe
W1017 04:33:50.849857 1 watcher.go:229] watch chan error: etcdserver: mvcc: required revision has been compacted
E1017 04:34:25.248175 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:57530->10.130.200.205:10250: write: broken pipe
E1017 04:34:50.534425 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:38346->10.130.200.205:10250: write: broken pipe
E1017 04:36:16.059895 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:45062->10.130.200.205:10250: write: broken pipe
E1017 04:37:04.330783 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:59104->10.130.200.205:10250: write: broken pipe
E1017 04:38:04.076606 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:57258->10.130.200.205:10250: write: broken pipe
E1017 04:39:21.980305 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:49284->10.130.200.205:10250: write: broken pipe
E1017 04:40:38.104596 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:53780->10.130.200.205:10250: write: connection reset by peer
E1017 04:40:38.951925 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:53812->10.130.200.205:10250: write: broken pipe
E1017 04:41:38.986494 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:44814->10.130.200.205:10250: write: broken pipe
E1017 04:41:46.921660 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:49916->10.130.200.205:10250: write: broken pipe
E1017 04:42:03.803831 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:41222->10.130.200.205:10250: write: broken pipe
E1017 04:42:20.729066 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:57502->10.130.200.205:10250: write: broken pipe
E1017 04:42:29.249557 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:58130->10.130.200.205:10250: write: broken pipe
E1017 04:42:42.443590 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:56690->10.130.200.205:10250: write: broken pipe
E1017 04:43:25.759812 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:59044->10.130.200.205:10250: write: broken pipe
W1017 04:43:38.023403 1 watcher.go:229] watch chan error: etcdserver: mvcc: required revision has been compacted
E1017 04:44:04.043033 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:38832->10.130.200.205:10250: write: broken pipe
E1017 04:44:08.526688 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:38894->10.130.200.205:10250: write: broken pipe
E1017 04:44:24.566848 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:53652->10.130.200.205:10250: write: broken pipe
E1017 04:44:29.892044 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:53750->10.130.200.205:10250: write: broken pipe
E1017 04:46:03.540266 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:43190->10.130.200.205:10250: write: connection reset by peer
E1017 04:47:03.233316 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:47018->10.130.200.205:10250: write: broken pipe
E1017 04:47:34.019569 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:54140->10.130.200.205:10250: write: broken pipe
E1017 04:48:46.590177 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:60486->10.130.200.205:10250: write: broken pipe
E1017 04:49:17.024853 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:50822->10.130.200.205:10250: write: broken pipe
E1017 04:49:46.976438 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:58512->10.130.200.205:10250: write: broken pipe
E1017 04:50:52.098814 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:39310->10.130.200.205:10250: write: broken pipe
W1017 04:50:56.716785 1 watcher.go:229] watch chan error: etcdserver: mvcc: required revision has been compacted
E1017 04:52:43.710655 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:53670->10.130.200.205:10250: write: broken pipe
E1017 04:53:01.404796 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:45056->10.130.200.205:10250: write: broken pipe
E1017 04:53:35.723988 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:55888->10.130.200.205:10250: write: broken pipe
E1017 04:53:38.963865 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:55896->10.130.200.205:10250: write: connection reset by peer
E1017 04:53:43.704862 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:55970->10.130.200.205:10250: write: broken pipe
E1017 04:54:14.293525 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:58620->10.130.200.205:10250: write: connection reset by peer
E1017 04:54:17.525153 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:58636->10.130.200.205:10250: write: broken pipe
E1017 04:54:34.669342 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:36564->10.130.200.205:10250: write: broken pipe
E1017 04:55:31.163440 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:49346->10.130.200.205:10250: write: broken pipe
E1017 04:55:34.590219 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:57554->10.130.200.205:10250: write: broken pipe
E1017 04:55:56.909730 1 upgradeaware.go:450] Error proxying data from backend to client: write tcp 10.130.200.205:6443->10.130.200.205:51080: write: connection reset by peer
E1017 04:56:48.331781 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:52450->10.130.200.205:10250: write: broken pipe
E1017 04:56:55.114327 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:51926->10.130.200.205:10250: write: broken pipe
E1017 04:57:39.131887 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:56180->10.130.200.205:10250: write: broken pipe
E1017 04:58:04.363832 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:44708->10.130.200.205:10250: write: broken pipe
E1017 04:59:13.636821 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:35446->10.130.200.205:10250: write: broken pipe
E1017 04:59:38.087953 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:59790->10.130.200.205:10250: write: connection reset by peer
E1017 04:59:56.946307 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:43934->10.130.200.205:10250: write: broken pipe
E1017 05:00:10.065387 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:58758->10.130.200.205:10250: write: broken pipe
E1017 05:00:10.426365 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:58764->10.130.200.205:10250: write: broken pipe
E1017 05:00:56.988203 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:43440->10.130.200.205:10250: write: broken pipe
E1017 05:04:26.661464 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:34122->10.130.200.205:10250: write: connection reset by peer
E1017 05:04:32.540373 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:34170->10.130.200.205:10250: write: broken pipe
W1017 05:04:56.726535 1 watcher.go:229] watch chan error: etcdserver: mvcc: required revision has been compacted
E1017 05:07:48.010198 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:58810->10.130.200.205:10250: write: broken pipe
E1017 05:08:06.036165 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:55658->10.130.200.205:10250: write: broken pipe
E1017 05:08:06.257285 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:55662->10.130.200.205:10250: write: connection reset by peer
E1017 05:11:27.902150 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:45212->10.130.200.205:10250: write: broken pipe
E1017 05:12:56.019111 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:50844->10.130.200.205:10250: write: broken pipe
W1017 05:13:36.924292 1 watcher.go:229] watch chan error: etcdserver: mvcc: required revision has been compacted
E1017 05:14:11.741564 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:35694->10.130.200.205:10250: write: broken pipe
E1017 05:14:54.985465 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:57952->10.130.200.205:10250: write: broken pipe
E1017 05:16:48.183282 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:49338->10.130.200.205:10250: write: connection reset by peer
E1017 05:16:48.390004 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:49352->10.130.200.205:10250: write: broken pipe
E1017 05:17:07.459494 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:47484->10.130.200.205:10250: write: broken pipe
E1017 05:18:15.552096 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:54172->10.130.200.205:10250: write: broken pipe
E1017 05:20:34.132598 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:59250->10.130.200.205:10250: write: broken pipe
E1017 05:20:34.982067 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:59302->10.130.200.205:10250: write: broken pipe
I1017 05:21:10.627320 1 trace.go:205] Trace[1908295088]: "Get" url:/api/v1/namespaces/kube-system/pods/kube-proxy-f94lh/log,user-agent:kubectl/v1.23.13 (linux/amd64) kubernetes/592eca0,audit-id:09e8c3c0-3687-4cc4-9d59-5c3bd43a676e,client:10.130.200.205,accept:application/json, */*,protocol:HTTP/2.0 (17-Oct-2022 05:20:08.019) (total time: 62607ms):
Trace[1908295088]: ---"Transformed response object" 62605ms (05:21:10.627)
Trace[1908295088]: [1m2.607901304s] [1m2.607901304s] END
E1017 05:21:24.668531 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:37376->10.130.200.205:10250: write: broken pipe
E1017 05:21:43.778186 1 upgradeaware.go:436] Error proxying data from client to backend: write tcp 10.130.200.205:36122->10.130.200.205:10250: write: broken pipe
I1017 05:21:45.975707 1 trace.go:205] Trace[560073527]: "Get" url:/api/v1/namespaces/kube-system/pods/kube-apiserver-hids/log,user-agent:kubectl/v1.23.13 (linux/amd64) kubernetes/592eca0,audit-id:d3d4511f-4cd3-450f-9ef4-8d8e4b738294,client:10.130.200.205,accept:application/json, */*,protocol:HTTP/2.0 (17-Oct-2022 05:21:24.385) (total time: 21590ms):
Trace[560073527]: ---"Transformed response object" 21587ms (05:21:45.975)
Trace[560073527]: [21.590263953s] [21.590263953s] END
I1017 05:22:00.491030 1 trace.go:205] Trace[1473039094]: "Get" url:/api/v1/namespaces/kube-system/pods/kube-apiserver-hids/log,user-agent:kubectl/v1.23.13 (linux/amd64) kubernetes/592eca0,audit-id:5fded3a9-8a1c-4b5b-9086-4c3afbb7dd68,client:10.130.200.205,accept:application/json, */*,protocol:HTTP/2.0 (17-Oct-2022 05:21:57.263) (total time: 3226ms):
Trace[1473039094]: ---"Transformed response object" 3224ms (05:22:00.490)
Trace[1473039094]: [3.226980928s] [3.226980928s] END
kubeadm config (init-config.yaml)
kind: InitConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
localAPIEndpoint:
advertiseAddress: 10.130.200.205
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
imagePullPolicy: IfNotPresent
name: k8s
taints: null
---
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: 1.23.13
imageRepository: k8s.gcr.io
apiServer:
timeoutForControlPlane: 4m0s
extraArgs:
service-node-port-range: "443-55000"
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
local:
dataDir: /var/lib/etcd
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}
---
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
mode: ipvs
ipvs:
syncPeriod: 30s
minSyncPeriod: 0s
scheduler: "lc"
excludeCIDRs: null
strictARP: false
install
yum install -y kubelet-1.23.13 kubeadm-1.23.13 kubectl-1.23.13
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
kubeadm init --config init-config.yaml
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl taint nodes --all node-role.kubernetes.io/master-
netstat
$ ss -natl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 4096 127.0.0.1:10257 *:*
LISTEN 0 4096 127.0.0.1:10259 *:*
LISTEN 0 4096 127.0.0.1:38267 *:*
LISTEN 0 4096 127.0.0.1:10248 *:*
LISTEN 0 4096 127.0.0.1:10249 *:*
LISTEN 0 4096 127.0.0.1:2379 *:*
LISTEN 0 4096 10.130.200.205:2379 *:*
LISTEN 0 4096 10.130.200.205:2380 *:*
LISTEN 0 4096 127.0.0.1:2381 *:*
LISTEN 0 4096 [::]:10256 [::]:*
LISTEN 0 4096 [::]:10250 [::]:*
LISTEN 0 4096 [::]:6443 [::]:*