Cluster information:
Kubernetes version: v1.14.3
Cloud being used:
Installation method: kubeadm
Host OS: centos 7
CNI and version: 0.3.0
CRI and version:
kubeadm init
–kubernetes-version=v1.14.3
–pod-network-cidr=10.244.0.0/16
–apiserver-advertise-address=10.10.102.190
docker images|grep k8s.gcr.io
k8s.gcr.io/kube-proxy v1.14.3 004666307c5b 2 weeks ago 82.1MB
k8s.gcr.io/kube-apiserver v1.14.3 9946f563237c 2 weeks ago 210MB
k8s.gcr.io/kube-controller-manager v1.14.3 ac2ce44462bc 2 weeks ago 158MB
k8s.gcr.io/kube-scheduler v1.14.3 953364a3ae7a 2 weeks ago 81.6MB
k8s.gcr.io/coredns 1.3.1 eb516548c180 5 months ago 40.3MB
k8s.gcr.io/etcd 3.3.10 2c4adeb21b4f 6 months ago 258MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 18 months ago 742kB
error:
journalctl --no-pager -xeu kubelet
6月 21 18:27:14 gfresh-master1 kubelet[19368]: I0621 18:27:14.971130 19368 kuberuntime_manager.go:210] Container runtime docker initialized, version: 18.09.4, apiVersion: 1.39.0
6月 21 18:27:14 gfresh-master1 kubelet[19368]: E0621 18:27:14.973641 19368 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSIDriver: Get https://10.10.102.190:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 10.10.102.190:6443: connect: connection refused
6月 21 18:27:14 gfresh-master1 kubelet[19368]: I0621 18:27:14.974000 19368 server.go:1054] Started kubelet
6月 21 18:27:14 gfresh-master1 kubelet[19368]: E0621 18:27:14.974066 19368 kubelet.go:1282] **Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache**
6月 21 18:27:14 gfresh-master1 kubelet[19368]: I0621 18:27:14.974116 19368 server.go:141] Starting to listen on 0.0.0.0:10250
6月 21 18:27:14 gfresh-master1 kubelet[19368]: E0621 18:27:14.974557 19368 event.go:200] Unable to write event: 'Post https://10.10.102.190:6443/api/v1/namespaces/default/events: dial tcp 10.10.102.190:6443: connect: connection refused' (may retry after sleeping)
6月 21 18:27:14 gfresh-master1 kubelet[19368]: I0621 18:27:14.975094 19368 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
6月 21 18:27:14 gfresh-master1 kubelet[19368]: I0621 18:27:14.975147 19368 status_manager.go:152] Starting to sync pod status with apiserver
6月 21 18:27:14 gfresh-master1 kubelet[19368]: I0621 18:27:14.975172 19368 kubelet.go:1806] Starting kubelet main sync loop.
6月 21 18:27:14 gfresh-master1 kubelet[19368]: I0621 18:27:14.975194 19368 kubelet.go:1823] skipping pod synchronization - [container runtime status check may not have completed yet., PLEG is not healthy: pleg has yet to be successful.]
6月 21 18:27:14 gfresh-master1 kubelet[19368]: I0621 18:27:14.975276 19368 server.go:343] Adding debug handlers to kubelet server.
6月 21 18:27:14 gfresh-master1 kubelet[19368]: I0621 18:27:14.975614 19368 volume_manager.go:248] Starting Kubelet Volume Manager
6月 21 18:27:14 gfresh-master1 kubelet[19368]: I0621 18:27:14.975719 19368 desired_state_of_world_populator.go:130] Desired state populator starts to run
6月 21 18:27:14 gfresh-master1 kubelet[19368]: E0621 18:27:14.975894 19368 controller.go:115] failed to ensure node lease exists, will retry in 200ms, error: Get https://10.10.102.190:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/gfresh-master1?timeout=10s: dial tcp 10.10.102.190:6443: connect: connection refused
6月 21 18:27:14 gfresh-master1 kubelet[19368]: E0621 18:27:14.976644 19368 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get https://10.10.102.190:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 10.10.102.190:6443: connect: connection refused
6月 21 18:27:15 gfresh-master1 kubelet[19368]: E0621 18:27:15.025557 19368 factory.go:340] devicemapper filesystem stats will not be reported: RHEL/Centos 7.x kernel version 3.10.0-366 or later is required to use thin_ls - you have "3.10.0-327.el7.x86_64"
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.026984 19368 clientconn.go:440] parsed scheme: "unix"
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.027015 19368 clientconn.go:440] scheme "unix" not registered, fallback to default scheme
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.027084 19368 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 <nil>}]
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.027103 19368 clientconn.go:796] ClientConn switching balancer to "pick_first"
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.027197 19368 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0xc000adf7b0, CONNECTING
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.027475 19368 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0xc000adf7b0, READY
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.075349 19368 kubelet.go:1823] skipping pod synchronization - container runtime status check may not have completed yet.
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.075846 19368 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach
6月 21 18:27:15 gfresh-master1 kubelet[19368]: E0621 18:27:15.075893 19368 kubelet.go:2244] node "gfresh-master1" not found
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.082415 19368 kubelet_node_status.go:72] Attempting to register node gfresh-master1
6月 21 18:27:15 gfresh-master1 kubelet[19368]: E0621 18:27:15.083132 19368 kubelet_node_status.go:94] Unable to register node "gfresh-master1" with API server: Post https://10.10.102.190:6443/api/v1/nodes: dial tcp 10.10.102.190:6443: connect: connection refused
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.131824 19368 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.135129 19368 cpu_manager.go:155] [cpumanager] starting with none policy
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.135157 19368 cpu_manager.go:156] [cpumanager] reconciling every 10s
6月 21 18:27:15 gfresh-master1 kubelet[19368]: I0621 18:27:15.135174 19368 policy_none.go:42] [cpumanager] none policy: Start
6月 21 18:27:15 gfresh-master1 kubelet[19368]: F0621 18:27:15.136451 19368 kubelet.go:1359] Failed to start ContainerManager failed to initialize top level QOS containers: failed to update top level Burstable QOS cgroup : failed to set supported cgroup subsystems for cgroup [kubepods burstable]: Failed to find subsystem mount for required subsystem: pids
6月 21 18:27:15 gfresh-master1 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
6月 21 18:27:15 gfresh-master1 systemd[1]: Unit kubelet.service entered failed state.
6月 21 18:27:15 gfresh-master1 systemd[1]: kubelet.service failed.