Cluster information:
Kubernetes version: v1.15.1
Cloud being used: bare-meta
Installation method: manual
Host OS: Ubuntu
CRI and version: v1.13.0
Hi guys, my cluster master node is on state NotReady and no kube-system pod is running properly on this node. Certificates are ok. Already tried restart kubelet service. this is the ouput.
NAME STATUS ROLES AGE VERSION
NODE1 Ready <none> 502d v1.15.1
NODE2 Ready <none> 502d v1.15.1
NODE3 Ready <none> 640d v1.15.1
NODE4 Ready <none> 640d v1.15.1
NODE5 NotReady master 640d v1.15.1
Jan 04 23:14:26 NODE5 kubelet[18166]: E0104 23:14:26.011737 18166 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
Jan 04 23:14:26 NODE5 kubelet[18166]: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.013443 18166 kuberuntime_manager.go:205] Container runtime docker initialized, version: 18.09.0, apiVersion: 1.39.0
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.015204 18166 server.go:1083] Started kubelet
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.015282 18166 server.go:144] Starting to listen on 0.0.0.0:10250
Jan 04 23:14:26 NODE5 kubelet[18166]: E0104 23:14:26.015312 18166 kubelet.go:1293] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.019445 18166 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.019510 18166 status_manager.go:152] Starting to sync pod status with apiserver
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.019539 18166 kubelet.go:1805] Starting kubelet main sync loop.
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.019566 18166 kubelet.go:1822] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.019607 18166 volume_manager.go:243] Starting Kubelet Volume Manager
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.019670 18166 desired_state_of_world_populator.go:130] Desired state populator starts to run
Jan 04 23:14:26 NODE5 kubelet[18166]: E0104 23:14:26.020819 18166 controller.go:115] failed to ensure node lease exists, will retry in 200ms, error: leases.coordination.k8s.io "NODE5" is forbidden: User "system:node:NODE5" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.021642 18166 server.go:350] Adding debug handlers to kubelet server.
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.073167 18166 clientconn.go:440] parsed scheme: "unix"
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.073187 18166 clientconn.go:440] scheme "unix" not registered, fallback to default scheme
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.073225 18166 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 <nil>}]
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.073240 18166 clientconn.go:796] ClientConn switching balancer to "pick_first"
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.073287 18166 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0xc00038a810, CONNECTING
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.073444 18166 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0xc00038a810, READY
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.119669 18166 kuberuntime_manager.go:928] updating runtime config through cri with podcidr 172.30.0.0/24
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.119726 18166 kubelet.go:1822] skipping pod synchronization - container runtime status check may not have completed yet
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.119803 18166 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.119898 18166 docker_service.go:353] docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:172.30.0.0/24,},}
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.120142 18166 kubelet_network.go:77] Setting Pod CIDR: -> 172.30.0.0/24
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.123021 18166 kubelet_node_status.go:72] Attempting to register node NODE5
Jan 04 23:14:26 NODE5 kubelet[18166]: E0104 23:14:26.124733 18166 kubelet_node_status.go:94] Unable to register node "NODE5" with API server: nodes "NODE5" is forbidden: node "NODE5" is not allowed to modify node "NODE5"
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.181598 18166 cpu_manager.go:155] [cpumanager] starting with none policy
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.181610 18166 cpu_manager.go:156] [cpumanager] reconciling every 10s
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.181620 18166 policy_none.go:42] [cpumanager] none policy: Start
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.182837 18166 plugin_manager.go:116] Starting Kubelet Plugin Manager
Jan 04 23:14:26 NODE5 kubelet[18166]: E0104 23:14:26.222054 18166 controller.go:115] failed to ensure node lease exists, will retry in 400ms, error: leases.coordination.k8s.io "NODE5" is forbidden: User "system:node:NODE5" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node
Jan 04 23:14:26 NODE5 kubelet[18166]: E0104 23:14:26.321752 18166 reflector.go:125] object-"kube-system"/"weave-net-token-sxdnv": Failed to list *v1.Secret: secrets "weave-net-token-sxdnv" is forbidden: User "system:node:NODE5" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "NODE5" and this object
Jan 04 23:14:26 NODE5 kubelet[18166]: E0104 23:14:26.321779 18166 reflector.go:125] object-"kube-system"/"kube-proxy": Failed to list *v1.ConfigMap: configmaps "kube-proxy" is forbidden: User "system:node:NODE5" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node "NODE5" and this object
Jan 04 23:14:26 NODE5 kubelet[18166]: E0104 23:14:26.321953 18166 reflector.go:125] object-"kube-system"/"kube-proxy-token-ztb4n": Failed to list *v1.Secret: secrets "kube-proxy-token-ztb4n" is forbidden: User "system:node:NODE5" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "NODE5" and this object
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.324899 18166 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Jan 04 23:14:26 NODE5 kubelet[18166]: I0104 23:14:26.328097 18166 kubelet_node_status.go:72] Attempting to register node NODE5
Jan 04 23:14:26 NODE5 kubelet[18166]: E0104 23:14:26.329667 18166 kubelet_node_status.go:94] Unable to register node "NODE5" with API server: nodes "NODE5" is forbidden: node "NODE5" is not allowed to modify node "NODE5"
Jan 04 23:14:26 NODE5 kubelet[18166]: W0104 23:14:26.330008 18166 status_manager.go:501] Failed to update status for pod "kube-proxy-ss68t_kube-system(5a737481-dc56-4270-8dd8-afddcc78a668)": failed to patch status "{\"status\":{\"startTime\":\"2021-01-04T14:14:26Z\"}}" for pod "kube-system"/"kube-proxy-ss68t": pods "kube-proxy-ss68t" is forbidden: node "NODE5" can only update pod status for pods with spec.nodeName set to itself
Jan 04 23:14:26 NODE5 kubelet[18166]: W0104 23:14:26.371837 18166 status_manager.go:501] Failed to update status for pod "weave-net-h495w_kube-system(7a206409-bead-41b6-9387-045ba38c8e7b)": failed to patch status "{\"status\":{\"$setElementOrder/conditions\":[{\"type\":\"Initialized\"},{\"type\":\"Ready\"},{\"type\":\"ContainersReady\"},{\"type\":\"PodScheduled\"}],\"conditions\":[{\"lastProbeTime\":null,\"lastTransitionTime\":\"2021-01-04T14:14:26Z\",\"status\":\"True\",\"type\":\"Initialized\"},{\"lastProbeTime\":null,\"lastTransitionTime\":\"2021-01-04T14:14:26Z\",\"message\":\"containers with unready status: [weave weave-npc]\",\"reason\":\"ContainersNotReady\",\"status\":\"False\",\"type\":\"Ready\"},{\"lastProbeTime\":null,\"lastTransitionTime\":\"2021-01-04T14:14:26Z\",\"message\":\"containers with unready status: [weave weave-npc]\",\"reason\":\"ContainersNotReady\",\"status\":\"False\",\"type\":\"ContainersReady\"}],\"containerStatuses\":[{\"image\":\"weaveworks/weave-kube:2.5.2\",\"imageID\":\"\",\"lastState\":{},\"name\":\"weave\",\"ready\":false,\"restartCount\":0,\"state\":{\"waiting\":{\"reason\":\"ContainerCreating\"}}},{\"image\":\"weaveworks/weave-npc:2.5.2\",\"imageID\":\"\",\"lastState\":{},\"name\":\"weave-npc\",\"ready\":false,\"restartCount\":0,\"state\":{\"waiting\":{\"reason\":\"ContainerCreating\"}}}],\"hostIP\":\"10.252.241.145\",\"podIP\":\"10.252.241.145\",\"startTime\":\"2021-01-04T14:14:26Z\"}}" for pod "kube-system"/"weave-net-h495w": pods "weave-net-h495w" is forbidden: node "NODE5" can only update pod status for pods with spec.nodeName set to itself