Asking for help? Comment out what you need so we can get more information to help you!
Cluster information:
Kubernetes version: 1.27.1
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: ubuntu 22.04
CNI and version: calico + multus
CRI and version: containerd
After nodes reboot, cluster is not working anymore, with following messages:
ubuntu@k8cp1:~$ kubectl get nodes -v=10
I0529 19:52:36.824715 2152 loader.go:373] Config loaded from file: /home/ubuntu/.kube/config
I0529 19:52:36.828228 2152 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/4c94112" 'https://k8cp.urbaman.it:6443/api?timeout=32s'
I0529 19:52:36.830701 2152 round_trippers.go:495] HTTP Trace: DNS Lookup for k8cp.urbaman.it resolved to [{10.0.50.64 }]
I0529 19:52:36.831831 2152 round_trippers.go:510] HTTP Trace: Dial to tcp:10.0.50.64:6443 succeed
I0529 19:52:36.832333 2152 round_trippers.go:553] GET https://k8cp.urbaman.it:6443/api?timeout=32s in 4 milliseconds
I0529 19:52:36.832366 2152 round_trippers.go:570] HTTP Statistics: DNSLookup 2 ms Dial 0 ms TLSHandshake 0 ms Duration 4 ms
I0529 19:52:36.832389 2152 round_trippers.go:577] Response Headers:
I0529 19:52:37.832766 2152 with_retry.go:234] Got a Retry-After 1s response for attempt 1 to https://k8cp.urbaman.it:6443/api?timeout=32s
ubuntu@k8cp1:~$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Mon 2023-05-29 19:50:52 CEST; 2min 32s ago
Docs: https://kubernetes.io/docs/home/
Main PID: 1923 (kubelet)
Tasks: 16 (limit: 38388)
Memory: 41.3M
CPU: 5.194s
CGroup: /system.slice/kubelet.service
└─1923 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/contain>
May 29 19:53:18 k8cp1 kubelet[1923]: I0529 19:53:18.887679 1923 kubelet_node_status.go:70] "Attempting to register node" node="k8cp1"
May 29 19:53:18 k8cp1 kubelet[1923]: E0529 19:53:18.890730 1923 kubelet_node_status.go:92] "Unable to register node with API server" err="Post \"https://k8cp.urbaman.it:6443/api/v1/nodes\": EOF" node="k8cp1"
May 29 19:53:23 k8cp1 kubelet[1923]: E0529 19:53:23.170755 1923 eviction_manager.go:262] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"k8cp1\" not found"
May 29 19:53:24 k8cp1 kubelet[1923]: E0529 19:53:24.051218 1923 controller.go:146] "Failed to ensure lease exists, will retry" err="Get \"https://k8cp.urbaman.it:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases>
May 29 19:53:24 k8cp1 kubelet[1923]: W0529 19:53:24.470764 1923 reflector.go:533] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.RuntimeClass: Get "https://k8cp.urbaman.it:6443/apis/node.k8s.io/v1/runtimecl>
May 29 19:53:24 k8cp1 kubelet[1923]: I0529 19:53:24.470987 1923 trace.go:219] Trace[1584291237]: "Reflector ListAndWatch" name:vendor/k8s.io/client-go/informers/factory.go:150 (29-May-2023 19:53:14.431) (total time: 10039ms):
May 29 19:53:24 k8cp1 kubelet[1923]: Trace[1584291237]: ---"Objects listed" error:Get "https://k8cp.urbaman.it:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": EOF 10039ms (19:53:24.470)
May 29 19:53:24 k8cp1 kubelet[1923]: Trace[1584291237]: [10.039787785s] [10.039787785s] END
May 29 19:53:24 k8cp1 kubelet[1923]: E0529 19:53:24.471050 1923 reflector.go:148] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https://k8cp.urbaman.it:>
May 29 19:53:24 k8cp1 kubelet[1923]: E0529 19:53:24.919084 1923 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"k8cp1.1763ae97e7475696", GenerateName:">