How to fix cluster upgrade?

General Discussions
1

Cluster information:

Kubernetes version: 1.23
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: Linux

I’m upgrading k8s cluster from 1.23 to 1.24 with kubeadm.
During node upgrade i run into exception: “nodes “kubernetes-admin” not found
failed to get corresponding node”. This error happened when kubelet tried to contact api-server, after upgrading control plane node.
I discovered that node name was taken from CN from client certificate from /etc/kubernetes/admin.conf.
How to fix it? Also, why node name is taken from admin config file, not from kubelet config ?

I follow the official guide: Upgrading kubeadm clusters | Kubernetes
Screenshot with an error:

image
image1018×435 19.1 KB

When i output certificate /var/lib/kubelet/pki/kubelet-client-current.pem I got CN with correct host in it

4:12

But when i output /etc/kubernetes/admin.conf i got CN with incorrect host: kubernetes-admin. Which is used in request resulting in 404 status

2

Hi, aleksaero

According to the output of the above command

k8s.io/kubernetes/cmd/kubeadm/app/util/config.GetNodeRegistration
                cmd/kubeadm/app/unti/config/cluster.go:140
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/ipgrade/node.runKubeletConfigPhase.func1
                cmd/kubeadm/app/cmd/phases/upgrade/node/kubeletconfig.go:112
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
                cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
               cmd/kubeadm/app/cmd/phases/workflow/runner.go:421

First, make sure that the kubelet configuration file (/var/lib/kubelet/config.yaml) exists and is intact.
cat /var/lib/kubelet/config.yaml

Run the following command to check and view the contents of this file:
If this file has been deleted or corrupted for any reason, you can recreate it.

sudo kubeadm reset
sudo rm -rf /var/lib/kubelet
sudo rm -rf /etc/kubernetes

Reconfigure kubelet and kubeadm
After cleaning, update the kubeadm and kubeadm packages to version 1.24 and start the process again:

sudo apt-get update
sudo apt-get install -y kubeadm=1.24.x-00 kubelet=1.24.x-00 kubectl=1.24.x-00
kubeadm upgrade plan
sudo kubeadm upgrade apply v1.24.x

Reconfigure kubelet configuration

sudo kubeadm init phase kubelet-start

if not, send more details