Couldn't get current server API group list - client certificate in /etc/kubernetes/kubelet.conf is expired

General Discussions
1

Hi, I have a kubernetes up and running since 1 year, this morning when I try to issue any command I get

E0305 11:10:30.992543 6055 memcache.go:238] couldn’t get current server API group list: Get “https://192.168.1.15:6443/api?timeout=32s”: dial tcp 192.168.1.15:6443: connect: connection refused
E0305 11:10:30.992839 6055 memcache.go:238] couldn’t get current server API group list: Get “https://192.168.1.15:6443/api?timeout=32s”: dial tcp 192.168.1.15:6443: connect: connection refused

Checking the journalctl I saw

Mar 05 11:15:15 kubernetes2 kubelet[6390]: E0305 11:15:15.940390 6390 bootstrap.go:265] part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2024-02-22 15:58:35 +0000 UTC

I followed instructions at Certificate Management with kubeadm | Kubernetes and updated all certificates with

kubeadm certs renew all

then rexecuted

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

restarted my VM, any command fails with the same error. Checking again the log I see the exact same error in spite of the fact when I run

kubeadm certs check-expiration

I see that all certificates are renewed as follows

image
image889×251 58.7 KB

So what am is missing or needs an update?

Cluster information:

Kubernetes version:
Client Version: v1.25.0
Kustomize Version: v4.5.7
Server Version: v1.26.4
Installation method: Bare-and-metal on a VM
Host OS: Ubuntu 22

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

2

Hi, can somebody help?

3

I check you thread and my scripts. What I always do when run cert renewall is moving kubelet.conf file from its location:
mkdir -p /tmp/backup/kubelet
mv /etc/kubernetes/kubelet.conf /tmp/backup/kubelet
mv /var/lib/kubelet/pki/kubelet-client* /tmp/backup/kubelet

and delete old certs and keys:
rm -rf /var/lib/kubelet/pki/kubelet.crt
rm -rf /var/lib/kubelet/pki/kubelet.key

After that we have to recreate this file from admin.conf making some changes and at the end restart kubelet:

export NODE=$(hostname)
mkdir /tmp/backup/kubelet/$NODE
cp /etc/kubernetes/admin.conf /tmp/backup/kubelet/
export APIIP=$(cat /etc/kubernetes/admin.conf | grep server | awk -F " " ‘{print $2}’ > /tmp/backup/kubelet/file | awk -F “//” ‘{print $2}’ /tmp/backup/kubelet/file| sed "s/:6443.//g")
kubeadm init phase kubeconfig admin --kubeconfig-dir /tmp/backup/kubelet/$NODE --apiserver-advertise-address $APIIP --cert-dir /etc/kubernetes/pki/
cp /tmp/backup/kubelet/admin.conf /etc/kubernetes/kubelet.conf
rm -rf /var/lib/kubelet/pki/
systemctl restart kubelet