Cluster information:
Kubernetes version: v1.31.0
Cloud being used: bare-metal
Installation method:
Host OS: Ubuntu 22.04.2 LTS
CNI and version: Calico:3.28.1
CRI and version: containerd://1.7.21
Hy everyone,
I’m running a bare-metal Kubernetes cluster created with kubeadm, consisting of 3 masters and 6 worker nodes.
Yesterday around 17:30, I suddenly encountered this error while executing kubectl get po:
E0521 17:38:10.747232 57844 memcache.go:265] “Unhandled Error” err=“couldn’t get current server API group list: the server has asked for the client to provide credentials. You must be logged in to the server”
My first assumption was that some certificates might have expired. However, upon checking the certificate expiration status with:
kubeadm certs check-expiration
I confirmed that everything is valid until September 12, 2025:
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Sep 12, 2025 07:18 UTC 112d ca no
apiserver Sep 12, 2025 07:18 UTC 112d ca no
apiserver-etcd-client Sep 12, 2025 07:18 UTC 112d etcd-ca no
apiserver-kubelet-client Sep 12, 2025 07:18 UTC 112d ca no
controller-manager.conf Sep 12, 2025 07:18 UTC 112d ca no
etcd-healthcheck-client Sep 12, 2025 07:18 UTC 112d etcd-ca no
etcd-peer Sep 12, 2025 07:18 UTC 112d etcd-ca no
etcd-server Sep 12, 2025 07:18 UTC 112d etcd-ca no
front-proxy-client Sep 12, 2025 07:18 UTC 112d front-proxy-ca no
scheduler.conf Sep 12, 2025 07:18 UTC 112d ca no
super-admin.conf Sep 12, 2025 07:18 UTC 112d ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca May 13, 2033 14:25 UTC 7y358d no
etcd-ca May 13, 2033 14:25 UTC 7y358d no
front-proxy-ca May 13, 2033 14:25 UTC 7y358d no
Checking journalctl around the affected time window, I found nothing unusual except for these entries:
May 21 17:19:25 k8s-master-02 systemd[1]: run-containerd-runc-...-runc.G6qTk9.mount: Deactivated successfully.
May 21 17:19:27 k8s-master-02 systemd[1]: run-containerd-runc-...-runc.0K6bpc.mount: Deactivated successfully.
May 21 17:20:25 k8s-master-02 systemd[1]: run-containerd-runc-...-runc.WimdH7.mount: Deactivated successfully.
May 21 17:20:25 k8s-master-02 systemd[1]: run-containerd-runc-...-runc.VM7yc0.mount: Deactivated successfully.
However, upon inspecting the client-certificate-data in my local kubeconfig and comparing it with the one in the admin.conf on the master node, I noticed they differ — although the expiration date remains the same.
Thanks in advance for any help or clarification.