Cluster information:
Kubernetes version: 1.29.1
Cloud being used: bare-metal
Installation method: apt
Host OS: Ubuntu
CNI and version: Calico, v3.23.2
CRI and version: containerd, 1.7.12-0ubuntu2~20.04.1
Yesterday all kubectl commands began to fail with the following error:
E0926 08:35:00.583312 22495 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0926 08:35:00.591308 22495 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0926 08:35:00.599525 22495 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0926 08:35:00.607297 22495 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0926 08:35:00.615521 22495 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)
A check of the certificates indicated that all were valid.
kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Feb 19, 2025 22:13 UTC 146d ca no
apiserver Feb 19, 2025 22:13 UTC 146d ca no
apiserver-etcd-client Feb 19, 2025 22:13 UTC 146d etcd-ca no
apiserver-kubelet-client Feb 19, 2025 22:13 UTC 146d ca no
controller-manager.conf Feb 19, 2025 22:13 UTC 146d ca no
etcd-healthcheck-client Feb 19, 2025 22:13 UTC 146d etcd-ca no
etcd-peer Feb 19, 2025 22:13 UTC 146d etcd-ca no
etcd-server Feb 19, 2025 22:13 UTC 146d etcd-ca no
front-proxy-client Feb 19, 2025 22:13 UTC 146d front-proxy-ca no
scheduler.conf Feb 19, 2025 22:13 UTC 146d ca no
!MISSING! super-admin.conf
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Sep 23, 2033 22:59 UTC 8y no
etcd-ca Sep 23, 2033 22:59 UTC 8y no
front-proxy-ca Sep 23, 2033 22:59 UTC 8y no
Even though the certs were valid, running the following resolved the issue:
kubeadm certs renew all
Can anyone explain why this happened and are there other certificates to check?