Unable to connect to the server: x509: certificate has expired or is not yet valid

General Discussions
1

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version:1.18
Cloud being used: (put bare-metal if not on a public cloud)
Installation method: kubeadm
Host OS: CentOS Linux release 7.8.2003
CNI and version: Weave
CRI and version: Docker 19.03.8

i have a 2 Master nodes and 3 worker nodes K8s Cluster, and i didn’t upgrade for an year now. Looks like now certs are expired and i am not able to run any kubectl commands. Is there a way to renew the certs without the downtime ? may i know the steps to renew the certs ?

2

So you’re saying you haven’t updated your control plane in over a year?

I found this stackoverflow article that seems to be the same issue as yours. The top comment points to this github issue.

3

Yes, these control plane nodes are not updated over an year. And trying to find out if there is any safe way to renew certs without any impacts/downtime.

4

Short of digging up the API calls that kubeadm certs renew would do and using emulating those requests while ignoring certificate check I don’t think you’ll find a way to get around doing this offline. I don’t even know if doing that would work either, it depends on if api-server also checks it’s certificate expiry.

After you get this certificate issue fixed, I recommend figuring out what toleration your cluster has for node loss. A cluster where you can’t survive a node dying defeats the purpose of even using k8s.

Also get your cluster components updated, you can’t just set software and forget software like that anymore. Failing to perform updates is how you end up with compromised systems.

5

Thank you. I updated the certs and all the nodes are active. However the worker nodes doesn’t show any ROLES assigned to them.

9:33:36 .helm # kubectl get nodes
STATUS ROLES AGE VERSION
Ready master 413d v1.18.2
Ready master 413d v1.18.2
Ready master 413d v1.18.2
Ready 413d v1.18.2
Ready 413d v1.18.2
Ready 413d v1.18.2
Ready 413d v1.18.2

Also Prometheus alerts “server returned HTTP status 401 Unauthorized” for below endpoints. Any help can i get here ? Rbac’s are already configured.

[https://kubernetes.default.svc:443/api/v1/nodes/worker-hostname/proxy/metrics](