We just got bit by hostname and kubelet case sensitivity

General Discussions
1

Cluster information:

Kubernetes version: 1.16
Cloud being used: bare-metal
Installation method: Handmade
Host OS: Ubuntu server

So, that’s the situation. we’ve installed our first bare-metal k8s on Ubuntu VM (unfortunately business wanted ubuntu instead of centos…) hosted by a third part.

We had to update from 1.16 to 1.20

We tested this update on another Ubuntu VM and, apart from updating a couple of yamls, everything went smooth.

But when we started to upgrade our primary Ubuntu VM we got the error message:
FATAL: failed to get node registration: failed to get corresponding node: nodes

If we run the following commands:

  • kubectl get nodes returns nodes with same name as the hostname of the VM but in lowercase
  • kubeadm upgrade plan is trying to get node registrations for nodes that have the same name as the hostname but in uppercase.

We tried to change the vm hostname and all the kubelet config files to use lowercase but this did not fix the issue so we went to k8s source codes and we found out that kubeadm upgrade plan it’s getting the node name from the CommonName of the Certificate inside the kubelet.config

Given that we can’t change the common name of the certificate without invalidating and regenerating them, and given that we cannot change the node names we are stuck as we can’t change anything.

We thought that a possibility would be to regenerate the certificates using the appropriate kubeadm phase but we are not sure that it would work and we don’t want to risk breaking the cluster. (Of course we have appropriate backups and all but we are trying to keep the downtime to a minimum).

Said that, I know that VM name should be a “valid DNS name” as stated here but unfortunately we did not thought about that when setting up the machine.

My question is: is there a way to work around the issue or we have to start from scratch ?

Additionally, shouldn’t kubeadm init stop or warn you if you pass a --node-name that contains uppercase letters ?

How to reproduce:

  • Have a machine with an host name with uppercase letters (e.g.: TEST123)
  • Run kubeadm init using --node-name set to the host name (e.g.: --node-name TEST123
  • Setup some nodes
  • Try to run kubadm upgrade plan