I’ve been having trouble automating microk8s VMs to form a cluster of 3 (mostly to support rolling updates without losing deployment state).
The process I’ve attempted so far is for when a new node is created, it SSHes to the other nodes to use microk8s remove node $VM --force. After it’s made it’s best effort to clean up any potential prior instances of itself, it does another ssh, microk8s add-node against the other nodes (beginning with what I think of as the primary node).
It appears that when I boot up the two non-primary nodes, one will begin adding itself to the primary node, and the second one will fail to connect to the primary node, but will instead begin connecting to the secondary node that it booted fresh along side. The result is that the joining nodes hang for a while and then print:
E0506 09:46:17.508817 1818037 request.go:1116] Unexpected error when reading response body: unexpected EOF
error: unexpected error when reading response body. Please retry. Original error: unexpected EOF
The primary node that was once functional as a stand-alone kubernetes node no longer responsive over kubectl.
If I boot the joining machines in a staggered manor it works making me think the problem is related to a joining node being able to create a join token for another node and allow it to join as it’s in the process of joining to a different cluster.
Is there a more specific protocol to follow to ensure automated cluster joins are handled correctly? So far I’ve just been following the docs here https://microk8s.io/docs/clustering