Unable to Connect a k3s Worker to a Remote k3s Cluster in a Cloud VM

General Discussions
1

My goal is to setup a lightweight kubernetes cluster using k3s with my server (master node) setup in a GCP virtual machine and be able to join remote agents (worker nodes) to it.

The process was successful when done with 2 VMs in the same GCP network but as soon as I attempt to join the cluster from outside of the LAN I end up with connection timeouts.

Here are the exact steps I took so far:

On the VM (k3s server):

1- Reserved a static external IP address

2- Added a firewall ingress rule for TCP:6443

4- Installed k3s as “server”: curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable-agent --write-kubeconfig-mode 644 --tls-san 35.198.x.x" sh -

5- Exported the node token located at: /var/lib/rancher/k3s/server/node-token

On the remote desktop machine (k3s agent):

1- Installed k3s as “agent”: curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig-mode 644" sh -

2- Attempted to join the cluster using the external IP of the VM and the exported token sudo k3s agent --server https://35.198.x.x:6443 --token K10a3x::node:8b62x

The output I receive after the last command is:

INFO[2019-10-10T15:12:54.171910055+01:00] Starting k3s agent v0.9.1 (755bd1c6) INFO[2019-10-10T15:12:54.172119361+01:00] Running load balancer 127.0.0.1:38669 -> [35.198.x.x:6443] INFO[2019-10-10T15:12:56.741810256+01:00] Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log INFO[2019-10-10T15:12:56.742150259+01:00] Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd INFO[2019-10-10T15:12:56.744248953+01:00] module br_netfilter was already loaded INFO[2019-10-10T15:12:56.744417346+01:00] module overlay was already loaded INFO[2019-10-10T15:12:56.744453854+01:00] module nf_conntrack was already loaded INFO[2019-10-10T15:12:56.929625073+01:00] Updating load balancer server addresses -> [10.156.0.4:6443 35.198.x.x:6443] INFO[2019-10-10T15:12:56.929915280+01:00] Connecting to proxy url="wss://10.156.0.4:6443/v1-k3s/connect" ERRO[2019-10-10T15:15:07.362512013+01:00] Failed to connect to proxy error="dial tcp 10.156.0.4:6443: connect: connection timed out" ERRO[2019-10-10T15:15:07.362608180+01:00] Remotedialer proxy error error="dial tcp 10.156.0.4:6443: connect: connection timed out"

The client indefinitely retries to connect but to no avail. It did find the correct local IP of the endpoint which shows up upon calling kubectl get endpoints on the VM:

NAME ENDPOINTS AGE kubernetes 10.156.0.4:6443 19m

Kindly let me know if this setup I’m trying to achieve is valid and if so, what could I be missing? Thank you.

2

I would try using K3sup

1 Like
3

Hey, thank you for your reply. I’ve tried this method and when I launch the “k3sup install” command it stops halfway through the k3s installation and times out. I’m starting to wonder it’s an issue with google cloud VMs in particular. I mean, this type of architecture is doable theoretically, right?

4

The above error tells me you have connectivity issue there. Please check the firewall attached to VM 10.156.0.4 to allow inboud connections to 6443 . After that please check if port 6443 is on LISTEN, you can do that with netstat -tenpl or equivalent command/option to your OS. If you have the right rules on your VM firewall and the port is on LISTEN state please check iptables on the VM. iptables -nvL. Would be helpful if we have the asked info posted here. Will gives us a better understanding.