Work node join cluster by public IP

General Discussions
1

hello,
I m facing some tricky thing, I have 2 kubernetes, one at A city and the other at B city.
and have VPN connected.
it’s VM and I snapshot before work node join.
my kubeadm kubectl, kubelet are both 1.14.0, and I use flannel as pod network.
master have public IP address “A”, node have public IP “B”
the work node can join the cluster by master node internal IP. but not able to join by master’s public IP.
I capture the packet when the join cluster by internal, and work node just need to communicate with master port 6443.
my work node able to telnet master port 6643 by master’s public IP.

I tcpdump and found there have network traffic between master (6443) and work node, but the work node screen hang

image
image.png1321×121 10.3 KB

2

the work node able to connected to master port 6443

image
image.png1150×167 28 KB

master ports allow access from internet:
80, 443, 2379-2380, 8285 , 8472 , 6443 , 9099 , 10250-10256, 30000-32767

work node ports allow access from internet
8285 , 8472 , 9099 , 10250 , 10254

3

Have you tried changing the Docker daemon settings to use systemd instead of cgroupfs?

4

no, I didn’t, do I must change to systemd?
and how to change? if there any guide line or something? thanks
:wink:

5

This should help you out, CRI installation - Kubernetes .

Let me know if you hit any speed bumps

6

thanks for the reply
I follow the doc procedure, and when I using kubeadn join …
it tell me
Found multiple CRI sockets, please use --cri-socket to select one: /var/run/dockershim.sock, /var/run/crio/crio.sock

so I add --cri-socket /var/run/crio/crio.sock at the end of the kubeadm join command
but still not able to join the cluster:

image
image.png1336×91 5.96 KB

I really want to know, if there any different setting when the work node join the cluster by public IP? since everytime I join the cluster by using master internal IP, but this time it’s really puzzled me :frowning:
thanks ~~~~

7

Does it just hang at the pre-flight check stage now? Did you start a fresh cluster or are you using the same join token as before?

8

correct, just hang there, and the token is get from kubeadm token list, still valid, not expired.
I follow the procedure at master and work node both, since the master is already init, I am not sure does that impact

9

below is kubelet log from work node:

Apr 03 14:49:21 K8S-Slave kubelet[1887]: F0403 14:49:21.474959 1887 server.go:193] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file “/var/lib/kubelet/config.yaml”, error: open /var/lib/kubelet/config.yaml: no such file or directory
Apr 03 14:49:21 K8S-Slave systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Apr 03 14:49:21 K8S-Slave systemd[1]: kubelet.service: Unit entered failed state.
Apr 03 14:49:21 K8S-Slave systemd[1]: kubelet.service: Failed with result ‘exit-code’.
Apr 03 14:49:31 K8S-Slave systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Apr 03 14:49:31 K8S-Slave systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
– Subject: Unit kubelet.service has finished shutting down
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

– Unit kubelet.service has finished shutting down.
Apr 03 14:49:31 K8S-Slave systemd[1]: Started kubelet: The Kubernetes Node Agent.
– Subject: Unit kubelet.service has finished start-up
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

– Unit kubelet.service has finished starting up.

– The start-up result is done.

it seems work node not able to get the file from master ?
some node, if I change the master IP from public IP to private IP, it work, and I am pretty sure the master port 6443 is open on the public IP.

10

If it works using the private IP then there might be some configuration setting with the public ip that needs looking at. I would be hesitant to expose the kube-api to public access though, unless it’s necessary.

11

make sure you have rite unit file defined for kubelet service whether its through tls bootstrap or through cert config.

crt bootstrap adding can be:

Create the kubelet-config.yaml configuration file:

worker-1$ cat <<EOF | sudo tee /var/lib/kubelet/kubelet-config.yaml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/var/lib/kubernetes/ca.crt"
authorization:
  mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
  - "10.96.0.10"
resolvConf: "/run/systemd/resolve/resolv.conf"
runtimeRequestTimeout: "15m"
EOF

cat <<EOF | sudo tee /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
  --config=/var/lib/kubelet/kubelet-config.yaml \\
  --image-pull-progress-deadline=2m \\
  --kubeconfig=/var/lib/kubelet/kubeconfig \\
  --tls-cert-file=/var/lib/kubelet/${HOSTNAME}.crt \\
  --tls-private-key-file=/var/lib/kubelet/${HOSTNAME}.key \\
  --network-plugin=cni \\
  --register-node=true \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

tls bootstrap

cat <<EOF | sudo tee /etc/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-proxy \\
  --config=/var/lib/kube-proxy/kube-proxy-config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF