Cluster information:
Kubernetes version: v1.20.4
Cloud being used: bare-metal
Installation method: kubespray
Host OS: Ubuntu OS
CNI and version: ?
CRI and version: ?
I’m using kubespray to install new cluster, and it’s going pretty good, most of the steps are behined me ( I was able to fix all the missing packages on my offline network ). But after running kubespray it’s keep crashing on 2 vms (from 6 total) and stopping on the step: kubernetes/control-plane : Joining control plane node to the cluster
now I can’t figure out what is wrong on those 2, they look fine, and I tested the other and kubectl get nodes saying nothing available.
Here’s some logs from crash site:
TASK [kubernetes/control-plane : Joining control plane node to the cluster.] ************************************************************************************
fatal: [k8s-cl3-m2]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["/usr/local/bin/kubeadm", "join", "--config", "/etc/kubernetes/kubeadm-controlplane.yaml", "--ignore-preflight-errors=all", "--skip-phases="], "delta": "0:05:00.106187", "end": "2022-07-07 16:16:51.666891", "msg": "non-zero return code", "rc": 1, "start": "2022-07-07 16:11:51.560704", "stderr": "error execution phase preflight: couldn't validate the identity of the API Server: configmaps \"cluster-info\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-public\"\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["error execution phase preflight: couldn't validate the identity of the API Server: configmaps \"cluster-info\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-public\"", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "[preflight] Running pre-flight checks", "stdout_lines": ["[preflight] Running pre-flight checks"]}
FAILED - RETRYING: [k8s-cl3-m3]: Joining control plane node to the cluster. (3 retries left).
FAILED - RETRYING: [k8s-cl3-m3]: Joining control plane node to the cluster. (2 retries left).
FAILED - RETRYING: [k8s-cl3-m3]: Joining control plane node to the cluster. (1 retries left).
fatal: [k8s-cl3-m3]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["/usr/local/bin/kubeadm", "join", "--config", "/etc/kubernetes/kubeadm-controlplane.yaml", "--ignore-preflight-errors=all", "--skip-phases="], "delta": "0:05:00.112363", "end": "2022-07-07 16:37:15.186975", "msg": "non-zero return code", "rc": 1, "start": "2022-07-07 16:32:15.074612", "stderr": "error execution phase preflight: couldn't validate the identity of the API Server: configmaps \"cluster-info\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-public\"\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["error execution phase preflight: couldn't validate the identity of the API Server: configmaps \"cluster-info\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-public\"", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "[preflight] Running pre-flight checks", "stdout_lines": ["[preflight] Running pre-flight checks"]}
NO MORE HOSTS LEFT **********************************************************************************************************************************************
PLAY RECAP ******************************************************************************************************************************************************
k8s-cl3-m1 : ok=601 changed=14 unreachable=0 failed=0 skipped=734 rescued=0 ignored=3
k8s-cl3-m2 : ok=558 changed=37 unreachable=0 failed=1 skipped=618 rescued=0 ignored=3
k8s-cl3-m3 : ok=558 changed=36 unreachable=0 failed=1 skipped=618 rescued=0 ignored=3
k8s-cl3-w1 : ok=426 changed=8 unreachable=0 failed=0 skipped=490 rescued=0 ignored=2
k8s-cl3-w2 : ok=426 changed=8 unreachable=0 failed=0 skipped=490 rescued=0 ignored=2
k8s-cl3-w3 : ok=426 changed=8 unreachable=0 failed=0 skipped=490 rescued=0 ignored=2
Few logs back:
TASK [kubernetes/control-plane : Check which kube-control nodes are already members of the cluster] *************************************************************
fatal: [k8s-cl3-m1]: FAILED! => {"changed": false, "cmd": ["/usr/local/bin/kubectl", "get", "nodes", "--selector=node-role.kubernetes.io/control-plane", "-o", "json"], "delta": "0:00:00.035690", "end": "2022-07-07 15:55:35.890984", "msg": "non-zero return code", "rc": 1, "start": "2022-07-07 15:55:35.855294", "stderr": "The connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server localhost:8080 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}
...ignoring
fatal: [k8s-cl3-m3]: FAILED! => {"changed": false, "cmd": ["/usr/local/bin/kubectl", "get", "nodes", "--selector=node-role.kubernetes.io/control-plane", "-o", "json"], "delta": "0:00:00.047637", "end": "2022-07-07 15:55:39.480390", "msg": "non-zero return code", "rc": 1, "start": "2022-07-07 15:55:39.432753", "stderr": "The connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server localhost:8080 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}
...ignoring
fatal: [k8s-cl3-m2]: FAILED! => {"changed": false, "cmd": ["/usr/local/bin/kubectl", "get", "nodes", "--selector=node-role.kubernetes.io/control-plane", "-o", "json"], "delta": "0:00:00.039208", "end": "2022-07-07 15:55:40.221929", "msg": "non-zero return code", "rc": 1, "start": "2022-07-07 15:55:40.182721", "stderr": "The connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server localhost:8080 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}
...ignoring
Anyone can give me a point to start with?