Kubespray stopping on kubernetes/control-plane : Joining control plane node to the cluster

Cluster information:

Kubernetes version: v1.20.4
Cloud being used: bare-metal
Installation method: kubespray
Host OS: Ubuntu OS
CNI and version: ?
CRI and version: ?

I’m using kubespray to install new cluster, and it’s going pretty good, most of the steps are behined me ( I was able to fix all the missing packages on my offline network ). But after running kubespray it’s keep crashing on 2 vms (from 6 total) and stopping on the step: kubernetes/control-plane : Joining control plane node to the cluster now I can’t figure out what is wrong on those 2, they look fine, and I tested the other and kubectl get nodes saying nothing available.
Here’s some logs from crash site:


TASK [kubernetes/control-plane : Joining control plane node to the cluster.] ************************************************************************************
fatal: [k8s-cl3-m2]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["/usr/local/bin/kubeadm", "join", "--config", "/etc/kubernetes/kubeadm-controlplane.yaml", "--ignore-preflight-errors=all", "--skip-phases="], "delta": "0:05:00.106187", "end": "2022-07-07 16:16:51.666891", "msg": "non-zero return code", "rc": 1, "start": "2022-07-07 16:11:51.560704", "stderr": "error execution phase preflight: couldn't validate the identity of the API Server: configmaps \"cluster-info\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-public\"\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["error execution phase preflight: couldn't validate the identity of the API Server: configmaps \"cluster-info\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-public\"", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "[preflight] Running pre-flight checks", "stdout_lines": ["[preflight] Running pre-flight checks"]}
FAILED - RETRYING: [k8s-cl3-m3]: Joining control plane node to the cluster. (3 retries left).

FAILED - RETRYING: [k8s-cl3-m3]: Joining control plane node to the cluster. (2 retries left).
FAILED - RETRYING: [k8s-cl3-m3]: Joining control plane node to the cluster. (1 retries left).
fatal: [k8s-cl3-m3]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["/usr/local/bin/kubeadm", "join", "--config", "/etc/kubernetes/kubeadm-controlplane.yaml", "--ignore-preflight-errors=all", "--skip-phases="], "delta": "0:05:00.112363", "end": "2022-07-07 16:37:15.186975", "msg": "non-zero return code", "rc": 1, "start": "2022-07-07 16:32:15.074612", "stderr": "error execution phase preflight: couldn't validate the identity of the API Server: configmaps \"cluster-info\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-public\"\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["error execution phase preflight: couldn't validate the identity of the API Server: configmaps \"cluster-info\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-public\"", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "[preflight] Running pre-flight checks", "stdout_lines": ["[preflight] Running pre-flight checks"]}

NO MORE HOSTS LEFT **********************************************************************************************************************************************

PLAY RECAP ******************************************************************************************************************************************************
k8s-cl3-m1                 : ok=601  changed=14   unreachable=0    failed=0    skipped=734  rescued=0    ignored=3   
k8s-cl3-m2                 : ok=558  changed=37   unreachable=0    failed=1    skipped=618  rescued=0    ignored=3   
k8s-cl3-m3                 : ok=558  changed=36   unreachable=0    failed=1    skipped=618  rescued=0    ignored=3   
k8s-cl3-w1                 : ok=426  changed=8    unreachable=0    failed=0    skipped=490  rescued=0    ignored=2   
k8s-cl3-w2                 : ok=426  changed=8    unreachable=0    failed=0    skipped=490  rescued=0    ignored=2   
k8s-cl3-w3                 : ok=426  changed=8    unreachable=0    failed=0    skipped=490  rescued=0    ignored=2

Few logs back:

TASK [kubernetes/control-plane : Check which kube-control nodes are already members of the cluster] *************************************************************
fatal: [k8s-cl3-m1]: FAILED! => {"changed": false, "cmd": ["/usr/local/bin/kubectl", "get", "nodes", "--selector=node-role.kubernetes.io/control-plane", "-o", "json"], "delta": "0:00:00.035690", "end": "2022-07-07 15:55:35.890984", "msg": "non-zero return code", "rc": 1, "start": "2022-07-07 15:55:35.855294", "stderr": "The connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server localhost:8080 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}
...ignoring
fatal: [k8s-cl3-m3]: FAILED! => {"changed": false, "cmd": ["/usr/local/bin/kubectl", "get", "nodes", "--selector=node-role.kubernetes.io/control-plane", "-o", "json"], "delta": "0:00:00.047637", "end": "2022-07-07 15:55:39.480390", "msg": "non-zero return code", "rc": 1, "start": "2022-07-07 15:55:39.432753", "stderr": "The connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server localhost:8080 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}
...ignoring
fatal: [k8s-cl3-m2]: FAILED! => {"changed": false, "cmd": ["/usr/local/bin/kubectl", "get", "nodes", "--selector=node-role.kubernetes.io/control-plane", "-o", "json"], "delta": "0:00:00.039208", "end": "2022-07-07 15:55:40.221929", "msg": "non-zero return code", "rc": 1, "start": "2022-07-07 15:55:40.182721", "stderr": "The connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server localhost:8080 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}
...ignoring

Anyone can give me a point to start with?