Kubelete tls hnadshake errors even though certs are good

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: 1.20
Cloud being used: VMWare
Installation method: kubeadm
Host OS: CentOS
CNI and version: containerd
CRI and version:

I am seeing kubelet errors in messages logs:

Jan 15 11:57:12 nalshsvrk8ss06 kubelet[27453]: I0115 11:57:12.283407 27453 log.go:181] http: TLS handshake error from 172.16.137.70:34756: remote error: tls: bad certificate
Jan 15 11:57:12 nalshsvrk8ss06 kubelet[27453]: I0115 11:57:12.290390 27453 log.go:181] http: TLS handshake error from 172.16.137.70:34760: remote error: tls: bad certificate

Also:
NAME STATUS MESSAGE ERROR
scheduler Unhealthy Get “http://127.0.0.1:10251/healthz”: dial tcp 127.0.0.1:10251: connect: connection refused
controller-manager Unhealthy Get “http://127.0.0.1:10252/healthz”: dial tcp 127.0.0.1:10252: connect: connection refused
etcd-0 Healthy {“health”:“true”}

CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jan 15, 2022 13:24 UTC 364d no
apiserver Jan 15, 2022 13:24 UTC 364d ca no
apiserver-etcd-client Jan 15, 2022 13:24 UTC 364d etcd-ca no
apiserver-kubelet-client Jan 15, 2022 13:24 UTC 364d ca no
controller-manager.conf Jan 15, 2022 13:24 UTC 364d no
etcd-healthcheck-client Dec 28, 2021 15:48 UTC 346d etcd-ca no
etcd-peer Dec 28, 2021 15:48 UTC 346d etcd-ca no
etcd-server Dec 28, 2021 15:48 UTC 346d etcd-ca no
front-proxy-client Jan 15, 2022 13:24 UTC 364d front-proxy-ca no
scheduler.conf Jan 15, 2022 13:24 UTC 364d no

CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Dec 26, 2030 15:48 UTC 9y no
etcd-ca Dec 26, 2030 15:48 UTC 9y no
front-proxy-ca Dec 26, 2030 15:48 UTC 9y no