How to get metric-server work without setting --kubelet-insecure-tls=true

General Discussions
1

I have a k8s 1.24 cluster built the hard way. Here is my setup
PKI Infra:
Kube-CA
All certs locally signed by Kube-CA including the Kubelet.

Created front-proxy-ca
Created certs as mentioned in kubernetes-the-hardway-virtualbox/15.Deploy-Metric-Server.md at master · ansilh/kubernetes-the-hardway-virtualbox · GitHub

Added to api-server and everything works as expected. Installed metrics-server using kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

It starts complaining about Certificate error:

E0403 01:39:41.376984       1 scraper.go:140] "Failed to scrape node" err="Get \"https://10.209.46.102:10250/metrics/resource\": x509: cannot validate certificate for 10.209.46.102 because it doesn't contain any IP SANs" node="worker003"

Had to disable cert checks with --kubelet-insecure-tls=true

Tried adding --kubelet-certificate-authority=/certs/ca.crt and later --requestheader-client-ca-file=/certs/front-proxy-ca.pem nothing looks to be working.

Wondering how hosted service providers can get everything going just with latest metric-server yaml applied without any changes to it? What am I missing?

2

Hi, did you solve it?

3

Not yet. Still waiting for a solution

4

I think you need to start the kubelet with --client-ca flag. Look here Kubelet authentication/authorization

5

I know it’s a long-time coming, but here is a solution that worked for me while fixing the problem in my self-managed k8s.
The solution:

Update the kubelet configuration with serverTLSBootstrap to use a cert that includes node IP in SAN. It enables the automated bootstrapping of the Kubelet’s serving certificate by requesting it from the Kubernetes API server.

Edit your kubelet configuration yaml, generally in /var/lib/kubelet/config.yaml and add the following line:

serverTLSBootstrap: true

If you are unsure, check with the kubelet service:

systemctl status kubelet | grep config

Get the path to your kubelet configuration yaml and add the line above.

After that, restart the kubelet service:

systemctl restart kubelet

Follow this on all nodes in your cluster.

On the control-plane, use the kubectl commands to approve the new certificates:

kubectl get csr
kubectl certificate approve <csr-name>

Once the new certs are approved, you may restart the kubelets again on all the nodes, including control-plane.

You should see the metric server now able to get the resource metrics with kubectl top nodes and kubectl top pods.

The problem in my case was:

  • Metrics Server scraped kubelet at https://:10250.
E0226 11:12:23.673917       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.128.0.20:10250/metrics/resource\": tls: failed to verify certificate: x509: cannot validate certificate for 10.128.0.20 because it doesn't contain any IP SANs" node="k8s-opl"
E0226 11:12:23.676050       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.128.0.108:10250/metrics/resource\": tls: failed to verify certificate: x509: cannot validate certificate for 10.128.0.108 because it doesn't contain any IP SANs" node="opl-20250702-100505"
111

  • Kubelet presented a cert that did not include node IP in SAN.

  • TLS verification failed.