I’m trying to improve monitoring of pod evictions / preemptions in our cluster. I see the metrics (even in 1.17.9 which we’re using in AKS) here:
However, these metrics don’t show up when I port-forward to prometheus and go to:
http://localhost:9090/api/v1/label/__name__/values
I have seen pods be preempted but these metrics are still missing.
Any ideas where things might be breaking down? The kubelet target is showing 6/6 up for 4 targets for kubelet:
- https://10.240.0.4:10250/metrics
- https://10.240.0.4:10250/metrics/cadvisor
- https://10.240.0.4:10250/metrics/probes
- https://10.240.0.4:10250/metrics/resource/v1alpha1
Cluster information:
Kubernetes version: 1.17.9
Cloud being used: AKS
Installation method: ARM template
You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.