Metrics for preemptions / evictions

General Discussions
1

I’m trying to improve monitoring of pod evictions / preemptions in our cluster. I see the metrics (even in 1.17.9 which we’re using in AKS) here:

However, these metrics don’t show up when I port-forward to prometheus and go to:
http://localhost:9090/api/v1/label/__name__/values

I have seen pods be preempted but these metrics are still missing.

Any ideas where things might be breaking down? The kubelet target is showing 6/6 up for 4 targets for kubelet:

Cluster information:

Kubernetes version: 1.17.9
Cloud being used: AKS
Installation method: ARM template

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

2

Would you try my test case in Quotas for Ephemeral Storage · Issue #1029 · kubernetes/enhancements · GitHub?

# HELP kubelet_evictions [ALPHA] Cumulative number of pod evictions by eviction signal
# TYPE kubelet_evictions counter
kubelet_evictions{eviction_signal="emptydirfs.limit"} 2

I get EvictionsKey but no eviction_stats_age_seconds.