List of prometheus metrics endpoints

General Discussions
1

Hi,

I’m looking to monitor a production kubernetes cluster with prometheus. I have a pretty solid grasp on prometheus - I have been using it for a while for monitoring various devices with node_exporter, snmp_exporter etc. I also found kubernetes_sd in prometheus and it seems it can discover nodes and pods via the k8s API.

However, I’d like to know where the actual metrics endpoints are. This is so I can scrape them manually with curl, understand what metrics are available, and decide which ones to include/exclude. What’s difficult is finding a simple list of the actual URLs!

For testing purposes I’m using microk8s. What I’ve discovered so far by poking around:

  1. There is a /metrics path at the main API endpoint (microk8s: https port 16443). This returns about 12,000 metrics, mainly the response times for the API and etcd split over lots of buckets:

    # curl -sk https://admin:<password>@localhost:16443/metrics

  2. There is an endpoint /api/v1/nodes/<nodename>/proxy/metrics, with kubelet stats, rest client stats

    There seems to be very little of interest there unless you’re debugging kubelet performance: looks like I still need node_exporter for full host stats.

  3. There is an endpoint /api/v1/namespaces/<namespace>/pods/<pod>/proxy/metrics, with things like:

    process_cpu_seconds_total 524.6
process_max_fds 65536
process_open_fds 24
process_resident_memory_bytes 2.7062272e+07
process_start_time_seconds 1.57953723378e+09
process_virtual_memory_bytes 1.33914624e+08
process_virtual_memory_max_bytes -1

    That looks to be very interesting at pod level. Each pod needs scraping separately though (kubernetes_sd will help there)

  4. I’ve seen mention of “cAdvisor” metrics collected by kubelet - and after more browsing I finally stumbled upon /api/v1/nodes/<nodename>/proxy/metrics/cadvisor

    Despite the metrics being prefixed container_*, this appears to be node-level info, e.g. counts of network bytes per interface, information on filesystems - all metrics are labelled with {container=""}.

    (Trying /api/v1/namespaces/<namespace>/pods/<pod>/proxy/metrics/cadvisor gives a 404)

    Note that microk8s uses containerd, not docker, as its runtime.

  5. If I install the metrics-server package (via microk8s.enable metrics-server) then I can find very basic node and pod metrics at /apis/metics.k8s.io/v1beta1/nodes/ and /apis/metics.k8s.io/v1beta1/pods/ in a single call - but these are JSON, not prometheus. The per-pod metrics are even fewer than the process_* ones shown above:

      "containers": [
    {
      "name": "speaker",
      "usage": {
        "cpu": "3m",
        "memory": "8012Ki"
      }
    }
  ]

  6. If I install kube-state-metrics (via kubectl apply -f examples/standard in the kube-state-metrics repo) then it installs a new service with a /metrics endpoint on port 8080. To get to it temporarily from outside the cluster I did:

    kubectl port-forward -n kube-system service/kube-state-metrics 8111:8080
...
curl localhost:8111/metrics

    Here I see info on the status of deployments, daemonsets, configmaps, services etc, and some resource info (kube_pod_container_resource_limits, kube_pod_container_resource_requests)

Questions:

  • Are there other core metrics that I’ve not listed above? Or a comprehensive list somewhere?
  • In particular, are there any more metrics under the /proxy/metrics/ prefix? I only discovered /proxy/metrics/cadvisor via a config I saw in a github issue.

Many thanks,

Brian.

UPDATE: on microk8s I’ve also discovered I can get some of these metrics directly from kubelet using <host>:10255/metrics and <host>:10255/metrics/cadvisor.

2

Hi Brian. Very useful. Thanks for sharing your knowledge.

I’ve also found “example” manifests for Prometeus operator (ServiceMonitors) with some endpoints: https://github.com/coreos/kube-prometheus/tree/master/manifests For example node-exporter installed as DeamonSet can monitor your nodes.

Metric Server - scrapes some K8s endpoints and exposes metrics in different format, more details: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/metrics-server.md

Regards,
Tomasz Prus

3

HI, I see you mentioned you have used node exporters. I need some help on how to scrape the node-exporter targets. I have got the node-exporter as part of VMWare TKZ grid and I have installed the Prometheus via helm chart and disable the node-exporter since I have one already.
After Prometheus is successfully running, I have created a service monitor to discover the node-exporter name space and its path /metrics. I can see the service discovery and it detected the 4 nodes but when I see the target it shows 0/4.

I have tried to add scrap_config via helm addition_scrapconfig but it was not detected the targets, so tried to exec the pod and thought of editing manually "targets: [“localhost:9100]”. But I have read-only option there.

I am new to Prometheus installations trying to get some help here.

Servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: node-exporter
namespace: prometheus-grafana
labels:
app: node-exporter
release: kube-prometheus-stack
spec:
jobLabel: node-exporter
static_configs:
- targets:
- localhost: 9100
namespaceSelector:
matchNames:
- demo
selector:
matchLabels:
app: node-exporter
release: kube-prometheus-stack
endpoints:

  • port: node-metrics
    path: /metrics

Prometheus /metrics output:
prometheus_sd_discovered_targets{config=“pks-system/node-exporter/0”,name=“scrape”} 9
prometheus_sd_discovered_targets{config=“prometheus-grafana/node-exporter/0”,name=“scrape”} 9
prometheus_target_metadata_cache_bytes{scrape_job=“pks-system/node-exporter/0”} 31443
prometheus_target_metadata_cache_bytes{scrape_job=“prometheus-grafana/node-exporter/0”} 31443
prometheus_target_metadata_cache_entries{scrape_job=“pks-system/node-exporter/0”} 684
prometheus_target_metadata_cache_entries{scrape_job=“prometheus-grafana/node-exporter/0”} 684
prometheus_target_scrape_pool_sync_total{scrape_job=“pks-system/node-exporter/0”} 53
prometheus_target_scrape_pool_sync_total{scrape_job=“prometheus-grafana/node-exporter/0”} 53
prometheus_target_scrape_pool_targets{scrape_job=“pks-system/node-exporter/0”} 4