How to get following metrics from k8s: Total and free cpus/mem in the cluster?

General Discussions
1

Hi,
We use a bespoke cluster autoscaler to scale our mesos clusters up and down. We do it based on total and available(free) cpus/mem in the cluster. Does k8s expose this? I read the docs, but could find any information pertaining to it.

Thanks,
Sagar

2

Kubectl describe node shows the cpu request and limit, same for mem, for each node.

Does that help? :slight_smile:

3

Sounds like you would like to know metrics about the cluster itself.
The metrics-server will make those available for consumption with a tool of your choice.
https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/#metrics-server

4

@rata, @RiRa12621, thanks for your reply!

I am looking to fetch these metrics in a script, so metrics-server seems like a better option.

My understanding is that metrics-server uses ‘stats/summary’ API of kubelet to get metrics. Looks like Kubelet API provides metrics pertaining to the actual cpu usage. I am looking for the cpu/mem reservation information. Am I missing something?

5

We use prometheus metrics to get the similar view, cadvisor metrics https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L102 combined with https://github.com/kubernetes/kube-state-metrics provides plenty of the information.

container_spec_cpu_shares from kubelet cadvisor can be used to get how much allocated and kube_node_status_allocatable_cpu_cores from kube-state-metrics provides how much available in the cluster.

6

@vnandha Thanks for sharing the information. https://github.com/kubernetes/kube-state-metrics/blob/master/internal/collector/node.go has all the metrics I was looking for. I feel these metrics are really critical from cluster administrator’s point of view and should be a little bit easier to discover.

7

@sagarp We also presented our work at Kubecon Copenhagen about complete monitoring suite and dashboards we build that are really crucial for day to day operation. You can check out the video and some links to dashboard source
Federated Prometheus Monitoring at Scale - Nandhakumar Venkatachalam & LungChih Tung ( https://www.youtube.com/watch?v=Zm43VM7nKdU )