Getting metrics from Metrics-server with a specific window parameter

General Discussions
1

I need to get the following metrics from the CLI in Kubernetes:

  • (MIN, MAX, AVG) of (CPU/MEMORY) for the API server within a specific window of time

This is for measuring the performance of the API server. I have a script which will make K8s API calls which I then need to get the metrics listed above within the time window that the script has run.

I have Prometheus and Grafana running as well. I believe there is no way to curl metrics from the CLI with Grafana. I’ve been trying with the metrics-server and Prometheus but I don’t know which one will give me the results I need.

This is what I’ve been using for the metrics-server:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-master-0.ocp9.pd.f5net.com" | jq

Here is the output:

{
  "kind": "PodMetrics",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "name": "kube-apiserver-master-0.ocp9.pd.f5net.com",
    "namespace": "openshift-kube-apiserver",
    "creationTimestamp": "2023-05-26T00:50:12Z",
    "labels": {
      "apiserver": "true",
      "app": "openshift-kube-apiserver",
      "revision": "18"
    }
  },
  "timestamp": "2023-05-26T00:50:12Z",
  "window": "5m0s",
  "containers": [
    {
      "name": "kube-apiserver-cert-syncer",
      "usage": {
        "cpu": "0",
        "memory": "24848Ki"
      }
    },
    {
      "name": "kube-apiserver-check-endpoints",
      "usage": {
        "cpu": "2m",
        "memory": "48964Ki"
      }
    },
    {
      "name": "kube-apiserver-insecure-readyz",
      "usage": {
        "cpu": "0",
        "memory": "13104Ki"
      }
    },
    {
      "name": "kube-apiserver",
      "usage": {
        "cpu": "292m",
        "memory": "3591000Ki"
      }
    },
    {
      "name": "kube-apiserver-cert-regeneration-controller",
      "usage": {
        "cpu": "3m",
        "memory": "34724Ki"
      }
    }
  ]
}

As you can see, the default ‘window’ is above 5m0s. How can I pass in my own value for ‘window’? Where can I find more information on the metrics parameters I can retrieve?

This is what I’ve been using for Prometheus:

curl -s "http://10.192.248.10:30000/api/v1/query?query=apiserver_delegated_authz_request_duration_seconds_sum" | jq

With the output:

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "apiserver_delegated_authz_request_duration_seconds_sum",
          "beta_kubernetes_io_arch": "amd64",
          "beta_kubernetes_io_os": "linux",
          "code": "201",
          "feature_node_kubernetes_io_network_sriov_capable": "true",
          "instance": "master-0.ocp9.pd.f5net.com",
          "job": "kubernetes-nodes",
          "kubernetes_io_arch": "amd64",
          "kubernetes_io_hostname": "master-0.ocp9.pd.f5net.com",
          "kubernetes_io_os": "linux",
          "node_openshift_io_os_id": "rhcos"
        },
        "value": [
          1685125713.993,
          "28.36041276300015"
        ]
      },
      {
        "metric": {
          "__name__": "apiserver_delegated_authz_request_duration_seconds_sum",
          "beta_kubernetes_io_arch": "amd64",
          "beta_kubernetes_io_os": "linux",
          "code": "201",
          "feature_node_kubernetes_io_network_sriov_capable": "true",
          "instance": "master-1.ocp9.pd.f5net.com",
          "job": "kubernetes-nodes",
          "kubernetes_io_arch": "amd64",
          "kubernetes_io_hostname": "master-1.ocp9.pd.f5net.com",
          "kubernetes_io_os": "linux",
          "node_openshift_io_os_id": "rhcos"
        },
        "value": [
          1685125713.993,
          "27.947552793000128"
        ]
      },
      {
        "metric": {
          "__name__": "apiserver_delegated_authz_request_duration_seconds_sum",
          "beta_kubernetes_io_arch": "amd64",
          "beta_kubernetes_io_os": "linux",
          "code": "201",
          "feature_node_kubernetes_io_network_sriov_capable": "true",
          "instance": "master-2.ocp9.pd.f5net.com",
          "job": "kubernetes-nodes",
          "kubernetes_io_arch": "amd64",
          "kubernetes_io_hostname": "master-2.ocp9.pd.f5net.com",
          "kubernetes_io_os": "linux",
          "node_openshift_io_os_id": "rhcos"
        },
        "value": [
          1685125713.993,
          "27.491909364000108"
        ]
      }
    ]
  }
}

Now for the above Prometheus command, you can also pass in the start and end time, and step parameters into the query. Here is an example of what it would look like:

&start=2023-05-25T03:52:15Z&end=2023-05-26T03:52:15Z&step=15s

and the whole Prometheus command will look like this:

curl -s "http://10.192.248.10:30000/api/v1/query?query=apiserver_delegated_authz_request_duration_seconds_sum&start=2023-05-25T03:52:15Z&end=2023-05-26T03:52:15Z&step=15s" | jq

When I run these two commands I get the same json output, and it doesn’t seem to query based on the time window provided.

Let me know of any questions, or if I need to explain anything in further detail.