Prometheus adapter HPA can't find custom metric

General Discussions
, ,
1

Kubernetes version:v1.25.12

I install promtheus server use yaml.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  namespace: prometheus
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    scrape_configs:
      - job_name: 'hello-app'
        static_configs:
          - targets: ['ip:port']
      - job_name: 'sample-app'
        static_configs:
          - targets: ['ip:port']

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-server
  namespace: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: config-volume
              mountPath: /etc/prometheus
      volumes:
        - name: config-volume
          configMap:
            name: prometheus-server-conf
            defaultMode: 420

I install prometheus adapter use helm.
helm install my-release prometheus-community/prometheus-adapter
then I change the prometheus_url in deployment. And update rules reference prometheus-adapter/docs/walkthrough.md at master · kubernetes-sigs/prometheus-adapter · GitHub . And rollout restart the dp.

apiVersion: v1
data:
  config.yaml: |
    rules:
    - seriesQuery: |
        {namespace!="",__name__!~"^container_.*"}
      resources:
        "template": "<<.Resource>>"
      name:
        "matches": "^(.*)_total"
        "as": ""
      metricsQuery: |
        sum by (<<.GroupBy>>) (
          irate (
            <<.Series>>{<<.LabelMatchers>>}[1m]
          )
        )
kind: ConfigMap

I can get the metric in prometheus.

image
image1872×675 44.5 KB

but I can’t get metric from apis.

 kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/" |jq   
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": []
}

Then I try updating the main.py in hello-pod which is for demo. I add the metric labels namespace and pod, and hard code the value namespace=default and pod=hello-pod. After this I get the callback:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/" |jq
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "jobs.batch/app_request_count",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "namespaces/app_request_count",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "pods/app_request_count",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "instances.kudo.dev/app_request_count",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

# but why this get no data?
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/app_request_count" |jq        
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": []
}

# and this get data
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/hello-pod/app_request_count" |jq                            
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "hello-pod",
        "apiVersion": "/v1"
      },
      "metricName": "app_request_count",
      "timestamp": "2024-05-30T09:10:31Z",
      "value": "5333m",
      "selector": null
    }
  ]
}

I trying create HPA but not work.

# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: hello-hpa 
  namespace: default
spec:
  minReplicas: 1
  maxReplicas: 3
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hello-dp
  metrics:
  - type: Pods
    pods:
      metric:
        name: app_request_count
      target:
        type: AverageValue
        averageValue: 4

  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric app_request_count: no metrics returned from custom metrics API
Events:
  Type     Reason                        Age               From                       Message
  ----     ------                        ----              ----                       -------
  Warning  FailedGetPodsMetric           1s (x6 over 76s)  horizontal-pod-autoscaler  unable to get metric app_request_count: no metrics returned from custom metrics API
  Warning  FailedComputeMetricsReplicas  1s (x6 over 76s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric app_request_count: no metrics returned from custom metrics API