HPA using a prometheus custom metric published by another pod

General Discussions
1

I have a need to scale on aws cloudwatch metrics.

I have a workload in my namespace that exports cloudwatch metrics such as SQS queue available messages and publishes that for all the queues that I am interested in. Went with this approach so that each microservice does not need to update their code to publish custom Prometheus stats. We anticipate using this to fetch other AWS cloudwatch stats to do scaling of workloads in K8s.

  1. I have confirmed that the metrics are being published and are available in kubernetes prometheus and to prometheu-adapter using the kubeclt --raw command
   {
      "name": "pods/aws_sqs_approximate_number_of_messages_visible_average",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },

Here’s the metric in prometheus dashboard.

aws_sqs_approximate_number_of_messages_visible_average{container="cloudwatch-metrics-exporter", endpoint="9106", exported_job="aws_sqs", instance="100.64.99.186:9106", job="cloudwatch-metrics-exporter", namespace="my-custom-namespace", pod="cloudwatch-metrics-exporter-deployment-8497bbfd4d-x5c7b", queue_name="my-test-queue.fifo", service="cloudwatch-metrics-exporter"}

  1. When I configure the HPA, I tried using the Pods metric type, but I think k8s fetches the above metric with filter for namespace and pods it is trying to scale, so this metrics is not available for those pods. (It kept failing to get metrics when I used Pods metric type in HPA definition). PS I have other HPA that scale on JVM threads and CPU usage and that works fine as long as the pod is publishing the JVM threads it is using.

  2. So attempted to use Object metric type and can’t seem to figure out how to specify the describeObject and I am not 100% sure if I can even do this type of metric, since this is not a kubernetes Object. I keep getting unable to fetch metrics in the events and in the status of the HPA.

  - type: Object
    object:
      describedObject:
        apiVersion: custom.metrics.k8s.io/v1beta1
        kind: MetricValueList
        name: aws_sqs_approximate_number_of_messages_visible_average
      metric:
        name: aws_sqs_approximate_number_of_messages_visible_average
      target:
        type: AverageValue
        averageValue: 1000

Is it possible to even scale using custom metrics of another pod? also was this ever possible?

Cluster information:

Kubernetes version: v1.30.8-eks-2d5f26
Cloud being used: AWS -EKS
Installation method: Terraform
Host OS: Rocky Linux
CNI and version: <Don’t have cluster level acces>
CRI and version: <Don’t have cluster level acces>

2

Ok, I think this way of describing the describedObject worked.

- type: Object
    object:
      describedObject:
        apiVersion: v1
        kind: service
        name: cloudwatch-metrics-exporter
      metric:
        name: aws_sqs_approximate_number_of_messages_visible_average
        selector: {matchLabels: {queue_name: my-queue.fifo}}
      target:
        type: AverageValue
        averageValue: 1000