HPA based Datadog metric

General Discussions
1

Hey everyone, I am last days reading and using HPA based Datadog metrics.
Under I put documentations

This is my Datadogmetric

apiVersion: datadoghq.com/v1alpha1
kind: DatadogMetric
metadata:
  name: deliver-rate
  namespace: beta
spec:
  query: rabbitmq.queue.messages.deliver.rate{environment:production,rabbitmq_queue:enqueue.x.callable}.as_count()

and this one is my HPA

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: test-beta2
  namespace: beta
spec:
  minReplicas: 1
  maxReplicas: 8
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 10
      policies:
      - type: Pods
        value: 2
        periodSeconds: 2
    scaleUp:
      stabilizationWindowSeconds: 10
      policies:
      - type: Pods
        value: 2
        periodSeconds: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: dealroom-x
  metrics:
    - type: External
      external:
        metric:
          name: 'datadogmetric@beta:deliver-rate'
        target:
          type: Value
          value: 200

I got two problems

1.

And all worked but didn’t in the right way scaling up each time when metric value upper than the defined target value. For example, under I will put the metricvalue/targetvalue and appropriate desiredReplicas

desiredReplicas: 4   252/200
desiredReplicas: 6   252/200
desiredReplicas: 6   252/200
desiredReplicas: 5   240/200
desiredReplicas: 6   240/200
desiredReplicas: 6   240/200
desiredReplicas: 8   240/200

Why the autoscaler adding pods more than 2

  behavior:
    scaleDown:
      stabilizationWindowSeconds: 10
      policies:
      - type: Pods
        value: 2
        periodSeconds: 2
    scaleUp:
      stabilizationWindowSeconds: 10
      policies:
      - type: Pods
        value: 2
        periodSeconds: 2

2.

Periodically I saw that datadogmetric’s validation is false, but at the same time datadogmetric fetched metric and showed the value

deliver-rate True False 252 beta/test-beta2 9s

this is the error message

HPA controller was able to get the target''s current scale"},{"type":"ScalingActive","status":"False","lastTransitionTime":"2021-05-15T14:32:01Z","reason":"FailedGetExternalMetric","message":"the
      HPA was unable to compute the replica count: unable to get external metric beta/datadogmetric@beta:deliver-rate/nil:
      unable to fetch metrics from external metrics API: Internal error occurred:
      DatadogMetric is invalid, err: Outdated result from backend, query: rabbitmq.queue.messages.deliver.rate{environment:production,rabbitmq_queue:enqueue.dealroom-php.callable}.as_count()"},{"type":"ScalingLimited","status":"True","lastTransitionTime":"2021-05-15T14:30:57Z","reason":"TooManyReplicas","message":"the
      desired replica count is more than the maximum replica count"}]'

Who can help me to solve these 2 problems?