Unable to scale pods based on https requests

General Discussions
1

Hi I am using public cloud [Microsoft Azure] with kubernetes version 1.19.11
I am trying to implement scaling of pods based on http requests. I am following this link https://dev.to/mraszplewicz/horizontal-pod-autoscaling-based-on-http-requests-metric-from-istio-cc3. I have deployed the hpa.yaml file. But the problem is that horizontal pod autoscaler is unable to fetch custom metrics from api. I am getting the below error.

 kubectl describe hpa -n dev
Name:                                   httpbin
Namespace:                              dev
Labels:                                 <none>
Annotations:                            <none>
CreationTimestamp:                      Tue, 29 Jun 2021 14:55:38 +0000
Reference:                              Deployment/httpbin
Metrics:                                ( current / target )
  "istio_requests_per_second" on pods:  <unknown> / 10
Min replicas:                           1
Max replicas:                           5
Deployment pods:                        1 current / 0 desired
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric istio_requests_per_second: unable to fetch metrics from custom metrics API: the server could not find the metric istio_requests_per_second for pods
Events:
  Type     Reason                        Age                   From                       Message
  ----     ------                        ----                  ----                       -------
  Warning  FailedComputeMetricsReplicas  13h (x21 over 13h)    horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric istio_requests_per_second: unable to fetch metrics from custom metrics API: the server could not find the metric istio_requests_per_second for pods
  Warning  FailedGetPodsMetric           13h (x41 over 13h)    horizontal-pod-autoscaler  unable to get metric istio_requests_per_second: unable to fetch metrics from custom metrics API: the server could not find the metric istio_requests_per_second for pods
  Warning  FailedGetPodsMetric           34m (x12 over 36m)    horizontal-pod-autoscaler  unable to get metric istio_requests_per_second: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)
  Warning  FailedComputeMetricsReplicas  34m (x12 over 36m)    horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric istio_requests_per_second: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)
  Warning  FailedComputeMetricsReplicas  21m (x49 over 33m)    horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric istio_requests_per_second: unable to fetch metrics from custom metrics API: the server could not find the metric istio_requests_per_second for pods
  Warning  FailedGetPodsMetric           113s (x128 over 33m)  horizontal-pod-autoscaler  unable to get metric istio_requests_per_second: unable to fetch metrics from custom metrics API: the server could not find the metric istio_requests_per_second for pods

@protosam @mrbobbytables Any help would be appreciated and let me know if any additional info is required.
Thanks in advance.

2

I have not implemented HPAs yet. Best I can recommend is checking out this documentation and aiming to understand how the feature works. The official documentation has this walkthrough that shows using a CPU metric. Edit, there’s a couple other metric examples too.

At a glance it just looks like it’s a matter of aggregating metrics and leveraging those metrics effectively. Troubleshooting would involve knowing your metrics source I assume.