We are trying to use HPA to scale our system based on load and we noticed that HPA works correctly in creating new pods. but our main issue is that there is no fair load balancing between the pods, and the old pods handles most of the requests as in the attached image.
As specified by the cluster information below, we use kube-proxy v1.21.2-eksbuild.2 in iptables mode as it is the only supported mode by bottlerocket. we also use AWS appmesh controller v 1.4.1 with envoy images 188.8.131.52 as service mesh.
We use ClusterIP service to point to the deployment, and the VirtualService is configured to use the DNS name of the service.
Is this behavior related happening because of kube-proxy working in iptables mode? or is it because of Appmesh and envoy?
Kubernetes version: v1.21.2-eks-0389ca3
Cloud being used: AWS EKS
Installation method: Managed
Host OS: Bottlerocket 1.2.0
CNI and version: AWS vpc-cni v1.9.0-eksbuild.1
CRI and version: containerd://1.4.8+bottlerocket