HPA is directing the load to one pod only

I have a golang server running on k8 before it was tcp but after converting it into udp, it is directing the load only to the one pod, and with less load creating multiple replicas, and restarting the pods again and again, is with UDP is there any limitation in HPA ?

Cluster information:

Kubernetes version: v1.22.2
Cloud being used: metlallb
Installation method: helm
Host OS: ubuntu20