K8s Performance Optimization

I have a problem. My environment is Kubernetes v1.34.10, using Volcano to submit large batches of multi-parameter training tasks. At one time, thousands of pod groups may be spawned for inference. Currently, both scheduling and task execution are relatively slow. Are there any optimization approaches to improve speed?