Scheduling not balanced

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: 1.20.11
Cloud being used: (put bare-metal if not on a public cloud) bare-metal
Installation method: kubeadm
Host OS: RedHat
CNI and version: calico 3.17.3
CRI and version: containerd 1.4.7

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

We have a weird behavior of the scheduler in our cluster.
Our cluster is composed of 4 worker nodes.
Currently we often have one node which is more loaded than the others so sometime I’m trying to reschedule workload to spread it elsewhere but most of the time it does not work (stays on the heavier node).
Even worse when creating new workload most of the time the scheduler launch it on this most used node.
We tried to start from scratch our load but it always end like that.
I don’t see anything in our cluster that could cause this bad behavior.
We don’t use priority class nor affinity nor selector for nodes.

If someone has some input that can help it would be much appreciate.

Thanks,
Thomas.