Service does not distribute traffic in a balanced manner

General Discussions
1

My K8S cluster has 14 Fluentd pods (DaemonSet) and 1 Fluentd service (NodePort). That means there are 14 Fluentd endpoints for processing inputs.
Also, there are about 100 pods using the Fluentd service.
I’ve found that some Fluentd pods have a load heavier than others. Not sure why Fluentd service does not distribute traffic in a balanced manner.
You can see the screenshot below, only 2 Fluentd pods have a heavy load, others are fine. Is it normal for Service? Are there any ways for configuring balanced traffic in Service? Please let me know if you need to see any inputs more.

image
image1812×638 132 KB

Cluster information:

Kubernetes version: v1.21.4
Cloud being used: bare-metal
Installation method: kubeadm init
Host OS: Ubuntu 18.04.2 LTS
CNI and version: calico v3.9.5
CRI and version: containerd 1.5.8

2

Each node makes independent decisions about service routing. This means the load is not going to be perfectly balanced, especially when the numbers are relatively small.