Statefulset not evenly distributed among nodes

General Discussions
1

I’m following the “Regular channel” of GKE Release channel, 5 days ago, the cluster has been updated, since that, I found that the 2 statefulsets are now running on only 2 of my 3 nodes, as these 2 statefulsets are database which require quite a lot of resources, it is causing those 2 nodes to have high CPU and Memory issue.

I’ve tried deleting the pods to hope that they can be re-created in the remaining node, but tried a few times, they were only re-created at those 2 nodes.

Kubernetes version: 1.17.13-gke.2600
Cloud being used: GKE

Please let me know what can I do to solve this issue. Thanks a lot.

2

You can configure pod affinity and anti-affinity rules that will inform the scheduler on how you’d like your workloads distributed. :+1:

As an FYI - Kubernetes by default does not take into account the placement of the pods belonging to a specific higher level type (Deployments, StatefulSets etc). It looks at it solely from a pod level, then filters and scores nodes to determine a pod’s placement. Part of that evaluation and scoring is looking at the pod affinity/anti-affinity rules :slightly_smiling_face:

3

We have similar challenge to place pods of redis shards/replicas across nodes.
We want pods of shard to be spread across nodes and each node should have same number of these pods.

Example, if we have 6 node cluster, 6 redis shards with 3 pods each (1 master, 2 replicas), we want to spread them perfectly across all 6 nodes.

We have added couple of pod anti-affinity rules. But since k8s scheduler places pods as and when they come up, we land in to scenarios where it becomes impossible to place the pods on the nodes (for the pods that are created towards the end).

Appreciate if we can solve this native facilities of k8s.

4

Jus use topology constraint: with right skew

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: "kubernetes.io/hostname"
      whenUnsatisfiable: DoNotSchedule # Or ScheduleAnyway
      labelSelector:
        matchLabels:
          <label_key>: <label_value>  # add your labels

Read about the definitions here: Pod Topology Spread Constraints | Kubernetes

thanks,
Manan

5

Thanks.
This does not give perfect balance of pods AFAIK. There can be a skew of 1 which is not acceptable for our use-case.

6

You can achieve distribution by combining Topology Spread Constraints (to balance the total pod count) with Pod Anti-Affinity (to prevent shard co-location).

looking at your example:

if we have 6 node cluster, 6 redis shards with 3 pods each (1 master, 2 replicas), we want to spread them perfectly across all 6 nodes.

  • Use topologySpreadConstraints with maxSkew: 1 and whenUnsatisfiable: DoNotSchedule or ScheduleAnyway if you have nodes scaling down.
  • Use topologyKey: kubernetes.io/hostname to target individual nodes.
  • Use a common label (e.g., app: redis) for the spread constraint to balance total volume.
  • Use required podAntiAffinity or podAffinity based on a shard-specific label (e.g., shard: shard-X) so you control where do the replicas of the same shard are scheduled.