Pod placement suggestion for Redis Cluster (KVRocks)

General Discussions
1

Hi All,

Need your suggestions on the below scenario we have encountered:

  • We have a Redis Cluster with multiple shards and replicas for each shard.
  • Number of nodes in cluster match with number of shards in the redis cluster
  • We create a statefulset for each shard

Our requirement needs pods to be placed such a way that:

  • No two pods of same shard scheduled on same node
  • All nodes should get equal number of redis cluster pods

We have tried this with two pod anti-affinity rules:

  • First: Statefulset Label with topology key as HostName ( to spread pods of same statefulset across different nodes)
  • Second: No two pods with same index across statefulsets should be scheduled on the same node (Statefulset Label, Pod-Index with hostname as topology key) (to place equal number of pods on each node)

This works well in some case, but creates impossible scenario where few pods remain in pending state forever.
The root-cause seems like the default scheduler places pods as and when they are created and this can create a scenario where scheduler does not find a suitable node for last few pods.

Is there a better way to handle this?

2

We are considering two approaches:

  • Separate statefulset for each pod and binding the pod to node with node affinity rules. PVC is created automatically with templates
  • Create pods directly in the operator and also create PVC manually.

With first approach, if we have to create multiple Redis Clusters in the futures, the number of statefulsets will explode. Not sure if that will be an issue. The number of pods will not change in both approaches.

3

Hello Bhaskar,

Why don’t you consider using PodAntiAffinity?
You can set pod anti affinity to make sure no other pod similar to it gets schedules on the same node.

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: <label key>
              operator: <operator - "in", "exists" "eq">
              values:
              - <label value>
          topologyKey: topology.kubernetes.io/zone

This will solve your use case.

Best regards,
Manan

4

Yes, we have tried some thing similar. We have tried with below two pod anti-affinity rules.
Challenge is that k8s places pods on nodes as and when pods are created. It does not have global view of the anti-affinity rules across all statefulsets.

With this, if we have tightly packed nodes, we can land in to a situation where it becomes impossible to place some of the pods created towards the end.

affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/kind: KVRocks
app.kubernetes.io/managed-by: Helm
kvrocks/name: strider-kvrocks
kvrocks/statefulset: strider-kvrocks-X
topologyKey: kubernetes.io/hostname

affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution
- labelSelector:
matchLabels:
app.kubernetes.io/kind: KVRocks
kvrocks/name: strider-kvrocks
matchLabelKeys:
- apps.kubernetes.io/pod-index
topologyKey: kubernetes.io/hostname

Below is one example where it becomes impossible to place S6/R1. S is shard, R is replica.

image
image2242×888 36.2 KB

5

But you mentioned that “Number of nodes in cluster match with number of shards in the redis cluster” so by that you should have an idea of how many replicas you can successfully deploy, right?
You can either configure node count or shards based on the requirements.

Total Nodes = Shards × Replicas

You can also control the replicas if you do not want to increase the nodes.
So I don’t see any problem here.

Thanks,
Manan

6

Reading through you other post, I think you want these shard replicas to be spread evenly across nodes, not just 1 per node but 1:1 distribution ratio.

You need to use topology constraint:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: "kubernetes.io/hostname"
      whenUnsatisfiable: DoNotSchedule # Or ScheduleAnyway
      labelSelector:
        matchLabels:
          <label_key>: <label_value>  # add your labels

Read about the definitions here: Pod Topology Spread Constraints | Kubernetes

7

Thanks for the reply.

Node count matches with Shard count, not the total pods count. Total number of pods we are is Shard Count * Replica per Shard.

For example, if we have 6 node cluster, we will have 6 shards, and our default config is to have 3 replicas per shard (so, total pods are 18).

  • Pods of each shard must be placed on different nodes
  • All nodes should have equal number of pods (3 in this case)

Not sure if we can achieve this with anti affinity rules. Pod placement seems to be done per pod as soon as the pod is created, which can result in impossible placement for later created pods.