How to ask service to deal traffic only when all pods are ready

General Discussions
1

I have case where lets say, a service selects 10 pods and serves traffic. There is constant heavy traffic into the pods … that if one pod breaks down, and by the time it restarts, traffic for another pods increases. So is there a way to tell the service to temporarily stop all traffic to all pods until this pod restart is done? Or more simply, is there a way to stop all traffic to all pods until all pods are ready (not just 9/10 pods being ready 10/10 ready)?

Kubernetes version: 1.18.9-eks

2

Seems like something a readinessProbe is meant to address.

Edit:
To clarify, for the 10/10 requirement, probes in general are probably what you need to investigate and implement.

Also if you can’t go below 10 pods, you may want to consider using more than 10 pods. Your application is under allocated based off of this description.

3

Thanks for the response @protosam

  • I don’t think readinessProbes are what I need. readinessProbes are to tell k8s whether that particular pod is ready to receive traffic or not.
  • If 6 out of 10 pods are ready, in this case, my service will deal traffic to those 6 pods … and when rest are up it will deal traffic to the rest also. But I need a method where until all the pods are ready, k8s wont deal traffic. So is there some way to tell if a set of pods are ready, rather than an individual pod being ready. If so how do I tell this to k8s or my service.
  • I can think of a way where all the pods communicate with each other (through another service or something) and design a readiness probe such that all pods go not-ready even if 1 goes not-ready. But it seems so complicated and undoable that I don’t want to go this way.
  • And yes I am also considering allocating more pods … but this decision is not entirely up to me … so I need another kind of solution.
  • A totally different approach to tackling my problem is also very much welcome.
4

You could add in livenessProbes that constantly checks availability after the initial readinessProbe.

Beyond that the only alternative I can think of is to write your own operator that manages the deployment and the service for you. While it is a more of a cloud-native approach, it seems like overkill for what you’re trying to achieve.

5
  • From my understanding, any probing at a pod level is not desirable for me because it only does readiness or liveness for that particular pod … (right?)
  • On a general note, a deployment itself gets ready status from all the pods being ready, right. So can I somehow ask the service to deal traffic only when the full deployment is ready?(instead of the individual pods being ready)
  • Your operator solution totally flew over my head :joy:. But it does seem complicated.
6

So, for a probe to be useful, the containers in the pod would probably need to be able to observe the deployment and maybe pod states. This would require a service account with role based access to be attached to the pod though.

7

What do you want to happen when, after all 10 become ready, one of them crashes or gets evicted? Should the service slam shut on all clients? That seems wrong to me. If not, what’s the difference between “9 available while #10 starts” and “9 available while #10 restarts”?

The only way I see to achieve your goal would be to have a controller which watches the pods and changes the service selector when any of them are not ready.

1 Like
8

Yes @thockin slamming shut on them for a while is what I want. Thanks for the advice.

  • You gave me another idea. I can have another pod/service running … acting like a reverse proxy + load balancer into my main set of pods/service, where (I will give it rbac and all so that) this pod get ready status only when all the main set of pods are ready … but this one pod will face the load of all the other pods combined right… so this doesn’t help
  • I, now, get your idea of having some sort of an external controller with control over the service… I will try implementing it.