I have a k8s service backed by a deployment with replicas of pods, which receive tasks and handle them asynchronously.
I want the pod to handle only one task at a time, thus I let my readinessProbe to track a variable status. When status equals BUSY, set the pod to NOT READY to avoid further request to this pod.
I also prefer when scaling down the deployment, pods that are not busy handling tasks are deleted first, so when I set status to BUSY, I set the controller.kubernetes.io/pod-deletion-cost to 100. When the task is finished, I set the controller.kubernetes.io/pod-deletion-cost back to 1.
When executing the system, I find that pods running task are still deleted first. This is because the priority of ready/not ready is higher than pod deletion cost value in the k8s pod deletion decision maker. Implementation can be find here (line 822 to 833).
Is there a way for me to:
determine whether a pod can be routed to from a service not by ready/not ready
or:
avoid use ready/not ready to determine which pod will be deleted
More info (for example how to reproduce the problem) can be find here. Thanks for your help!