How can I gracefully drain a node with deployments with minReplicas of 1.
Adding any PodDisruptionBudget just prevents the node from being drained:
evicting pod foo/bar-6c46845466-5rxq2
error when evicting pod "bar-6f94d57b66-hbz68" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I want a PodDisruptionBudget for all my deployments, but to reduce cost, I don’t require HA for all my deployments (I don’t want regular maintenance to cause down time, but unexpected node failures can cause downtime).
I also do not want to lose any desired pods on my critical deployments, but again, PodDisruptionBudget of maxUnavailable 0 just breaks node drain.
Basically I’m forced to run n+1 desired pods and have no single instance services to avoid impact of regular maintenance node rotations.
Is there anything planned to improve this?
In terms of custom solutions, is there a way I can respond to node drain or pod eviction and force my Deployments to make a no-op change so they can get scheduled elsewhere and allow the PodDisruptionBudget to pass eventually without human intervention?
How can I gracefully drain a node with deployments with minReplicas of 1.
Adding any PodDisruptionBudget just prevents the node from being drained:
I’m still Kubernetes rookie, but after having hard time while upgrading Kubernetes on AKS, ended up with manual deletion of PodDisruptionBudget-s, kubectl get pdb; kubectl delete pdb <pdb name>.
In this thread, you can find similar problem of mine about not being able to evict pods: