How to properly delete failed Pods from Deployment without affecting running Pods?

#1

Hello,

I ran into this issue several times and haven’t really found a good answer after googling around. The scenario looks like this – We have an early version of helloworld deployed and it’s serving production traffic. Now the next version comes and it has some issues (ImagePullBackOff). As a result, the old pods are still running while the new ones are stuck there until the next deployment kicks in.

Pods:

pod/helloworld-664bcd66cc-52zms 1/1 Running 0 30d
pod/helloworld-664bcd66cc-cwrfs 1/1 Running 0 30d
pod/helloworld-664bcd66cc-dpq7c 1/1 Running 0 30d
pod/helloworld-664bcd66cc-s9q7v 1/1 Running 0 30d
pod/helloworld-664bcd66cc-svrgd 1/1 Running 0 30d
pod/helloworld-84b6864b4-dtvg9 0/1 ImagePullBackOff 0 5m
pod/helloworld-84b6864b4-g488w 0/1 ImagePullBackOff 0 5m
pod/helloworld-84b6864b4-rw6kd 0/1 ImagePullBackOff 0 5m

RelicaSets:

replicaset.apps/helloworld-664bcd66cc 5 5 5 30d
replicaset.apps/helloworld-84b6864b4 3 3 0 5m

My question is how I’m supposed to cancel the problematic deployment (replicaset.apps/hhvm-84b6864b4 in this example). I tried to delete “replicaset.apps/hhvm-84b6864b4” using kubectl delete but then a new one was spawned. Deleting the Deployment source could solve this problem but it will obviously cause production downtime.

I’m aware that this action is not necessary as bad pods are not in rotation anyway. Just wanted to learn more about how Deployment/ReplicaSet/Pod work together.

Thanks!

#2

Have you tried rolling back the Deployment?

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#checking-rollout-history-of-a-deployment