# Pods terminating on downed node

**URL:** <https://discuss.kubernetes.io/t/pods-terminating-on-downed-node/5668>\
**Category:** General Discussions\
**Created:** [March 26, 2019, 7:41pm UTC](https://discuss.kubernetes.io/t/pods-terminating-on-downed-node/5668 "2019-03-26T19:41:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![CrankyCoder](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/crankycoder/32/2373_2.png) [@CrankyCoder](https://discuss.kubernetes.io/u/CrankyCoder)\
**Post date:** [March 26, 2019, 7:41pm UTC](https://discuss.kubernetes.io/t/pods-terminating-on-downed-node/5668/1 "2019-03-26T19:41:16Z")

</div>

So last night something on my esxi lab environment happened and one of my nodes vm completely shut off. Not gracefully.

When i did a kubectl get pods i saw that the pods that had been running on that node showed terminating, and new pods had spun up on the other node. However, since the previous nodes were still “terminating” it appears they were still be sucked in to the svc. So the svc was not fully functional because it was trying to route traffic to pods that were terminating.

The system restored stability when the node booted back up. the pods completed their termination and the svc’s came back online.

Is there something I can configure that would force delete of terminating pods after X amount of time? or if the node is gone?

Thanks!

---

<div class="post-metadata">

**Author:** ![macintoshprime](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/macintoshprime/32/7121_2.png) [@macintoshprime](https://discuss.kubernetes.io/u/macintoshprime)\
**Post date:** [March 26, 2019, 7:50pm UTC](https://discuss.kubernetes.io/t/pods-terminating-on-downed-node/5668/2 "2019-03-26T19:50:12Z")

</div>

Try this, it’s worked for me in the past  
`kubectl delete pod NAME --grace-period=0 --force`

---

<div class="post-metadata">

**Author:** ![CrankyCoder](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/crankycoder/32/2373_2.png) [@CrankyCoder](https://discuss.kubernetes.io/u/CrankyCoder)\
**Post date:** [March 26, 2019, 8:22pm UTC](https://discuss.kubernetes.io/t/pods-terminating-on-downed-node/5668/3 "2019-03-26T20:22:28Z")

</div>

I was wondering if there is a way to do it automatically instead of force killing it. Seems counter productive for terminating pods that aren’t accessible to impact the rest of the service. I figured I must have missed something.

---

<div class="post-metadata">

**Author:** ![macintoshprime](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/macintoshprime/32/7121_2.png) [@macintoshprime](https://discuss.kubernetes.io/u/macintoshprime)\
**Post date:** [March 27, 2019, 3:38pm UTC](https://discuss.kubernetes.io/t/pods-terminating-on-downed-node/5668/4 "2019-03-27T15:38:05Z")

</div>

ya sadly I haven’t come across anything that automatically does that. There is an open ticket for it here if you want to track it [https://github.com/kubernetes/kubernetes/issues/51835](https://github.com/kubernetes/kubernetes/issues/51835)

---

<div class="post-metadata">

**Author:** ![rata](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/rata/32/18244_2.png) [@rata](https://discuss.kubernetes.io/u/rata)\
**Post date:** [March 28, 2019, 4:17am UTC](https://discuss.kubernetes.io/t/pods-terminating-on-downed-node/5668/5 "2019-03-28T04:17:44Z")

</div>

> ![](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/crankycoder/45/2373_2.png "CrankyCoder")
> 
> [CrankyCoder](https://discuss.kubernetes.io/u/crankycoder)
> 
> ```
> March 26
> 
> ```
> 
> So last night something on my esxi lab environment happened and one of my nodes vm completely shut off. Not gracefully.

> When i did a kubectl get pods i saw that the pods that had been running on that node showed terminating, and new pods had spun up on the other node. However, since the previous nodes were still “terminating” it appears they were still be sucked in to the svc. So the svc was not fully functional because it was trying to route traffic to pods that were terminating.

That seems more surprising. The pods in the terminating state were still shown in the endpoints? We’re the pods marked as ready too?

If that is the case, it smells like something worth reporting (if not already report, please check before opening a new issue).

> The system restored stability when the node booted back up. the pods completed their termination and the svc’s came back online.

> Is there something I can configure that would force delete of terminating pods after X amount of time? or if the node is gone?

I’m really not sure if this is a bug to report, that might be fixable, or if there is something you can tune.

I’d consider looking for open issues, reading documentation heavily and try to reproduce with different kubernetes versions (if easy for you to change them).