# Node "not ready" state when sum of all running pods exceed node capacity **URL:** \ **Category:** General Discussions\ **Created:** [September 27, 2019, 9:53pm UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137 "2019-09-27T21:53:37Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![niketanand](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@niketanand](https://discuss.kubernetes.io/u/niketanand)\ **Post date:** [September 27, 2019, 9:53pm UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/1 "2019-09-27T21:53:37Z")
I have 5 nodes running in k8s cluster and with around 30 pods. Some of the pods usually take high memory. At one stage we found a node went to “not ready” state when the sum of memory of all running pods exceeded node memory. Anyhow, I increased the resource request memory to high value for high memory pods but shouldn’t node controller kill all the pods and restarts all instead of making a node to “not ready” state? Suppose 4 pods were already running in a node and scheduler allowed another pod to get added in that node as resource request memory is within the node left memory capacity. Now over a period of time due to some reason all pods memory started increasing and although each pod memory is still under the individual resource memory limit value but sum of all pods memory exceeds the node memory and this causes the node to “not ready” state. Is there any way to overcome this situation? Due to this all pods get shifted to other node or some pods to pending as it has higher resource request value. Please help me how to handle this. ### Cluster information: Kubernetes version: 1.10.6 Cloud being used: (put bare-metal if not on a public cloud) AWS Installation method: Host OS: CNI and version: CRI and version: ---
**Author:** ![rata](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/rata/32/18244_2.png) [@rata](https://discuss.kubernetes.io/u/rata)\ **Post date:** [September 27, 2019, 10:36pm UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/2 "2019-09-27T22:36:27Z")
Not sure I follow. Pod mem usage is RSS men usage? Are you running with swap disabled, right? In that case, I don’t see how process (pods) can use more mem than the node has. Am I missing something? Is it possible that some other thing happened? ---
**Author:** ![niketanand](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@niketanand](https://discuss.kubernetes.io/u/niketanand)\ **Post date:** [September 27, 2019, 11:05pm UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/3 "2019-09-27T23:05:14Z")
Thanks Rata for quick response. _Not sure I follow. Pod mem usage is RSS men usage?_ - Yes, pod mem usage is RSS mem usage. _Are you running with swap disabled, right? In that case, I don’t see how process (pods) can use more mem than the node has._ - Yes, running with swap disabled. Suppose 3 pods are running in a 32 GB memory node and each pod currently taking 8 GB RSS mem with each resource memory limit 10GB. Now going to add a new pod with resource request value 1GB and resource memory limit 10GB, so all 4 pods are running fine for a while. After a while, RSS memory of a new pod started increasing and reached to 8GB. Now total RSS memory of pods = 32 GB which is equal to node memory capacity. At this stage, if any pod memory goes further (still less than limit 10GB) then it was making that node to “not ready” state. I was seeing this and speculating this scenario. If the pods memory trying to exceed node then ideally what should happen? Looks to me this situation is making the node to “not ready” state instead of restarting all the pods. ---
**Author:** ![rata](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/rata/32/18244_2.png) [@rata](https://discuss.kubernetes.io/u/rata)\ **Post date:** [September 28, 2019, 9:41am UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/4 "2019-09-28T09:41:47Z")
Ohh, I see now. Thanks! If that happens, the kernel OOM killer should kill processes to bring mem usage in the node back to normal. That is the ““ideal”” (because it is, of course, not perfect nor very nice). Do you see in the kernel logs (or dmesg if haven’t rebooted) output that says OOM or similar? Of course processes can have different scores and adjustments for the OOM and you might be in a weird situation where that is not very effective? Sounds weird, though. What happens after a while the system is not ready? Does it recover? Maybe the OOM is starting late or something, bit after a while it kills some processes and frees RAM and the node recovers? If those process are killed, will they start with small amount of RAM usage? Or grow big fast? If they will grow big fast, they can be killed and restarted very soon and the node does not recover? And just curious, can you reproduce if you write a simple program that uses 32GB and run it in the node? (Like, just a malloc and write something to every byte so it is actually allocated by the kernel). You can try with 30/32/34, something close to node mem capacity. Now, regarding the kubernetes layer, why do you have so big difference between request and limit? Is that in purpose? ---
**Author:** ![BhargavB](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/bhargavb/32/3660_2.png) [@BhargavB](https://discuss.kubernetes.io/u/BhargavB)\ **Post date:** [September 28, 2019, 11:49am UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/5 "2019-09-28T11:49:57Z")
I am trying to understand your situation better. Node has 32GB in total. 3 Pods, Pods 1 to 3: Has following request & limit 8GB Request & 10 GB Limits 4th Pod has 1 GB Request & 10 GB Limits. When 4th pod memory reaches 8GB, nodes goes in to Not Ready state? Question 1. Is above summary of your situation right ? 2. How do you know the nodes crashes when 4th pod memory reaches 8GB ? ---
**Author:** ![niketanand](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@niketanand](https://discuss.kubernetes.io/u/niketanand)\ **Post date:** [September 28, 2019, 12:04pm UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/6 "2019-09-28T12:04:15Z")
Yes, the well described situation is right. I m seeing that the node is in not ready state and pods r moved to other node and now this node is under pressure. ---
**Author:** ![BhargavB](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/bhargavb/32/3660_2.png) [@BhargavB](https://discuss.kubernetes.io/u/BhargavB)\ **Post date:** [September 28, 2019, 12:08pm UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/7 "2019-09-28T12:08:07Z")
1. Can you do a describe for that node on working and non-working state ? 2. How do you know that when pod-4 reaches 8GB, node crashes ? ---
**Author:** ![niketanand](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@niketanand](https://discuss.kubernetes.io/u/niketanand)\ **Post date:** [September 28, 2019, 12:21pm UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/8 "2019-09-28T12:21:16Z")
Node is in running state as seen in aws console but k8s is not running any pods on it. I have datadog alert setup for each pod ---
**Author:** ![BhargavB](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/bhargavb/32/3660_2.png) [@BhargavB](https://discuss.kubernetes.io/u/BhargavB)\ **Post date:** [September 28, 2019, 12:25pm UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/9 "2019-09-28T12:25:05Z")
1. Can you do a describe on the node and paste the output ? 2. Can also throw some light on the metric you are throwing alert on ? ---
**Author:** ![niketanand](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@niketanand](https://discuss.kubernetes.io/u/niketanand)\ **Post date:** [September 28, 2019, 12:47pm UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/10 "2019-09-28T12:47:15Z")
Unfortunately I can’t provide describe node now as we have restarted the k8s cluster with 2 additional nodes and increased memory request value to a high value for some memory intensive pods. You can see this [https://medium.com/retailmenot-engineering/what-happens-when-a-kubernetes-pod-uses-too-much-memory-or-too-much-cpu-82165022f489](https://medium.com/retailmenot-engineering/what-happens-when-a-kubernetes-pod-uses-too-much-memory-or-too-much-cpu-82165022f489) Here he is having only one pod running in a node and making node to notready state. ---
**Author:** ![niketanand](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@niketanand](https://discuss.kubernetes.io/u/niketanand)\ **Post date:** [September 29, 2019, 2:00am UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/11 "2019-09-29T02:00:32Z")
Yes, I had seen OOM error in the log but it made the node to notready state. Ideally, it should have killed the pods to restart. I have increased the request memory value now for high memory intensive pods but trying to fix at kubernetes config level. Do you know how can we make OOM to trigger fast so that pods could be killed and at least not to go to notready state? ---
**Author:** ![rata](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/rata/32/18244_2.png) [@rata](https://discuss.kubernetes.io/u/rata)\ **Post date:** [September 29, 2019, 8:24am UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/12 "2019-09-29T08:24:03Z")
Oh, if you are seeing the OOM, it makes sense for me. But is the node able to become ready again after some time? Please have a look at this documentation doc, that explains this a little more and some best practices: [https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#node-oom-behavior](https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#node-oom-behavior) I think that seems very useful, and maybe the flags to the kubelet can do the trick 🙂 ---
**Author:** ![niketanand](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@niketanand](https://discuss.kubernetes.io/u/niketanand)\ **Post date:** [September 29, 2019, 1:15pm UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/13 "2019-09-29T13:15:30Z")
Thanks. This is a great help. I will also try to upgrade my k8s version from 1.10.6 to 1.16.0 ---
**Author:** ![rata](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/rata/32/18244_2.png) [@rata](https://discuss.kubernetes.io/u/rata)\ **Post date:** [September 29, 2019, 1:52pm UTC](https://discuss.kubernetes.io/t/node-not-ready-state-when-sum-of-all-running-pods-exceed-node-capacity/8137/14 "2019-09-29T13:52:32Z")
Please keep us posted on your findings! If some situation is still not good, we can try to open an issue and even create some patches if we come up with ideas to improve it 🙂 I’ve not seen that myself (the OOM kills the big mem using process in my clusters), but I’m definitely interested to see what is possible if your cluster setups have different requirements and things don’t bode so well there