Identifying containers that go beyond their requested memory

General Discussions
#1

Kubernetes version: 1.16
Cloud being used: Bare Metal

Here is the problem I am trying to solve in K8 version 1.16. I have a namespace in K8, which has a hard limit (say 5 GB). The namespace itself has made reservations for 3 GB on memory. In the namespace, I have a few containers. The sum of all the memory requests from these containers adds up to 3 GB.

I want to get an email every time a container overshoots its memory requirements. And anytime the namespace memory requests go beyond 3GB, I need to know the offending containers that have gone beyond their memory requirements.

Since the hard limit is 5GB (and beyond the namespace requirements), I see the offending containers performing with no penalties.

How do I go about solving this problem in K8? K8 documentation is vague on this aspect (in fact K8 says, the containers will be allowed to run if there is memory on the nodes)

#2

K8s itself doesn’t have any built in capability to alert on that sort of thing natively, but it does provide the hooks for you to plugin a monitoring solution to do so. I’d check out Prometheus, it’s one of, if not the most popular solutions.

If you’re curious about how how these things fit in with each other, the first few min of this video should covers why k8s doesn’t handle the metrics+reporting itself and instead allows an external system to do it:

1 Like
#3

If you go for Prometheus and you are also interested in having a visual description of your K8s’ state (CPU/Memory usage, etc), you may be interested in implementing Grafana as well.
Although Prometheus does have graphs, Grafana provides you with better visualization tools. You can also add different datasources (such as Loki ) and integrate them on a single dashboard.
I’m pretty new to Prometheus/Grafana (and k8s as well), but I think that if you choose to monitor Kubernets’ metrics with another tool different from Prometheus, Grafana might be compatible too.