A container walks up to 84% of its memory limit. Nothing restarts. The limit just
gets bigger, four times in a row, and the job finishes.
That’s a real run, not a mockup — 256Mi → 448 → 640 → 832 → 1Gi, requests and
limits climbing together, RESTARTS sitting at 0 the whole way, and the closing
frames show the kubectl command that reads the restart count straight from pod
status so you don’t have to take my word for it.
k8s-oom-watchdog is a sidecar that makes this automatic. It samples the target
container’s cgroup v2 working set every 100ms and, when usage crosses 80% of the
limit, patches pods/resize to add headroom — then gives the memory back once the
burst passes. Kubernetes 1.33 graduated that subresource to GA, so the container is
never restarted and a six-hour job never loses its progress.
The part I’m proud of
Borrowing memory is easy. Borrowing it without ever putting the node at risk is
the interesting problem, and there are three layers:
- requests move with limits. Borrowed memory lands in scheduler accounting, so
nothing new is scheduled into the headroom, and when node allocatable runs out the
kubelet refuses the resize instead of the node overcommitting. The failure
direction is always “this container might OOM” — never “the node goes down”. - A host red line. A scale-up only proceeds while the new limit plus everything
else on the node stays under 90% of physical memory. Short on room, the step
degrades adaptively and takes whatever is safely available. - RBAC narrowed at admission. A Role cannot express “only your own pod”, so the
token is projected into the sidecar alone — and a shipped
ValidatingAdmissionPolicy pins every write it can make to seven CEL checks: the
labelled pod only, labels immutable, two annotations, memory only, never CPU.
And when a resize is accepted but never applied, it doesn’t wait forever: the spec
rolls back to what’s actually in effect, an alert fires, and scale-ups pause for ten
minutes.
Why not VPA
Fair question — InPlaceOrRecreate has been GA since VPA 1.6.0. The difference is
when each acts. VPA’s recommender raises memory after a container has been
OOMKilled, driven by a metrics pipeline with its own lag: exactly right for getting
sizes right over weeks, and by construction it can’t prevent the first kill. This
reads the cgroup on the node and reacts in seconds. They compose nicely — let VPA
own the baseline, let this borrow above it during a burst.
Getting it
docker pull ghcr.io/mryhm/k8s-oom-watchdog:0.1.0
Apache-2.0, one Python file over the standard library plus the Kubernetes client,
~70Mi resident, amd64 + arm64. examples/ has complete applyable manifests for a
Deployment and a Job — no Helm chart, because a sidecar has nothing to install on its
own. demo/ reproduces the GIF above on any 1.33+ cluster in two commands.
One caveat worth stating up front: a burst can outrun the rescue path. The limiting
factor is how much one step adds versus how long detect + PATCH + kubelet-apply takes
(1–5s in my measurements), so very sharp allocations still get killed — raising
memoryStep is the lever that helps most. The docs work through the arithmetic
rather than hiding it.
Two things I’d love input on from people who know the kubelet side better than I do:
- How variable is resize-apply latency on a loaded node? That number bounds
everything this can do. - Is there a cleaner signal than polling
PodResizePendingto learn a resize was
accepted but not yet applied?