We host a handful of client web apps and a couple of AI-powered tools (chatbots, small internal automation services) on a shared cluster rather than spinning up separate infrastructure per client, mainly to keep costs sane at our scale. Each client gets its own namespace, which handles logical separation fine, but we had an incident recently where one client’s app got a sudden traffic spike (a marketing campaign went out) and ended up starving CPU for a couple of unrelated client namespaces on the same nodes.
Went back and set ResourceQuotas and LimitRanges per namespace after that, plus tightened requests/limits on the deployments themselves so the scheduler has better information to work with. Also looking at whether it’s worth setting up separate node pools for higher-tier clients versus everyone sharing the same pool with just quotas as the boundary.
For anyone running a similar multi-tenant setup for external clients rather than internal teams, did node pool separation end up being worth the added complexity, or did quotas plus proper requests/limits end up being enough on their own once actually configured correctly? Also curious how others handle HPA in this kind of setup without one client’s autoscaling behavior indirectly affecting node availability for another.
We’re a small dev team, Semark Global (semarkglobal.com), so keeping this efficient without over-provisioning per client is a real cost concern for us.