Kubernetes version: 1.18.4 Cloud being used: bare-metal Host OS: Ubuntu
I have been setting ResourceQuotas and LimitRanges on my single-node kubernetes to ensure that a certain namespace never uses more than 42 cpus of the 48 available, as I wanted to be sure other namespaces always have at least 6 cpu for them.
In particular, I have set:
apiVersion: v1 kind: ResourceQuota metadata: name: mem-cpu-demo spec: hard: requests.cpu: "40" requests.memory: 192Gi limits.cpu: "42" limits.memory: 192Gi
I expected to be able to schedule up to 40 single-container pods with a CPU request of 1 and that they would be individually be able to burst to up to 24 cores by setting their limits.cpu to 24.
However, if I do that the second pod I create is rejected because:
Error creating: pods “sim-nv87acee-n4hzm” is forbidden: exceeded quota: mem-cpu-demo, requested: limits.cpu=24, used: limits.cpu=24, limited: limits.cpu=42
Contrary to what I believed, the 42 cpu limit is not enforced at runtime, but it is enforced at pod creation, because the sum of all the pods limits must never exceed 42.
So, if I specify any limit for my pods (which I do want to do, because I want them to be able to burst so limit=request would not be good) then this results in the fact that I can no longer use the full 42 cpus.
Actually, in the above mentioned case, with pod limit.cpu=24, I can only run 1 pod, so most of the time I would only use 2 cpu with all the rest of the server being inutilized.
Is there a way to cap namespace total cpu usage at runtime (leaving the ability for each pod to burst)?