Asking for help? Comment out what you need so we can get more information to help you!
Cluster information:
Kubernetes version: 1.24 (EKS)
Cloud being used: Amazon
Installation method: EKS \ Terragrunt
Host OS: Linux
CNI and version: Amazon VPC-cni-plugin - amazon-k8s-cni:v1.13.2-eksbuild.1
CRI and version: Containerd -1.6.19
Hi All,
I know that the topic for CPU limits and cfs quota has received a lot of attention since 2017 and in a multiple threads on kubernetes, a kernal fix was issued and much more. I’m hoping to find some solace here so i’m quite puzzled and would like help on the subject.
We’ve come across something we really cant find any reason for and maybe someone here knows.
We are checking out some throttling issue and came across the notion of either having cpu limits vs no cpu limits set.
as such, we decided to test both and see what best works for us.
We took a pod and removed its limits - but what we found out was - that now the period, quota and stat are empty
cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
-1
cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
100000
cat /sys/fs/cgroup/cpu/cpu.stat
nr_periods 0
nr_throttled 0
throttled_time 0
When we ran htop within the pod we clearly saw cpu being used, when we checked /proc/stat - its full of data about the cpu.
The metrics in grafana - container_cpu_usage_seconds_total - shows metrics.
yet ‘container_cpu_cfs_throttled_periods_total’ and ‘container_cpu_cfs_periods_total’ - are empty for that same pod
when we put limits back in, those fields and files are populated.
we’re quite phased by this, because it means that we cannot monitor any potential throttling on pods that dont have any limits on the CPU.
or is there some other method to monitor throttling when those values are zeroed?
is this by design? has anyone come across this?
any input would be appreciated, thanks