Share HugePages across node

Cluster information:

Kubernetes version: 1.21
Cloud being used: Tested on AWS and GCP
Installation method: kops
Host OS: Ubuntu 20.04 (also tested on CentOS)
CNI and version: -
CRI and version: CRI-O


Hi,

We are developing an application deployed on Kubernetes, for which we use hugetlbfs as shared memory across pods within a node.

This was working fine but for some time now, most modern distributions have now support of hugetlb cgroup controller and Kubernetes has HugePages management, so unless we miss something:

  • a pod must have HugePages resources limit/request set, otherwise it cannot create hugetlb allocations at all
  • this resources cannot overcommit, so this limit/request must be split across all pods.

We use the hugetlbfs backed with files, that any pod can create, so there is no “by-pod” management here, and we don’t figure out how to configure this properly.
For now, we disabled the hugetlb cgroup controller on the host. It looks like a bad idea to us to have to customise the host.

Is there any good pattern to handle that on Kubernetes?

Thanks in advance for any advice on this.

1 Like