GKE emptydir size limit value ignored in kubeflow

General Discussions
1

Hello,

In our team, we are using a GKE cluster in google cloud platform (version : 1.19.9-gke.1400) to deploy Kubeflow to run some machine learning pipelines. We have some memory intensive tasks and we would like to increase shm size by mounting it to empty dir but it increase it by the half of available memory which is not the desired since we are setting sizeLimit parameter to “1Gi”.
When running the command : df -h COMMAND, I can see that my limit is ignored and the half of memory is booked.

How could we set such limitation in GKE ? is it a bug ?

Thank you for your help.

2

There’s a note about a feature gate in this documentation here.

Note:
If the SizeMemoryBackedVolumes feature gate is enabled, you
can specify a size for memory backed volumes.

If no size is specified, memory backed volumes are sized to 50%
of the memory on a Linux host.

Looking at the feature gates page that note links to, I think the default for SizeMemoryBackedVolumes is False, which leads me to believe you won’t be able to control this.

You can check the enabled feature gates with kubectl -n kube-system describe pod kube-apiserver-<nodename> and checking the --feature-gates flag in the Command section.

Edit: a word.

3

Hello,

Thank you for your reply. The SizeMemoryBackedVolumes is only available for Kubernetes 1.20 and above.
In GCP, we don’t have access to feature-gates to check their values.

Any other ideas to set the sizelimit in a gke cluster ?