Hello Kubernetes Community,
I am currently leveraging JupyterHub to dynamically allocate containers within a Kubernetes cluster for running my machine learning notebooks. My machine learning datasets, which are essential for my projects, require persistent storage to manage and store data effectively. The challenge I’m facing is integrating storage with these dynamically created containers to ensure data persistence across sessions.
To address this, I’m considering using Minio as a flexible, S3-compatible storage solution that can cater to my storage needs. The key requirement here is to dynamically provision storage for each Jupyter notebook instance created through JupyterHub. This setup would ideally allow each notebook to have its dedicated storage space, facilitating seamless data management and accessibility.
I’m in search of a Container Storage Interface (CSI) solution for Minio that supports dynamic provisioning. The goal is to mount Minio storage volumes into my Kubernetes cluster, enabling automatic storage allocation for each new Jupyter notebook instance. This would significantly streamline the management of machine learning datasets and storage resources within my cluster.
Could anyone please recommend a dynamic provisioning solution or a CSI driver for S3 (compatible with Minio) that meets these criteria? Any guidance, experiences, or recommendations on setting up a robust, dynamic storage provisioning system with Minio for JupyterHub in a Kubernetes environment would be greatly appreciated.
Thank you for your support and suggestions.
Cluster information:
Kubernetes version: 1.28.0
Cloud being used: bare-metal
Installation method: kuberspray
Host OS: RHEL7
CNI and version: calico v3.26.4
CRI and version: containerd v1.7.13