Good storage provisioner for on-premises (non-cloud) cluster?

General Discussions
1

We have an simple on-premises kubernetes cluster here and are looking for the best way to provision persistent volumes. The cluster runs on KVM – not OpenStack. Currently we use the nfs provisioner here https://github.com/kubernetes-incubator/external-storage which has been working well for us, but for databases, the performance over NFS is like 6X slower than a local disk.

Is there a better solution? What do non-cloud kubernetes people use?

I found this “local” provisioner:

but it is static only – no dynamic provisioning. Less than ideal, but workable.

Any suggestions are appreciated.

2

Unfortunately, the answer is always going to be “It depends.” That said, how large do you expect your clusters to be, how many applications do expect to run, and what is your expected storage footprint in terms of size? There are open-source block storage solutions that could run within your cluster (Nexenta, OpenEBS, Ceph/Rook, etc.) but there are management overheads there.

We know folks using local storage for their Kubernetes clusters but then you do want to use tools like Kanister (https://github.com/kanisterio/kanister) to extract data for backup/recovery/migration/etc.

Hope this helps but happy to answer additional questions if needed.

3

Since you are on KVM and not on OpenStack or vSphere or something more sophisticated like that, I definitely recommend you take a look at Rook, which seems specifically developed for this use case.

https://rook.io/

Please report back with your findings and experience, as this is something that quite a few people in the community would be interested in.

2 Likes
4

We will be taking a hard look at rook – thank you both for your feedback.

5

@dvree We are in a similar position. Can you share what worked for you?

6

Rook - Ceph that utilize KVM pass-through SSD drives.

Steep learning curve, but works very well

7

+1 for rook

8

Hello. I am fairly new to Kubernetes and have setup a on premise 3 node cluster on VMs in VMware. I want to use dynamic provisioning on local storage. I tried Azure storage but since the Ubuntu version 20.04 that K8 is installed on doesn’t have support for Azure-Cli I didn’t get very far. My question is generally the same as above. I have had no luck with what dvree used. I am unable to install it, having difficulty following the instructions. If nothing else is there a fast way to create persistent volumes instead of writing the yaml file like 20 times. I would like to pre-create a bunch. Or if there is a new easy to follow document on some sort of dynamic provisioning I’d love that link. Thanks.