Addon: OpenEBS

General Discussions microk8s
1

Homepage: https://openebs.io/
From MicroK8s version: 1.21+
Supported arch: amd64, arm64 (1.22+)

OpenEBS, is the most widely deployed and easy to use open-source storage solution for Kubernetes. can be enabled with:

microk8s enable openebs

The addon includes the following StorageClass

  • openebs-hostpath and
  • openebs-jiva-default

The openebs-hostpath is recommended when using on a laptop or a single node cluster. Use openebs-jiva-default StorageClass for multi-node cluster.

Note: Using openebs-jiva-default requires to have 3 replicas.

Using OpenEBS is as easy as creating a PersistentVolumeClaim.

On a single node MicroK8s

When using OpenEBS with a single node MicroK8s, it is recommended to use the openebs-hostpath StorageClass

To create a PersistentVolumeClaim utilizing the openebs-hostpath StorageClass

kind: PersistentVolumeClaim 
apiVersion: v1
metadata:
  name: local-hostpath-pvc
spec:
  storageClassName: openebs-hostpath
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5G

On multi-node MicroK8s

If you are planning to use OpenEBS with multi nodes, you can use the openebs-jiva-csi-default StorageClass.
For example:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jiva-volume-claim
spec:
  storageClassName: openebs-jiva-csi-default
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5G

Note: Running versions of Kubernetes prior to 1.23 will create the openebs-jiva-default storage class instead of openebs-jiva-csi-default

Disabling

To disable openebs,

microk8s disable openebs
or
microk8s disable openebs:force

2

What do you mean by 3 replicas? 3 nodes?

3

Yes some components of openebs runs as daemonset. Which provides HA. But you can always create a new StorageClass with 1 replica using the jiva storage.

More information here Jiva User Guide ·

But if you are running on your laptop or single node, i usually use the localpv.

4

I am planning to use it in microk8s cluster but failed to add node to cluster. Everything worked well until i tried to add node to a cluster. Not sure if i did something wrong or microk8s cluster is not production-ready yet. Looking for alternative software to microk8s.

5

Thanks @Jaanus for using MicroK8s. Sorry for the experience you’re having. If it is possible, would you be able to provide a bit more details on what happened when u add a node?

6

Microk8s fails to start after joining a cluster.

jaanus@nova-testnode1:~$ sudo microk8s status
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    ha-cluster           # Configure high availability on the current node
  disabled:

output truncated to keep post shorter.

jaanus@nova-testnode1:~$ sudo microk8s join 192.168.200.10:25000/af44dc0e26411748cbef9640fcb4c785/75f336ce9282
Contacting cluster at 192.168.200.10
Waiting for this node to finish joining the cluster. .. .. .. .. .. .. .. .. .. ..  
jaanus@nova-testnode1:~$ sudo microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
jaanus@nova-testnode1:~$ sudo microk8s.start
Started.
jaanus@nova-testnode1:~$ sudo microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.

On the master node testnode1 is not visible:

[13:10:02] jaanus@nova1:/srv/<my project name>$ microk8s kubectl get node
NAME    STATUS   ROLES    AGE    VERSION
nova1   Ready    <none>   189d   v1.21.0-3+121713cef81e03

UPDATE:
I opened port 25000 for the master node but didnt open 19001. After opening it everything started to work.

Open issue here:

7

Happy to hear its working.
The lists of ports are here.

8

Hey, any chance it’s going to be available for ARM64? It would be so awesome for my Raspberry Pi cluster. Thanks!

2 Likes
9

First I didn’t know openebs supports arm64. :grinning:
Second, at that time, i didn’t have the chance to test it.
Will probably need to do some test.

1 Like
10

Hi @balchua1 - I work at MayaData. We built, and continue to build with other contributors, the OpenEBS project on the CNCF.

A note about our implementation of cStor - admittedly, the performance is not that great. We see far more hostpath/Local PV usage of OpenEBS than cStor.

That said, we’re actively building a storage layer called MayaStor, that focuses on performance, and can be used instead of cStor. MayaStor specifically works with NVMe’s (open source NVMeOF) and we’re documenting how to use this @ Welcome to Mayastor! - Mayastor Reference

11

@kelly_mayadata thanks for pointing me out to mayastor. I am aware of Mayastor. :blush:. But i have not tried it at all. Now that you have reminded me, i will surely give this a try and perhaps add it to MicroK8s.
One question, is this something that will live outside OpenEBS or it will be part of OpenEBS?
Btw, great work on OpenEBS!!!

12

Thank you for the kind words and thank you for your efforts in integrating OpenEBS into MicroK8s!

We’re still a bit early on MayaStor. OpenEBS LocalPV is very mature. That said, there’s a planned June release (first/second week) that may be better to use in evaluation. A number of the folks that work on OpenEBS hang out on the CNCF’s Slack server under #openebs or I can connect to make your life easier :slight_smile:

As the MayaStor repo is under OpenEBS, the intent is to keep it with that project.

13

I will surely give this a try as soon as i can. :+1:
Will reach out to you guys on slack.

14

Btw if you spot something isn’t right or could be done better on the current OpenEBS addon in MicroK8s, please do let us know.
Thanks!!

15

Any chance to support storageClassName: openebs-device in addition to hostpath for Local-PV ? It would be nice to have for installing Minio using block devices. Thanks.

16

What do you mean by openebs-device?

17

From OpenEBS documentation.

There are 2 ways to use OpenEBS Local PV.

  • openebs-hostpath - Using this option, it will create Kubernetes Persistent Volumes that will store the data into OS host path directory at: /var/openebs/<“minio-pv-name”>/. Select this option, if you don’t have any additional block devices attached to Kubernetes nodes. You would like to customize the directory where data will be saved, create a new OpenEBS Local PV storage class using these instructions.
  • openebs-device - Using this option, it will create Kubernetes Local PVs using the block devices attached to the node. Select this option when you want to dedicate a complete block device on a node to a MinIO node. You can customize which devices will be discovered and managed by OpenEBS using the instructions here.
18

Thanks @Dragnell
I don’t know a lot about the openebs device. But i suppose the user can create their own StorageClass to suit their needs.
Its probably not easy to make this as a default, since not all microk8s installations have spare device to use. Wdyt?
Thanks