Using OpenEBS for your Prometheus data

General Discussions microk8s
1

With the inclusion of OpenEBS addon, one can now use OpenEBS to serve your Prometheus’ persistent volume.

This has the advantage of keeping your Prometheus data in place when one of the node goes down.

In the steps below, we will be using jiva StorageClass for simplicity

After installing MicroK8s:

  1. Enable OpenEBS,
$ microk8s enable openebs
  1. Set the openebs-jiva-default StorageClass as your default.
$ kubectl patch storageclass open-jiva-default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
  1. Enable the prometheus addon
$ microk8s enable prometheus
  1. Next we are going to patch the Prometheus CR to use the openebs-jiva-default storage class.
  2. Create a file, ex: /tmp/patch-prometheus.yaml 

spec:
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: openebs-jiva-default
        resources:
          requests:
            storage: 5Gi
  1. Apply the patch using merge, because strategic patch does not work with CRDs.
kubectl -n monitoring patch prometheus k8s --patch "$(cat /tmp/patch-prometheus.yaml)" --type=merge

You should start seeing the pod prometheus-k8s-0 restarting and Persistent Volume Claim using the openebs-jiva-default storage class.

 kubectl get pvc -n monitoring
NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
prometheus-k8s-db-prometheus-k8s-0   Bound    pvc-a8923b60-bd4c-4fe0-b5a9-aa0c74d2939d   5Gi        RWO            openebs-jiva-default   13m

Now if one of your nodes goes down, prometheus can be started on a different node while retaining its data.