MicroK8s, Observability, Grafana Persistant Storage

General Discussions microk8s
,
1

Hello,

I am newish to Observability, so I added it to my MicroK8s using the addon.

I edited the congifmap to fix the URL endpoint and add an SMTP mail server, and everything is working. I can invite users to the Dashboard, mail goes how, and the user clicks invite; all works.

I have added an ingress to access externally.

I can create Dashboards etc.

In the minutes I do a rollout restart (Change in the config map), all the users and dashboards are lost.

The grafana.db does not persist by default; if I look at the deployment yaml in place, it’s configured with emptydir() volumes.

I have searched high and low; I find references to the manual installation of Prometheus but not much information on the MicroK8s version.

I find it hard to believe I’m the first to have this issue. Is there an article or example of adding PV & PVC to the MicroK8s install?

I would like to persist the grafana.db

thanks in advance

2

I have a working work-around

Well, I gave up trying to persist the grafana.db.

Using MySQL

Please Note: This is for educational use only; I offer nor inply any guarantees

I already have MySQL running in three different namespaces, so I configured Grafana to use one of those.

I will do my best to document it here if someone else wants to try.

Please Note: I changed some values to protect the innocent.
I’m also using an alias of kubectrl → k

Namespaces

Grafana lives in the observability namespace, and there is an instance of MySQL server in each of the following namespaces:

  • mynamespace-dev
  • mynamespace-uat
  • mynamespace-prod

I did not want another instance of MySQL running; it is possible to create a service (ExternalName) in one namespace that can talk to another service in another namespace.

Does this break the idea of Namespace isolation?

External Service

Create a service of the type ExternalName in the observability namespace, then in the spec:externalName, reference the service in the other namespace.

This will use the DNS server to connect to the service in the other namespace.
service.name.other-namespace.svc.cluster.local

grafana-mysql-ext-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: observability
  labels:
    app: mysql
spec:
  externalName: mysql.mynamespace-uat.svc.cluster.local
  ports:
  - port: 3306
    protocol: TCP
    targetPort: 3306
  sessionAffinity: None
  type: ExternalName

Start this service.

Edit Grafana ConfigMap.

List your config maps and find the Grafana one.

k get configmaps -n observability

Of course, you can edit the config map in place. However, I prefer to save my work as a YAML so I can rerun it later if needed.

k get configmaps  kube-prom-stack-grafana -n observability -o yaml

I copy the output and paste into a text editor

Add the database section

    [database]
    type = "mysql"
    host = "mysql"
    name = "grafana"
    user = "mysqluser"
    password = """k******gk"""  

  • type = "mysql" # Either `mysql` , `postgres` or `sqlite3` , it’s your choice.

  • host = "mysql" # Name of the ExternalName service you just created

  • name = "grafana" # Name of the database in MySQL

  • user = "mysqluser" #MySQL User

  • password = """k******gk"""   #MySQL UserPassword

Save config and apply

!important

You must create the database and grant access before restarting Grafana
In MySQL console, create data and grant access. If you do not have these two things in place the Grafana deployment will fail to restart.

CREATE DATABASE grafana;
GRANT ALL PRIVILEGES ON grafana.* TO 'mysqluser'@'%';

Restart Deployment

Now we have the service in place, and updated the Grafana config map to use MySQL.
!important, Created grafana database and granted access to the user in MySQL.

Time to restart grafana
Note
When you switch the database store, you will start with a default database, no users except admin, and no custom dashboards will be carried over.

kubectl rollout restart deployment kube-prom-stack-grafana -n observability

Refresh the Web page for Grafana, and you should now see a fresh Grafana instance.

If you go back into MySQL, the Grafana tables should now be created with default values.

Bonus

To save you port forwarding, here is my Ingress for Grafana

grafana-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  name: connect-analytics
  namespace: observability
spec:
  rules:
    - host: analytics.<yourdomain.com>
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: kube-prom-stack-grafana
                port:
                  number: 80
  tls: 
    - hosts:
        - analytics.<yourdomain.com>
      secretName: analytics.<yourdomain.com>-cert