Microk8s storage configuration

General Discussions
1

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: 1.16.3
Cloud being used: edis.at
Installation method: microk8s
Host OS: ubuntu 18.04
CNI and version:
CRI and version:

The documentation is not clear on how I can configure the storage path.

I enabled the storage with microk8s.enable storage, and by default it uses /var/snap/microk8s/common/default-storage. I would like to change it to /data/, for example, because my root is small (10G). How can I change the pv-volume to be used by the internal registry (microk8s.enable registry)?

2

I looked around to see how you can override SNAP_COMMON env variable on a per-snap basis, but all the recommendations I found suggested you simply bind mount onto the /var/snap/microk8s/common/default-storage/ path

I take that to mean that you could completely shutdown/disable microk8s, save any data that’s currently in /var/snap/microk8s/common/default-storage/ , then mount your larger partition onto /var/snap/microk8s/common/default-storage/ and restore the previously saved data (if any)

Note that you cannot symlink /var/snap/microk8s/common/default-storage/ to another location, you must use a mount.

3

I have attempted this by uninstalling microk8s completely, then mounting a folder on a large partition (/dev/sda1)

sudo mount --bind /media/DATA/kubernetes_default_snap_storage/ /var/snap/microk8s/common/default-storage

Then reinstalled microk8s and enabled the registry addon which in turn uses the storage addons and a persistent volume claim of 20 GB with a volume at:

kubectl describe -n kube-system pod/hostpath-provisioner-75fdc8fccd-t4ttk
...
Volumes:
  pv-volume:
    Type:          HostPath (bare host directory volume)
    Path:          /var/snap/microk8s/common/default-storage

I can confirm the mount seems to be working fine:

(thriai37) ben@ben-asus /media/DATA/kubernetes_default_snap_storage $ df -h /var/snap/microk8s/common/default-storage
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       1.7T  1.5T  195G  89% /var/snap/microk8s/common/default-storage

But despite this the registry/storage plugins are failing to successfully to load for what seems to be due to insufficient space based on the below:

(thriai37) ben@ben-asus ~ $ kubectl describe -n kube-system pod/hostpath-provisioner-75fdc8fccd-t4ttk
Name:           hostpath-provisioner-75fdc8fccd-t4ttk
Namespace:      kube-system
Priority:       0
Node:           ben-asus/
Start Time:     Thu, 14 May 2020 10:58:17 +0900
Labels:         k8s-app=hostpath-provisioner
                pod-template-hash=75fdc8fccd
Annotations:    <none>
Status:         Failed
Reason:         Evicted
Message:        The node was low on resource: ephemeral-storage. Container hostpath-provisioner was using 64Ki, which exceeds its request of 0. 
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/hostpath-provisioner-75fdc8fccd
Containers:
  hostpath-provisioner:
    Image:      cdkbot/hostpath-provisioner-amd64:1.0.0
    Port:       <none>
    Host Port:  <none>
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
      PV_DIR:     /var/snap/microk8s/common/default-storage
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from microk8s-hostpath-token-sc7xn (ro)
      /var/snap/microk8s/common/default-storage from pv-volume (rw)
Volumes:
  pv-volume:
    Type:          HostPath (bare host directory volume)
    Path:          /var/snap/microk8s/common/default-storage
    HostPathType:  
  microk8s-hostpath-token-sc7xn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  microk8s-hostpath-token-sc7xn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  2m39s (x3 over 2m49s)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Normal   Scheduled         2m35s                  default-scheduler  Successfully assigned kube-system/hostpath-provisioner-75fdc8fccd-t4ttk to ben-asus
  Normal   Pulling           2m34s                  kubelet, ben-asus  Pulling image "cdkbot/hostpath-provisioner-amd64:1.0.0"
  Normal   Pulled            2m30s                  kubelet, ben-asus  Successfully pulled image "cdkbot/hostpath-provisioner-amd64:1.0.0"
  Normal   Created           2m30s                  kubelet, ben-asus  Created container hostpath-provisioner
  Normal   Started           2m29s                  kubelet, ben-asus  Started container hostpath-provisioner
  Warning  Evicted           100s                   kubelet, ben-asus  The node was low on resource: ephemeral-storage. Container hostpath-provisioner was using 64Ki, which exceeds its request of 0.
  Normal   Killing           100s                   kubelet, ben-asus  Stopping container hostpath-provisioner

Any idea what could be going on here? Thanks!

4

Aha I think I have found a completely different solution for this based on https://github.com/ubuntu/microk8s/issues/463#issuecomment-491285745

I.e. stop microk8s, edit that file (/var/snap/microk8s/current/args/containerd) to where you want to put the data, then start microk8s. I still had issue of 0 capacity being reported when I first tried this but I believe this was because docker was using overlay2 driver which required XFS formatted partition, so I made a new partiion which was XFS and used this to solve that, now seems to be working.

5

This does not change the PVC/PV capacity/capacity requests however. It also seems that even after doing this; when I load lots of images into a local registry (enable.registry) extension, then the root drive partition containing snap expands dramatically in size, so I ran into space issues regardless.

6

Has anyone found an actual working solution to this issue? I tried using a symlink and that sort of works, but not really.

Does the mount actually work?