Question about overlay rootfs, shm and /dev/loopX

General Discussions microk8s
1

Env:
snap 2.43.3-1.el7
snapd 2.43.3-1.el7
series 16
centos 7
kernel 3.10.0-1062.18.1.el7.x86_64

Microk8s: v1.18.9

During evaluating of Microk8s we observed something we don’t understand in the output of df -h:
/dev/loop1 193M 193M 0 100% /var/lib/snapd/snap/microk8s/1609
/dev/loop0 191M 191M 0 100% /var/lib/snapd/snap/microk8s/1702
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/67f3dde8e087dfdc3abbfdcf615a3ed89821c364ec93bb4ef4bc32b381b0ff4d/shm
overlay 18G 13G 4.4G 75% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/67f3dde8e087dfdc3abbfdcf615a3ed89821c364ec93bb4ef4bc32b381b0ff4d/rootfs
overlay 18G 13G 4.4G 75% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/3a7991d1dd4615629b73a77878cc2cb791b5afa92cb061c04f019205c29d9fd2/rootfs
overlay 18G 13G 4.4G 75% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/ec49b9de2fb84bc8692072b08fc60fe2b6a1f3c7bd2057812dac3fa20945f644/rootfs overlay 18G 13G 4.4G 75% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/ed0f5f8d04f40c580f239db0dd6557385b988aed20b6f064fcaa5aaa785f20b3/rootfs

There are many of those entries and the summed up size is beyond the disk space we have there. We’ll really appreciate if someone can help explain why. Also thankful for pointing out how we can clean this up or configure it correctly.

We were not able to find similar questions and apologize if it is a duplicated question, but please tell us the link to the other one. And it could be a question of snapd, not of microk8s.

2

If my memory serves me right, these are the images, container metadata, logs etc used by the running containers.

For more info, you can get it from here

3

Thanks for the reply. Just tested: (let mk be microk8s)

  • mk stop: those entries still there
  • mk start: entries there, but less

It seems that some of them only get cleaned up after restarting. Wondering if we can tell mk to use the locally installed docker?

4

When pods are killed, containerd will do garbage collection to clean up the old data then create new ones.
What you are seeing is probably transient.

Yes you can use your own docker, just like any kubernetes. You may have to change the kubelet args in /var/snap/microk8s/current/args/kubelet.
I’ve never done it before though.

5

I wish for better documentation of mk as Christmas present this year. Is that what you mentioned documented anywhere?

6

Its not in Microk8s docs. Probably some googling can help.
Just some point to remember, all control plane components arguments are in /var/snap/microk8s/current/args. While the certificates are here /var/snap/microk8s/current/certs. In case you will need them.