Question about overlay rootfs, shm and /dev/loopX

kenix · November 18, 2020, 11:12am

Env:
snap 2.43.3-1.el7
snapd 2.43.3-1.el7
series 16
centos 7
kernel 3.10.0-1062.18.1.el7.x86_64

Microk8s: v1.18.9

During evaluating of Microk8s we observed something we don’t understand in the output of df -h:
/dev/loop1 193M 193M 0 100% /var/lib/snapd/snap/microk8s/1609
/dev/loop0 191M 191M 0 100% /var/lib/snapd/snap/microk8s/1702
shm 64M 0 64M 0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/67f3dde8e087dfdc3abbfdcf615a3ed89821c364ec93bb4ef4bc32b381b0ff4d/shm
overlay 18G 13G 4.4G 75% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/67f3dde8e087dfdc3abbfdcf615a3ed89821c364ec93bb4ef4bc32b381b0ff4d/rootfs
overlay 18G 13G 4.4G 75% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/3a7991d1dd4615629b73a77878cc2cb791b5afa92cb061c04f019205c29d9fd2/rootfs
overlay 18G 13G 4.4G 75% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/ec49b9de2fb84bc8692072b08fc60fe2b6a1f3c7bd2057812dac3fa20945f644/rootfs overlay 18G 13G 4.4G 75% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/ed0f5f8d04f40c580f239db0dd6557385b988aed20b6f064fcaa5aaa785f20b3/rootfs

There are many of those entries and the summed up size is beyond the disk space we have there. We’ll really appreciate if someone can help explain why. Also thankful for pointing out how we can clean this up or configure it correctly.

We were not able to find similar questions and apologize if it is a duplicated question, but please tell us the link to the other one. And it could be a question of snapd, not of microk8s.

balchua1 · November 18, 2020, 11:34am

If my memory serves me right, these are the images, container metadata, logs etc used by the running containers.

For more info, you can get it from here

github.com

containerd/containerd/blob/master/docs/ops.md

# containerd for Ops and Admins

containerd is meant to be a simple daemon to run on any system.
It provides a minimal config with knobs to configure the daemon and what plugins are used when necessary.

```
NAME:
   containerd -
                    __        _                     __
  _________  ____  / /_____ _(_)___  ___  _________/ /
 / ___/ __ \/ __ \/ __/ __ `/ / __ \/ _ \/ ___/ __  /
/ /__/ /_/ / / / / /_/ /_/ / / / / /  __/ /  / /_/ /
\___/\____/_/ /_/\__/\__,_/_/_/ /_/\___/_/   \__,_/

high performance container runtime


USAGE:
   containerd [global options] command [command options] [arguments...]

This file has been truncated. show original

kenix · November 18, 2020, 12:17pm

Thanks for the reply. Just tested: (let mk be microk8s)

mk stop: those entries still there
mk start: entries there, but less

It seems that some of them only get cleaned up after restarting. Wondering if we can tell mk to use the locally installed docker?

balchua1 · November 18, 2020, 1:27pm

When pods are killed, containerd will do garbage collection to clean up the old data then create new ones.
What you are seeing is probably transient.

Yes you can use your own docker, just like any kubernetes. You may have to change the kubelet args in /var/snap/microk8s/current/args/kubelet.
I’ve never done it before though.

kenix · November 18, 2020, 1:31pm

I wish for better documentation of mk as Christmas present this year. Is that what you mentioned documented anywhere?

balchua1 · November 18, 2020, 1:41pm

Its not in Microk8s docs. Probably some googling can help.
Just some point to remember, all control plane components arguments are in /var/snap/microk8s/current/args. While the certificates are here /var/snap/microk8s/current/certs. In case you will need them.