Using microk8s 1.18.2 on Ubuntu 20.04
Q1. Does microk8s keep track of all the change history? If so how to clean up all the old obsolete objects?
Q2. Is it normal that microk8s reset last a long time? (Didn’t finished since +2 hour as of this writing)
This sounds like a bug so I filed an issue here: https://github.com/ubuntu/microk8s/issues/1237
Details for the context of the questions above:
If I played around with the cluster for a while, creating Custom Resouce Definition, namespaces, deploy delete deploy multiple times. Could microk8s keep track of the deleted pods, containers, etc around?
I suspect microk8s hold a lots of old stuffs somewhere. I have made some mistakes playing around with addons. Made some addon update outside of microk8s and ended up with partial update. My deployment no longer work. I wanted to restart fresh by reinstalling from scratch by:
$ microk8s reset
$ sudo snap remove microk8s
It has been 30 minutes now and microk8s reset is not yet finished. The logs on the console showed all the past deployments:
event "demo-deployment-8d687867c-xrzbn.1612be9efc4cdab5" deleted
event "demo-deployment-8d687867c-xrzbn.1612be9f09e9545f" deleted
event "demo-deployment-8d687867c-xrzbn.1612be9f09fde2e2" deleted
... etc ...
(around hundreds lines like that)
namespace "mydemo" deleted
(console cursor blinking here)
I suppose the process is till working b/c the harddrive led is still blinking non stop.
MicroK8s is a Kubernetes distribution. Other than the history kept by Kubernetes, MicroK8s does not keep track of old objects.
No it is not normal for microk8s.reset to run for such a long time. The reset command can be found in [1] and is a script that would disable all addons and the cycle through all namespaces cleaning up any resources it can find. It is possible that one of these operations takes longer than expected. If you intend to remove the snap you do not need to do a reset first. I could look into why reset got stuck if I had a way to recreate the state the cluster was in.
[1] https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/wrappers/microk8s-reset.wrapper
What is the history kept by Kubernetes? Events or logs, it is certainly OK. But I hope Kubernetes won’t keep around the deleted objects which consumes large storage. Because I have limited storage space in my test cluster.
For example, I deploy a service that create a pod with having 3 containers. Then I delete it and redeploy it again. Something like that
$ kubectl apply -f myservice.yaml
$ kubectl delete -f myservice.yaml
$ kubectl apply -f myservice.yaml
It is mostly the container images that consume disk space. Kubernetes will not immediately reclaim the disk space of terminated pods instead it will garbage collect them later. In [1] you can see how to configure this mechanism. In MicroK8s the kubelet arguments are available in /var/snap/microk8s/current/args/kubelet. After updating them you will need to restart MicroK8s with microk8s.stop and microk8s.start.
[1] Configuring kubelet Garbage Collection - Kubernetes
Hello,
this interesting because i have removed the snap, and as i was debugging failure to start the latest snap on ubuntu 21.10, I decided to do a cleanup just in case, I launched microk8s reset command and it- is now removing add-ons that I installed for resting purpose in the previous install of the snap? is that normal behavior?
The easiest to do a clean microk8s is to do snap remove microk8s --purge.
Yes reset will remove addons
I did this first snap remove microk8s --purge and installed the snap again and without installing one single addon, just wanted to start a barebone install of microk8s but couldn’t successfully start it. as a kitchen-sink approach to debugging, I just ran microk8s reset and I was surprised to see it disabling addons that I tested a while ago on the previous and removed install of microk8s.
is snap remove microk8s --purge supposed to remove everything associated with microk8s? or are there stuff left behind that we should remove to do a completely clean install
It should remove everything associated to microk8s.