Microk8s cluster stuck after installing OLM 0.14.1

General Discussions microk8s
#1

Hi,

Using microk8s 1.18.6 stable. I tried to run this execise Cassandra in Kubernetes. Part of the requirement is to install Operator Lifecycle Manager version 0.14.1 using this script

Once I installed OLM 0.14.1, I noticed that the microk8s has significantly slowed down. Took about 2 hours for the operator pod to be stabilized in running status. A few hours later, all pods fails with an error like this (seen by kubectl describe pod the-pod-id)

  Warning  FailedMount  17m (x140 over 4h48m)  kubelet, silverbullet  MountVolume.SetUp failed for volume "cass-operator-token-fjmnd" : failed to sync secret cache: timed out waiting for the condition

OR this

Events:
  Type     Reason            Age                   From                   Message
  ----     ------            ----                  ----                   -------
  Warning  FailedScheduling  4m21s (x39 over 19m)  default-scheduler      0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Warning  FailedMount       18s (x8 over 89s)     kubelet, silverbullet  MountVolume.SetUp failed for volume "grafana-operator-token-p45h6" : failed to sync secret cache: timed out waiting for the condition

Attempting to cleanup, I tried to delete all the resources created by the OLM operators and also delete the namespace which was used for the exercise. But the namespaces are stuck in a Terminating status.

$ kubectl get namespaces

NAME                 STATUS        AGE
cass-operator        Terminating   11h
olm                  Terminating   9h
operators            Terminating   9h
... skip remaining Active namespaces ...

QUESTION: is there a way to get microk8s cluster back to its normal working status and to finish the deletion of the namespaces?

I have the tarball of microk8s inspect (Google Drive) can you suggest a proper way to submit attachment in this board? Or would it be better I file an issue in https://github.com/ubuntu/microk8s/issues and attach the inspection report there?

How to update authentication info of ~/kube/config after re-install of microk8s?
#2

OLM requires to have rbac enabled. I remember an issue regarding OLM. Usually namespaces stuck in Terminating state is caused by CRDs or Apiservice which are not deleted.
If it is ok for you to do sudo snap remove microk8s then sudo snap install microk8s --channel=<the channel> --classic.

#3

I have had RBAC enabled. I have attempted to delete the CRDs many times. Even stop/start microk8s or the rebooting the machine didn’t help.

I can of course remove and reinstall microk8s. But it would be nice to know the root cause or a way to force the deletion of CRDs. I don’t know if this kind of problem would occur often. But the solution to delete the entire cluster is quite time consuming.

#4

I sometimes follow this article https://www.ibm.com/support/knowledgecenter/SSBS6K_3.2.0/troubleshoot/ns_terminating.html
It is an old one, but seems to point me to the right direction.