# Recover from lost quorum **URL:** \ **Category:** microk8s\ **Tags:** docs\ **Created:** [February 23, 2022, 3:00pm UTC](https://discuss.kubernetes.io/t/recover-from-lost-quorum/19160 "2022-02-23T15:00:37Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![evilnick](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/evilnick/32/3481_2.png) [@evilnick](https://discuss.kubernetes.io/u/evilnick)\ **Post date:** [February 23, 2022, 3:00pm UTC](https://discuss.kubernetes.io/t/recover-from-lost-quorum/19160/1 "2022-02-23T15:00:37Z")
By default, a three-node MicroK8s cluster automatically becomes highly available (HA). In HA mode the default datastore (dqlite) implements a [Raft](https://raft.github.io/) based protocol where an elected leader holds the definitive copy of the database. Under normal operation copies of the database are maintained by two more nodes. If you permanently lose the majority of the cluster members that serve as database nodes (for example, if you have a three-node cluster and you lose two of them), the cluster will become unavailable. However, if at least one database node has survived, you will be able to recover the cluster with the following manual steps. > ⓘ **Note:** that the following recovery process applies to clusters using the default (dqlite) datastore of MicroK8s only. This process does not recover any data you have in PVs on the lost nodes. ## Stop dqlite on all nodes Stopping MicroK8s is done with: ```bash microk8s stop ``` You must also make sure the lost nodes that used to form the cluster will not come back alive again. Any lost nodes that can be reinstated will have to re-join the cluster with the `microk8s add-node` and `microk8s join` process (see the [documentation on clusters](https://discuss.kubernetes.io/t/create-a-microk8s-cluster/11276)). ## Backup the database Dqlite stores data and configuration files under `/var/snap/microk8s/current/var/kubernetes/backend/`. To make a safe copy of the current state log in to a surviving node and create tarball of the dqlite directory: ```auto tar -cvf backup.tar /var/snap/microk8s/current/var/kubernetes/backend ``` ## Set the new state of the database cluster Under `/var/snap/microk8s/current/var/kubernetes/backend` the file `cluster.yaml` reflects the state of the cluster as dqlite sees it. Edit this file to remove the lost nodes leaving only the ones available. For example, let’s assume a three-node cluster with nodes `10.211.205.12`, `10.211.205.253` and `10.211.205.221` where `10.211.205.12` and `10.211.205.221` are lost. In this case the `cluster.yaml` will look like this: ```auto - Address: 10.211.205.122:19001 ID: 3297041220608546238 Role: 0 - Address: 10.211.205.253:19001 ID: 9373968242441247628 Role: 0 - Address: 10.211.205.221:19001 ID: 3349965773726029294 Role: 0 ``` By removing the lost nodes `10.211.205.122` and `10.211.205.221` the `cluster.yaml` should be left with only the `10.211.205.253` entry. For this example, we convert to a single node. You may choose to include more nodes in the new cluster. ```auto - Address: 10.211.205.253:19001 ID: 9373968242441247628 Role: 0 ``` ## Reconfigure dqlite MicroK8s comes with a dqlite client utility for node reconfiguration. The command to run is: ```auto sudo /snap/microk8s/current/bin/dqlite \ -s 127.0.0.1:19001 \ -c /var/snap/microk8s/current/var/kubernetes/backend/cluster.crt \ -k /var/snap/microk8s/current/var/kubernetes/backend/cluster.key \ k8s ".reconfigure /var/snap/microk8s/current/var/kubernetes/backend/ /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml" ``` The `/snap/microk8s/current/bin/dqlite` utility needs to be called with `sudo` and takes the following arguments: - the endpoint to the (now stopped) dqlite service. We have used `-s 127.0.0.1:19001` for this endpoint in the example above. - the private and public keys needed to access the database. These keys are passed with the `-c` and `-k` arguments and are found in the directory where dqlite keeps the database. - the name of the database. For MicroK8s the database is `k8s`. - the operation to be performed in this case is “reconfigure” - the path to the database we want to reconfigure is the current database under `/var/snap/microk8s/current/var/kubernetes/backend` - the end cluster configuration we want to recreate is reflected in the `cluster.yaml` we edited in the previous step. ## Update the rest of the cluster nodes Copy the `cluster.yaml`, `snapshot-abc-abc-abc`, `snapshot-abc-abc-abc.meta` and segment files `00000abcxx-00000abcxx`) from the node where you ran the `reconfigure` command in the previous step on all other nodes mentioned in the `cluster.yaml` file. Further, on all nodes create an `info.yaml` that is in line with the `info.yaml` file you created previously. > _WARNING_: Make sure to delete any leftover `snapshot-abc-abc-abc`, `snapshot-abc-abc-abc.meta`, segment (`00000abcxx-000000abcxx`, `open-abc`) and `metadata{1,2}` files that it contains. This is important otherwise the nodes will fail to cleanly rejoin the node. ## Restart MicroK8s services It should now be possible to bring the cluster back online with: ```auto microk8s start ``` ## Cleanup the lost nodes from Kubernetes The lost nodes are registered in Kubernetes but should be reporting as `NotReady` in: ```auto microk8s kubectl get no ``` To remove the lost nodes use: ```auto microk8s remove-node ``` ## Restore HA High availability will be reattained when there are three or more nodes in the MicroK8s cluster. If the original failed nodes have been revived, or new nodes created, these can be joined to the cluster to restore high availability. See the [documentation on clusters](https://discuss.kubernetes.io/t/create-a-microk8s-cluster/11276) for instructions on adding nodes. ## References - [GitHub issue](https://github.com/canonical/microk8s/issues/2898#issuecomment-1026708893) ---
**Author:** ![canerbis](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/canerbis/32/9637_2.png) [@canerbis](https://discuss.kubernetes.io/u/canerbis)\ **Post date:** [March 23, 2022, 4:33am UTC](https://discuss.kubernetes.io/t/recover-from-lost-quorum/19160/2 "2022-03-23T04:33:30Z")
Thank you for your proceduers for recovering. You archived the dqlite database file with tar. ex) tar -cvf backup.tar /var/snap/microk8s/current/var/kubernetes/backend When do you extract the archived dqlite database file ? ---
**Author:** ![kjackal](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/kjackal/32/1750_2.png) [@kjackal](https://discuss.kubernetes.io/u/kjackal)\ **Post date:** [March 23, 2022, 2:05pm UTC](https://discuss.kubernetes.io/t/recover-from-lost-quorum/19160/3 "2022-03-23T14:05:31Z")
Hi @canerbis, the tarball we create at the very beginning is for safety reasons. We do not use it in the recovery process, we just keep it in case we need to revert to our initial state. ---
**Author:** ![canerbis](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/canerbis/32/9637_2.png) [@canerbis](https://discuss.kubernetes.io/u/canerbis)\ **Post date:** [March 24, 2022, 12:02am UTC](https://discuss.kubernetes.io/t/recover-from-lost-quorum/19160/4 "2022-03-24T00:02:59Z")
> [@kjackal](#): > > the tarball we create at the very beginnin I understood it mean tarball’s job is backup to prepare for failure. Thanks a lot. ---
**Author:** ![phong-phan](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/phong-phan/32/17956_2.png) [@phong-phan](https://discuss.kubernetes.io/u/phong-phan)\ **Post date:** [January 4, 2026, 6:55am UTC](https://discuss.kubernetes.io/t/recover-from-lost-quorum/19160/5 "2026-01-04T06:55:23Z")
I think there some missing typing here in the “Set the new state of the database cluster” section: For example, let’s assume a three-node cluster with nodes `10.211.205.12`, `10.211.205.253` and `10.211.205.221` where `10.211.205.12` and `10.211.205.221` are lost. In this case the `cluster.yaml` will look like this It should be “10.211.205.122” instead of “10.211.205.12”