# High Availability (HA) **URL:** https://discuss.kubernetes.io/t/high-availability-ha/11731 **Category:** microk8s **Tags:** docs **Created:** [July 3, 2020, 4:14pm UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731 "2020-07-03T16:14:34Z") **Posts on this page:** 20 **Page:** 1
### Author: ![evilnick](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/evilnick/32/3481_2.png) [@evilnick](https://discuss.kubernetes.io/u/evilnick) #### Post date: [July 3, 2020, 4:14pm UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/1 "2020-07-03T16:14:34Z")
> High availability is automatically enabled on MicroK8s for clusters with three or more nodes. A highly available Kubernetes cluster is a cluster that can withstand a failure on any one of its components and continue serving workloads without interruption. There are three components necessary for a highly available Kubernetes cluster: 1. There must be more than one node available at any time. 2. The control plane must be running on more than one node so that losing a single node would not render the cluster inoperable. 3. The cluster state must be in a datastore that is itself highly available. This documentation describes the steps needed to form an HA cluster in MicroK8s and to check its state. ## Setting up HA for MicroK8s To achieve HA, you will need: 1. To install a 1.19+ version of MicroK8s 2. At least three nodes. For testing on a single machine, please see the documentation for [installing on LXD](https://discuss.kubernetes.io/t/microk8s-in-lxd/11520) ### Install the first node On **Linux** , you can install any 1.19+ by specifying a channel: ```auto sudo snap install microk8s --classic --channel=1.19/stable ``` or update an existing installation with: ```auto sudo snap refresh microk8s --classic --channel=1.19/stable ``` For **Windows** and **macOS** , you can update your installation with: ```auto multipass exec microk8s -- sudo snap refresh microk8s --classic --channel=1.19/stable ``` (see the [install docs](https://discuss.kubernetes.io/t/alternative-installs-macos-windows-10-multipass/11257) for Windows and macOS if you need to install MicroK8s.) ### Add at least two other nodes As before, install a 1.19+ version of MicroK8s on at least two additional machines (or LXD containers). Follow the usual procedure for clustering (described in the [clustering documentation](https://discuss.kubernetes.io/t/clustering-with-microk8s/11276)): On the initial node, run: ```auto microk8s add-node ``` This will output a command with a generated token such as `microk8s join 10.128.63.86:25000/567a21bdfc9a64738ef4b3286b2b8a69`. Copy this command and run it from the next node. It may take a few minutes to successfully join. Repeat this process (generate a token, run it from the joining node) for the third and any additional nodes. ### Set failure domains > Available with 1.20+ To make MicroK8s failure domain aware associate an integer to each failure domain and update the `/var/snap/microk8s/current/args/ha-conf` with it. A restart of MicroK8s in the updated nodes is required (`microk8s.stop; microk8s.start`). For example: ```auto echo "failure-domain=42" > /var/snap/microk8s/current/args/ha-conf microk8s.stop microk8s.start ``` ### Check the status Run the status command: ```auto microk8s status ``` From MicroK8s version 1.19, this will now inform you of the HA status and the addresses and roles of additional nodes. For example: ```auto microk8s is running high-availability: yes datastore master nodes: 10.128.63.86:19001 10.128.63.166:19001 10.128.63.43:19001 datastore standby nodes: none ``` ## Working with HA All nodes of the HA cluster run the master control plane. A subset of the cluster nodes (at least three) maintain a copy of the Kubernetes [dqlite](https://dqlite.io/) database. Database maintenance involves a voting process through which a leader is elected. Apart from the voting nodes there are non-voting nodes silently keeping a copy of the database. These nodes are on standby to take over the position of a departing voter. Finally, there are nodes that neither vote nor replicate the database. These nodes are called `spare`. To sum up, the three node roles are: **voters** : replicating the database, participating in leader election **standby** : replicating the database, not participating in leader election **spare** : not replicating the database, not participating in leader election Cluster formation, database syncing, voter and leader elections are all transparent to the administrator. The state of the current state of the HA cluster is shown with: ```auto microk8s status ``` The output of the HA inspection reports: - If HA is achieved or not. - The voter and stand-by nodes. Since all nodes of the HA cluster run the master control plane the `microk8s *` commands are now available everywhere. Should one of the nodes crash we can move to any other node and continue working without much disruption. Almost all of the HA cluster management is transparent to the admin and requires minimal configuration. The administrator can only add or remove nodes. To ensure the health of the cluster the following timings should be taken into account: - If the leader node gets “removed” ungracefully, e.g. it crashes and never comes back, it will take up to 5 seconds for the cluster to elect a new leader. - Promoting a non-voter to a voter takes up to 30 seconds. This promotion takes place when a new node enters the cluster or when a voter crashes. To remove a node gracefully, first run the `leave` command on the departing node: ```auto microk8s leave ``` The node will be marked as ‘NotReady’ (unreachable) in Kubernetes. To complete the removal of the departing node, issue the following on any of the remaining nodes: ```auto microk8s remove-node ``` In the case we are not able to call `microk8s leave` from the departing node, e.g. due to a node crash, we need to call `microk8s remove-node` with the `--force` flag: ```auto microk8s remove-node --force ``` ### Add-ons on an HA cluster Certain add-ons download and “install” client binaries. These binaries will be available only on the node the add-on was enabled from. For example, the helm client that gets installed with `microk8s enable helm` will be available only on the node the user issued the `microk8s enable` command. ## Upgrading an existing cluster If you have an existing cluster, you have to refresh all nodes to at least v1.19, for example: ```auto sudo snap refresh microk8s --channel=1.19/stable ``` You then need to enable HA clustering on the master node: ```auto microk8s enable ha-cluster ``` Any machines which are already nodes in a cluster will need to exit and rejoin in order to establish HA. To do so, cycle through the nodes to drain, remove, and rejoin them: ```auto microk8s kubectl drain --ignore-daemonsets ``` On the node machine, force it to leave the cluster with: ```auto microk8s leave ``` Then enable HA with `microk8s enable ha-cluster` and re-join the node to the cluster with a `microk8s add-node` and `microk8s join` issued on the master and node respectively. ## What about an etcd based HA? MicroK8s ships the upstream Kubernetes so an etcd HA setup is also possible, see the upstream documentation on how this can be achieved: [Kubernetes HA topology docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/) The etcd approach is more involved and outside the scope of this document. Overall you will need to maintain your own etcd HA cluster. You will then need to configure the API server and flannel to point to that etcd. Finally you will need to provide a load balancer in front of the nodes acting as masters and configure the workers to reach the masters through the load-balanced endpoint. ---
### Author: ![Alex\_Chalkias](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/alex_chalkias/32/5498_2.png) [@Alex\_Chalkias](https://discuss.kubernetes.io/u/Alex_Chalkias) #### Post date: [July 9, 2020, 9:31am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/2 "2020-07-09T09:31:28Z")
The output of the add-node cmd can be improved to avoid people expecting both master and worker nodes. I suggest the following: “This will output a command with a generated token such as ‘microk8s join ip-address:port/token’. Copy this command and run it on the second node you want to distribute the control plane to. It may […]” ---
### Author: ![evilnick](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/evilnick/32/3481_2.png) [@evilnick](https://discuss.kubernetes.io/u/evilnick) #### Post date: [July 9, 2020, 11:22am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/3 "2020-07-09T11:22:24Z")
there is already a PR for this: [https://github.com/ubuntu/microk8s/pull/1389](https://github.com/ubuntu/microk8s/pull/1389) ---
### Author: ![Alex\_Chalkias](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/alex_chalkias/32/5498_2.png) [@Alex\_Chalkias](https://discuss.kubernetes.io/u/Alex_Chalkias) #### Post date: [July 9, 2020, 11:36am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/4 "2020-07-09T11:36:30Z")
Thanks. Should I add my comment there then? ---
### Author: ![kjackal](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/kjackal/32/1750_2.png) [@kjackal](https://discuss.kubernetes.io/u/kjackal) #### Post date: [July 9, 2020, 11:40am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/5 "2020-07-09T11:40:52Z")
For sure the PR is needed. But I think the ask from Alex is slightly different. In our docs we say “… a generated token such as microk8s join master:25000/DDOkUupkmaBezNnMheTBqFYHLWINGDbf” but in an HA setup there is no master, all nodes are acting as masters. Furthermore, the fact that we say “master” implies that there is also a worker which is not right. ---
### Author: ![evilnick](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/evilnick/32/3481_2.png) [@evilnick](https://discuss.kubernetes.io/u/evilnick) #### Post date: [July 9, 2020, 11:46am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/6 "2020-07-09T11:46:59Z")
Yeah, I agree. I think the confusion about nodes should be addressed in the output from the command as the PR currently does. The master/worker issue should probably be resolved in the docs. In future will ALL clusters be ha-enabled? is it easy to change the output of add-node depending on whether HA is enabled? ---
### Author: ![kjackal](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/kjackal/32/1750_2.png) [@kjackal](https://discuss.kubernetes.io/u/kjackal) #### Post date: [July 9, 2020, 11:52am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/7 "2020-07-09T11:52:04Z")
Only pre-1.19 clusters will have a master node. It is relatively easy to change the output of add-node depending on whether HA is enabled. I see we do not mention the work “master” in the add-node output. What do you have in mind? ---
### Author: ![evilnick](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/evilnick/32/3481_2.png) [@evilnick](https://discuss.kubernetes.io/u/evilnick) #### Post date: [July 9, 2020, 12:01pm UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/8 "2020-07-09T12:01:59Z")
From 1.19 what happens in the case of only two nodes then? are they master/worker or a not-ready HA cluster? If HA becomes the only type of cluster then we merge the HA and cluster docs (possibly keep a legacy page for pre-1.19) and most of this can be settled there, then we only need to have one output command. ---
### Author: ![evilnick](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/evilnick/32/3481_2.png) [@evilnick](https://discuss.kubernetes.io/u/evilnick) #### Post date: [July 9, 2020, 12:04pm UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/9 "2020-07-09T12:04:22Z")
@Alex_Chalkias I updated the example command in this doc from the current output 👍 ---
### Author: ![kjackal](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/kjackal/32/1750_2.png) [@kjackal](https://discuss.kubernetes.io/u/kjackal) #### Post date: [July 9, 2020, 12:05pm UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/10 "2020-07-09T12:05:05Z")
> [@evilnick](#): > > From 1.19 what happens in the case of only two nodes then? are they master/worker or a not-ready HA cluster? We report `high-availability: no`. Both nodes can act as K8s masters but only one of them is the datastore master. > [@evilnick](#): > > If HA becomes the only type of cluster then we merge the HA and cluster docs (possibly keep a legacy page for pre-1.19) and most of this can be settled there, then we only need to have one output command. +1 makes sense. ---
### Author: ![balchua1](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/balchua1/32/5372_2.png) [@balchua1](https://discuss.kubernetes.io/u/balchua1) #### Post date: [July 10, 2020, 7:21am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/11 "2020-07-10T07:21:27Z")
Wanted to check here, is there plan to have an HA control plane while maintaining worker nodes separately? The advantage of having a “worker” only nodes is for constrained environments. There’s no need to run control plane components. ---
### Author: ![balchua1](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/balchua1/32/5372_2.png) [@balchua1](https://discuss.kubernetes.io/u/balchua1) #### Post date: [July 10, 2020, 10:29am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/12 "2020-07-10T10:29:05Z")
From 1.19+ MicroK8s version, there will no need to do `microk8s enable ha-cluster`? Which means if i understand correctly, microk8s will form an HA cluster as soon as the user performs the following. ```auto $ microk8s add-node ``` Followed by (on a different node) ```auto $ microk8s join ..... ``` Thanks! ---
### Author: ![evilnick](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/evilnick/32/3481_2.png) [@evilnick](https://discuss.kubernetes.io/u/evilnick) #### Post date: [July 10, 2020, 11:04am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/13 "2020-07-10T11:04:20Z")
> [@balchua1](#): > > From 1.19+ MicroK8s version, there will no need to do `microk8s enable ha-cluster`? Yes, that’s the plan. You will need 3 nodes for HA. ---
### Author: ![balchua1](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/balchua1/32/5372_2.png) [@balchua1](https://discuss.kubernetes.io/u/balchua1) #### Post date: [July 10, 2020, 11:39am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/14 "2020-07-10T11:39:48Z")
Thanks. 👍 So HA is available as long as there are more than 2 nodes? ---
### Author: ![evilnick](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/evilnick/32/3481_2.png) [@evilnick](https://discuss.kubernetes.io/u/evilnick) #### Post date: [July 13, 2020, 11:46am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/15 "2020-07-13T11:46:33Z")
> [@balchua1](#): > > So HA is available as long as there are more than 2 nodes? That is the intention, yes. ---
### Author: ![balchua1](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/balchua1/32/5372_2.png) [@balchua1](https://discuss.kubernetes.io/u/balchua1) #### Post date: [August 14, 2020, 5:54am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/16 "2020-08-14T05:54:09Z")
Quick question, does using dqlite over etcd still imposes the 1MB resource object size limit? I am guessing no, since sqlite can have records larger than that. Just checking. ---
### Author: ![VladoPortos](https://avatars.discourse-cdn.com/v4/letter/v/9e8a1a/32.png) [@VladoPortos](https://discuss.kubernetes.io/u/VladoPortos) #### Post date: [October 17, 2020, 5:52pm UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/17 "2020-10-17T17:52:19Z")
Is this different method form standard 3 node master and rest worked mode ? How can I manually define which nodes are “datastore master” and “datastore standby” ? ---
### Author: ![kjackal](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/kjackal/32/1750_2.png) [@kjackal](https://discuss.kubernetes.io/u/kjackal) #### Post date: [October 18, 2020, 4:47am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/18 "2020-10-18T04:47:28Z")
Hi @VladoPortos There is currently no way to set the role of each node. In future releases we will allow users to specify failure domains and that would indirectly affect where data store masters are located. With MicroK8s (at least for now) we try to remain true to zero-ops. If you need more flexibility you may want to look into other K8s distributions. Canonical ships [Charmed Kubernetes](https://ubuntu.com/kubernetes/features), you may want to look into that or anything else that suits your needs. ---
### Author: ![VladoPortos](https://avatars.discourse-cdn.com/v4/letter/v/9e8a1a/32.png) [@VladoPortos](https://discuss.kubernetes.io/u/VladoPortos) #### Post date: [October 18, 2020, 10:43am UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/19 "2020-10-18T10:43:43Z")
Yes, i get it, for developers its great, copy paste 2 commands and kubernetes cluster is up and running… but for the infrastructure / OS admins ( or somebody who wants to learn kubernetes platform ) its not ideal since as it is now its not something you would do in production. I’ll switch back to K3s or K8s for learning kubernetes it self. Thanks for reply 🙂 I’ll keep an eye in MicroK8s in the future though 🙂 ---
### Author: ![JensF](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/jensf/32/6491_2.png) [@JensF](https://discuss.kubernetes.io/u/JensF) #### Post date: [December 16, 2020, 10:33pm UTC](https://discuss.kubernetes.io/t/high-availability-ha/11731/20 "2020-12-16T22:33:47Z")
Dear all, I am running a 4-node Raspberry PI4 (4GB RAM) Cluster and used to work with kubespray and Kubernetes v1.19 It is a playground right now to get myself firm in working and configuring K8S as well as workloads running on top of it. I wanted to try the latest version v1.20 and came back to MicroK8S since I really like the seamless installation and speed over using Kubespray (Raspis are a bit slow, especially only 1 node has a SSD attached). Now while HA is awesome, I would like to configure it. 2 Nodes out of my cluster could be the HA-pair for the K8S-Master and any other nodes should be workers. Unfortunatelly it seems that K8S HA on Raspi4 is killing performance significantly (simple web server deploys take easily 5 minutes) and I regularly get databsae is locked error messages. Any means I can manually “disable” HA and use my cluster as I did in previous installtions? Trying to remove nodes and re-adding does not help since it automatically jumps back to HA as soon as 3 nodes become present. Thx for help. JF [Next page](https://discuss.kubernetes.io/t/high-availability-ha/11731.md?page=2)