Using an external etcd with MicroK8s

neoaggelos · December 8, 2021, 10:12am

This guide will take you through the process of using an external etcd as the underlying data store for MicroK8s. This makes sense depending on your use case, for example:

In production environments, you want your data store to run in designated nodes (for example, nodes with SSD disks).
You want to manage the data store yourself, instead of depending on MicroK8s for backup and restore functionality.

It is important to note that while migrating a running MicroK8s cluster from dqlite to an external etcd should be theoretically possible, it is highly discouraged. The instructions below are meant to be followed in a fresh MicroK8s installation.

Further, any existing resources and workloads that might be running in the cluster will be lost.

Deploy an external etcd cluster

The first step is to deploy the etcd cluster you wish to use. If you are installing it manually or using an existing installation, the section below details the information you will need to gather from the cluster. Alternatively, you can use Juju to deploy etcd, also detailed below.

Deploy manually

Deploy your etcd cluster using your method of choice. There are no limitations other than making sure that your etcd members are reachable from MicroK8s.

After deploying, you will need the following:

List of addresses of the etcd members, for example https://10.10.0.1:2379,https://10.10.0.2:2379,https://10.10.0.3:2379.
CA certificate for the etcd cluster, we will refer to this as ca.crt.
Client certificate for connecting to the etcd cluster, we will refer to this as client.crt.
Client key for connecting to the etcd cluster, we will refer to this as client.key.

Deploy etcd in local LXD containers using Juju

Use the following commands to deploy a simple etcd cluster using Juju, running in LXD containers in a local machine:

If not already installed, you should install and configure LXD:

sudo snap install lxd
sudo lxd init --auto
lxc network set lxdbr0 ipv6.address none

Similarly, install Juju via snap if it isn’t already installed:

sudo snap install juju --classic

Juju can then be used to deploy an etcd cluster

juju bootstrap lxd
juju deploy containers-easyrsa
juju deploy etcd -n 3
juju integrate easyrsa etcd

You can then check the status of the install using:

watch -d -c juju status etcd --color

After etcd is deployed successfully, create client credentials for use in MicroK8s:

juju run etcd/leader package-client-credentials --wait=2m
juju scp etcd/leader:etcd_credentials.tar.gz ./
tar xvzf etcd_credentials.tar.gz

This leaves us with a directory named etcd_credentials, with the following contents:

etcd_credentials/
etcd_credentials/client.key
etcd_credentials/client.crt
etcd_credentials/ca.crt
etcd_credentials/README.txt

Also, note the IPs of the etcd cluster members. These can be found with the juju status etcd command. For example, in the output below …

Unit     Workload  Agent  Machine  Public address  Ports     Message
etcd/0*  active    idle   1        10.139.219.7    2379/tcp  Healthy with 3 known peers
etcd/1   active    idle   2        10.139.219.148  2379/tcp  Healthy with 3 known peers
etcd/2   active    idle   3        10.139.219.100  2379/tcp  Healthy with 3 known peers

… the IP addresses of the etcd cluster would be 10.139.219.7, 10.139.219.148, and 10.139.219.100.

Install MicroK8s

Install a new MicroK8s cluster with:

sudo snap install microk8s --classic

Do not add any extra nodes at this point.

Switch MicroK8s to external etcd

Copy the etcd certificates from the previous step to the machine where MicroK8s is running if required (e.g. via scp). Move the cert files into the MicroK8s certificates directory. Make sure to prefix them with etcd- to avoid overwriting existing files:
```
mv etcd_credentials/client.key /var/snap/microk8s/current/certs/etcd-client.key
mv etcd_credentials/client.crt /var/snap/microk8s/current/certs/etcd-client.crt
mv etcd_credentials/ca.crt /var/snap/microk8s/current/certs/etcd-ca.crt
```
Edit /var/snap/microk8s/current/args/kube-apiserver to remove any existing --etcd-servers arguments (for MicroK8s 1.23+), or --storage-backend=dqlite (MicroK8s 1.22 and earlier).

Append the following lines in /var/snap/microk8s/current/args/kube-apiserver. Make sure to replace the IP addresses with the IP addresses of your etcd cluster members.

--etcd-servers=https://10.139.219.7:2379,https://10.139.219.148:2379,https://10.139.219.100:2379
--etcd-cafile=${SNAP_DATA}/certs/etcd-ca.crt
--etcd-certfile=${SNAP_DATA}/certs/etcd-client.crt
--etcd-keyfile=${SNAP_DATA}/certs/etcd-client.key

Restart MicroK8s
```
sudo snap restart microk8s
```

Re-apply the Calico CNI:

microk8s kubectl apply -f /var/snap/microk8s/current/args/cni-network/cni.yaml

That’s all!

Congratulations! Your MicroK8s installation is now running using your external etcd as a data store!

Notes for MicroK8s cluster

Adding worker only nodes (MicroK8s 1.23+)

When using an external data store, adding new worker-only nodes works as is:

In the existing MicroK8s node, run the add-node command:
```
microk8s add-node
```

In the new node, join as worker:

microk8s join 10.0.1.1:25000/XXXXXXXXXXXXXXXXXXXXXXXXXXX/YYYYYYYYYY --worker

Adding more control plane nodes

When adding new control plane nodes to the cluster, an extra manual step is required:

In an existing MicroK8s node, run the add-node command:
```
microk8s add-node
```

In the new node, join the cluster:

microk8s join 10.0.1.1:25000/XXXXXXXXXXXXXXXXXXXXXXXXXXX/YYYYYYYYYY

Copy the etcd certificates to the new MicroK8s node (e.g. via scp). Then, move them into the MicroK8s certificates directory. Make sure to prefix them with etcd- to avoid overwriting existing files:

mv etcd_credentials/client.key /var/snap/microk8s/current/certs/etcd-client.key
mv etcd_credentials/client.crt /var/snap/microk8s/current/certs/etcd-client.crt
mv etcd_credentials/ca.crt /var/snap/microk8s/current/certs/etcd-ca.crt

Edit /var/snap/microk8s/current/args/kube-apiserver and remove any existing --etcd-servers arguments (for MicroK8s 1.23+), or --storage-backend=dqlite (MicroK8s 1.22 and earlier).