Backup and Restore ETCD Database

General Discussions
1
  • Are this the correct commands to backup the ETCD DB:

ETCDCTL_API=3 etcdctl help

ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /tmp/snapshot-pre-boot.db

  • Are this the correct commands to Restore the ETCD DB:

ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt
–name=master
–cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
–data-dir /var/lib/etcd-from-backup
–initial-cluster=master=https://127.0.0.1:2380
–initial-cluster-token=etcd-cluster-1
–initial-advertise-peer-urls=https://127.0.0.1:2380
snapshot restore /tmp/snapshot-pre-boot.db

  • Update ETCD manifest file with correct backup directory and token:

  • –data-dir=/var/lib/etcd-from-backup

  • –initial-cluster-token=etcd-cluster-1

  • Update ETCD manifest file with correct hostpath and volume:
    volumeMounts:

    • mountPath: /var/lib/etcd-from-backup
      name: etcd-data
    • mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs

    volumes:

    • hostPath:
      path: /var/lib/etcd-from-backup
      type: DirectoryOrCreate
      name: etcd-data
    • hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
2

You can visit this post. ETCD - backup and restore management

3

Above steps looks good, are you facing any issue?

5

First, you need to install etcd-client: sudo apt install etcd-client

6

can we restore etcd using pem format cert and keys? if so, is it possible only from master node?

ETCD backup is as below:
ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt
–name=master
–cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/etcd-backup.db

ETCD restore using below:

cert-/opt/etcd-client-ca.pem key=/opt/etcd-client-key.pem restore from /opt/previous_backup.db

10

Can anyone post the commands to take backup and restore HA ETCD cluster. and also please let me knoe, after restore ,we need reboot any kubeadm tools ?

12

#get certs
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd

#take a snapshot
ETCDCTL_API=3 etcdctl snapshot save /tmp/etcd-backup.db
–cacert /etc/kubernetes/pki/etcd/ca.crt
–cert /etc/kubernetes/pki/etcd/server.crt
–key /etc/kubernetes/pki/etcd/server.key

#restore a snapshot
ETCDCTL_API=3 etcdctl snapshot restore /tmp/etcd-backup.db
–data-dir /var/lib/etcd-backup
–cacert /etc/kubernetes/pki/etcd/ca.crt
–cert /etc/kubernetes/pki/etcd/server.crt
–key /etc/kubernetes/pki/etcd/server.key

17

Back Up the etcd Data

  1. From the terminal, log in to the etcd server:
    ssh etcd1
  2. Back up the etcd data:
    ETCDCTL_API=3 etcdctl snapshot save /home/cloud_user/etcd_backup.db
    –endpoints=https://etcd1:2379
    –cacert=/home/cloud_user/etcd-certs/etcd-ca.pem
    –cert=/home/cloud_user/etcd-certs/etcd-server.crt
    –key=/home/cloud_user/etcd-certs/etcd-server.key
    Restore the etcd Data from the Backup
  3. Stop etcd:
    sudo systemctl stop etcd
  4. Delete the existing etcd data:
    sudo rm -rf /var/lib/etcd
  5. Restore etcd data from a backup:
    sudo ETCDCTL_API=3 etcdctl snapshot restore /home/cloud_user/etcd_backup.db
    –initial-cluster etcd-restore=https://etcd1:2380
    –initial-advertise-peer-urls https://etcd1:2380
    –name etcd-restore
    –data-dir /var/lib/etcd
  6. Set database ownership:
    sudo chown -R etcd:etcd /var/lib/etcd
  7. Start etcd:
    sudo systemctl start etcd
  8. Verify the system is working:
    ETCDCTL_API=3 etcdctl get cluster.name
    –endpoints=https://etcd1:2379
    –cacert=/home/cloud_user/etcd-certs/etcd-ca.pem
    –cert=/home/cloud_user/etcd-certs/etcd-server.crt
    –key=/home/cloud_user/etcd-certs/etcd-server.key
18

Backup:

ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/snapshot-pre-boot.db

To view status

ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot status /opt/snapshot-pre-boot.db –write-out=table

Before restore , kindly follow below steps

mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/

ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt
–cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
–data-dir /var/lib/etcd-from-backup
snapshot restore /opt/snapshot-pre-boot.db

And edit etcd.yaml file

–data-dir /var/lib/etcd-from-backup

volumeMounts:

  • mountPath: /var/lib/etcd-from-backup
    name: etcd-data
  • mountPath: /etc/kubernetes/pki/etcd
    name: etcd-certs

volumes:

-hostPath:
path: /etc/kubernetes/pki/etcd
type: DirectoryOrCreate
name: etcd-certs

-hostPath:
path: /var/lib/etcd-from-backup
type: DirectoryOrCreate
name: etcd-data

mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests

sudo systemctl restart kubelet