# ETCD backup !ssues

**URL:** <https://discuss.kubernetes.io/t/etcd-backup-ssues/8304>\
**Category:** General Discussions\
**Created:** [October 13, 2019, 7:12am UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304 "2019-10-13T07:12:47Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![raj.152](https://avatars.discourse-cdn.com/v4/letter/r/b782af/32.png) [@raj.152](https://discuss.kubernetes.io/u/raj.152)\
**Post date:** [October 13, 2019, 7:12am UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/1 "2019-10-13T07:12:47Z")

</div>

Hi

I am using below commands for etcd backup and restore. ofcourse they successfully worked for cluster created by kubeadm however they are not working for the cluster created by Hardway. please provide your suggestions

**Backup:**

ETCDCTL\_API=3 etcdctl snapshot save mysnapshot.db --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key

**Restore:**

ETCDCTL\_API=3 etcdctl snapshot restore mysnapshot.db --name ip-172-31-27-180 --initial-cluster ip-172-31-27-180=https://172.31.27.180:2380 --initial-advertise-peer-urls [https://172.31.27.180:2380](https://172.31.27.180:2380)

**List members:**  
ETCDCTL\_API=3 etcdctl member list --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key

ETCDCTL\_API=3 etcdctl member list --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key

---

<div class="post-metadata">

**Author:** ![nirajtolia](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/nirajtolia/32/3939_2.png) [@nirajtolia](https://discuss.kubernetes.io/u/nirajtolia)\
**Post date:** [October 19, 2019, 4:16pm UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/2 "2019-10-19T16:16:30Z")

</div>

This doesn’t answer your question directly, but I wouldn’t recommend etcd backup as a way of protecting a Kubernetes cluster. Look at [https://stateful.kubernetes.sh/](https://stateful.kubernetes.sh/) for other options.

---

<div class="post-metadata">

**Author:** ![Mach\_3](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/mach_3/32/3981_2.png) [@Mach\_3](https://discuss.kubernetes.io/u/Mach_3)\
**Post date:** [October 27, 2019, 4:46pm UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/3 "2019-10-27T16:46:27Z")

</div>

I got it working on mine by creating a new data directory:

sudo ETCDCTL\_API=3 etcdctl snapshot restore latest-snapshot-test.db   
–name master   
–initial-cluster master=http://127.0.0.1:2380   
–initial-cluster-token etcd-cluster-1   
–initial-advertise-peer-urls [http://127.0.0.1:2380](http://127.0.0.1:2380)   
–data-dir=/var/lib/etcd-new

then updating the etcd.service to use this new directory then restarted and it worked. This was just me testing and running through hardway steps

---

<div class="post-metadata">

**Author:** ![satya](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@satya](https://discuss.kubernetes.io/u/satya)\
**Post date:** [October 30, 2019, 5:16am UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/4 "2019-10-30T05:16:53Z")

</div>

```auto
ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
     --name=master \
     --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
     --data-dir /var/lib/etcd-from-backup \
     --initial-cluster=master=https://127.0.0.1:2380 \
     --initial-cluster-token etcd-cluster-1 \
     --initial-advertise-peer-urls=https://127.0.0.1:2380 \ 
After running above command then deplend on etcd state where pod or as service make sure to change the config passing initial-cluster-token and data directory path accordingly and restart
     snapshot restore /tmp/snapshot-pre-boot.db

```

---

<div class="post-metadata">

**Author:** ![PavanRaga](https://avatars.discourse-cdn.com/v4/letter/p/e480ec/32.png) [@PavanRaga](https://discuss.kubernetes.io/u/PavanRaga)\
**Post date:** [February 2, 2020, 12:20pm UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/5 "2020-02-02T12:20:44Z")

</div>

FYI for authentication and auth of new users -

1. Create a CSR with openssl -

2. Copy the o/p of

3. Paste in csr.yaml file(it is mentioned where to paste in file)

4. Create the CSR k8 obj -

5. Approve the CSR -

6. now get the certificate that is approved -

7. Now to create the kubeconfig, we need CA cert, ops cert(one we have above) -

8. Running the below will create a kubeconfig template with current CA’s certificate -

9. This will add ops cert into the kubeconfig -

10. This will add context mapping for the ops user

11. kubectl config use-context ops --kubeconfig=ops-k8s-config

#Now pass on the “ops-k8s-config” as kubeconfig file for the ops team!

1. apply role and rolebinding  
kubectl apply -f role.yaml  
kubectl apply -f rolebinding.yaml

Role:

```
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: create-pod
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create", "delete"]

```

RoleBinding:

```
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: role-grantor-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: create-pod
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: ops

```

CSR object template

```
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ops-k8s-access
spec:
  groups:
  - system:authenticated
  request: #replace with output from shell command: cat ops-k8s.csr | base64 | tr -d '\n'
  usages:
  - client auth
```

---

<div class="post-metadata">

**Author:** ![PavanRaga](https://avatars.discourse-cdn.com/v4/letter/p/e480ec/32.png) [@PavanRaga](https://discuss.kubernetes.io/u/PavanRaga)\
**Post date:** [February 12, 2020, 1:53pm UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/6 "2020-02-12T13:53:59Z")

</div>

ETCDCTL\_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --name=master --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --data-dir /var/lib/etcd-from-backup --initial-cluster=master=https://127.0.0.1:2380 --initial-cluster-token etcd-cluster-1 --initial-advertise-peer-urls=https://127.0.0.1:2380 snapshot restore /tmp/snapshot-pre-boot.db

---

<div class="post-metadata">

**Author:** ![draghuram](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/draghuram/32/3529_2.png) [@draghuram](https://discuss.kubernetes.io/u/draghuram)\
**Post date:** [February 13, 2020, 5:48pm UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/7 "2020-02-13T17:48:33Z")

</div>

Hi,

Please check out the following project that simplifies etcd backup. It obviates the need to explicitly create etcd snapshots and also provides the benefit of automatically backing up the snapshot file to any S3 bucket.

[https://github.com/catalogicsoftware/kubedr](https://github.com/catalogicsoftware/kubedr)

Just as a full disclosure, the project is released by Catalogic Software where I work.

Thanks,

Raghu

---

<div class="post-metadata">

**Author:** ![suman\_saurabh](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/suman_saurabh/32/4747_2.png) [@suman\_saurabh](https://discuss.kubernetes.io/u/suman_saurabh)\
**Post date:** [March 13, 2020, 7:31pm UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/10 "2020-03-13T19:31:24Z")

</div>

BackUp

```auto
ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
     --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
     snapshot save /tmp/snapshot-pre-boot.db

```

Restore

```auto
ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
     --name=master \
     --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
     --data-dir /var/lib/etcd-from-backup \
     --initial-cluster=master=https://127.0.0.1:2380 \
     --initial-cluster-token etcd-cluster-1 \
     --initial-advertise-peer-urls=https://127.0.0.1:2380 \
     snapshot restore /tmp/snapshot-pre-boot.db

```

# Modify /etc/kubernetes/manifests/etcd.yaml

Update ETCD POD to use the new data directory and cluster token by modifying the pod definition file at `/etc/kubernetes/manifests/etcd.yaml` . When this file is updated, the ETCD pod is automatically re-created as thisis a static pod placed under the `/etc/kubernetes/manifests` directory.

Update --data-dir to use new target location

```auto
--data-dir=/var/lib/etcd-from-backup

```

Update new initial-cluster-token to specify new cluster

```auto
--initial-cluster-token=etcd-cluster-1

```

Update volumes and volume mounts to point to new path

```auto
    volumeMounts:
    - mountPath: /var/lib/etcd-from-backup
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /var/lib/etcd-from-backup
      type: DirectoryOrCreate
    name: etcd-data
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs

```

---

<div class="post-metadata">

**Author:** ![Vinod\_Melekkandy](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/vinod_melekkandy/32/4753_2.png) [@Vinod\_Melekkandy](https://discuss.kubernetes.io/u/Vinod_Melekkandy)\
**Post date:** [March 14, 2020, 10:20am UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/12 "2020-03-14T10:20:18Z")

</div>

This one worked for me.

```
ETCDCTL_API=3 etcdctl \

  --endpoints=https://[127.0.0.1]:2379 \

  --cacert=/etc/kubernetes/pki/etcd/ca.crt \

  --name=master \

  --cert=/etc/kubernetes/pki/etcd/server.crt \

  --key=/etc/kubernetes/pki/etcd/erver.key \

  --data-dir /var/lib/etcd-from-backup \

  --initial-cluster=master=https://127.0.0.1:2380 \

  --initial-cluster-token etcd-cluster-1 \

  --initial-advertise-peer-urls=https://127.0.0.1:2380 \

  snapshot restore /tmp/snapshot-pre-boot.db
```

---

<div class="post-metadata">

**Author:** ![ranbir](https://avatars.discourse-cdn.com/v4/letter/r/9de0a6/32.png) [@ranbir](https://discuss.kubernetes.io/u/ranbir)\
**Post date:** [April 6, 2020, 4:02am UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/14 "2020-04-06T04:02:04Z")

</div>

Thanks you, solution worked for me, Was also able to verify also  
ETCDCTL\_API=3 etcdctl --write-out=table snapshot status /tmp/snapshot-pre-boot.db

---

<div class="post-metadata">

**Author:** ![A\_K](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/a_k/32/5014_2.png) [@A\_K](https://discuss.kubernetes.io/u/A_K)\
**Post date:** [April 27, 2020, 7:18am UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/15 "2020-04-27T07:18:13Z")

</div>

Thank you, i like to have everything on one line:

`ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /tmp/etcd-backup.db`

Then to verify:

`ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot status -w table /tmp/etcd-backup.db`

---

<div class="post-metadata">

**Author:** ![stefano\_terzon](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/stefano_terzon/32/5235_2.png) [@stefano\_terzon](https://discuss.kubernetes.io/u/stefano_terzon)\
**Post date:** [June 2, 2020, 12:40pm UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/16 "2020-06-02T12:40:42Z")

</div>

…To complete your procedure

See if the container process is back on

docker ps -a | grep etcd

* * *

see if the cluster members have been recreated

ETCDCTL\_API=3 etcdctl member list --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --endpoints=127.0.0.1:2379

* * *

see if pods, deployments and services have been recreated

kubectl get pods,svc,deployments

---

<div class="post-metadata">

**Author:** ![lindostech](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/lindostech/32/5472_2.png) [@lindostech](https://discuss.kubernetes.io/u/lindostech)\
**Post date:** [July 6, 2020, 7:36am UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/17 "2020-07-06T07:36:10Z")

</div>

**Copy paste the complete script below… Then run the command to verify the same.**

cat \<\< EOF \> etcd\_snapshot\_backup.sh

#How can I save the etcd-backup Snapshot in a single command  
#Author :Lindos\_tech\_geeks

_ **echo -n "Please enter the location to save the backup : ";read loc;ETCDCTL\_API=3 etcdctl snapshot save $loc --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key** _  
_ **echo "You are verifying the output of the saved snapshot $loc"** _  
_ **ETCDCTL\_API=3 etcdctl --write-out=table snapshot status $loc** _

EOF

#Then run the command

sh etcd\_snapshot\_backup.sh

If you are using a browser based shell sometimes the cat based creation of file adds some junk characters, in that case please copy only the bold part of command to a vim editor

# _Sample O/P_

master $ vim etcd\_snapshot\_backup.sh

master $ sh etcd\_snapshot\_backup.sh

Please enter the location to save the backup : /root/etcdbackup.db  
Snapshot saved at /root/etcdbackup.db  
You are verifying the output of the saved snapshot /root/etcdbackup.db  
±---------±---------±-----------±-----------+  
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |  
±---------±---------±-----------±-----------+  
| 4743dec6 | 3245 | 1466 | 3.4 MB |  
±---------±---------±-----------±-----------+  
master $

---

<div class="post-metadata">

**Author:** ![abhinavsinha1991](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/abhinavsinha1991/32/5684_2.png) [@abhinavsinha1991](https://discuss.kubernetes.io/u/abhinavsinha1991)\
**Post date:** [August 12, 2020, 12:04pm UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/20 "2020-08-12T12:04:03Z")

</div>

Here, this works across an etcd cluster of 3 too:

> <https://github.com/abhinavsinha1991/multi-etcd-restore/blob/master/README.md>

I have created the steps and tested the same when couldn’t find anything on the internet.

Abhinav

---

<div class="post-metadata">

**Author:** ![swaroopcs88](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/swaroopcs88/32/6584_2.png) [@swaroopcs88](https://discuss.kubernetes.io/u/swaroopcs88)\
**Post date:** [December 29, 2020, 3:22pm UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/21 "2020-12-29T15:22:32Z")

</div>

Hi Suman,

What is that I am doing incorrectly here?   
 ![image](https://us1.discourse-cdn.com/flex016/uploads/kubernetes/original/2X/3/35aa678cd54fef0e38e260baa32f9db57b0c38f6.png)  
–data-dir is this the value from the backup snapshot saved location?  
thanks

---

<div class="post-metadata">

**Author:** ![tej-singh-rana](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/tej-singh-rana/32/5921_2.png) [@tej-singh-rana](https://discuss.kubernetes.io/u/tej-singh-rana)\
**Post date:** [December 30, 2020, 8:55am UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/22 "2020-12-30T08:55:19Z")

</div>

Hello, @swaroopcs88  
Use **server.key** and **server.crt** instead of **apiserver-etcd-client.crt** and **apiserver-etcd-client.key**.  
**server.key** and **server.crt** files located at **/etc/kubernetes/pki/etcd/**.

---

<div class="post-metadata">

**Author:** ![swaroopcs88](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/swaroopcs88/32/6584_2.png) [@swaroopcs88](https://discuss.kubernetes.io/u/swaroopcs88)\
**Post date:** [December 30, 2020, 10:52am UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/23 "2020-12-30T10:52:11Z")

</div>

thank you, Tej, what is the --data-dir value? will it be the snapshot saved location?  
I am not seeing --initial-cluster-token in etcd.yaml file.  
please advise.  
Update new initial-cluster-token to specify new cluster

```auto
--initial-cluster-token=etcd-cluster-1

```

![image](https://us1.discourse-cdn.com/flex016/uploads/kubernetes/original/2X/0/06824503c75ab282c93d34eec504f817f0d7514d.png)

---

<div class="post-metadata">

**Author:** ![swaroopcs88](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/swaroopcs88/32/6584_2.png) [@swaroopcs88](https://discuss.kubernetes.io/u/swaroopcs88)\
**Post date:** [January 1, 2021, 5:09am UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/24 "2021-01-01T05:09:18Z")

</div>

hi all,  
can you please check and advise here?  
thanks  
Swaroop

---

<div class="post-metadata">

**Author:** ![swaroopcs88](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/swaroopcs88/32/6584_2.png) [@swaroopcs88](https://discuss.kubernetes.io/u/swaroopcs88)\
**Post date:** [January 1, 2021, 6:02am UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/25 "2021-01-01T06:02:11Z")

</div>

never mind, I figured it out.  
Thank you!

---

<div class="post-metadata">

**Author:** ![shaik\_zillani](https://avatars.discourse-cdn.com/v4/letter/s/f9ae1b/32.png) [@shaik\_zillani](https://discuss.kubernetes.io/u/shaik_zillani)\
**Post date:** [February 21, 2021, 4:12pm UTC](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304/26 "2021-02-21T16:12:16Z")

</div>

Simple backup and restore that works!

```auto
ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/opt/ca.crt --cert=/opt/etcd-client.crt --key=/opt/etcd-client.key snapshot save /srv/data/etcd-snapshot.db

ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/opt/ca.crt --cert=/opt/KUIN00601/etcd-client.crt --key=/opt/etcd-client.key snapshot restore /srv/data/etcd-snapshot-previous.db

```

[Next page](https://discuss.kubernetes.io/t/etcd-backup-ssues/8304.md?page=2)