# Etcdserver: mvcc: database space exceeded

**URL:** <https://discuss.kubernetes.io/t/etcdserver-mvcc-database-space-exceeded/22806>\
**Category:** General Discussions\
**Created:** [January 23, 2023, 4:29pm UTC](https://discuss.kubernetes.io/t/etcdserver-mvcc-database-space-exceeded/22806 "2023-01-23T16:29:46Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ankitkaushik94](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@ankitkaushik94](https://discuss.kubernetes.io/u/ankitkaushik94)\
**Post date:** [January 23, 2023, 4:29pm UTC](https://discuss.kubernetes.io/t/etcdserver-mvcc-database-space-exceeded/22806/1 "2023-01-23T16:29:46Z")

</div>

Asking for help? Comment out what you need so we can get more information to help you!

### Cluster information:

Kubernetes version: 1.23.6  
Cloud being used: (put bare-metal if not on a public cloud) : Bare-metal  
Installation method: Using Kubeadm  
Host OS: Ubuntu  
CNI and version:  
CRI and version:

Hi Team,  
This is very URGENT!!  
The whole cluster is down due to the error “etcdserver: mvcc: database space exceeded” .  
Nothing is working.  
Could you please help me in this.  
Hoping for solutions.

Regards,  
Ankit

---

<div class="post-metadata">

**Author:** ![chethan](https://avatars.discourse-cdn.com/v4/letter/c/6bbea6/32.png) [@chethan](https://discuss.kubernetes.io/u/chethan)\
**Post date:** [March 10, 2023, 12:12pm UTC](https://discuss.kubernetes.io/t/etcdserver-mvcc-database-space-exceeded/22806/2 "2023-03-10T12:12:12Z")

</div>

@ankitkaushik94 have you able to fix this issue i am also getting same issue

---

<div class="post-metadata">

**Author:** ![bmiag](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/bmiag/32/12038_2.png) [@bmiag](https://discuss.kubernetes.io/u/bmiag)\
**Post date:** [March 13, 2023, 11:47am UTC](https://discuss.kubernetes.io/t/etcdserver-mvcc-database-space-exceeded/22806/3 "2023-03-13T11:47:58Z")

</div>

Hello,  
we had the same issue today and I was able to solve it as follows:

On my cluster manager node:

```auto
ETCDCTL_API=3 etcdctl \
	--endpoints=https://127.0.0.1:2379 \
	--cacert=/etc/kubernetes/pki/etcd/ca.crt \
	--cert=/etc/kubernetes/pki/etcd/peer.crt \
	--key=/etc/kubernetes/pki/etcd/peer.key \
	endpoint status --write-out=table

```

Output looks like this:

```auto
+------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://127.0.0.1:2379 | 123456789ABCDEFG | 3.5.6 | 2.1 GB | true | 985 | 269601165 |
+------------------------+------------------+---------+---------+-----------+-----------+------------+

```

etcd database is really big, exceeds quota. Check the alarm message of etcd:

```auto
ETCDCTL_API=3 etcdctl \
	--endpoints=https://127.0.0.1:2379 \
	--cacert=/etc/kubernetes/pki/etcd/ca.crt \
	--cert=/etc/kubernetes/pki/etcd/peer.crt \
	--key=/etc/kubernetes/pki/etcd/peer.key \
	alarm list

```

Output shows the alarm: NOSPACE:

```auto
memberID:123456789ABCDEFG alarm:NOSPACE

```

Get the current revision and compact it and defrag the database:

```auto
REVISION=$(ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9].*')
ETCDCTL_API=3 etcdctl \
	--endpoints=https://127.0.0.1:2379 \
	--cacert=/etc/kubernetes/pki/etcd/ca.crt \
	--cert=/etc/kubernetes/pki/etcd/peer.crt \
	--key=/etc/kubernetes/pki/etcd/peer.key \
	compact ${REVISION}
ETCDCTL_API=3 etcdctl \
	--endpoints=https://127.0.0.1:2379 \
	--cacert=/etc/kubernetes/pki/etcd/ca.crt \
	--cert=/etc/kubernetes/pki/etcd/peer.crt \
	--key=/etc/kubernetes/pki/etcd/peer.key \
	defrag

```

Output should look like this:

```auto
compacted revision 111111111
Finished defragmenting etcd member[https://127.0.0.1:2379]

```

When you now repeat the status request from the beginning, it shoul look like this:

```auto
+------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://127.0.0.1:2379 | 123456789ABCDEFG | 3.5.6 | 228 MB | true | 985 | 269601165 |
+------------------------+------------------+---------+---------+-----------+-----------+------------+

```

Note that the DB SIZE is now a lot smaller again.  
After that, my cluster worked again.  
We will now activate auto compacting and to be sure, increase the quota. See details for etcd maintenance here: [etcd maintenance docs](https://etcd.io/docs/v3.5/op-guide/maintenance/)

Hope this helps someone solve their problems, too.

Kind regards  
Timo (b+m Informatik AG)