# Kubernetes multpiple control plane nodes cluster, not working when one control plane node fails **URL:** \ **Category:** General Discussions\ **Created:** [September 5, 2023, 10:14am UTC](https://discuss.kubernetes.io/t/kubernetes-multpiple-control-plane-nodes-cluster-not-working-when-one-control-plane-node-fails/25318 "2023-09-05T10:14:47Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Roberto\_D\_Maggi](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/roberto_d_maggi/32/9565_2.png) [@Roberto\_D\_Maggi](https://discuss.kubernetes.io/u/Roberto_D_Maggi)\ **Post date:** [September 5, 2023, 10:14am UTC](https://discuss.kubernetes.io/t/kubernetes-multpiple-control-plane-nodes-cluster-not-working-when-one-control-plane-node-fails/25318/1 "2023-09-05T10:14:47Z")
Hi you all, I’m in a little puzzling problem. I created, just for test purpose, a 2 control plane and 2 worker node cluster. everything’s fine until I decide to turn off one cp-node2 and the cp-node1 stops to respond, turned the node2 on and the cluster turns back. is this normal? this is the contrary of fail over… Could someone help me to understand this behavior ? thanks in advance ### Cluster information: Kubernetes version: Cloud being used: (put bare-metal if not on a public cloud) Installation method: kubeadm Host OS: ubuntu CNI and version: calico CRI and version: containerd [containerd.io](http://containerd.io) 1.6.21 3dce8eb055cbb6872793272b4f20ed16117344f8 You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read. ---
**Author:** ![mrbobbytables](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/mrbobbytables/32/7_2.png) [@mrbobbytables](https://discuss.kubernetes.io/u/mrbobbytables)\ **Post date:** [September 5, 2023, 11:01am UTC](https://discuss.kubernetes.io/t/kubernetes-multpiple-control-plane-nodes-cluster-not-working-when-one-control-plane-node-fails/25318/2 "2023-09-05T11:01:25Z")
A 2 node control plane cluster is already in a degraded state. Etcd requires an odd # of nodes for HA. > **[Options for Highly Available Topology](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/)** > > This page explains the two options for configuring the topology of your highly available (HA) Kubernetes clusters. > You can set up an HA cluster: > With stacked control plane nodes, where etcd nodes are colocated with control plane nodes With external... ---
**Author:** ![Roberto\_D\_Maggi](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/roberto_d_maggi/32/9565_2.png) [@Roberto\_D\_Maggi](https://discuss.kubernetes.io/u/Roberto_D_Maggi)\ **Post date:** [September 7, 2023, 8:01am UTC](https://discuss.kubernetes.io/t/kubernetes-multpiple-control-plane-nodes-cluster-not-working-when-one-control-plane-node-fails/25318/3 "2023-09-07T08:01:15Z")
Hi @mrbobbytables and thanks for your reply, I knew this but I didn’t aspect that the the whole cluster would stop. I "gracefully excluded the node cordoning it etc … but after some minutes that the 2nd node wasn’t responding the cluster stuck and after a reboot the kubernetes control plane ( single ) node didn’t want to restart, becoming unable to contact itself root@k-master-1:~# kg po E0907 07:40:24.503955 2136 memcache.go:265] couldn’t get current server API group list: Get “[https://k-master-1.cluster:6443/api?timeout=32s](https://k-master-1.cluster:6443/api?timeout=32s)”: dial tcp 192.168.1.110:6443: connect: connection refused E0907 07:40:24.504203 2136 memcache.go:265] couldn’t get current server API group list: Get “[https://k-master-1.cluster:6443/api?timeout=32s](https://k-master-1.cluster:6443/api?timeout=32s)”: dial tcp 192.168.1.110:6443: connect: connection refused E0907 07:40:24.505445 2136 memcache.go:265] couldn’t get current server API group list: Get “[https://k-master-1.cluster:6443/api?timeout=32s](https://k-master-1.cluster:6443/api?timeout=32s)”: dial tcp 192.168.1.110:6443: connect: connection refused E0907 07:40:24.509766 2136 memcache.go:265] couldn’t get current server API group list: Get “[https://k-master-1.cluster:6443/api?timeout=32s](https://k-master-1.cluster:6443/api?timeout=32s)”: dial tcp 192.168.1.110:6443: connect: connection refused E0907 07:40:24.510938 2136 memcache.go:265] couldn’t get current server API group list: Get “[https://k-master-1.cluster:6443/api?timeout=32s](https://k-master-1.cluster:6443/api?timeout=32s)”: dial tcp 192.168.1.110:6443: connect: connection refused The connection to the server k-master-1.cluster:6443 was refused - did you specify the right host or port? Could it be an etcd problem? How could I investigate it? and, more important, how could I solve this? If it would happen in a production environment, would be a “little” problem to reset a cluster. thanks in advance ---
**Author:** ![mrbobbytables](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/mrbobbytables/32/7_2.png) [@mrbobbytables](https://discuss.kubernetes.io/u/mrbobbytables)\ **Post date:** [September 7, 2023, 9:42am UTC](https://discuss.kubernetes.io/t/kubernetes-multpiple-control-plane-nodes-cluster-not-working-when-one-control-plane-node-fails/25318/4 "2023-09-07T09:42:19Z")
> [@Roberto\_D\_Maggi](#): > > Is this normal? > this is the contrary of fail over Yes, it is completely normal. HA requires at least 3 nodes. A 3 node etcd cluster can take 1 failure as long as 2 of can still communicate. You have created an instance where the cluster was already in a degraded state and stopping one put the other into thinking it was the failed node. Please read the docs on etcd, K8s and HA. > **[FAQ](https://etcd.io/docs/v3.5/faq/#why-an-odd-number-of-cluster-members)** > > Frequently asked questions ---
**Author:** ![Roberto\_D\_Maggi](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/roberto_d_maggi/32/9565_2.png) [@Roberto\_D\_Maggi](https://discuss.kubernetes.io/u/Roberto_D_Maggi)\ **Post date:** [September 7, 2023, 9:50am UTC](https://discuss.kubernetes.io/t/kubernetes-multpiple-control-plane-nodes-cluster-not-working-when-one-control-plane-node-fails/25318/5 "2023-09-07T09:50:19Z")
thanks a lot! it was puzzling me a lot