Kubernetes apiserver not working during etcd leader goes down

General Discussions
1

Hello,

We are using 1.13.1 Kubernetes and etcd 3.3.9. As we are doing negative testing for production readiness. During the test we observed that whenever etcd leader goes down abruptly, all the calls coming to nginx which we are running on K8 are failing. After some time the all the calls are flowing fine.

During that time we are not able to execute “kubectl get pods” commad as well. APIserver was freezed for few seconds. Is this expected behaviour with apiserver when etcd leader goes down. ?

Can some one help on this issue.

etcd - 3 node cluster
K8 - 6 node cluster.

Thanks,
Niranjan

2

Yes, this is expected. You can see the failure situation with etcd here:

1 Like
3

How to minimize the failures.?
I see etcd with in a same second it is electing the leader, but apiserver not able to communicate with the new leader.

4

you can tune etcd time settings, following the guidance on this page

the documentation is pretty extensive, just make sure the timeouts are not too low, so you’re not re-electing on any sort of “hiccup”

1 Like
5

Thanks, @RiRa12621 & @ejc3. We are going with default timeouts.