Let’s share how Kubernetes has failed for us and what we’ve learned from it. It will probably help more than sharing our successes. I’ll go first. Last year I had 2 etcd nodes go down. I wasn’t monitoring etcd and the cluster went into read only mode and was unhealthy. The first sign of a problem …

Hi Justin - what did you do to fix your etcd issue? We recently had a master node go down due to an issue on the underlying hardware - a new master came up automatically and there were no issues apart from a delay in the etcd volumes detaching from the old master. We’ve also seen a problem with ku…

In my case the etcd cluster was external from the kubernetes cluster and the services were just not running. The data was still available so all I had to do was restart failed etcd containers and they automatically re-synced the data and kubernetes was happy again once etcd was healthy.

I have two: one for Kops / AWS a year ago (between k8s 1.4 to 1.6), another one happens one month ago. For the one happened in one year ago, I had checked all masters and nodes finding no clue at that time. Killing nodes and arranging another won’t help - the nodes just stop working with network co…

Cluster installed a little over a year ago. Recently we learned that the kubernetes api certificate was setup to only last 12 months; it was a bunch of fuss to recover from.

When HPA is cooled down say cpu utilisation comes down to 80% ( say stable for 5 min). Downscale happens. What happens to the other process/jobs running on downscaled pod ? Is is moves to the active pod?

How has Kubernetes failed for you?

General Discussions

hjacobs January 19, 2019, 11:28am 6

Hi Justin,

I just started a compilation of links to Kubernetes Failure Stories: https://github.com/hjacobs/kubernetes-failure-stories

The list is rather short so far, please contribute with your failure stories / outages

Thanks!

2 Likes

Topic		Replies	Views
Kubernetes apiserver not working during etcd leader goes down General Discussions	4	905	March 5, 2019
Etcd and kube-apiserver pods in CrashLoopBackOff state after node reboot General Discussions	5	15422	December 29, 2022
Etcd member down after chaning OS from Centos7 to Rocky Linux9 General Discussions	0	328	September 21, 2023
How to restore master failure in kubernetes General Discussions	0	1558	June 9, 2020
Kubernetes Cluster Downtime Issue After Resetting Old Master Node Following Replacement with New Master Node General Discussions	0	62	July 22, 2024

How has Kubernetes failed for you?

Related topics