I am trying to set up so that when our app crashes, it will generate core dump file in a specified location that is on a persisted volume, and I am able to make it work.
My question is, if the core dump file to be generate is huge, will Kube controller wait for the the core dump file to be generated and then kill the container? Or the container might get killed and restarted before the core dump file is finishedd?
The liveness probe will fail when our application’s process crashes. My gut feeling is that the container might get restarted before the core dump file is finished if it is huge, but need to confirm with people with more experience on it. And if that’s the case, how do you deal with it? Making liveness probe smart enough to not fail when core dump being generated?
A update:
I don’t think kubelet will wait for the core dump file to be generated. It will kill the container and create a new one. This is because our current liveness probe will fail during core dump generation
I tried to increase our memory to about 20G+, and then triggered a core dump (kill -s SIGSEGV ), it only take about 10 seconds before the container was killed and the generated core dump file is only about 1G+.
So I will try to make our liveness probe smarter in this situation and see what happens.
Hi
I m facing similar issue as sometimes I m getting corrupted coredump may be it’s truncated dump. Could u please guide how u r making liveness probe smarter