Kubernetes version: 1.13
Cloud being used: EKS
Installation method: Cloudformation
Host OS: Amazon Linux
Hi all, I have a console application that does background processing tasks. Every now and then the application fails in such a way that it doesn’t generate an exit code, it just pauses/freezes. The only way to know this has happened is by inspecting the pod output and checking for the lack of any recent output. I would really like to handle this situation with a liveliness check and have kubernetes restart the pod rather than passing it down the line to a monitoring platform. I cant currently see any way of getting a liveliness probe to access the pods stdout to get this data. Is this in any way possible to achieve?
Hi @rata, the problem is that the application is already running in the pod from its entrypoint. I need to check the output from that application rather than the output of a command executed separately.
But can’t you execute a command and it should answer? If that is the case, you can use that. Right?
There are other hacks you can do, but you should really add that notion of health check to the application for being"cloud native" and taking advantage of that.
Unfortunately not as it’s a background process that just ticks along without any intervention or input. I know that the application should technically exit when it fails but I was hoping that a liveliness check would be a solution for that problem.
Ups. You can do hacks, like using a sidecar container that reads a tick on a shared volume, or use a service account and check the logs… But I’d strongly advise not to.
If the application is stalled, doesn’t crash and can’t be queried to see if it’s still alive… Then I would really suggest to break any of those things to fix it (make it crash, fix the bug or make it possible to see if it’s alive).
Livenessprobe are ways to check the app is running (http get, run a command, tcp connection IIRC, etc.), but if your app doesn’t have that… Then, I think there is no reasonable way to do it, IMHO.
Sort of going along those lines, you could either redirect or send a copy of stdout to a file, the exec command to check / call a script within the container to check. Probably wouldn’t be a good idea for a very noisy app though.
When you start your entrypoint in your container, you can pipe its stdout through some program that will analyze your logs and put some flag into your container /tmp/status for example, then write itself the logs to stdout. Then your liveness probe just needs to get the contents of your status file.