Hi All, We are using kuberenetes in AWS.
When we perform rolling update, we agree that kubernetes will always make sure rolling update is performed based on the maxSurge and maxUnavailable counts. But how kubernetes will decide when a pod should get terminated. For example, if a pod is writing something writing a big insert/update statement to DB, the connection from the pod to the DB is still active only from that particular pod, during this case, how kubernetes will decide whether the particular pod should get terminated or not.
Kubernetes sends a signal, so your process should handle that. Then, it waits a customizable time of seconds for your process to terminate.
In the deployment yaml, there is a field called gracePeriodSeconds that is how many seconds kubernetes will wait for your pod to terminate. So, you can tune that to make it as big as you need. But a deadline is enforced because kubernetes can’t stop deployments if your app is not responding
Thanks @rata for the reply.
So even if the gracePeriodSeconds crosses, if the new pod is not ready, deployment will still not terminate the old pod right, if we mention the maxUnavailable to 0.
No, sorry I wasn’t clear.
If the gracePeriodSeconds is elapsed, it will be killed forcedly.
The process needs to end some time so new deployments can work. You can choose how much time to wait max, that is maxGracePeriodSeconds, but after that if it didn’t end, it will be ended by force (kill -9 and the likes).
Thanks @rata for the reply.
If I am doing a deployment, with maxSurge 1 and maxUnavailable 0, the deployment will first try to provision a new pod of the updated version, if the pod takes more time to become ready(eg. takes more time than maxGracePeriodSeconds), will deployment start terminating the existing pod or it will wait until the new pod to get ready before terminating the old pod
gracePeriodSeconds applies to evictions, which happens after you’re at 6/5 and we need to go back to 5/5.
More generally, your app should be written with the assumption that individual nodes, pods, processes could die at any time - the DB will get a connection reset and terminate the big insert’s transaction, which if you designed your app well leaves it in a consistent state