We are trying to look into rolling upgrade for statefulsets with zero-downtime for our cluster and we are running into a problem. Our statefulsets deployments are generally 2 pods but will be scaled to 3 or more pods in the future. With 2 pods, Rolling Upgrade with no downtime is doable without any problem. With 3 or more pods we run into a problem of k8s route traffic to pods with different version. For example, when we roll out our update for our, lets say, nginx, with 3 replicas from version 1.1 to 1.2, nginx-3 will get updated first and started with 1.2 then k8s will perform the upgrade on nginx-2. Since we use readiness probe, now we have nginx-1 (1.1) and nginx-3 (1.2) both in ready state and k8s will be routing request traffic to these 2 pods in round robin style which create a problem due to mismatch of server version. This problem is bigger if we have more than 3 pods. My question is there any straight forward way or k8s api that will rectify this problem?
Not really that I know of out of the box. It’d be possible with some other scripting during your update process with an additional label in the selector targeting version. Other option would be to use something a little more advances than a default kubernetes service for your application routing e.g. Istio.
Most things update like this. Are you sure this is a problem in your case?
And easy way to work around this might be doing a blue green deployment. But it might be tricky because of the state, it really depends on your case.
Is it possible on your case to use blue green deployment?
yes because it creates problem for session handling between our pods if one of them are of different version.
Im not sure if blue-green deployment will ensure the zero-downtime upgrade. We will certainly have to try that out.
Please share what you find 