I’m looking for ways to “synchronize” (or serialize) some kubernetes operations done via kubectl.
Motivation:
Sometimes 2 developers merge to master within a couple minutes of each other. Whoever merges second will often get a helm failure to deploy due to “another operation (install/upgrade/rollback) is in progress”. This can happen because our CI/CD does not currently support queueing deployment jobs (unlike spinnaker, eg, which does).
So I’m looking for ways to make a helm deploy job wait till current one is done.
But I’m posting here because this challenge is not specific to helm: what if I wanted to run kubectl apply -f from CI/CD, and wait till the deployment has rolled out before making a new one (instead of just issuing a new apply and let kubernetes decide what to do if a rollout is already in progress)?
Solution:
One trick I think should work, is to use kubectl create secret (or configmap) in a loop which exits only if the secret could be created. Only then would my script run the “synchronized” kubectl command, and once done (regardless of any failures), it would delete that secret.
Does anyone foresee any gotchas with this approach?
Cluster information:
Kubernetes version: 1.20
Kubectl version: 1.21
Host OS (for running kubectl): linux
(all other cluster info is n/a to this question)