Incremental Release Infrastructure

Cluster information:

Kubernetes version: 1.18
Cloud being used: AWS, self-hosted k8s cluster
Installation method: kops
Host OS: flatcar
CNI and version: weave

Hi guys, our software is comprised of several microservices. They are being deployed and upgraded thru helm charts.
I was wondering If there is a tool or some scripts that takes care of incrementally increasing the blast radius of a newly deployed version. That is:

  • deploy the new version as a single pod for a particular subtenant
  • upgrade all pods for a particular subtenant
  • upgrade all pods for a particular tenant

Once a single POD with the latest version is deployed to the cluster of a particular subtenant we are monitoring some SLI and overall health of the new version. If everything is OK then we manually trigger deployment to all tenants with full number of PODs/replicas.