Ways to manage data for many microservices serving many tenants

Given (in the future, not at present):

  • 100 multi-tenant microservices
  • 5000 customers (or tenants)
  • each microservice must be able to store data per tenant
  • data is managed in relational databases, because ACID transactions are required (for whatever reasons)
    • I fully realize the scaling limitation of such databases. Let us assume that the per tenant data volume as well as request intensity does not warrant advanced scaling requirements.
    • Azure cloud (it is what it is :-()

From these I make the following conclusions:

  1. Database per tenant.

    Although we assume that per tenant data volume and request intensity does not warrant advanced scaling, if more than one tenant is “stuffed” into a single database that assumption may no longer be true.

  2. N databases per DB server

    Theoretically, DB servers may hold thousands of databases. However, we want to minimize the amount of downed tenants if “fit hits the shan”. Hence N would not be thousands. But probably tens.

  3. Each DB server has dedicated microservice deployment associated only with it

    This way the DB server X may serve requests from the microservice A of version 1 and the DB server Y may serve requests from the microservice A of version 2. The DB schema may be different between these versions.

So these conclusions lead to multiple questions:

  1. What is N?
    • 50 ?
    • 100 ?
  2. What database to use?
    • Azure Database for PostgreSQL
    • Azure Database for MySQL
    • Azure SQL Database
  3. 5000/N DB servers to cover all the tenants for one microservice. For 100 microservices it results in 500,000/N DB servers. E.g.:
    • N = 50 → 10,000 DB servers (deployments)
    • N = 100 → 5,000 DB servers (deployments)

This is a lot of DB servers and a lot of microservice deployments. When adding geo redundancy this translates to doubling the number of web server nodes as well as DB servers (to be able to failover to another region). This is crazy amount of DB servers and web server nodes.

I wonder if anyone has experience with such systems. Our current system is a big multi-tenant enterprise application, but we have started creating small micro services around it and trying to figure out how the future would look like.

Your thoughts are very appreciated and welcome.

P.S.
I also asked the question here - databases - Ways to manage data for many microservices serving many tenants - DevOps Stack Exchange