We have exceeded the limit on the number of pods and need a way to enable kubernetes to support running more containers

General Discussions
1

Cluster information:

k8s 1.17
Cloud being used: eks

In order to serve users, we need to create some jobs for each user, these jobs will create pods which require only a small amount of resources.

The problem is that because k8s has a limit on the number of pods per node, it cannot exert node performance. It is also difficult for us to maintain a large number of nodes.

So I’m wondering if there are any open source workloads where a pod can run multiple containers, and containers can be easily created and deleted, similar to a thread pool.

Hope you can give some advice.

grateful

2

You can not practically add/remove containers on a running pod. There are ways to add more pods to nodes, but it does require configuration and prep (e.g. allocating enough IPs). The better answer is more-but-smaller nodes.

3

Thanks for your insight.

But our number of users is already quite large, and the number of pods in the production environment is already in the tens of thousands, and it is still increasing, so if we just add more nodes, I am afraid that the current problem cannot be completely solved.

It would be great if there is a better solution.

4

At least some providers have demonstrated massively large clusters. But perhaps multiple clusters is a better answer for you.

What is the relationship of these pods to each other? Do they communicate?

5

These pods do not communicate with each other, they are similar.

So I’m thinking there should be a way to abstract them to use resources efficiently.
I’ve noticed that there are actor models that seem to fit our needs, but there doesn’t seem to be a solution that can be applied directly to containers and pods

6

You havea many degrees of freedom, but there is no magic bullet

You can make each pod serve multiple tenants, which amortizes the overhead but is more complex or you can run one pod per tenant (so called multi-single-tenant) which is simple but has more overhead.

You can run fewer nodes with more pods per node or more nodes with fewer pods per node.

You can run fewer clusters with more nodes or more clusters with fewer nodes.