I am curious if anyone is tracking pod level resource constraints? We have a workload that needs to run 5 pretty heavy duty containers. However the load is balanced, when one container is busy the others tend to be idle. We would like to run them so they can borrow CPU and memory from each other. With container resource constraints, we have to provision a pod that is quite a bit bigger than we would like.
Some are pushing to push all of these processes into one large container, so we can constrain that. I worry that is a bad idea, but Docker best practices have softened on this. Any one have any reasons why this is good or bad idea in Kubernetes? I worry that the sub processes are opaque to Kubernetes and to our other monitoring tools. Overall it seems hacky and less composable at deploy time but want to make sure I am not just making a argument solely for the sake of purity.