We are working on adding Istio’s sidecar to our workloads, and are running into a stumbling block because it requires NET_ADMIN capability due to iptables interactions. While we can work around this by moving Istio iptables manipulation out of tenant pods (which OpenShift does), that adds complexity. It is far easier to manage cohesive pods than coordinating between pods and host level software (dependent versions can skew, communication channels need to be hardened and rolling out upgrades is much riskier).
What I think would be ideal would be the notion of a privileged container that can be injected into pods that the tenant has no access to. This would be analogous to the privileged container running in a different namespace. E.g. the tenant cannot ssh into this container, even if we permit ssh’ing into other containers in their pod.
Any suggestions on this would be appreciated.
Thanks,
EJ