Hi all!
We have several geographically separated K8s clusters (-> with different IP ranges), and we are figuring out how to efficiently secure communication between them.
We are using Calico for the network, but its encryption/decryption has a significant performance drop (in the range of high tens of percent). Additionally, we are concerned about latency. If we disable encryption in Calico, we need to find another solution, but not a full mesh.
Another option is deploying WireGuard, but that would lead to a full mesh again.
If we tried SRv6 tunnels or something else, we still have to do encryption on Linux, perhaps at the IPsec level, which is not ideal.
In principle, with a bit of manual work, we can configure WireGuard outside of Calico (a bit ugly, but it could work), enable the use of VXLAN in cross-subnet mode, and tell VXLAN to use the interface from WireGuard. Then it does exactly what we want → encrypting only between locations. It’s nice that each node encrypts itself, so the aggregated performance isn’t too tragic. It just needs a bit of fine-tuning of the MTU, ideally to have the same MTU for both local packets and those going through VXLAN/WireGuard, but I think it could work.
Or use Istio as a service mesh, but that may be a bit convoluted. I would say that what I wrote above (the previous paragraph) is, in principle, the optimal solution, but there might be a better and easier solution out there.
So, how to address encryption between K8s networks without having a full mesh? Any encrypted tunnel between networks where we can send everything? Is there generally an elegant way to solve this problem?
Thank you!
Michal