I have used Azure Kubernetes Service recently, it’s quite good and now I want to deploy a k8s cluster on-premise (by kubeadm - HA cluster , with stacked control plane nodes) for migrating some internal services to k8s and and also give the dev team a dev environment. I don’t have well known about network , I read documentations , I have a lot of questions , I look forward to hearing from your experience.
My environment : I want the master and worker nodes to be spread over the 2 sites/locations head site (192.168.7.x/24) and data center site (192.168.0.x/24) (already connected by vpn site to site) , and my k8s cluster can be integrated (VPN site to site) with AKS later, actually , AKS has been connected VPN site to site with data center site (192.168.0.x/24) already (for my app LDAP authentication), is it possible ?
I will use MetalLB for Services of type LoadBalancer, about address pool, maybe 10 address from each range 192.168.7.x/24 and 192.168.0.x/24.
Services at head site (192.168.7.x/24) will be NAT by a nginx ingress like 192.168.7.200
Services at data center site (192.168.0.x/24) will be NAT by a nginx ingress like 192.168.0.200
The rest is just for backup when needed.
Pods for internal services at head site or data center sites will be assigned to respective nodes at each site by the nodeSelector label.
About Container Network Interface (CNI): which one should I choose ?
I want pods are also assigned IP address and I can connect to pods from outside k8s cluster
I want to use Azure CNI for my k8s cluster , is it possible ?
If cannot then I want to use Calico or Weave ? because I want to deploy network policy in future.
About pod-network-cidr and service-cidr : I want to use 192.168.176.x/21 for them
pod-network-cidr 192.168.176.0/22 = 192.168.176.0 - 192.168.179.255 (max 1024 pods)
service-cidr 192.168.180.0/23 = 192.168.180.0 - 192.168.181.255 (max 512 services)
The remains 192.168.182.0/23 = 192.168.182.0 - 192.168.183.255 (512 IP addresses) is for backup.
Which MetalLB mode layer 2 or BGP should I use ? I read documentation about their limitation but I don’t quite understand how these 2 modes affect my network design above.
About domain for cluster and coredns , I want to use my internaldomain.local (Windows Active Directory) so dns queries between k8s cluster and internal network can work smoothly, for ex :
k8s.internaldomain.local for control plane API server
*.k8s.internaldomain.local for k8s cluster services .
About storage for persistent volume I attempt to use the glusterfs operator with heketi, is it good ? I am graceful for reviews from people who have used it or other recommendations.