Need advice for implementation k8s on-premise

General Discussions
1

Hi all,
I have used Azure Kubernetes Service recently, it’s quite good and now I want to deploy a k8s cluster on-premise (by kubeadm - HA cluster , with stacked control plane nodes) for migrating some internal services to k8s and and also give the dev team a dev environment. I don’t have well known about network , I read documentations , I have a lot of questions , I look forward to hearing from your experience.

My environment : I want the master and worker nodes to be spread over the 2 sites/locations head site (192.168.7.x/24) and data center site (192.168.0.x/24) (already connected by vpn site to site) , and my k8s cluster can be integrated (VPN site to site) with AKS later, actually , AKS has been connected VPN site to site with data center site (192.168.0.x/24) already (for my app LDAP authentication), is it possible ?

I will use MetalLB for Services of type LoadBalancer, about address pool, maybe 10 address from each range 192.168.7.x/24 and 192.168.0.x/24.
Services at head site (192.168.7.x/24) will be NAT by a nginx ingress like 192.168.7.200
Services at data center site (192.168.0.x/24) will be NAT by a nginx ingress like 192.168.0.200
The rest is just for backup when needed.
Pods for internal services at head site or data center sites will be assigned to respective nodes at each site by the nodeSelector label.

About Container Network Interface (CNI): which one should I choose ?
I want pods are also assigned IP address and I can connect to pods from outside k8s cluster
I want to use Azure CNI for my k8s cluster , is it possible ?
If cannot then I want to use Calico or Weave ? because I want to deploy network policy in future.

About pod-network-cidr and service-cidr : I want to use 192.168.176.x/21 for them
pod-network-cidr 192.168.176.0/22 = 192.168.176.0 - 192.168.179.255 (max 1024 pods)
service-cidr 192.168.180.0/23 = 192.168.180.0 - 192.168.181.255 (max 512 services)
The remains 192.168.182.0/23 = 192.168.182.0 - 192.168.183.255 (512 IP addresses) is for backup.

Which MetalLB mode layer 2 or BGP should I use ? I read documentation about their limitation but I don’t quite understand how these 2 modes affect my network design above.

About domain for cluster and coredns , I want to use my internaldomain.local (Windows Active Directory) so dns queries between k8s cluster and internal network can work smoothly, for ex :
k8s.internaldomain.local for control plane API server
*.k8s.internaldomain.local for k8s cluster services .

About storage for persistent volume I attempt to use the glusterfs operator with heketi, is it good ? I am graceful for reviews from people who have used it or other recommendations.

2

My biggest piece of advise is to not stretch your cluster. You can search this forum and elsewhere about it, but k8s is really not meant to be stretched over a wide area network. People HAVE done it, but it requires serious planning and a deeper understanding of k8s and how all the pieces fit together.

I’d suggest starting with a basic bare metal cluster and moving up from there.

1 Like
3

Hi @mrbobbytables , thank you very much for your advice.
You’re right , stretch my cluster over multi locations is too much for me right now, I will start with basic bare metal cluster. I’m just curious , have you done that ? And what is your solutions ? kubefed v2 ?
What do you think about my other concerns ?

So I will create 1 cluster at data center site (192.168.0.x/24) , maybe 10 nodes (192.168.0.120 - 192.168.0.129)
I have simple setups with Calico, I can just install and configure MetalLB (layer 2 mode) as usual and enjoy.
pod-network-cidr 10.100.0.0/16
service-cidr 10.101.0.0/16
An I don’t have to worry about extend pod/service cidr range in furture.

4

I would avoid kubefed, it was a bit more of an experiment and isn’t practical in many cases.
There are a few different options out there depending on your needs, Cilium, Submariner, Crossplane etc They all have their pros and cons depending on the needs of your Application/Infrastructure.

EDIT: There are other options being developed in upstream, but like the Multi Cluster Services API, but not a lot supports them right now.

1 Like
5

Disclaimer: I work for Netris.

You should consider looking at Netris.ai, as this problem is exactly what our product solves. You can use it for free and it is easy to deploy. It is an alternative to MetalLB and will automatically configure an L4LB if that is what your application deployment needs.

Stretching your cluster is generally a bad idea.

1 Like