Hi all,
I am completely new to Kubernetes and have a few doubts regarding networking in Kubernetes.
From my understanding, Kubernetes has three networks: one for pods, one for the nodes, and one for the outside world (WAN). Is my understanding correct? If so, when assigning IP addresses in these networks, should we allocate different networks/subnets for these three network components, or can we assign IPs from the same network? Would this create any problems?
Thank you in advance.
The words “network” and “subnet” are vague in this context. I’ll re-state in a way I think is clearer.
Kubernetes is concerned with 3 sets of IPs: IPs assigned to nodes, IPs assigned to pods, and IPs assigned to services. How those map to subnets or networks is not really relevant.
Every node (VM or bare metal) needs at least on IP for itself (e.g. which Kubelet can use to talk to an apiserver). Some nodes have multiple interfaces and IPs. How a node is created and how IPs are assigned are outside the purvey of Kubernetes.
Every pod (minus hostNetwork: true pods) is also assigned at least one IP address. Exactly how that is managed is context-dependent, but the USUAL model is that each node is given a single CIDR block (e.g. a /26) and on-node agents allocate IPs for pods from that range. The ranges can be assigned to nodes by a kubernetes controller (node-ipam-controller) or by some other mechanism, and they might come from a single large CIDR block (e.g. a /20 can hold 64 nodes with /26 each) or from a set of discontiguous blocks. It’s even possible that every node’s range is discontiguous to every other node! Kubernetes does not define that. Note that I said this is the USUAL approach. It’s also possible that the system dynamically allocates IPs for each pod on the fly, with no CIDR block(s) per node. All Kubernetes cares about is that these IPs are reachable and unique - they can be on the same or different subnet as the node IPs themselves.
Every service (minus some special cases) is also assigned an IP. Kubernetes handles this iitself, and requires that you tell it which CIDR block(s) (singular for now, soon plural) to allocate from. These IPs are usually virtual (they are never observed “on the wire” between nodes), but that is, again, an implementation detail. All Kubernetes cares about is that they are not used for any other purpose.
Thank you very much. I am trying to understand this with relation to my simple setup of just two nodes. I still find it difficult to relate it to the practical. What should I consider when I initialise a Kubernetes cluster and assigning IPs.
To start with the servers/nodes we assign IP either manually or through DHCP. Then we initialise a Kubernetes cluster, at this stage onwards what should we do when it comes to IP addresses is still not clear to me. Sorry if this is dumb question. Don’t we have to consider the node IP at all as if it does not exists when we do anything within kubernetes? Could you please elaborate the simple steps?
Adding a bit more specific questions:
I understand that one can setup the networks as they want for security or any other reason so, there are no one way to do it. But I wanna get things started with kubernetes as a basic beginner. A basic simple setup requirements and the reasoning behind it would be of great help to understand now.
Thank you again for answering in very detailed manner.
To start with the servers/nodes we assign IP either manually or through DHCP.
That’s fine.
Then we initialise a Kubernetes cluster, at this stage onwards what should we do when it comes to IP addresses is still not clear to me.
What you do here depends on how you want your network to operate. Different people want different modes.
Some people use a “flat” network, where every node is assigned an IP range from the “real” network, and those IPs are used for pods.
Some people use an “island” network, where there’s an overlay defined (e.g.VXLAN) or some other form of non-advertised routing.
You need to determine which mode you want to use (depends on your underlying network!) and which network driver does what you need. Cilium is a popular choice, as is Calico.
Thank you,
I am trying to use Cilium and with separate networks/island network. If you do not mind, if I do the following would that work?
k8s service host: 192.168.30.19
k8s service port: 6443
pod network cidr: 10.50.0.0/19
apiserver advertise address: 192.168.30.19
I also use Google’s DNS in my /etc/resolve.conf. And coredns pods stuck with crashloopbackoff. I am not sure whether the networks provided or something else is the reason for the problem.
Thank you.