Configure host interfaces used by MicroK8s

General Discussions microk8s
,
1

This document assumes basic knowledge of Kubernetes, and is only intended for advanced use cases.

By default, MicroK8s will use the default host interface for all control plane (e.g. kube-apiserver) and data plane (e.g. Calico vxlan network) services. For production deployments with multiple interfaces or designated VLANs, cluster administrators may want to configure which host interfaces are in use by MicroK8s.

NOTE: For the rest of this document, the term default interface refers to the host interface that includes a default gateway route.

Control Plane

This section explains how to configure the host interfaces used by the Kubernetes control plane services.

kube-apiserver

By default, kube-apiserver will bind to all host interfaces and advertise the default interface. You can configure it by editing /var/snap/microk8s/current/args/kube-apiserver and setting the following arguments:

# /var/snap/microk8s/current/args/kube-apiserver
--advertise-address=10.10.10.10
--bind-address=0.0.0.0
--secure-port=16443

Apply the changes with:

sudo snap restart microk8s

NOTE: MicroK8s assumes that kube-apiserver is accessible from the local interface 127.0.0.1 for some of its default configuration. If you change the bind-address, you may want to update the address of the apiserver in the kubeconfig files in /var/snap/microk8s/current/credentials accordingly.

kube-controller-manager

By default, kube-controller-manager binds to the default host interface. You can configure it by editing /var/snap/microk8s/current/args/kube-controller-manager and setting the following arguments:

# /var/snap/microk8s/current/args/kube-controller-manager
--bind-address=0.0.0.0
--secure-port=10257

Apply the changes with:

sudo snap restart microk8s

NOTE: The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients.

kube-scheduler

By default, kube-scheduler binds to the default host interface. You can configure it by editing /var/snap/microk8s/current/args/kube-scheduler and setting the following arguments:

# /var/snap/microk8s/current/args/kube-scheduler
--bind-address=0.0.0.0
--secure-port=10259

Apply the changes with:

sudo snap restart microk8s

NOTE: The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients.

kube-proxy

By default, kube-proxy binds to localhost for its health endpoint, and binds NodePort services to all host interfaces. You can configure it by editing /var/snap/microk8s/current/args/kube-proxy and setting the following arguments:

# /var/snap/microk8s/current/args/kube-proxy
--bind-address=0.0.0.0
--healthz-bind-address=127.0.0.1

Apply the changes with:

sudo snap restart microk8s

kubelet

By default, kubelet binds to all host interfaces and advertises the default host interface. You can configure it by editing /var/snap/microk8s/current/args/kubelet and setting the following arguments:

# /var/snap/microk8s/current/args/kubelet
--address=0.0.0.0
--node-ip=10.10.10.10
--healthz-bind-address=127.0.0.1

Apply the changes with:

sudo snap restart microk8s

NOTE: The address must be reachable by the rest of the cluster.

NOTE: The address set in --node-ip is used as the InternalIP of the node, as shown in microk8s kubectl get node -o wide

dqlite

By default, dqlite will bind to localhost (127.0.0.1). When forming a MicroK8s cluster, dqlite will be updated to use the address that was used in the microk8s join command.

Data Plane

Calico VXLAN interface

Calico is the default CNI for MicroK8s, and VXLAN overlay networks are used to configure pod networking. By default, Calico uses the default host interface for pod networking(IP_AUTODETECTION_METHOD=first-found). When forming a MicroK8s cluster, Calico is updated to use address that was used in the microk8s join command (IP_AUTODETECTION_METHOD=can-reach=10.10.10.10). You can configure it by editing /var/snap/microk8s/current/args/cni-network/cni.yaml and setting the following configuration parameters:

     - name: IP_AUTODETECTION_METHOD
       value: "first-found"

Apply the changes with:

microk8s kubectl apply -f /var/snap/microk8s/current/args/cni-network/cni.yaml

NodePort services

By default, kube-proxy will bind NodePort services to all host interfaces. This can be configured by editing /var/snap/microk8s/current/args/kube-proxy and setting the following arguments:

--nodeport-addresses=10.10.10.10

Apply the changes with:

sudo snap restart microk8s
2

Hello,

Thank you for this summary.

I’ve been trying to form a cluster on a 2nd interface different from the default interface, where I want the API servers to use this interface.

But whenever you configure the --advertise-address of the kube-apiserver, joining a node will break the server certificate and the node will never show up on kubectl get nodes even though it joins the dqlite cluster.
The csr.cert will say “changeme” and it won’t be possible to update it using refresh-certs.

It seems to only work if you join the node before changing the advertise-address