Changing servicecidrs

General Discussions microk8s
1

I was able to change the POD cidrs at my cluster:

>  $ microk8s kubectl get po -A -o wide
> NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE   IP              NODE            NOMINATED NODE   READINESS GATES
> kube-system   calico-kube-controllers-79949b87d-w4pwj   1/1     Running   4          46m   192.1.182.101   hsc-ctsc-kub1   <none>           <none>
> kube-system   calico-node-w5jbx                         1/1     Running   4          46m   10.234.70.212   hsc-ctsc-kub1   <none>           <none>
> kube-system   coredns-ccd8f67bc-zf2pj                   1/1     Running   5          17h   192.1.182.103   hsc-ctsc-kub1   <none>           <none>
> kube-system   metrics-server-64fc948c75-d59rm           1/1     Running   4          46m   192.1.182.100   hsc-ctsc-kub1   <none>           <none>

$ microk8s kubectl describe pod calico-node-w5jbx  -n kube-system | egrep 'CALICO_IPV4POOL_CIDR:'
      CALICO_IPV4POOL_CIDR:               192.1.0.0/16

$ cat /var/snap/microk8s/current/args/kube-proxy | egrep 'cluster-cidr='
--cluster-cidr=192.1.0.0/16

I also wanted the service-cluster-ip-range and for this I changed the file

/var/snap/microk8s/current/args/kube-apiserver

To have:

$ cat /var/snap/microk8s/current/args/kube-apiserver | egrep 'service-cluster-ip-range='
--service-cluster-ip-range=192.152.183.0/24

I re-started the cluster using “microk8s stop” and “microk8s start” and this setting is not being updated since:

$ microk8s kubectl get servicecidrs
NAME         CIDRS             AGE
kubernetes   10.152.183.0/24   17h

A consequence I have is on joining a node I get the warning:

$ microk8s join 10.234.70.212:25000/xxxxXXXXXXXXXXxxxxxxxx --worker
Contacting cluster at 10.234.70.212
WARNING: Joining a cluster that has a different CIDR. The kube-proxy CIDR configuration will be overwritten.
Cluster CIDR: 192.1.0.0/16 -- Node CIDR: 10.1.0.0/16(will be overwritten)

Why is that? Thanks in advance

2

MicroK8s doesn’t update the ServiceCIDR when you change the --service-cluster-ip-range flag because the ServiceCIDR is an immutable object created during cluster initialization. The manual edit to the kube-apiserver config file isn’t enough, as MicroK8s may overwrite it during restarts.

Why the CIDR Isn’t Updating

  • Immutability: The ServiceCIDR object, which manages IP allocations for services, is created only once when the cluster first starts up. After that, it cannot be changed.

  • MicroK8s Configuration: MicroK8s automatically regenerates some configuration files during restarts and upgrades, pulling from its internal data store instead of your manual edits. This can cause your changes to be lost.

  • Worker Node Warnings: The warnings about mismatched CIDRs occur because the worker nodes are trying to join with the cluster’s original network settings, which have not been updated by your manual change. The worker nodes’ kube-proxy and calico pods are configured for the old CIDR.

Kubernetes doesn’t support changing the service CIDR of an existing cluster. Here are the available options, with the recommended one first.

Option 1: Recreate the Cluster (Recommended)

This is the cleanest and most reliable method.

  1. Stop and reset the cluster: microk8s reset

  2. Start a new cluster with the correct CIDRs:
    microk8s start
    –cluster-cidr=192.1.0.0/16
    –service-cidr=192.152.183.0/24

  3. If you need to save your workloads, back them up before resetting the cluster and redeploy them afterward.

    Note Attempt to Patch the Configuration (Not Recommended). This is a riskier approach and not guaranteed to work.
    Migrate Services to a New CIDR (Complex and Unreliable)

1 Like
3

Thanks for your answer…

Here it is the option I was given:

1 Like
4

You can change the ServiceCIDR of a cluster since 1.33 Kubernetes Default ServiceCIDR Reconfiguration | Kubernetes