MicroK8s IPv6 DualStack HOW-TO

Install Micro Kubernetes 1.20

sudo snap install microk8s --classic --channel=1.20/stable
sudo sudo snap alias microk8s.kubectl kubectl
microk8s enable dns

Configure firewall permissions

sudo sysctl -w net.ipv6.conf.all.forwarding=1
sudo iptables -P FORWARD ACCEPT
sudo ufw allow in on vxlan.calico
sudo ufw allow out on vxlan.calico

Activate IPv6DualStack feature in Kubernetes

Look at sample IPv6 CIDR for pods (fd01::/64) and services (fd98::/108) and minimum prefixes:

sudo patch /var/snap/microk8s/current/args/kube-proxy << EOF
2c2,3
< --cluster-cidr=10.1.0.0/16
---
> --cluster-cidr=10.1.0.0/16,fd01::/64
> --feature-gates="IPv6DualStack=true"
EOF
sudo patch /var/snap/microk8s/current/args/kube-apiserver << EOF
2c2
< --service-cluster-ip-range=10.152.183.0/24
---
> --service-cluster-ip-range=10.152.183.0/24,fd98::/108
18a19
> --feature-gates="IPv6DualStack=true"
EOF
sudo patch /var/snap/microk8s/current/args/kube-controller-manager << EOF
7a8,11
> --feature-gates="IPv6DualStack=true"
> --service-cluster-ip-range=10.152.183.0/24,fd98::/108
> --cluster-cidr=10.1.0.0/16,fd01::/64
> 
EOF
sudo patch /var/snap/microk8s/current/args/kubelet << EOF
16a17
> --feature-gates="IPv6DualStack=true"
EOF

Calico

Enable ipv6 in calico config

Add "assign_ipv6": "true" in the config map:

cat << EOF > calico-config.patch
data:
  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "calico",
          "log_level": "info",
          "datastore_type": "kubernetes",
          "nodename_file_optional": true,
          "nodename": "__KUBERNETES_NODE_NAME__",
          "mtu": __CNI_MTU__,
          "ipam": {
              "type": "calico-ipam",
              "assign_ipv4": "true",
              "assign_ipv6": "true"
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        },
        {
          "type": "bandwidth",
          "capabilities": {"bandwidth": true}
        }
      ]
    }
  typha_service_name: none
  veth_mtu: "1440"
EOF
kubectl patch -n kube-system configmaps/calico-config --patch-file=calico-config.patch

Enable IPv6 in calico node

cat << EOF > calico-node.patch
spec:
  template:
    spec:
      containers:
      - env:
        - name: IP6
          value: autodetect
        - name: IP6_AUTODETECTION_METHOD
          value: can-reach=www.google.com
        - name: CALICO_IPV6POOL_CIDR
          value: fd01::/64
        - name: FELIX_IPV6SUPPORT
          value: "true"
        name: calico-node
EOF
kubectl patch -n kube-system daemonset/calico-node --patch-file=calico-node.patch

TIP: You need an interface with a valid IPv6 address that can reach the Internet.
Otherwise change or remove auto-detection method, and provide an explicit address in IP6.
Check kubectl logs -n kube-system daemonset/calico-node to see if IPv6 auto-detection works.

Install calicoctl

Deploy a container with the calicotl tool (other options available too):

kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml

Configure calico IPv6 pool for egress traffic

Enable natOutgoing: true in calico IPv6 pool, so pods can reach Internet:

kubectl exec -i -n kube-system calicoctl -- /calicoctl replace -f - << EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: default-ipv6-ippool
spec:
  blockSize: 122
  cidr: fd01::/64
  ipipMode: Never
  nodeSelector: all()
  vxlanMode: Never
  natOutgoing: true
EOF

Test IPv6

Restart Micro Kubernetes so changes are applied:

microk8s stop
microk8s start

Test internal connectivity

Deploy a container to run commands:

kubectl run  -it --rm ipv6 --image=busybox -- sh

Check interface configuration

Run ifconfig eth0 to see the inet6 assigned address in the CIDR pool ( fd01: prefix for pods):

eth0      Link encap:Ethernet  HWaddr 6E:22:EF:86:09:D5  
          inet addr:10.1.200.131  Bcast:0.0.0.0  Mask:255.255.255.255
          inet6 addr: fd01::75b5:37a:e343:ba08/128 Scope:Global
          inet6 addr: fe80::6c22:efff:fe86:9d5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1440  Metric:1
          RX packets:17 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:1 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2786 (2.7 KiB)  TX bytes:822 (822.0 B)

Check default IPv6 route

Run route -A inet6:

Kernel IPv6 routing table
Destination                                 Next Hop                                Flags Metric Ref    Use Iface
fd01::75b5:37a:e343:ba08/128                ::                                      U     256    1        0 eth0    
fe80::/64                                   ::                                      U     256    1        0 eth0    
::/0                                        fe80::ecee:eeff:feee:eeee               UG    1024   2        4 eth0    
::1/128                                     ::                                      Un    0      2        0 lo      
fd01::75b5:37a:e343:ba08/128                ::                                      Un    0      4        6 eth0    
fe80::6c22:efff:fe86:9d5/128                ::                                      Un    0      3        2 eth0    
ff00::/8                                    ::                                      U     256    7       18 eth0    
::/0                                        ::                                      !n    -1     1        1 lo      

Check that egress traffic is working

ping6 ipv6.google.com

PING ipv6.google.com (2800:3f0:4002:800::200e): 56 data bytes
64 bytes from 2800:3f0:4002:800::200e: seq=0 ttl=115 time=11.254 ms
64 bytes from 2800:3f0:4002:800::200e: seq=1 ttl=115 time=10.840 ms
64 bytes from 2800:3f0:4002:800::200e: seq=2 ttl=115 time=20.023 ms
64 bytes from 2800:3f0:4002:800::200e: seq=3 ttl=115 time=11.181 ms
^C
--- ipv6.google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 10.840/13.324/20.023 ms

wget ipv6.google.com

Connecting to ipv6.google.com ([2800:3f0:4002:800::200e]:80)
saving to 'index.html'
index.html           100% |*************************************| 12746  0:00:00 ETA
'index.html' saved

IPv6 service

Deploy nginx dual stack to test services (note ipFamilies to use IPv6):

kubectl apply -f - << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginxdualstack
spec:
  selector:
    matchLabels:
      run: nginxdualstack
  replicas: 1
  template:
    metadata:
      labels:
        run: nginxdualstack
    spec:
      containers:
      - name: nginxdualstack
        image: rocks.canonical.com/cdk/diverdane/nginxdualstack:1.0.0
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx6
  labels:
    run: nginxdualstack
spec:
  type: NodePort
  ipFamilies:
  - IPv6
  ipFamilyPolicy: RequireDualStack
  ports:
  - port: 80
    protocol: TCP
  selector:
    run: nginxdualstack
EOF

Check kubernetes services

Execute kubectl get svc that it is being served in IPv6 ( fd98: prefix for services):

NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.152.183.1   <none>        443/TCP        131m
nginx6       NodePort    fd98::6935     <none>        80:31639/TCP   13m

Check that internal traffic (services) is working

/ # wget nginx6 -O -
Connecting to nginx6 ([fd98::6935]:80)
saving to 'index.html'
index.html           100% |*************************************|   362  0:00:00 ETA
'index.html' saved

Accessing the service externally

At this point you should be able to open your browser at http://[public IPv6]:port/ to get a nice IPv6 page:

<!DOCTYPE html>
<html>
<head>
<title>Kubernetes IPv6 nginx</title> 
</head>
<body>
<h1>Welcome to nginx on IPv6 Kubernetes!</h1>
<p>Pod: nginxdualstack-7986d8df8d-9g448</p>
</body>
</html>

References:

3 Likes