This guide explains how to add Windows worker nodes to MicroK8s. This how-to guide explains how to join a Windows Server 2022 to an existing MicroK8s cluster running Calico.
Note: Currently Windows nodes may be affected by a bug in the Windows container code for Server 2022/July update. Please see this issue for more details and a workaround:Container networking on Kubernetes broken after Server 2022 July 2024 / KB5040437 (OS Build 20348.2582) update · Issue #516 · microsoft/Windows-Containers · GitHub
Requirements
- A node running Windows (tested with Windows Server 2022).
- A MicroK8s cluster using the Calico CNI (enabled by default since 1.19).
Prepare the MicroK8s cluster
-
Determine the exact version of Kubernetes running in the cluster, e.g.
1.27.1. You can use the following command:microk8s kubectl get node -o wide -
Determine the exact version of Calico running in the MicroK8s cluster, e.g.
3.25.0. For this, you can inspect the image used by thecalico-nodecontainers:microk8s kubectl get ds/calico-node -n kube-system -o jsonpath='{.spec.template.spec.containers[?(.name=="calico-node")].image}{"\n"}' -
Generate a
kubeconfigfile for the MicroK8s cluster. You will need this to runcalicoctlcommands, and later copy it to the Windows node.mkdir -p ~/.kube microk8s config > ~/.kube/config -
In order for Windows pods to schedule, strict affinity must be set to true. This is required to prevent Linux nodes from borrowing IP addresses from Windows nodes. This can be set with the calicoctl binary. Install the
calicoctlbinary with:CALICO_VERSION="3.25.0" curl -L https://github.com/projectcalico/calico/releases/download/v$CALICO_VERSION/calicoctl-linux-amd64 -o calicoctl chmod +x ./calicoctl(set the environment variable to the appropriate version of Calico for your system)
Then, set strict affinity to true with the following command:./calicoctl ipam configure --strictaffinity=true --allow-version-mismatch
Prepare the Windows node
All the commands shown next have to be run in a PowerShell window as Administrator.
-
Install the
containerdcontainer runtime. The machine might restart during this operation.Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/microsoft/Windows-Containers/Main/helpful_tools/Install-ContainerdRuntime/install-containerd-runtime.ps1" -o install-containerd-runtime.ps1 .\install-containerd-runtime.ps1 -
Create directory
c:\k, create the kubeconfig filec:\k\config. After creating, open the file with notepad, paste the contents of thekubeconfigfile of the MicroK8s cluster and save.mkdir c:\k New-Item c:\k\config notepad c:\k\config
Install Calico
-
Retrieve the
install-calico-windows.ps1script from the Calico GitHub releases page. NOTE: do not worry about the calico version in the URL, we will pick the Calico version to install later.Invoke-WebRequest -Uri https://github.com/projectcalico/calico/releases/download/v3.25.1/install-calico-windows.ps1 -OutFile c:\k\install-calico-windows.ps1 -
Download Calico and Kubernetes binaries using the following command (replace
1.27.1with the Kubernetes version and3.25.0with the Calico version running in the cluster):c:\k\install-calico-windows.ps1 -ReleaseBaseURL "https://github.com/projectcalico/calico/releases/download/v3.25.0" -ReleaseFile "calico-windows-v3.25.0.zip" -KubeVersion "1.27.1" -DownloadOnly "yes" -ServiceCidr "10.152.183.0/24" -DNSServerIPs "10.152.183.10" -
Configure the CNI bin and configuration directories and then install the Calico services. If the vSwitch is not yet created, this will temporarily affect network connectivity for a few seconds.
$ENV:CNI_BIN_DIR="c:\program files\containerd\cni\bin" $ENV:CNI_CONF_DIR="c:\program files\containerd\cni\conf" c:\calicowindows\install-calico.ps1 c:\calicowindows\start-calico.ps1If successful, the output should look like this:
Starting Calico... This may take several seconds if the vSwitch needs to be created. Waiting for Calico initialisation to finish... Waiting for Calico initialisation to finish...StoredLastBootTime , CurrentLastBootTime 5/21/2023 8:21:24 AM Waiting for Calico initialisation to finish...StoredLastBootTime , CurrentLastBootTime 5/21/2023 8:21:24 AM Calico initialisation finished. Done, the Calico services are running: Status Name DisplayName ------ ---- ----------- Running CalicoFelix Calico Windows Agent Running CalicoNode Calico Windows Startup -
Install Kubernetes services (kubelet and kube-proxy):
c:\calicowindows\kubernetes\install-kube-services.ps1 -
Add a firewall rule for incoming connections to the Windows kubelet node service. This is for
kubectl logsandkubectl execcommands to work with pods running in Windows nodes:New-NetFirewallRule -Name 'Kubelet-In-TCP' -DisplayName 'Kubelet (node)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 10250 -
Configure the network adapter name expected by the kubelet service.
Run
ipconfigto retrieve the IP configuration of the machine’s network adapters. An example output is shown below, where the network adapter name isvEthernet (tapeb67a82f-a1):Windows IP Configuration Ethernet adapter vEthernet (tapeb67a82f-a1): Connection-specific DNS Suffix . : teststack.internal Link-local IPv6 Address . . . . . : fe80::d17b:a2e:c02c:1163%8 IPv4 Address. . . . . . . . . . . : 172.16.1.177 Subnet Mask . . . . . . . . . . . : 255.255.255.0 Default Gateway . . . . . . . . . : 172.16.1.1Then, open
c:\calicowindows\kubernetes\kubelet-service.ps1with notepad and edit the default value of theInterfaceNameparameter to match the one above.# c:\calicowindows\kubernetes\kubelet-service.ps1 Param( [string]$NodeIp="", [string]$InterfaceName="vEthernet (tapeb67a82f-a1)" )Save the file and exit notepad.
-
Clean service arguments for kubelet. Open
c:\calicowindows\kubernetes\kubelet-service.ps1with notepad and ensure that:- For MicroK8s version 1.26 or newer, remove the argument
--logtostderr=true - For MicroK8s version 1.27 or newer, remove the argument
--container-runtime=remote
Save the file and exit notepad.
- For MicroK8s version 1.26 or newer, remove the argument
-
Start Kubernetes services
Start-Service kubelet Start-Service kube-proxy
Verify the Windows node
-
Ensure that the Windows node appears in the output of
microk8s kubectl get nodes -o wide, for example:NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME u1 Ready <none> 3h3m v1.27.1 172.16.1.49 <none> Ubuntu 20.04.5 LTS 5.4.0-139-generic containerd://1.6.15 w1 Ready <none> 105m v1.27.1 172.16.1.177 <none> Windows Server 2022 Datacenter Evaluation 10.0.20348.587 containerd://1.6.6 -
Start a
nanoserverpod on the Windows node and wait for it to come up.microk8s kubectl run test-pod -it --image mcr.microsoft.com/windows/nanoserver:ltsc2022 --overrides '{"spec": {"nodeSelector": {"kubernetes.io/os": "windows"}}}'The default timeout for the pod to come up is 60 seconds, which might not be enough for the nanoserver image to be fetched and the pod to be started. In that case, simply wait for the pod to start and then attach using
microk8s kubectl attach -it test-pod -
Ensure that you can curl the
kubernetesservice, e.g. withcurl https://kubernetes --insecureMicrosoft Windows [Version 10.0.20348.1726] (c) Microsoft Corporation. All rights reserved. C:\>curl https://kubernetes --insecure { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "Unauthorized", "reason": "Unauthorized", "code": 401 }
Uninstall the Windows worker node
On the Windows machine:
-
Stop and uninstall Kubernetes services:
c:\calicowindows\kubernetes\uninstall-kube-services.ps1 -
Stop and uninstall Calico services:
c:\calicowindows\uninstall-calico.ps1
On the MicroK8s machine:
-
Remove the node from MicroK8s (replace
w1with the name of the Windows node):microk8s kubectl delete node w1
Upgrade
There is no specific upgrade path for a Windows node. If you have upgraded MicroK8s to a later version, and wish the Windows node to remain compatible, you should first uninstall the node components (see above), then re-install the newer version.
