There are situations where it is necessary or desirable to run MicroK8s on a
machine not connected to the internet. This is possible, but there are a few
extra things to be aware of, and some different strategies depending on the extend of separation from the network. This guide explains the necessary preparation required and the steps required for the potential scenarios.
- Install MicroK8s in airgap environments
- Prepare for deployment
- 1. Download MicroK8s snap
- 2. Networking Requirements
- 3. Images
- Deploy MicroK8s cluster
- Prepare for deployment
The main things to consider when deploying MicroK8s in an airgap environment are:
From a machine that has access to the internet, download the
microk8s snaps and assertion files.
NOTE: For MicroK8s versions 1.26 or earlier, the
core18snap is required instead.
sudo snap download microk8s --channel 1.27 sudo snap download core20 sudo mv microk8s_*.snap microk8s.snap sudo mv microk8s_*.assert microk8s.assert sudo mv core20_*.snap core20.snap sudo mv core20_*.assert core20.assert
We will use
microk8s.snap to install MicroK8s on the next steps. The
microk8s.assert are the snap assertion files, required to verify the integrity of the snap packages.
Air-gap deployments are typically associated with a number of constraints and restrictions with the networking connectivity of the machines. Below we discuss the requirements that the deployment needs to fulfil.
Make sure that all cluster nodes are reachable from each other. Refer to Services and ports used for a list of all network ports used by MicroK8s.
Kubernetes services use the default interface of the machine for discovery reasons:
kubelite) uses the default interface to advertise this address to other nodes in the cluster. Starting
kube-apiserverwithout a default route will fail.
kubelite) uses the default interface to pick the node InternalIP address.
- A default gateway greatly simplifies the process of setting up the Calico CNI.
In case your airgap environment does not have a default gateway, you can add a dummy default route on interface
eth0 using the following command:
ip route add default dev eth0
NOTE: The dummy gateway will only be used by the Kubernetes services to know which interface to use, actual connectivity to the internet is not required.
NOTE: Make sure that the dummy gateway rule survives a node reboot.
This is only required if an HTTP proxy (e.g.
squid) is used to allow limited access to image registries (e.g.
rocks.canonical.com, etc) (see the Access to upstream registries via an HTTP proxy section below).
Ensure that all nodes can use the proxy to access the registry. For example, if using
http://squid.internal:3128 to access
docker.io, an easy way to test connectivity is:
export https_proxy=http://squid.internal:3128 curl -v https://registry-1.docker.io
All workloads in a Kubernetes cluster are running as an OCI image. Kubernetes needs to be able to fetch these images and load them into the container runtime, otherwise the cluster will be unable to run any workload. For a MicroK8s deployment, you will need to fetch the images used by the MicroK8s core (calico, coredns, etc) as well as any images that are needed to run your workloads.
For airgap deployments, there are 3 main options, ordered by ease of use.
NOTE: For a list of all images used by MicroK8s, see images.txt. This is the list of core images required to bring up MicroK8s (e.g. CoreDNS, Calico CNI, etc). Make sure that you also include any images for the workloads that you intend to run on the cluster.
NOTE: Depending on the use case, more than one of the methods below may be required.
In many cases, the nodes of the airgap deployment may not have direct access to upstream registries, but can reach them through the use of an HTTP proxy.
In case regulations and/or network constraints do not allow the cluster nodes to access any upstream image registry, it is typical to deploy a private registry mirror. This is an image registry service that contains all the required OCI Images (e.g. registry, Harbor or any other OCI registry) and is reachable from all cluster nodes.
This requires three steps:
- Deploy and secure the registry service. This is out of scope for this document, please follow the instructions for the registry that you want to deploy.
- Load all images from the upstream source and push to our registry mirror.
- Configure the MicroK8s container runtime (
containerd) to load images from the private registry mirror instead of the upstream source. This will be described in the Configure registry mirrors section.
In order to load images into the private registry, you need a machine with access to both the upstream registry (e.g.
docker.io) and the internal one. Loading the images is possible with
For the examples below we assume that a private registry mirror is running at
On the machine with access to both registries, first install
ctr. For Ubuntu hosts, this can be done with:
sudo apt-get update sudo apt-get install containerd
Then, pull an image:
NOTE: For DockerHub images, prefix with
export IMAGE=library/nginx:latest export FROM_REPOSITORY=docker.io export TO_REPOSITORY=10.100.100.100:5000 # pull the image and tag ctr image pull "$FROM_REPOSITORY/$IMAGE" ctr image convert "$FROM_REPOSITORY/$IMAGE" "$TO_REPOSITORY/$IMAGE"
Finally, push the image (see
ctr image push --help for a complete list of supported arguments):
# push image ctr image push "$TO_REPOSITORY/$IMAGE" # OR, if using HTTP and basic auth ctr image push "$TO_REPOSITORY/$IMAGE" --plain-http -u "$USER:$PASS" # OR, if using HTTPS and a custom CA (assuming CA certificate is at `/path/to/ca.crt`) ctr image push "$TO_REPOSITORY/$IMAGE" --ca /path/to/ca.crt
Make sure to repeat the steps above (pull, convert, push) for all the images that you need.
On the machine with access to both registries, first install
docker. For Ubuntu hosts, this can be done with:
sudo apt-get update sudo apt-get install docker.io
If needed, login to the private registry:
sudo docker login $TO_REGISTRY
Then pull, tag and push the image:
export IMAGE=library/nginx:latest export FROM_REPOSITORY=docker.io export TO_REPOSITORY=10.100.100.100:5000 sudo docker pull "$FROM_REPOSITORY/$IMAGE" sudo docker tag "$FROM_REPOSITORY/$IMAGE" "$TO_REPOSITORY/$IMAGE" sudo docker push "$TO_REPOSITORY/$IMAGE"
Repeat the pull, tag and push steps for all required images.
Image side-loading is the process of loading all required OCI images directly into the container runtime, so that they do not have to be fetched at runtime. If the image side-loading option is chosen, you then need a bundle of all the OCI images that will be used by the cluster.
See the Image side-loading page for more information on how to create a bundle of OCI images. As an example, to create a bundle of all OCI images currently in use by a MicroK8s instance and store it into
images.tar, you can use:
microk8s images export-local > images.tar
core20.assert files into the target node, then install with:
sudo snap ack core20.assert && sudo snap install ./core20.snap sudo snap ack microk8s.assert && sudo snap install ./microk8s.snap --classic
Repeat the above for all nodes of the cluster.
NOTE: This step is not required for single-node deployments.
On one of the nodes, run the following command:
microk8s add-node --token-ttl 3600
This will print the command that needs to be used by all other nodes to join the cluster, for example:
microk8s join 10.0.0.10:25000/asd6fa8sd67857a587dsa65f87a/fg6sdf87g65
After a while, you should be able to see all the cluster nodes showing up in the output of the
microk8s kubectl get node. The nodes will most likely be in
NotReady state, since we still need to ensure the container runtime can fetch images.
/var/snap/microk8s/current/args/containerd-env and set
no_proxy. For example, if your proxy is at
http://squid.internal:3128, append the following lines:
HTTP_PROXY=http://squid.internal:3128 HTTPS_PROXY=http://squid.internal:3128 NO_PROXY=10.0.0.0/8,192.168.0.0/16,127.0.0.1,172.16.0.0/12
Then restart MicroK8s with:
sudo snap restart microk8s
NOTE: For more information, see Installing behind a proxy.
This requires that you have already setup a registry mirror, as explained in Use a private registry mirror.
Assuming the registry mirror is at
/var/snap/microk8s/current/args/certs.d/docker.io/hosts.toml and make sure it looks like this:
# /var/snap/microk8s/current/args/certs.d/docker.io/hosts.toml [host."http://10.100.100.100:5000"] capabilities = ["pull", "resolve"]
You will have to specify the registry CA certificate as well. Copy the certificate to
/var/snap/microk8s/current/args/certs.d/docker.io/ca.crt, then add
# /var/snap/microk8s/current/args/certs.d/docker.io/hosts.toml [host."https://10.100.100.100:5000"] capabilities = ["pull", "resolve"] ca = "/var/snap/microk8s/current/args/certs.d/docker.io/ca.crt"
For MicroK8s 1.25 or newer, copy the
images.tar file to one of the cluster nodes and run the following command:
microk8s images import < images.tar
In older MicroK8s versions, copy the
images.tar on all nodes and run the following on each node:
microk8s ctr image import - < images.tar
NOTE: See the image side-loading page for more details.