TL;DR the offline/airgap instructions are incomplete and you can’t get a functioning cluster today with strict microk8s and cis-hardening applied offline without significant engineering effort… but it IS possible.
Did you install the cis-hardening addon by any chance? This ensures (amongst many other things) that the admission controller enables AlwaysPullImages (as this is effectively the only way to check that this image is still authorised to be ran on a node at runtime).
If you are sideloading images before you have a container registry this of course fails even if you have the image present on every node. So you need to:-
- do a fresh vanilla microk8s (classic or strict) installation with nothing but dns and cis-hardening enabled
- disable this setting in /var/snap/microk8s/current/args/kube-apiserver (Search for ‘admission’ in there and remove AlwaysPullImages) - I have a sed one liner to do this.
- restart microk8s (so the setting is applied)
- sideload images for any addons you intend to enable before (and including) a container repository
- install/enable any addons
Note: If a pod configuration is already deployed then changing the above setting won’t have any effect - it only gets mutated when a deployment happens. So make the above changes before you enable any other addons. Calico and CoreDNS seem fine doing the above. I’ve never had issues with either.
Once the above is done you can sideload images for a container repository (including ALL core component images like calico cni), enable the container repository, connect the k8s nodes to this repository, and then re-enable this admission controller setting to ensure this cis-hardening rule is enforced for apps later loaded into the cluster. (You may want to try zot - it’s very lightweight. I’ve not tested it in anger but it looks good so far)
Note that none of the core addon images (or indeed required calico-cni images, weirdly) are included in the microk8s snap - you have to manually fetch these yourself. Note also that the Microk8s offline/airgap installation instructions do not mention the fact that some core addons (Mayastor in particular) have external helm / image dependencies which are not included in the files shipped with the microk8s snap. So you have to do ‘helm dependency update’ on each on an internet connected system (!!!) and copy the tgz files in addons/mayastor/chart/charts to your target system alongside the images for sideloading yourself, and copy them to the appropriate target folder once the microk8s snap is installed.
You’re looking at a few hundred MB of offline files for something with coreDNS, calico, mayastor (and thus also its rbac and other dependencies), and metalLB.
I really like microk8s but more thought does need putting into the offline mode, about shipping at the very least the calico, DNS, etc images and auto sideloading them during install, and more testing needs doing with strict mode and cis-hardening enabled alongside ‘addon’ (but, lets face it, required) components.