How to use a local registry

Kubernetes manages containerised applications based on images. These images can be created locally, or more commonly are fetched from a remote image registry. The following documentation explains how to use MicroK8s with local images, or images fetched from public or private registries.

A familiarity with building, pushing and tagging container images will be helpful. These examples use Docker but you can use your preferred container tool chain.

To install Docker on Ubuntu 18.04:

sudo apt-get install docker.io

Add the user to the docker group:

sudo usermod -aG docker ${USER}

Open a new shell for the user, with updated group membership:

su - ${USER}

The Dockerfile we will be using is:

FROM nginx

To build the image tagged with mynginx:local, navigate to the directory where Dockerfile is and run:

docker build . -t mynginx:local

This will generate a new local image tagged mynginx:local.

Working with locally built images without a registry

When an image is built it is cached on the Docker daemon used during the build. Having run the docker build . -t mynginx:local command, you can see the newly built image by running:

docker images

This will list the images currently known to Docker, for example:

REPOSITORY          TAG                 IMAGE ID            SIZE
mynginx             local               1fe3d8f47868        16.1MB

The image we created is known to Docker. However, Kubernetes is not aware of the newly built image. This is because your local Docker daemon is not part of the MicroK8s Kubernetes cluster. We can export the built image from the local Docker daemon and “inject” it into the MicroK8s image cache like this:

docker save mynginx > myimage.tar
microk8s ctr image import myimage.tar

Note that when we import the image to MicroK8s we do so under the k8s.io namespace (in versions on MicroK8s prior to 1.17 it was necessary to specify ‘-n k8s.io’ with these commands).

Now we can list the images present in MicroK8s:

microk8s ctr images ls

At this point we are ready to microk8s kubectl apply -f a deployment with this image:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: mynginx:local
        imagePullPolicy: Never
        ports:
        - containerPort: 80

ⓘ Note: Make sure the imagePullPolicy is set to Ǿever as shown above, otherwise MicroK8s will continue to try and pull from Dockerhub, even if the image is present in the local registry.

We reference the image with image: mynginx:local. Kubernetes will behave as though there is an image in docker.io (the Dockerhub registry) for which it already has a cached copy. This process can be repeated any time changes are made to the image. Note that containerd will not cache images with the latest tag so make sure you avoid it.

ImagePullPolicy: Never doesn’t seem to work on the current version. This document https://itnext.io/working-with-image-registries-and-containerd-in-kubernetes-63c311b86368 more or less mirrors the documentation but omits that.

Thanks for the feedback @serversonic
That does seem to be the correct policy for using local images, but I’ll check into it

What do you mean by that? Does the pod crash or is it failing silently? Also, by current version you mean Kubernetes v1.18.2? Thank you.

microk8s version is 1.18.2, installed as per the docs.
I followed this page: https://microk8s.io/docs/registry-images

I can see from microk8s ctr images ls that I have an image as follows:
ee6992c2ed0f4628fcef75751048927bcd6b1cee89c79f6acb63ca5474d5a 11.2 MiB linux/amd64,linux/arm,linux/arm64,linux/ppc64le,linux/s390x io.cri-containerd.image=managed docker.io/library/mynginx:local application/vnd.oci.image.manifest.v1+json sha256:49b

But creating a deployment using the YAML given results in this:

kubectl get pods nginx-deployment-6b96d9cb5d-ldg5k 0/1 ErrImageNeverPull 0 54m

I can see the image is being written to syslog here:
Jun 9 10:18:37 cluster01 microk8s.daemon-containerd[4486]: time="2020-06-09T10:18:37.824339360Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/library/mynginx:local,Labels:map[string]string{io.cri-containerd.image: managed,},}"

While I’ve been writing this I deleted the pod that had failed, and then the deployment, then tried it again and now it’s worked. The error I was getting in syslog was along the lines of the image not available in the local registry and policy meaning it wouldn’t be pulled externally.

I’m looking at this as a experienced sysadmin who is new to kubernetes
and starting from scratch, and it seems like a troubleshooting issue which would be (well, is) frustrating if the docs are a tutorial but on the other hand I can understand that microk8s is meant to be a ‘just works’ version of kubernetes and I’m fine with that too, so it feels like a documentation issue rather than a bug but I’m not really sure what a viable resolution would be.

Thanks again for mentioning this - glad you resolved your issue.
I’m going to run through this page and see what useful extra detail and troubleshooting we can add

It’s a bit strange that this article does not go any further. It stops at deploying the image, but does not explain how to verify that the container is actually running, and how to reach the Nginx service that was deployed


Via a separate tutorial I installed the dashboard, but otherwise I wouldn’t be able to figure out how to actually access/use the deployed service.

BTW, I’m new to k8s and MicroK8s. Earlier I tried deploying a local image using Minikube, but got stuck trying to use a local container repository (which itself needs to run inside Minikube). So I love how much easier this is with MicroK8s!

How do these options work when you have a multi-node cluster? Will you be able to use the “local images” option? And if using the built-in registry, can it be authenticated or limited to certain ip addresses?

Some of the features of MicroK8s are better suited for local development. The image importing, the build in registry and the storage addons are such features. In a cluster setup the registry addon or the image importing approach impose certain challenges. To work with locally build images without a registry in a cluster you would need to import the images on all nodes. That is not so convenient, right? Another point you should be aware of is that the registry addon uses local storage to store images and that may lead to problems in a multi node setup where the registry pod gets rescheduled to other nodes.

In a cluster setup you are expected not to import images directly to containerd and also to provide your own registry.

These are good points which I think are obvious to people who understand how the cluster works, but are perhaps non-obvious to people in general, particularly those who are new to MicroK8s. I will work in some wording to the registry docs to make this more clear, thanks!

I suggest highlighting the importance of ImagePullPolicy: Never in the closing paragraph, otherwise, MicroK8s will pull the image from dockerhub despite having it in the local registry, I had a pre-existing yaml to deploy my Pod and missed this detail the first time

Thanks! done

1 Like

Thank you!

I noticed this behavior when using the local registry and updating a previously imported image:

  1. I generate the image again with docker, but no change on the version tag
  2. Import the image into microk8s registry, works as expected
  3. kubectl rollout restart, expecting to load the new image from local registry, but keeps using the old one

When using dockerhub, if I upload a new image and then execute a rollout restart, microk8s detects that there is a new image version despite the tag being the same as the current one in use, but this is not happening when using the local registry, I guess the ImagePullPolicy: Never affects this behavior? but if I change this attribute then the image will be retrieved from dockerhub, I wonder if there is a workaround for those cases when you change the image but not the tag.

The title says “How to use a local registry” but I think it should be " How to use a local image "