DockerHub download rate limits

kjackal · January 14, 2022, 11:18am

By default, MicroK8s defaults to DockerHub as a source for pulling images that are required for the Kubernetes cluster to work as expected. DockerHub enforces a rate limit to image downloads. When this limit is reached pods are blocked in the ImagePullBackOff state (shown in microk8s kubectl get po) and describing the blocked pods reports a Too Many Requests error message similar to:

Warning  Failed     19m (x2 over 19m)   kubelet            Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

In order to avoid hitting the DockerHub rate limits, the following workarounds can be applied.

1. Login to DockerHub

The DockerHub rate limits are less strict for authenticated users, though they may still cause problems depending on your usage. Providing a username and password can be achieved by some minor edits to the MicroK8s configuration, or by setting up a secret and configuring your pods to use it. Both are described below.

Option 1 : Configure containerd

ⓘ NOTE: For MicroK8s clusters, you need to repeat these steps for all nodes.

It is possible to configure your DockerHub credentials in the configuration of containerd, so that they are used automatically when pulling images from DockerHub, without users having to specify an image pull secret manually for each container.

To do this, edit /var/snap/microk8s/current/args/containerd-template.toml and add the following section (the configuration is in TOML format, so indentation does not matter):

# containerd-template.toml

[plugins."io.containerd.grpc.v1.cri".registry.configs."registry-1.docker.io".auth]
username = "DOCKERHUB_USERNAME"
password = "DOCKERHUB_PASSWORD"

Afterwards, restart MicroK8s with:

microk8s stop
microk8s start

WARNING: This configuration will allow any user of the cluster to pull images using your DockerHub credentials. If this is not desired, consider using “image pull secrets” instead (described below).

Option 2: Using Image Pull Secrets

Kubernetes allows you to create a secret containing DockerHub credentials and then make use of it for pulling images. See the following upstream documentation:

2. Use an alternative source for images

Use microk8s kubectl edit to edit the deployments and source images from other public registries, such as rocks.canonical.com, gcr.io or quay.io. This involves manually editing the spec of the pods running in your cluster.

For example, to change the image used by the Calico CNI, you can do:

microk8s kubectl edit deploy/calico-kube-controllers
microk8s kubectl edit daemonset/calico-node

Both commands will open up an editor, make sure to change the images:

docker.io/calico/cni:$TAG    -->   quay.io/calico/cni:$TAG
docker.io/calico/node:$TAG   -->   quay.io/calico/node:$TAG
docker.io/calico/kube-controllers:$TAG   -->   quay.io/calico/kube-controllers:$TAG
docker.io/calico/pod2daemon-flexvol:$TAG   -->   quay.io/calico/pod2daemon-flexvol:$TAG

Then, exit the editor and Kubernetes will apply your changes, pulling the Calico CNI images from quay.io instead.

3. Use a private image registry to mirror DockerHub

For production environments, it is highly recommended to use a private image registry to mirror DockerHub. In this setup when you specify an image from docker.io (e.g. nginx, or docker.io/nginx), MicroK8s will retrieve the image nginx from your private registry instead of DockerHub.

ⓘ Note!: This means that your private registry should contain at least an nginx image.

Assuming your private registry is available at https://my.registry.internal:5000, the required configuration depends on your MicroK8s version.

For MicroK8s version 1.23 or newer

MicroK8s 1.23 and newer versions use separate hosts.toml files for each image registry. For docker.io, this can be found in /var/snap/microk8s/current/args/certs.d/docker.io/hosts.toml.

Edit the file so that the contents look like this:

# /var/snap/microk8s/current/args/certs.d/docker.io/hosts.toml
server = "https://my.registry.internal:5000"

[host."my.registry.internal:5000"]
capabilities = ["pull", "resolve"]

Then, restart MicroK8s with:

microk8s stop
microk8s start

For MicroK8s version 1.22 or older

Find the [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] section in /var/snap/microk8s/current/args/containerd-template.toml and edit it as follows:

# containerd-template.toml

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://my.registry.internal:5000"]