Kubernetes - limiting count and frequency of image pulls cluster wide

General Discussions
1

I am getting image pull outage due to reaching the image pull count limit from docker hub, see “Events” section of the output of pod describe below.

It seems a pod in the kubernetes cluster I have setup is not finding the image on docker hub and hence failing the pod construction phase. Kubernetes then tries to construct a new pod for with for the same image and attempts to pull the same image once more.

In Kubernetes, is there a way to limit the number of times images are pulled from an image repository.

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  6m3s                 default-scheduler  Successfully assigned calico-system/calico-node-rq98n to centos7-03-10
  Warning  Failed     5m15s                kubelet            Failed to pull image "docker.io/calico/pod2daemon-flexvol:v3.24.0": rpc error: code = Unknown desc = initializing source docker://calico/pod2daemon-flexvol:v3.24.0: reading manifest v3.24.0 in docker.io/calico/pod2daemon-flexvol: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Normal   Pulling    108s (x4 over 6m3s)  kubelet            Pulling image "docker.io/calico/pod2daemon-flexvol:v3.24.0"
  Warning  Failed     62s (x4 over 5m15s)  kubelet            Error: ErrImagePull
  Warning  Failed     62s (x3 over 3m53s)  kubelet            Failed to pull image "docker.io/calico/pod2daemon-flexvol:v3.24.0": rpc error: code = Unknown desc = reading manifest v3.24.0 in docker.io/calico/pod2daemon-flexvol: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Normal   BackOff    21s (x7 over 5m15s)  kubelet            Back-off pulling image "docker.io/calico/pod2daemon-flexvol:v3.24.0"
  Warning  Failed     21s (x7 over 5m15s)  kubelet            Error: ImagePullBackOff

Cluster information:

Kubernetes version: v1.24.2
Deployment type: on-premises using VMs (libvirt)
Installation method: kubeadm
Host OS: Centos 7.2009
CNI and version: Calico v3.24.0
CRI and version: CRI-O v1.24.2

2

I would advise creating a local docker repository using nexus and push all your images there, then direct your deployments to pull it from that repo - no need to limit.

3

you can add " imagePullPolicy: IfNotPresent "

4

| haitang5445
August 30 |

  • | - |

you can add " imagePullPolicy: IfNotPresent "

Visit Topic or reply to this email to respond.

To unsubscribe from these emails, click here.

Thank you. Your answer is very brief. I assume you mean setting the value to “IfNotPresent” of the imagePullPolicy field of Kubernetes object such as

  1. Pod, at the “{.pod.spec.containers.imagePullPolicy)”
  2. ReplicationController. at the “{.spec.template.spec.containers[*].imagePullPolicy}” location.

But this will require knowledge of the containers and their deployment objects (Pods, ReplicaSets, Deployments, Jobs, etc) that are being deployed on the cluster so I may include or modify the value of imagePullPolicy to “IfNotPresent” of each such top level object.

For example some of the Calico CNI objects to be installed on the cluster mentions (as per the official documentation “https://projectcalico.docs.tigera.io/getting-started/kubernetes/self-managed-onprem/onpremises”) the use of the command below.

kubectl create -f [https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/tigera-operator.yaml](https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/tigera-operator.yaml)

To set the imagePullPolicy at the container level of the Pod, Deployment or other object(s) object mentioned in the above yaml file, I will need to first download the yaml file and modify it accordingly by setting the imagePullPolicy each containers[*] section therein.

I would like to avoid this and simply set some sort of “cluster wide imagePullPolicy” to “IfNotPresent” hence will work even for future deployments too.

This functionality, if it does not already exist, may qualify for “feature request” to the kubernetes developers.

I see not having some sort of “cluster wide imagePullPolicy” feature may lead to a kind of denial of service as follows.

If an object definition file is to be deployed directly on to the cluster without first passing through some kind of inhouse verification CI/CD pipeline. And where such a definition file mentions images which cannot be pulled from the repository for some reason on the current setup.

This will lead to the respective controller trying over and over again to pull the image, finally leading to some kind of “blacklisting” by the image repository for the given kubernetes cluster.

Here subsequent image pulls of other images from the image repository by the given kubernetes cluster will also lead to failure and thereby extending the “blacklisting” period for this cluster at the image repository.

Allan.

6

Thank you. Setting up a “clone” image repository having the images to be pulled by the kubernetes cluster the pointing deployments to use that repository solves the issue of being blacklisted by the source image repositories such as docker hub.

I am new to kubernetes and I have just heard (I am yet to learn) of “nexus” from your email.

Hence my understanding and explanation of these products may be wrong, corrections are welcome.

Populating such a “clone” image repository would require envisaging the images that will be pulled by deployments on my kubernetes.
And may require some configuration at the kubernetes objects definition files as explained here “Pull an Image from a Private Registry | Kubernetes”.
May be the configuration would entail.

As I mentioned in my response to the use of containers[*].imagePullPolicy: IfNotPresent solution, this will require knowledge of the containers and their deployment objects (Pods, ReplicaSets, Deployments, Jobs, etc) that are being deployed on the cluster so I may configure the use of the “private repository” in the definition files of those objects to be constructed.

For example some of the Calico CNI objects to be installed on the cluster mentions (as per the official documentation “Install Calico networking and network policy for on-premises deployments”) the use of the command below.

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/tigera-operator.yaml

To set the private repostory at the container level of the Pod, Deployment or other object(s) object mentioned in the above yaml file, I will need to first download the yaml file and modify it accordingly to use the local repository. In pull failure I would manually “copy the image” from docker hub and into the local repository.

I would like to avoid this.

I may preceding email on this matter contains my explanation for wanting to avoid per deployment reconfiguration.

Below is the explanation I provided.

This functionality, if it does not already exist, may qualify for “feature request” to the kubernetes developers.

I see not having some sort of “cluster wide image pull limit” feature may lead to a kind of denial of service as follows.

This may be brought about by deployments I miss to change the image definitions for point to the local repository where image definition remains pointing to docker hub or public image repository where the image may not be successfully pulled for one reason or another.

Here subsequent image pulls of other images from the image repository by the given kubernetes cluster will also lead to failure and thereby extending the “blacklisting” period for this cluster at the image repository.

Allan.