How to configure contrainerd to use PAUSE image from the local container runtime?

General Discussions
1

Cluster information:

Kubernetes version: v1.28.7
Cloud being used: Virtualbox
Installation method: Kubernetes Cluster VirtualBox
Host OS: Ubuntu 22.04.3 LTS
CNI and version: calico
CRI and version: containerd://1.7.2

Containerd pulls pause image from remote registry:
I can tell that by looking at /etc/containerd/config.toml and the following line:

sandbox_image = "registry.k8s.io/pause:3.8"

I want contrainerd to use pause image from the local container runtime:

$ sudo crictl image
IMAGE                                     TAG                 IMAGE ID            SIZE
docker.io/calico/cni                      v3.26.0             5d6f5c26c6554       93.3MB
docker.io/calico/kube-controllers         v3.26.0             45ae357729e3a       33.8MB
docker.io/calico/node                     v3.26.0             44f52c09decec       87.6MB
registry.k8s.io/coredns/coredns           v1.10.1             ead0a4a53df89       16.2MB
registry.k8s.io/etcd                      3.5.10-0            a0eed15eed449       56.6MB
registry.k8s.io/kube-apiserver            v1.28.7             eeb80ea665767       34.7MB
registry.k8s.io/kube-controller-manager   v1.28.7             4d9d9de55f196       33.4MB
registry.k8s.io/kube-proxy                v1.28.7             123aa721f941b       28.1MB
registry.k8s.io/kube-scheduler            v1.28.7             309c26d006295       18.6MB
registry.k8s.io/pause                     3.8                 4873874c08efc       311kB   <-- here
registry.k8s.io/pause                     3.9                 e6f1816883972       322kB

What I already tried was to replace sandbox_image as this:
sandbox_image = "pause:3.8"

This however prevented containerd to start:

Apr 05 21:49:01 master containerd[1803]: time="2024-04-05T21:49:01.366173162Z" level=info msg="StopPodSandbox for \"b0a9341d7bfc349d757c9952ae5a06f03b37144351239f832bf4d0cfd6049d40\""
Apr 05 21:49:01 master containerd[1803]: time="2024-04-05T21:49:01.366989045Z" level=info msg="Container to stop \"6fd03b15fbf189c42f20361efaf40315b6b0d0e637bfbca1951664880afc1dfb\" must be in running or unknown state, current state \"CONTAINER_EXITED\""
Apr 05 21:49:01 master containerd[1803]: time="2024-04-05T21:49:01.367756574Z" level=info msg="TearDown network for sandbox \"b0a9341d7bfc349d757c9952ae5a06f03b37144351239f832bf4d0cfd6049d40\" successfully"
Apr 05 21:49:01 master containerd[1803]: time="2024-04-05T21:49:01.368472394Z" level=info msg="StopPodSandbox for \"b0a9341d7bfc349d757c9952ae5a06f03b37144351239f832bf4d0cfd6049d40\" returns successfully"
Apr 05 21:49:01 master containerd[1803]: time="2024-04-05T21:49:01.374337028Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-controller-manager-master,Uid:64fbe8c343e7e6423571e9033e6b3037,Namespace:kube-system,Attempt:22,}"
Apr 05 21:49:02 master containerd[1803]: time="2024-04-05T21:49:02.226758066Z" level=info msg="trying next host" error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry-1.docker.io
Apr 05 21:49:02 master containerd[1803]: time="2024-04-05T21:49:02.236924313Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-controller-manager-master,Uid:64fbe8c343e7e6423571e9033e6b3037,Namespace:kube-system,Attempt:22,} failed, error" error="failed to get sandbox image \"pause:3.8\": failed to pull image \"pause:3.8\": failed to pull and unpack image \"docker.io/library/pause:3.8\": failed to resolve reference \"docker.io/library/pause:3.8\": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Apr 05 21:49:02 master containerd[1803]: time="2024-04-05T21:49:02.238655583Z" level=info msg="stop pulling image docker.io/library/pause:3.8: active requests=0, bytes read=5069"
Apr 05 21:49:04 master containerd[1803]: time="2024-04-05T21:49:04.364949549Z" level=info msg="StopPodSandbox for \"d623d75553cec418467f72cbaa31eae6df269d7ba95fd7b6c66a529ac73d8f88\""

Question:
How to configure contrainerd to use pause image from the local container runtime?

2

UPDATE:
I just sorted out that by using image ID it worked.
Does anyone know why by specifying NAME it doesnt work?

3

I have noticed a warning related to this topic while creating my test cluster v1.29.3 using kubeadm and containerd runtime.

W0410 04:00:45.178045 5461 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.8" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.

Any recommendations about how to remove this warning ? Thanks

4

Hi,

Check /var/lib/kubelet/kubeadm-flags.env file
It contains pod-infra-container-image parameter. Try to configure it to use 3.9.

KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9"
5

If you want to keep using image from remote registry, simply replace the image version/tag in the config.toml (as root on every node including masters)

/etc/containerd/config.toml 
sandbox_image = "registry.k8s.io/pause:3.8" > sandbox_image = "registry.k8s.io/pause:3.9"

In my case, to use image from LOCAL CONTAINER RUNTIME the solution/workaround it to give it ID (taken from: crictl image).
What is still not clear to me is why cant i use name?

I did not check what @fox-md proposed (might be also an option AND might solve also my question - will post update once i have time to try it).