Kubernetes with containerd and nerdctl - K8s does not find locally built images

Kubernetes doesn’t see the locally built container images, despite the images being visible in the k8s.io namespace - tested both with nerdctl and crictl commands:

crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
docker.io/library/mytestimage             latest              f240b96f676e1       2.81MB
docker.io/library/nginx-test              1                   5e34a0811eefb       56.9MB
docker.io/library/nginx-test              latest              5e34a0811eefb       56.9MB
docker.io/nick/nginx                      latest              5e34a0811eefb       56.9MB
docker.io/registry-nick/nginx-test        latest              5e34a0811eefb       56.9MB

nerdctl --namespace=k8s.io images
REPOSITORY                                 TAG        IMAGE ID        CREATED         PLATFORM       SIZE         BLOB SIZE
mytestimage                                latest     dfb4f2a8f7c9    5 hours ago     linux/amd64    5.6 MiB      2.7 MiB
nginx-test                                 1          7f7e7eb6b3c2    2 days ago      linux/amd64    146.7 MiB    54.3 MiB
registry                                   latest     3f71055ad7c4    2 days ago      linux/amd64    23.7 MiB     8.8 MiB
registry-nick/nginx-test                   latest     7f7e7eb6b3c2    3 days ago      linux/amd64    146.7 MiB    54.3 MiB

The Pod YAML file:

Note: The commented out “#image:” lines are things I tried and all of them failed with the same [error](### The pod fails to start with ErrImageNeverPull error:). I tried accessing the image directly. Because that didn’t work, I also tried by running a private registry.

apiVersion: v1
kind: Pod
  name: nginx-test-pod
  - name: nginx-test-crp
    image: mytestimage
    #image: docker.io/library/nginx-test:1
    #image: registry-nick/nginx-test
    #image: localhost:5000/nginx-test
    imagePullPolicy: Never

I create the pod using the command kubectl apply -f pod-test.yaml

The pod fails to start with ErrImageNeverPull error:

kubectl get pod -o wide --all-namespaces
default            nginx-test-pod                             0/1     ErrImageNeverPull   0               124m    alma-kube2   <none>  

kubectl logs nginx-test-pod
Error from server (BadRequest): container "nginx-test-crp" in pod "nginx-test-pod" is waiting to start: ErrImageNeverPull

kubectl describe pod nginx-test-pod
  Type     Reason             Age                   From     Message
  ----     ------             ----                  ----     -------
  Warning  ErrImageNeverPull  58s (x630 over 136m)  kubelet  Container image "local-nick/bokluk" is not present with pull policy of Never

Obviously K8s can’t see the locally built image. How can I fix this problem?

Cluster information:

Kubernetes version: 1.26.1
Cloud being used: bare-metal
Installation method: manual
Host OS: AlmaLinux 9.1
CNI and version: Calico v3.24.5
CRI and version: v1, Containerd v1.6.16
Container Runtime: Containerd v1.6.16

[root@alma-kube ~]# kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
alma-kube    Ready    control-plane   62d   v1.26.1
alma-kube2   Ready    worker          61d   v1.26.0

I found out the problem was that the pod is scheduled on the worker node which doesn’t have the image. The fix was to copy the image to the worker node:
alma-kube ~]# nerdctl --namespace=k8s.io image save nginx-test:1 -o nginx-test-1.tar
alma-kube2 ~]# nerdctl --namespace=k8s.io load -i nginx-test-1.tar
Is there an automated way to shage image cache between all cluster nodes?

Thanks for that. This is one of those issues where the answer is obvious, once you know it. I spent days on it, before finding your post.