Kubernetes with containerd and nerdctl - K8s does not find locally built images

General Discussions
1

Kubernetes doesn’t see the locally built container images, despite the images being visible in the k8s.io namespace - tested both with nerdctl and crictl commands:

crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
docker.io/library/mytestimage             latest              f240b96f676e1       2.81MB
docker.io/library/nginx-test              1                   5e34a0811eefb       56.9MB
docker.io/library/nginx-test              latest              5e34a0811eefb       56.9MB
docker.io/nick/nginx                      latest              5e34a0811eefb       56.9MB
docker.io/registry-nick/nginx-test        latest              5e34a0811eefb       56.9MB

nerdctl --namespace=k8s.io images
REPOSITORY                                 TAG        IMAGE ID        CREATED         PLATFORM       SIZE         BLOB SIZE
...
mytestimage                                latest     dfb4f2a8f7c9    5 hours ago     linux/amd64    5.6 MiB      2.7 MiB
nginx-test                                 1          7f7e7eb6b3c2    2 days ago      linux/amd64    146.7 MiB    54.3 MiB
registry                                   latest     3f71055ad7c4    2 days ago      linux/amd64    23.7 MiB     8.8 MiB
registry-nick/nginx-test                   latest     7f7e7eb6b3c2    3 days ago      linux/amd64    146.7 MiB    54.3 MiB

The Pod YAML file:

Note: The commented out “#image:” lines are things I tried and all of them failed with the same [error](### The pod fails to start with ErrImageNeverPull error:). I tried accessing the image directly. Because that didn’t work, I also tried by running a private registry.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-test-pod
spec:
  containers:
  - name: nginx-test-crp
    image: mytestimage
    #image: docker.io/library/nginx-test:1
    #image: registry-nick/nginx-test
    #image: localhost:5000/nginx-test
    imagePullPolicy: Never

I create the pod using the command kubectl apply -f pod-test.yaml

The pod fails to start with ErrImageNeverPull error:

kubectl get pod -o wide --all-namespaces
default            nginx-test-pod                             0/1     ErrImageNeverPull   0               124m   192.168.48.11    alma-kube2   <none>  

kubectl logs nginx-test-pod
Error from server (BadRequest): container "nginx-test-crp" in pod "nginx-test-pod" is waiting to start: ErrImageNeverPull

kubectl describe pod nginx-test-pod
Events:
  Type     Reason             Age                   From     Message
  ----     ------             ----                  ----     -------
  Warning  ErrImageNeverPull  58s (x630 over 136m)  kubelet  Container image "local-nick/bokluk" is not present with pull policy of Never

Obviously K8s can’t see the locally built image. How can I fix this problem?

Cluster information:

Kubernetes version: 1.26.1
Cloud being used: bare-metal
Installation method: manual
Host OS: AlmaLinux 9.1
CNI and version: Calico v3.24.5
CRI and version: v1, Containerd v1.6.16
Container Runtime: Containerd v1.6.16

[root@alma-kube ~]# kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
alma-kube    Ready    control-plane   62d   v1.26.1
alma-kube2   Ready    worker          61d   v1.26.0
2

I found out the problem was that the pod is scheduled on the worker node which doesn’t have the image. The fix was to copy the image to the worker node:
alma-kube ~]# nerdctl --namespace=k8s.io image save nginx-test:1 -o nginx-test-1.tar
alma-kube2 ~]# nerdctl --namespace=k8s.io load -i nginx-test-1.tar
Is there an automated way to shage image cache between all cluster nodes?

3

Thanks for that. This is one of those issues where the answer is obvious, once you know it. I spent days on it, before finding your post.