Microk8s kubernetes cannot pull local image

General Discussions microk8s
1

Hello all,
I have a remote node with private registry (IP: 10.34.104.50:32000) that only builds and store images, deployment is made on another node (IP: 10.34.104.5) that pulls the image from remote private registry.

if I remove the image from the remote private registry and attempt to run a pod with the same image on a node where it HAS been pulled previously, Kubernetes will try to pull from remote registry even if it has image locally, and the pull will fail,

    imagePullPolicy: IfNotPresent is set

Tried to change imagePullPolicy to Always and Never, it still ignores local image and tries to pull from remote

Pulling image "10.34.104.50:32000/service-name:stage-66"

Failed to pull image "10.34.104.50:32000/service-name:stage-66": rpc error: code = NotFound desc = failed to pull and unpack image "10.34.104.50:32000/service-name:stage-66": failed to resolve reference "10.34.104.50:32000/service-name:stage-66": 10.34.104.50:32000/service-name:stage-66: not found
 Error: ErrImagePull
Back-off pulling image "10.34.104.50:32000/service-name:stage-66"
Error: ImagePullBackOff

but if i execute on local node -

microk8s ctr i ls -q |grep service-name
10.34.104.50:32000/service-name:stage-66
10.34.104.50:32000/service-name@sha256:27bbdae5d6cc49d18f8ffed260faa21a88e15a2204a27e834a86568710ac8dbd

it exists

here’s some config to pull from remote private registry

/var/snap/microk8s/current/args/certs.d/10.34.104.50:32000/hosts.toml 

server = "http://10.34.104.50:32000"

[host."http://10.34.104.50:32000"]
capabilities = ["pull", "resolve"]

/var/snap/microk8s/current/args/certs.d/localhost:32000/hosts.toml

server = "http://localhost:32000"

[host."http://localhost:32000"]
capabilities = ["pull", "resolve"]
2

Add skip_verify = true to your containerd hosts.toml, like this:

server = “http://10.34.104.50:32000”

[host.“http://10.34.104.50:32000”]
capabilities = [“pull”, “resolve”]
skip_verify = true

and restart containerd

3

RESOLVED. The problem was in AlwaysPullImages in admission control.
I had configured cis hardening and forgot about that setting. So that’s the reason why imagePullPolicy: IfNotPresent was ignored. After disabling this setting problem was resolved.
More info here:

Securing Kubernetes: A Deep Dive into AlwaysPullImages Admission Control – The Linux Notes