Hello all,
I’ve got a working Microk8s 1.29 installation on an Ubuntu Noble nightly from 8 days ago and am trying to export then import images for use in an offline / airgapped installation. This is for a customer project.
After doing microk8s images export-local > images-node03.tar
I decided to test the export by immediately doing a local import (single node, as you would need to do on initial installation). This fails saying a container image is missing. This image is consistently reported as missing when repeating this locally on each of my three nodes, and isn’t reported as being exported when doing the export.
Below is a screenshot of the issue.
As you can see, the export doesn’t mention this SHA, and doing a local ctr list
doesn’t reveal this SHA either.
As there is no option in ctr image import
to skip non found images I’m rather stuck on producing an airgapped installation.
There is no mention of this issue in either the airgapped or side loading pages on the Microk8s website. I’m presuming the export-local command erroneously culls some images that are in fact required, but I’m at a loss of how to work around this.
Can someone please advise a way forward?
Thanks in advance.
Regards,
Adam.
I think I’ve managed to answer my own question. For the benefits of anyone in future, the answer lies in the fact that each image supports multiple platforms but I’m installing on a single platform. This can be seen in this answer here:-
opened 08:52AM - 11 May 23 UTC
closed 11:04PM - 17 Jun 23 UTC
bug
Stale
### Before creating an issue, make sure you've checked the following:
- [X] Y… ou are running the latest released version of k0s
- [X] Make sure you've searched for existing issues, both open and closed
- [X] Make sure you've searched for PRs too, a fix might've been merged already
- [X] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.
### Platform
```shell
Probed on Red Hat Enterprise Linux 9.2 x86_64 and Red Hat Enterprise Linux 8.6 x86_64
```
### Version
v1.27.1+k0s.0
### Sysinfo
<details><summary>`k0s sysinfo`</summary>
<pre>
Machine ID: "14230b582d3f48455e84f0c2f8c38ad35d1e519fb226e8edf6a4d404d615cf96" (from machine) (pass)
Total memory: 963.4 MiB (warning: 1.0 GiB recommended)
Disk space available for /var/lib/k0s: 6.8 GiB (pass)
Operating system: Linux (pass)
Linux kernel release: 5.14.0-284.11.1.el9_2.x86_64 (pass)
Max. file descriptors per process: current: 65536 / max: 65536 (pass)
Executable in path: modprobe: /usr/sbin/modprobe (pass)
/proc file system: mounted (0x9fa0) (pass)
Control Groups: version 2 (pass)
cgroup controller "cpu": available (pass)
cgroup controller "cpuacct": available (via cpu in version 2) (pass)
cgroup controller "cpuset": available (pass)
cgroup controller "memory": available (pass)
cgroup controller "devices": available (assumed) (pass)
cgroup controller "freezer": available (assumed) (pass)
cgroup controller "pids": available (pass)
cgroup controller "hugetlb": available (pass)
cgroup controller "blkio": available (via io in version 2) (pass)
CONFIG_CGROUPS: Control Group support: built-in (pass)
CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
CONFIG_CPUSETS: Cpuset support: built-in (pass)
CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
CONFIG_NAMESPACES: Namespaces support: built-in (pass)
CONFIG_UTS_NS: UTS namespace: built-in (pass)
CONFIG_IPC_NS: IPC namespace: built-in (pass)
CONFIG_PID_NS: PID namespace: built-in (pass)
CONFIG_NET_NS: Network namespace: built-in (pass)
CONFIG_NET: Networking support: built-in (pass)
CONFIG_INET: TCP/IP networking: built-in (pass)
CONFIG_IPV6: The IPv6 protocol: built-in (pass)
CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: built-in (pass)
CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
CONFIG_NETFILTER_NETLINK: module (pass)
CONFIG_NF_NAT: module (pass)
CONFIG_IP_SET: IP set support: module (pass)
CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
CONFIG_IP_VS: IP virtual server support: module (pass)
CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
CONFIG_NF_DEFRAG_IPV4: module (pass)
CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
CONFIG_NF_DEFRAG_IPV6: module (pass)
CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
CONFIG_LLC: module (pass)
CONFIG_STP: module (pass)
CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: module (pass)
CONFIG_PROC_FS: /proc file system support: built-in (pass)
</pre>
</details>
### What happened?
k0s ctr image push command shows a content digest not found error with some containers images
### Steps to reproduce
1. Execute: **k0s ctr image pull registry.k8s.io/ingress-nginx/controller:v1.7.1**
2. Execute: **k0s ctr image tag registry.k8s.io/ingress-nginx/controller:v1.7.1 my-harbor-registry.mydomain/ingress-nginx/controller:v1.7.1**
3. Execute: **k0s ctr image push -k --user "user:password" my-harbor-registry.mydomain/ingress-nginx/controller:v1.7.1**
4. I get this error: **Error: content digest sha256:31953c360bca8311437640b9488fe7715384a34babc4055e0920920d2959bf09: not found**
### Expected behavior
k0s ctr image push command should work as with other container images
### Actual behavior
k0s ctr image push command fails with content digest not found error with some container images like:
registry.k8s.io/ingress-nginx/opentelemetry:v20230312-helm-chart-4.5.2-28-g66a760794
registry.k8s.io/ingress-nginx/controller:v1.7.1
k0s ctr image push works fine if I've pulled previously running k0s ctr image pull --all-platforms
### Screenshots and logs
[root@practicas ~]# journalctl -x -u k0scontroller|grep controller:v1.7.1
may 11 09:32:00 practicas.localdomain k0s[740]: time="2023-05-11 09:32:00" level=info msg="time=\"2023-05-11T09:32:00.101447529+02:00\" level=info msg=\"ImageCreate event name:\\\"registry.k8s.io/ingress-nginx/controller:v1.7.1\\\"\"" component=containerd stream=stderr
may 11 09:32:00 practicas.localdomain k0s[740]: time="2023-05-11 09:32:00" level=info msg="time=\"2023-05-11T09:32:00.131090513+02:00\" level=info msg=\"ImageUpdate event name:\\\"registry.k8s.io/ingress-nginx/controller:v1.7.1\\\" labels:{key:\\\"io.cri-containerd.image\\\" value:\\\"managed\\\"}\"" component=containerd stream=stderr
may 11 09:32:17 practicas.localdomain k0s[740]: time="2023-05-11 09:32:17" level=info msg="time=\"2023-05-11T09:32:17.536020919+02:00\" level=info msg=\"ImageCreate event name:\\\"registry.k8s.io/ingress-nginx/controller:v1.7.1@sha256:7244b95ea47bddcb8267c1e625fb163fc183ef55448855e3ac52a7b260a60407\\\"\"" component=containerd stream=stderr
may 11 09:32:17 practicas.localdomain k0s[740]: time="2023-05-11 09:32:17" level=info msg="time=\"2023-05-11T09:32:17.567217746+02:00\" level=info msg=\"ImageUpdate event name:\\\"registry.k8s.io/ingress-nginx/controller:v1.7.1@sha256:7244b95ea47bddcb8267c1e625fb163fc183ef55448855e3ac52a7b260a60407\\\" labels:{key:\\\"io.cri-containerd.image\\\" value:\\\"managed\\\"}\"" component=containerd stream=stderr
### Additional context
This command works fine with images like echoserver:
[root@practicas ~]# k0s ctr image pull registry.k8s.io/echoserver:1.10
registry.k8s.io/echoserver:1.10: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e0962580d8254d0b1ef35006d7e2319eb4870e63dc1f9573d2406c7c47d442d2: downloading |--------------------------------------| 0.0 B
/754.0 B
layer-sha256:bc391bffe5907b0eaa04e96fd638784f77d39f1feb7fbe438a1dae0af2675205: downloading |--------------------------------------| 0.0 B
registry.k8s.io/echoserver:1.10: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e0962580d8254d0b1ef35006d7e2319eb4870e63dc1f9573d2406c7c47d442d2: downloading |--------------------------------------| 0.0 B
/754.0 B
layer-sha256:bc391bffe5907b0eaa04e96fd638784f77d39f1feb7fbe438a1dae0af2675205: downloading |--------------------------------------| 0.0 B
/172.0 B
layer-sha256:2d3e25b9e93ad26878862abee5ed02683206f6f6d57e311cdd1dedf3662b61c8: downloading |--------------------------------------| 0.0 B
registry.k8s.io/echoserver:1.10: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e0962580d8254d0b1ef35006d7e2319eb4870e63dc1f9573d2406c7c47d442d2: downloading |++++++++++++++++++++++++++++++++++++++| 754.0 B
registry.k8s.io/echoserver:1.10: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e0962580d8254d0b1ef35006d7e2319eb4870e63dc1f9573d2406c7c47d442d2: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bc391bffe5907b0eaa04e96fd638784f77d39f1feb7fbe438a1dae0af2675205: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:2d3e25b9e93ad26878862abee5ed02683206f6f6d57e311cdd1dedf3662b61c8: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:365ec60129c5426b4cf160257c06f6ad062c709e0576c8b3d9a5dcc488f5252d: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:06679f57dba70a6875e4ae5843ba2483ecab6ec48182ca8720ddc5b1863bad52: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:28c6282d04f63710146ace6c7be14a40c7ee6a71a2f91316928469e4aafe0d92: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d5157969118932d522396fe278eb722551751c7aa7473e6d3f03e821a74ee8ec: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:76d46396145f805d716dcd1607832e6a1257aa17c0c2646a2a4916e47059dd54: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7fd34bf149707ca78b3bb90e4ba68fe9a013465e5d03179fb8d3a3b1cac8be27: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4b12f3ef8e65aaf1fd77201670deb98728a8925236d8f1f0473afa5abe9de119: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:b0e3c31807a2330c86f07d45a6d80923d947a8a66745a2fd68eb3994be879db6: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 5.1 s total: 44.1 M (8.6 MiB/s)
unpacking linux/amd64 sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229...
done: 1.932794771s
[root@practicas ~]# k0s ctr image tag registry.k8s.io/echoserver:1.10 my-harbor-registry.mydomain/ingress-nginx/echoserver:1.10
[root@practicas ~]# k0s ctr image push -k --user "user:password" my-harbor-registry.mydomain/ingress-nginx/echoserver:1.10
manifest-sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:365ec60129c5426b4cf160257c06f6ad062c709e0576c8b3d9a5dcc488f5252d: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 7.2 s total: 6.3 Ki (889.0 B/s)
**Ingress-nginx images are pushed fine if I pulled them with k0s ctr image pull --all-platforms**
So the correct import command needs to be:-
microk8s ctr images import --platform amd64 - < ./offline/containers/images-post-node03.tar
This works fine as expected.