I am using Jetson Xavier NX.
I rewrote config.toml as follows and restarted containerd.
However, when I execute describe to gpu pod, it seems to be warnning.
The pod is in penging state all the time.
Edge Device: Jetson Xavier NX(control-plane and worker node node1)
kubernetes plugin: k8s-device-plugin
Construction method: kuberspray
Runtime: containerd
■ sudo cat /etc/containerd/config.toml
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0
[grpc]
max_recv_message_size = 16777216
max_send_message_size = 16777216
[debug]
level = "info"
[metrics]
address = ""
grpc_histogram = false
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "nvidia"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
privileged_without_host_devices = false
runtime_engine = ""
runtime_root = ""
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
BinaryName = "/usr/bin/nvidia-container-runtime"
■ kubectl describe pod gpu-pod
Name: gpu-pod
Namespace: default
Priority: 0
Node: <none>
Labels: <none>
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
cuda-container:
Image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
Port: <none>
Host Port: <none>
Limits:
nvidia.com/gpu: 1
Requests:
nvidia.com/gpu: 1
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vmr27 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-vmr27:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
nvidia.com/gpu:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 30s default-scheduler 0/2 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }, 1 node(s) were unschedulable, 2 Insufficient nvidia.com/gpu. preemption: 0/2 nodes are available: 1 No preemption victims found for incoming pod, 1 Preemption is not helpful for scheduling.
■ cat gpuTest2.yaml
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
spec:
restartPolicy: Never
containers:
- name: cuda-container
image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
resources:
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
■ sudo nerdctl run --rm --gpus all nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
FATA[0008] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: src: /usr/lib/aarch64-linux-gnu/libcudnn.so.8, src_lnk: libcudnn.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn.so.8, dst_lnk: libcudnn.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libcudnn.so, src_lnk: /etc/alternatives/libcudnn_so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn.so, dst_lnk: /etc/alternatives/libcudnn_so
src: /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8, src_lnk: libcudnn_ops_infer.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8, dst_lnk: libcudnn_ops_infer.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so, src_lnk: /etc/alternatives/libcudnn_ops_infer_so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so, dst_lnk: /etc/alternatives/libcudnn_ops_infer_so
src: /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8, src_lnk: libcudnn_ops_train.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8, dst_lnk: libcudnn_ops_train.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so, src_lnk: /etc/alternatives/libcudnn_ops_train_so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so, dst_lnk: /etc/alternatives/libcudnn_ops_train_so
src: /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8, src_lnk: libcudnn_adv_infer.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8, dst_lnk: libcudnn_adv_infer.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so, src_lnk: /etc/alternatives/libcudnn_adv_infer_so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so, dst_lnk: /etc/alternatives/libcudnn_adv_infer_so
src: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8, src_lnk: libcudnn_cnn_infer.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8, dst_lnk: libcudnn_cnn_infer.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so, src_lnk: /etc/alternatives/libcudnn_cnn_infer_so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so, dst_lnk: /etc/alternatives/libcudnn_cnn_infer_so
src: /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8, src_lnk: libcudnn_adv_train.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8, dst_lnk: libcudnn_adv_train.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so, src_lnk: /etc/alternatives/libcudnn_adv_train_so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so, dst_lnk: /etc/alternatives/libcudnn_adv_train_so
src: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8, src_lnk: libcudnn_cnn_train.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8, dst_lnk: libcudnn_cnn_train.so.8.2.1
src: /usr/include/cudnn_adv_infer.h, src_lnk: /etc/alternatives/cudnn_adv_infer_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_adv_infer.h, dst_lnk: /etc/alternatives/cudnn_adv_infer_h
src: /usr/include/cudnn_adv_train.h, src_lnk: /etc/alternatives/cudnn_adv_train_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_adv_train.h, dst_lnk: /etc/alternatives/cudnn_adv_train_h
src: /usr/include/cudnn_backend.h, src_lnk: /etc/alternatives/cudnn_backend_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_backend.h, dst_lnk: /etc/alternatives/cudnn_backend_h
src: /usr/include/cudnn_cnn_infer.h, src_lnk: /etc/alternatives/cudnn_cnn_infer_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_cnn_infer.h, dst_lnk: /etc/alternatives/cudnn_cnn_infer_h
src: /usr/include/cudnn_cnn_train.h, src_lnk: /etc/alternatives/cudnn_cnn_train_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_cnn_train.h, dst_lnk: /etc/alternatives/cudnn_cnn_train_h
src: /usr/include/cudnn.h, src_lnk: /etc/alternatives/libcudnn, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn.h, dst_lnk: /etc/alternatives/libcudnn
src: /usr/include/cudnn_ops_infer.h, src_lnk: /etc/alternatives/cudnn_ops_infer_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_ops_infer.h, dst_lnk: /etc/alternatives/cudnn_ops_infer_h
src: /usr/include/cudnn_ops_train.h, src_lnk: /etc/alternatives/cudnn_ops_train_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_ops_train.h, dst_lnk: /etc/alternatives/cudnn_ops_train_h
src: /usr/include/cudnn_version.h, src_lnk: /etc/alternatives/cudnn_version_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_version.h, dst_lnk: /etc/alternatives/cudnn_version_h
src: /etc/alternatives/libcudnn, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_v8.h
src: /etc/alternatives/libcudnn_adv_infer_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_adv_infer_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8
src: /etc/alternatives/libcudnn_adv_train_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_adv_train_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8
src: /etc/alternatives/libcudnn_cnn_infer_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_cnn_infer_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8
src: /etc/alternatives/libcudnn_cnn_train_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_cnn_train_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8
src: /etc/alternatives/libcudnn_ops_infer_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_ops_infer_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8
src: /etc/alternatives/libcudnn_ops_train_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_ops_train_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8
src: /etc/alternatives/libcudnn_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn.so.8
src: /etc/alternatives/cudnn_adv_infer_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_adv_infer_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_adv_infer_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_adv_infer_v8.h
src: /etc/alternatives/cudnn_backend_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_backend_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_backend_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_backend_v8.h
src: /etc/alternatives/cudnn_cnn_train_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_cnn_train_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_cnn_train_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_cnn_train_v8.h
src: /etc/alternatives/cudnn_ops_train_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_ops_train_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_ops_train_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_ops_train_v8.h
src: /etc/alternatives/cudnn_adv_train_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_adv_train_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_adv_train_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_adv_train_v8.h
src: /etc/alternatives/cudnn_cnn_infer_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_cnn_infer_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_cnn_infer_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_cnn_infer_v8.h
src: /etc/alternatives/cudnn_ops_infer_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_ops_infer_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_ops_infer_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_ops_infer_v8.h
src: /etc/alternatives/cudnn_version_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_version_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_version_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_version_v8.h
src: /usr/lib/aarch64-linux-gnu/libcudnn_static.a, src_lnk: /etc/alternatives/libcudnn_stlib, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_static.a, dst_lnk: /etc/alternatives/libcudnn_stlib
src: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8, src_lnk: libnvinfer.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvinfer.so.8, dst_lnk: libnvinfer.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.8, src_lnk: libnvinfer_plugin.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.8, dst_lnk: libnvinfer_plugin.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvparsers.so.8, src_lnk: libnvparsers.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvparsers.so.8, dst_lnk: libnvparsers.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvonnxparser.so.8, src_lnk: libnvonnxparser.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvonnxparser.so.8, dst_lnk: libnvonnxparser.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvinfer.so, src_lnk: libnvinfer.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvinfer.so, dst_lnk: libnvinfer.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so, src_lnk: libnvinfer_plugin.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so, dst_lnk: libnvinfer_plugin.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvparsers.so, src_lnk: libnvparsers.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvparsers.so, dst_lnk: libnvparsers.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvonnxparser.so, src_lnk: libnvonnxparser.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvonnxparser.so, dst_lnk: libnvonnxparser.so.8
src: /etc/vulkan/icd.d/nvidia_icd.json, src_lnk: /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/vulkan/icd.d/nvidia_icd.json, dst_lnk: /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json
src: /usr/lib/aarch64-linux-gnu/libcuda.so, src_lnk: tegra/libcuda.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcuda.so, dst_lnk: tegra/libcuda.so
src: /usr/lib/aarch64-linux-gnu/libdrm_nvdc.so, src_lnk: tegra/libdrm.so.2, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libdrm_nvdc.so, dst_lnk: tegra/libdrm.so.2
src: /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999, src_lnk: tegra/libnvv4l2.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999, dst_lnk: tegra/libnvv4l2.so
src: /usr/lib/aarch64-linux-gnu/libv4lconvert.so.0.0.999999, src_lnk: tegra/libnvv4lconvert.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libv4lconvert.so.0.0.999999, dst_lnk: tegra/libnvv4lconvert.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so, src_lnk: ../../../tegra/libv4l2_nvargus.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so, dst_lnk: ../../../tegra/libv4l2_nvargus.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so, src_lnk: ../../../tegra/libv4l2_nvcuvidvideocodec.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so, dst_lnk: ../../../tegra/libv4l2_nvcuvidvideocodec.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvidconv.so, src_lnk: ../../../tegra/libv4l2_nvvidconv.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvidconv.so, dst_lnk: ../../../tegra/libv4l2_nvvidconv.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so, src_lnk: ../../../tegra/libv4l2_nvvideocodec.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so, dst_lnk: ../../../tegra/libv4l2_nvvideocodec.so
src: /usr/lib/aarch64-linux-gnu/libvulkan.so.1.2.141, src_lnk: tegra/libvulkan.so.1.2.141, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libvulkan.so.1.2.141, dst_lnk: tegra/libvulkan.so.1.2.141
src: /usr/lib/aarch64-linux-gnu/tegra/libcuda.so, src_lnk: libcuda.so.1.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/tegra/libcuda.so, dst_lnk: libcuda.so.1.1
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so, src_lnk: libnvbufsurface.so.1.0.0, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so, dst_lnk: libnvbufsurface.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so, src_lnk: libnvbufsurftransform.so.1.0.0, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so, dst_lnk: libnvbufsurftransform.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so, src_lnk: libnvbuf_utils.so.1.0.0, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so, dst_lnk: libnvbuf_utils.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so, src_lnk: libnvdsbufferpool.so.1.0.0, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so, dst_lnk: libnvdsbufferpool.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so, src_lnk: libnvid_mapper.so.1.0.0, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so, dst_lnk: libnvid_mapper.so.1.0.0
src: /usr/share/glvnd/egl_vendor.d/10_nvidia.json, src_lnk: ../../../lib/aarch64-linux-gnu/tegra-egl/nvidia.json, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/share/glvnd/egl_vendor.d/10_nvidia.json, dst_lnk: ../../../lib/aarch64-linux-gnu/tegra-egl/nvidia.json
src: /usr/lib/libvisionworks_sfm.so, src_lnk: libvisionworks_sfm.so.0.90, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/libvisionworks_sfm.so, dst_lnk: libvisionworks_sfm.so.0.90
src: /usr/lib/libvisionworks_sfm.so.0.90, src_lnk: libvisionworks_sfm.so.0.90.4, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/libvisionworks_sfm.so.0.90, dst_lnk: libvisionworks_sfm.so.0.90.4
src: /usr/lib/libvisionworks.so, src_lnk: libvisionworks.so.1.6, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/libvisionworks.so, dst_lnk: libvisionworks.so.1.6
src: /usr/lib/libvisionworks_tracking.so, src_lnk: libvisionworks_tracking.so.0.88, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/libvisionworks_tracking.so, dst_lnk: libvisionworks_tracking.so.0.88
src: /usr/lib/libvisionworks_tracking.so.0.88, src_lnk: libvisionworks_tracking.so.0.88.2, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/libvisionworks_tracking.so.0.88, dst_lnk: libvisionworks_tracking.so.0.88.2
, stderr: nvidia-container-cli: mount error: stat failed: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/local/cuda-10.2/compat/libcuda.so.440.118.02: no such file or directory: unknown