I installed microk8s (v1.29.14) on Ubuntu 22.04 LTS. I also installed nvidia driver and cuda compiler by:
sudo ubuntu-drivers autoinstall
sudo apt-get install nvidia-cuda-toolkit
After I enable gpu by:
microk8s enable gpu
I got CrashLookBackOff
administer@test1:~$ microk8s kubectl get pods -A
................
gpu-operator-resources nvidia-container-toolkit-daemonset-br79b 0/1 Init:CrashLoopBackOff 37 (106s ago) 165m
................
The log of the pod provides message:
administer@test1:~$ microk8s kubectl logs nvidia-container-toolkit-daemonset-br79b -n gpu-operator-resources
Defaulted container "nvidia-container-toolkit-ctr" out of: nvidia-container-toolkit-ctr, driver-validation (init)
Error from server (BadRequest): container "nvidia-container-toolkit-ctr" in pod "nvidia-container-toolkit-daemonset-br79b" is waiting to start: PodInitializing
Any idea?
Here are the output of nvcc and nvidia-smi:
administer@test1:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0
administer@test1:~$ nvidia-smi
Mon Mar 17 12:20:27 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.120 Driver Version: 550.120 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX 6000 Ada Gene... Off | 00000000:47:00.0 Off | Off |
| 30% 34C P8 16W / 300W | 101MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA RTX 6000 Ada Gene... Off | 00000000:5E:00.0 Off | Off |
| 30% 32C P8 10W / 300W | 12MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA RTX 6000 Ada Gene... Off | 00000000:75:00.0 Off | Off |
| 30% 30C P8 3W / 300W | 12MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA RTX 6000 Ada Gene... Off | 00000000:A3:00.0 Off | Off |
| 30% 39C P8 10W / 300W | 12MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 4714 G /usr/lib/xorg/Xorg 50MiB |
| 0 N/A N/A 4898 G /usr/bin/gnome-shell 39MiB |
| 1 N/A N/A 4714 G /usr/lib/xorg/Xorg 4MiB |
| 2 N/A N/A 4714 G /usr/lib/xorg/Xorg 4MiB |
| 3 N/A N/A 4714 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------------------+