Asking for help? Comment out what you need so we can get more information to help you!
Cluster information:
Kubernetes version:
Client Version: v1.28.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
bare-metal used
Installation method: Rancher
Host OS:
NAME=“Ubuntu”
VERSION_ID=“22.04”
VERSION=“22.04.3 LTS (Jammy Jellyfish)”
I have test k8s env with 4 nodes, and periodically some nodes show notification kubelet has disk pressure:Resource is Ready (DiskPressure True 1.5 mins ago [KubeletHasDiskPressure] kubelet has disk pressure)
k get nodes
NAME STATUS ROLES AGE VERSION
l-k8s-t-01 Ready controlplane,etcd,worker 2y250d v1.25.6
l-k8s-t-02 Ready controlplane,etcd,worker 2y250d v1.25.6
l-k8s-t-03 Ready etcd,worker 2y250d v1.25.6
l-k8s-t-04 Ready worker 2y250d v1.25.6
k describe node l-k8s-t-04
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 770m (19%) 0 (0%)
memory 1059356672 (3%) 4G (11%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ImageGCFailed 47m kubelet wanted to free 5130594713 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "k8s/rancher/rke-tools:v0.1.87" (must force) - container 9f299a403f40 is using its referenced image caffe885434d, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "gitlab.energo.lv:k8s/rancher/hyperkube:v1.25.6-rancher4" (must force) - container 59e629984e37 is using its referenced image 2f531e92f214]
Warning ImageGCFailed 37m kubelet wanted to free 4827474329 bytes, but freed 1121830134 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "gitlab.energo.lv:k8s/rancher/rke-tools:v0.1.87" (must force) - container ed8d872fbe16 is using its referenced image caffe885434d, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "k8s/rancher/hyperkube:v1.25.6-rancher4" (must force) - container cd3a22a5b57b is using its referenced image 2f531e92f214]
Warning ImageGCFailed 32m kubelet wanted to free 5083625881 bytes, but freed 414002967 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "gitlab.energo.lv:k8s/rancher/rke-tools:v0.1.87" (must force) - container ed8d872fbe16 is using its referenced image caffe885434d, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "gitlab.energo.lv:k8s/rancher/hyperkube:v1.25.6-rancher4" (must force) - container 59e629984e37 is using its referenced image 2f531e92f214]
Warning EvictionThresholdMet 18m (x2498 over 35d) kubelet Attempting to reclaim ephemeral-storage
Normal NodeHasNoDiskPressure 13m (x427 over 35d) kubelet Node l-k8s-t-04 status is now: NodeHasNoDiskPressure
Warning ImageGCFailed 2m2s (x1402 over 7d9h) kubelet (combined from similar events): wanted to free 5178964377 bytes, but freed 1121830134 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "gitlab.energo.lv:/rancher/rke-tools:v0.1.87" (must force) - container ed8d872fbe16 is using its referenced image caffe885434d, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "gitlab.energo.lv:rancher/hyperkube:v1.25.6-rancher4" (must force) - container 59e629984e37 is using its referenced image 2f531e92f214]
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3,2G 3,7M 3,2G 1% /run
/dev/mapper/vgubuntu-root 100G 80G 16G 84% /
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/loop0 56M 56M 0 100% /snap/core18/1988
/dev/loop4 219M 219M 0 100% /snap/gnome-3-34-1804/66
/dev/loop1 56M 56M 0 100% /snap/core18/1997
/dev/loop3 32M 32M 0 100% /snap/snapd/11036
/dev/loop2 65M 65M 0 100% /snap/gtk-common-themes/1514
/dev/loop5 33M 33M 0 100% /snap/snapd/11588
/dev/loop6 52M 52M 0 100% /snap/snap-store/518
/dev/sda1 511M 4,0K 511M 1% /boot/efi
/dev/mapper/db_disk-db 49G 4,9G 42G 11% /mnt/l_montak8s_t_04_db_disk
tmpfs 3,2G 0 3,2G 0% /run/user/657455455
Where an how to delete files to resolve a problem with disk pressure