Log spam: "Cleaned up orphaned pod volumes dir" and "Path does not exist"

General Discussions
1

Cluster information:

Kubernetes version: v1.25.5+k3s2
Cloud being used: bare-metal
Installation method: k3s install script
Host OS: Ubuntu 20 on some VMs, Ubuntu 22 on others.
CNI and version: ??? I don’t know how to get this.
CRI and version: containerd://1.6.14-k3s1

The problem

/var/log/syslog is being spammed with the following lines repeatedly. The log file is growing very fast, and the disk is getting full.

Jan 18 13:32:40 pixie k3s[1219]: I0118 13:32:40.287414    1219 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/7cc052d1-71b7-4754-8b97-29b3a20bceea/volumes"
Jan 18 13:32:40 pixie k3s[1219]: I0118 13:32:40.287490    1219 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/7cc052d1-71b7-4754-8b97-29b3a20bceea/volumes"
Jan 18 13:32:40 pixie k3s[1219]: I0118 13:32:40.287799    1219 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=7cc052d1-71b7-4754-8b97-29b3a20bceea path="/var/lib/kubelet/pods/7cc052d1-71b7-4754-8b97-29b3a20bceea/volumes"
Jan 18 13:32:40 pixie k3s[1219]: I0118 13:32:40.288286    1219 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/91ac99d0-0a39-41d2-9fa4-f2212117c148/volumes"
Jan 18 13:32:40 pixie k3s[1219]: I0118 13:32:40.288331    1219 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/91ac99d0-0a39-41d2-9fa4-f2212117c148/volumes"
Jan 18 13:32:40 pixie k3s[1219]: I0118 13:32:40.288629    1219 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=91ac99d0-0a39-41d2-9fa4-f2212117c148 path="/var/lib/kubelet/pods/91ac99d0-0a39-41d2-9fa4-f2212117c148/volumes"

What I have tried

  • systemctl restart k3s-agent — Makes no difference.
  • reboot — Makes no difference.
  • systemctl stop k3s-agent && mkdir -p ... && mkdir -p ... && systemctl start k3s-agent — I though that creating the missing folders would bypass the error, but it makes no difference. The folders are removed shortly after starting k3s-agent, and then the same log spam cycle continues.

Issues I have read

None of these seem like the exact same problem as mine, but I might have misunderstood much of what is being said here:

What may have caused this

I may have inadvertently deleted a few folders from /var/lib/kubelet/pods/. Those pods were not running at the time of deletion. I thought they were left-over folders from some bug, perhaps.

Please help

Let me know if I need to post more info. :slight_smile: