Cluster information:
Kubernetes version: master 1.19.10, worker 1.18.8
Cloud being used: Baremetal
Installation method: manual
Host OS: Master: Ubuntu 18.04 Worker: 18.04
CNI and version: Flannel 0.11.0-amd64
CRI and version: Docker 19.03.13
Looks like suddenly garbage collection does not work anymore. It used to work just fine earlier. When checking kubelet logs there is a lot of the following
kubelet[3039]: E0204 00:32:45.969154 3039 kuberuntime_image.go:122] Remove image “sha256:9d9b90970da9b4349eab8b4068e2702b514b6c6e647e8f7b76a033daf0fe109f” failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 9d9b90970da9 (must be forced) - image is being used by stopped container e25863e58df1
kubelet[3039]: I0204 00:32:45.969193 3039 image_gc_manager.go:375] [imageGCManager]: Removing image “sha256:d3aa03c9eeeb68d370a81b949a2a9445af88d06498acd449e3da88e2e6a971e2” to free 3194885757 bytes
kubelet[3039]: E0204 00:32:45.979779 3039 remote_image.go:135] RemoveImage “sha256:d3aa03c9eeeb68d370a81b949a2a9445af88d06498acd449e3da88e2e6a971e2” from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete d3aa03c9eeeb (must be forced) - image is being used by stopped container e5885d7b367f
kubelet[3039]: E0204 00:32:45.979816 3039 kuberuntime_image.go:122] Remove image “sha256:d3aa03c9eeeb68d370a81b949a2a9445af88d06498acd449e3da88e2e6a971e2” failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete d3aa03c9eeeb (must be forced) - image is being used by stopped container e5885d7b367f
kubelet[3039]: I0204 00:32:45.979839 3039 image_gc_manager.go:375] [imageGCManager]: Removing image “sha256:6721d0f0cbe0b307d54ba0caaca6e2f67ddf804d3ae09c892932fc2a9057d01b” to free 6655781391 bytes
kubelet[3039]: E0204 00:32:45.983743 3039 remote_image.go:135] RemoveImage “sha256:6721d0f0cbe0b307d54ba0caaca6e2f67ddf804d3ae09c892932fc2a9057d01b” from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 6721d0f0cbe0 (must be forced) - image is being used by stopped container c668ea5d104f
However when checking from docker, stopped containers with such id’s do not exist, so it’s somehow out of sync. This is causing the disk to fill up quickly. Does somebody have any tips how to fix this?