Zfs list got invoked every few seconds and makes cpu usage to 100%

General Discussions
1

One of the node in my Kubernetes cluster is using ZFS. I put all my docker images on one of the pool. I found Kubelet is invoking zfs list on that pool every few seconds which makes CPU usage goes to 100%.

Here is the command it’s invoking:

zfs list -r -t all -Hp -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/root/docker

I’m wondering where this behavior comes from? Is there any configuration to prevent it or make it less frequent?

Cluster information:

Kubernetes version:

-> kubectl version                                 
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"archive", BuildDate:"2022-04-29T06:39:1
6Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:43:11Z
", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}

Cloud being used: (put bare-metal if not on a public cloud)
bare-metal

Installation method:
kubeadm

Host OS:
Arch Linux

CNI and version:

CRI and version:

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

1 Like
2

I have the same issue on my server. It takes the majority of my CPU resources.

3

I have created a new PR to fix this issue: Add flags to kubelet to disable cadvisor metrics by wb14123 · Pull Request #110669 · kubernetes/kubernetes · GitHub