Disk performance in pod 10x lower then host. Cant find solution

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: 1.23.6
Cloud being used: bare metal cloud VmWare
Installation method: manual
Host OS: Centos 7
CNI and version: containterd 1.6.4
CRI and version: calico 3.23.0

Hi! we are facing the problem that we cant resolve for 2 weeks for now. Disk performance in pod is 10x slower then on host

Host Machine

[alex@k8s-node-03 ~]$ dd if=/dev/zero of=/tmp/test1.img bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB) copied, 0,0952872 s, 1,1 GB/s

[alex@k8s-node-03 ~]$ dd if=/dev/urandom of=/tmp/test1.img bs=100M count=1
0+1 records in
0+1 records out
33554431 bytes (34 MB) copied, 0,223062 s, 150 MB/s

Inside of the pod

/app # dd if=/dev/zero of=/tmp/test1.img bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 1.00406 s, 104 MB/s
/app # dd if=/dev/urandom of=/tmp/test1.img bs=100M count=1
0+1 records in
0+1 records out
33554431 bytes (34 MB, 32 MiB) copied, 2.29815 s, 14.6 MB/s