I'm on a VMWare backed VM with a disk connected using virtual SCSI connector (PVSCSI). The storage is a Dell emc powervault me4024 connected to the ESXi hosts (2 node cluster) with a direct FC connection.
I'm observing a prblem I never saw on any other system. A simple signle process application isn't able to read a file at the full speed of the underlying sotrage device.
I managed to track down the issue to the difference between 2 `fio` commands:
With:
fio --name=read_throughput --directory=. --numjobs=1 --size=10G --time_based --runtime=60s --ramp_time=2s --direct=0 --verify=0 --bs=64K --iodepth=64 --rw=read --group_reporting=1
I'm getting only a very low read throughput:
READ: bw=529MiB/s (554MB/s), 529MiB/s-529MiB/s (554MB/s-554MB/s), io=30.0GiB (33.3GB), run=60001-60001msec
But with multiple processes I'm able to reach nearly the maximum speed of the underlying storage system as expected:
fio --name=read_throughput --directory=. --numjobs=8 --size=10G --time_based --runtime=60s --ramp_time=2s --direct=0 --verify=0 --bs=64K --iodepth=64 --rw=read --g
roup_reporting=1
READ: bw=3238MiB/s (3395MB/s), 3238MiB/s-3238MiB/s (3395MB/s-3395MB/s), io=190GiB (204GB), run=60001-60001msec
I checked the gest OS settings that in my opinion could have an impact on such behavior but I don't see anyting problematic:
# cat /sys/block/sda/queue/scheduler
[mq-deadline] none
# cat /sys/block/sda/device/queue_depth
254
The only one that could be problematic was:
# cat /sys/block/sda/queue/rotational
1
I forced it to 0 but the results of the single thread fio test were the same.
I checked the RAM and CPU during the single threaded fio test and the VM is definitely not nor RAM nor CPU bound.
As a reference this is what I'm getting on my personal pretty average laptop with a standard m2 SSD:
fio --name=read_throughput --directory=. --numjobs=1 --size=1G --time_based --runtime=60s --ramp_time=2s --direct=0 --verify=0 --bs=64K --iodepth=64 --rw=read --group_reporting=1
READ: bw=2003MiB/s (2101MB/s), 2003MiB/s-2003MiB/s (2101MB/s-2101MB/s), io=117GiB (126GB), run=60001-60001msec
fio --name=read_throughput --directory=. --numjobs=8 --size=1G --time_based --runtime=60s --ramp_time=2s --direct=0 --verify=0 --bs=64K --iodepth=64 --rw=read --group_reporting=1
READ: bw=2277MiB/s (2387MB/s), 2277MiB/s-2277MiB/s (2387MB/s-2387MB/s), io=133GiB (143GB), run=60038-60038msec
Any ideas on how to move forward?