ESXI 8 & Gen4 NVME - Slow

[Michael Osmolski]

Hello

I have an AsrockRack SIENAD8-2L2T motherboard and Sk Hynix P41 2TB Gen4 nvme in m.2 slot 1.

Installed ESXI 8.0 u3d and decided to do some disk benchmarking.

~] time dd if=/dev/zero of=/vmfs/volumes/67d462e8-929ecfdb-0f12-9c6b004f248f/testfile bs=10M count=30240 conv=fsync

esxtop MBWRTN/s tops out at 1638 which equates to 1.6GB/s which is way under the 5-6GB/s top write speed of the drive.

I know the VMFS layer and vmware native nvme driver hinders the performance of the drive but is it really that bad?

The queue depth is 1023 and everything else checks out OK for the drive so not sure if I am doing anything wrong or this is expected peformance on consumer gen4 drives.

Please advise.

Thanks!

[Andrea Consalvi]

Hi, yes, 1.6 GB/s is definitely lower than what you'd expect from a Gen4 NVMe, but ESXi adds some overhead, especially with VMFS. Have you tried writing directly to the raw device instead of through VMFS? Something like:

time dd if=/dev/zero of=/dev/nvme0n1 bs=10M count=30240 conv=fsync 

That would at least show if VMFS is the bottleneck. Also, which NVMe driver are you using? You can check with:

esxcli storage core adapter list 

If you're using VMware’s native nvme_pcie, it tends to prioritize latency over raw speed, so that could be part of it.

[Duncan Epping]

It is using the HPP? As that could make a difference, more details here: https://4sysops.com/archives/vmware-vsphere-nvme-high-performance-plug-in-hpp/