VSAN 6.5 write speed really slow on NVMe disks

[sebtesta007]

Hi

We are experiencing difficulties to write data fast from a 2 nodes cluster vSAN 6.5 (with external witness), executing only one virtual machine.

Write rate is kinda limited to 600-800ko/s where the vSAN is mounted on a full flash NVMe PCI4 disks with a theorical write rate of 3500 Mo/s

The virtual machine is running CentOS last stable.

We already asked IBM product level 3 support and found nothing with them, regarding parameters of their software Aspera.

But I found this value in performance meters and it keeps my attention.

Does anyone know something about such a trouble ?

Best regards

Thomas

[TheBobkin]

, Is there a reason you are on vSAN 6.5? You will get a better experience in general on 7.0 U3 - if you are bound to 6.x for whatever reason then at least update to latest 6.7 U3.

 

Have you checked the overall cluster performance?

Start with VM/frontend:

Cluster > Monitor > vSAN > Performance

If frontend is high (e.g. > 10ms latency) then check Backend performance graphs - if you have just high frontend latency then it is likely a network issue, if you have high backend then it is likely a storage or other vSAN issue.

[sebtesta007]

Thanks for the reply.

The performance drop was relying on software platform which did not managed correctly the link to the storage and we didn't find a solution, even with our partner L3 support.

Latency was less than 0.3ms in front but average >120ms in the write task to NVMe...

Seems that the software deploy a docker container for each instance and for an unexplained reason performance drop dramaticaly down when we reach 20 simultaneous instances... We needed more than 1 hundred concurrent instances...

The product was IBM Aspera (with no rate limit licence). For the next production event, we will have to setup a physical machine as this software is not running with VMware, as they said.