vSAN Network Performance Test

[orddie]

two node cluster running 7.0.3D

hosts are directly connected to one another via Mellanox MCX314A-BCCT 40Gb Ethernet.

 

Going into speed test Proactive Network Performance Test via the monitor section Im seeing 10GBE connections between the hosts. I was expecting to see closer to 40.

 

pictures attached

 

[TheBobkin]

 Are you positive the 40Gbps links are connected to the vSwitch and hence vmk enabled for vSAN?

 

Are you sure you don't have some multihoming configuration issue whereby the vSAN traffic is actually going out the network connected to the 10Gbps via another vmk? https://kb.vmware.com/s/article/2010877

Are you sure these are using 40Gbps-capable cables connected to a 40Gbps switch-port that is connected end-to-end to the other vSAN nodes? (Don't ever assume from your network documentation, test this methodically from hop to hop)

 

I vaguely recall some Mellanox driver/firmware versions/issues that result in these defaulting to 10Gbps though I can't recall the specifics off the top of head (as not really a vSAN-specific issue) - have a look on the Mellanox forums/release notes in relation to this (and/or validate which driver+firmware are on and are reasonably latest and supported pair per ESXi HCL).

 

If all of the above checks out then I would advise you figure out exactly where+what is limiting this to 10Gbps and if cannot figure this out with your team then open a case with VMware GS networking team to investigate further.

[orddie]

Thank you for the response and sorry for the delayed response here.

 

In talking with others, it looks like I'm getting the projected speed as MB/s does come out to faster than the 40 GBE ability.

In the attached screen shot, we are getting 9,474.35 MB/s, which is about 75 GBIT/s.  This assumes the test is adding both sending and receive numbers together.

 

I  have also attached screen shots showing the configuration of host 6. Host 6 and 5 are directly connected to one another. No switch.

[TheBobkin]

 , No that test is showing output in Mbps not MBps.

 

I was having a further research on this test last week and seemingly on previous engineering engagements it was informed that this test was capped in some way at 10Mbps - now, that was quite a while ago so I am unaware of whether this test was later increased to be  capable of working fully with NICs >10Gbps.

 

I'll try aim to have another look but in the mean time I would advise checking the points I mentioned.

[orddie]

Thanks for calling that out! I looked back and confirmed it does show Mb/s and NOT MB/s

 

would be very interested in know what you find out or anyone in the community shows a different speed.

[orddie]

did you get a chance to review the speed results with peers?

 

Thank you.

[TheBobkin]

 The iPerf process used for Proactive Network Test in 6.x/7.x looks to be single threaded and only check up to max of 10Gbps so this looks to be expected. If you want to test this for network >10Gbps then you can manually start multiple concurrent iPerf threads, example of how to do this here https://www.dell.com/support/kbdoc/en-ie/000187640/dell-emc-vxrail-vsan-hosts-network-performance-test-does-not-exceed-10-000mb-s-bandwidth-on-25gb-or-higher-network-infrastructure

 

This looks to have been updated in vSAN 8.x so that it can evaluate 25Gbps networks via the proactive test.

[orddie]

Great.  Thank you!