Virtual machines slow network performance

[Infraestructura ANV]

Hello everyone,

Im trying to figure out the reason why all of my virtual machines are not getting the speeds that I expect. I give you a brief summary about my infraestructure:

- 4 esxi Host cluster

- 120+ VMs running on the cluster

Hosts are interconnected with a 10Gbit Aruba switch. Iperf test between hosts shows 9.8+Gbps speeds, so speed between hosts seems normal. Hosts are running vsphere 7, some are in a newer patch because im in the middle of an update process.

Virtual machines, almost all of them running Windows server 2016 or newer. VMWare tools installed, and im using VMXNET3 network adapters.

Inside the guest OS (Windows server), if i go to the adapter options i can see it shows the correct speed (10Gbps). All the VMs are interconnected using a standard switch at the esxi virtual network level.

But when i run a IPERF test between two virtual machines running on the same host i get speeds between 2.5Gbps and 3.5Gbps (with luck).

If VMs are running on different hosts, iperf test show speeds slower than 2Gbps.

 

Looking on the forums i saw that some kind of network performance degradation is expected, but im not getting even the 50% of the speed i should have. The aruba switch ports show no error/drops/discarded and the utilization is under 5% on every port. Cables are cat6.

 

Could someone help me with this?

 

Thanks in advance.

[Davoud Teimouri]

Hi,

Did you same test between two Linux virtual machines?

Also I recommend that do test by QPERF as well.

If result of the test was good between Linux machines, so there is no problem about infrastructure and you should focus on Windows servers and do some tuning or applying best practices.

[lancelot5502]

i did that same on linux machines with no issues 9,8 Gbit/s.

running windows server 2022 on esx 7u3  2,9 Gbit/s

[Infraestructura ANV]

Yes, the test i did no so long ago (1 year maybe) showed the same results: Speed seems normal on linux virtual machines, but windows machines show less than 40% of the speed that they should have. Tried every combination of nic settings (Jumbo, Offloading, etc) with no luck.

[VMHero4Ever]

Do you have any AV-software enabled? 

Please also test with two windows VMs and 16+ GB memory.

[lancelot5502]

Did you get any solution for this issue ?

[D_G_Tal]

In windows VM used robocopy to used multithreaded.

robocopy C:\source\folder\path\ D:\destination\folder\path\ /S /E /Z /ZB /R:5 /W:5 /TBD /NP /V /MT:32

 

[cism1385]

 Hi InfraANV, i have the same problem with windows,  cloud you resolve this problem?

regards

[Infraestructura ANV]

Hello,

 

No, i did not find a solution to this problem yet. I forgot about it because it was taking too much of my time and i needed to focus on other stuff that had to be done.

Its seems that most of the time, iperf3 or Windows servers in general have problems reaching near link speeds. That problem seem to not be present if you are using linux boxes.