Windows VM Guest Bandwidth 80% that given by Ubuntu

[OmenIV]

Hello all,

I'd like your thoughts on the following please, using iperf and the command "iperf.exe -c <server_ip> -P 1 -i 1 -p 5001 -w 64K -f m -t 10" between two Windows Guest VMs on two different Vsphere ESXi 4.1 Hosts in the same cluster gives an average result of around 550 Mbits/sec, where as between two Ubuntu Guest VMs on the same two Vsphere ESXi 4.1 Hosts gives an average result of 870 Mbits/sec. Which is quite a diference! Also note this result is the same when using a crossover lead between the two ESXi Hosts as when testing over the LAN. These are two new ESXi Hosts with shared iSCSI storage and there are no other VM Guests on the hosts.

The answer we have had from VMware support to explain this difference in bandwith's is as follows, does this sound correct, I find it surprising that there is such a large jump?

To measure the performance with iPerf/jPerf and crossover cable between two ESX servers (for VM Network traffic), will give you the most accurate I/O results you can get.

The fact that you are seeing the same results with or without the physical network involved, points to an performance restriction on the guest OS level and the way the OS is managing its resources such as CPU, memory or NIC.

The reason you see different results between Linux and Windows VM is due to different handling of the resources between these two guests.

From VMware prospective the same virtual HW is presented to WIndows and to Linux VMs and is only up to the guest OS to manage the resources presented.

In my opinion the results you are seeing are pretty good, even if we take into account the additional virtualization layer.

[DSTAVERT]

Welcome to the communities

There have been many benchmark tests over the years comparing Linux and Windows network performance. BSD used to come out on top. I don't know how it fairs today. Do a Google search. I don't see anything out of line. Try your tests directly on the hardware.

Are you experiencing issues?

[OmenIV]

Thanks DSTAVERT, tests directly on the hardware with a crossover cable using Ubuntu gives results of around 1100 Mbit/Sec. The real issue is that we want our Windows servers to match or at least come near the same bandwidth seen by the Ubuntu VM guests.

We are not experiencing issues as this is a VMWare Proof of Concept, however, the issue is holding up the project since we run predominantly Windows 2008 R2 Servers.

[rickardnobel]

Which Windows Server version are you using? There was a major rewrite of the TCP code in Windows 2008 and it should be able to get a lot more than 550 Mbit/s.

Here is a screenshot of an ordinary Windows 7 (physical) getting a throughput of almost 980 Mbit/sec.

[OmenIV]

Thanks ricnob,  we are using Windows Server 2008 R2 and Windows 7 guest VMs to test this issue, both give similar results of 550 Mbits/Sec.

[rickardnobel]

Just to rule out some issues with the different options on iperf, have you tried to just copy some large iso file or something between the two Windows hosts using native SMB?

[f10]

Yes I agree with rincob lets try copying a large file between the two VMs.

Also have you tried VMXNET3 over E1000, I have seen VMXNET3 provide better performance. How many vCPUs are configured for the Windows 2008 VM, run esxtop on the ESX/ESXi host and check the %RDY and other CPU stats to check for any scheduling issues? What is the result if you run the same test with both the VMs on the same vSwitch eliminating any external route? Do we have the latest VMware tools installed?

I hoe one of these questions answer your question :smileywink:

Regards,

Arun

VCP3/4, HPCP, HP UX CSA http://kb.vmware.com/

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

[OmenIV]

Hi ricnob and f10,

Both Servers are using VMXNET3, I have copied a 2.49 GB iso file from one server to the other with an approx average 56 MB/Sec (using a crossover lead).

Both servers are 1vCPU and 2GB RAM.

Latest VMWare tools is installed.

I'll have to check esxtop next week, however, the VMWare network support guy did already look at statistics using this.

I'll answer some more of the questions you posted next week.

Many thanks for your help!

[f10]

Can you disable TCP Offload Engine on the Windows Guest and check the performance, refer to http://kb.vmware.com/kb/1006619 for more information, though this KB is linked for Workstation VMs but I have seen this work on vSphere VMs also.

All the best :smileyhappy:

[rickardnobel]

Have you tried to locate both Windows VMs on the same host, just to see if there is any difference in throughput when all in RAM?

Another thing to to could be to check if Windows "Receive Side Scaling" is enabled, that is the possibility for the OS to spread network load on several CPUs, which could be very useful in situations with high network IO.

Check with:

netsh int tcp show global

and look for "Receive-Side Scaling State", it should be enabled and if not use:

netsh int tcp set global rss=enabled

Then you have to enable it on the VMXNET3 adapter inside the guest too, go to Advanced Settings, find the RSS option and set it to Enabled. See if this would spread the CPU load inside the two Windows guests.

[OmenIV]

Hi all,

With the two servers on the same  VMWare Host via the same vSwitch we get excellent results of circa 1300 Mbits/Sec.

I will check Receive Side Scaling now and then let you know the result.

[OmenIV]

Hi f10,

We have already tried disabling TCP Offload Engine and unfortunately it had no effect.

Many thanks

[OmenIV]

Hi ricnob,

With regards to the Receive-Side Scaling State, this was already enabled on both servers using the netsh, but RSS was not enabled on the NICs. Therefore I enabled RSS on both, getting results of appox 660 Mbits/Sec. Both of these VMs only had one vCPU to begin with so I have added an addition vCPU to each. The results are now circa 600 Mbits/Sec. :smileycry:

[rickardnobel]

OmenIV wrote:

Hi ricnob,

With regards to the Receive-Side Scaling State, this was already enabled on both servers using the netsh, but RSS was not enabled on the NICs. Therefore I enabled RSS on both, getting results of appox 660 Mbits/Sec. Both of these VMs only had one vCPU to begin with so I have added an addition vCPU to each. The results are now circa 600 Mbits/Sec.:smileycry:

So the throughput first went up from 550 to 660 with the RSS enabling, and then down 50 Mbit with the second vCPU? :smileyhappy:

Very strange also that you get good performance when doing VM to VM inside the same host. For the Windows guest they see the same virtual hardware and should not know if a physical NIC is used or not. Do you see any latency when doing ordinary ping between the hosts?

Do you see any difference in host or guest CPU when testing internal to the host and external?

Perhaps you should try to enable more offload to the network card?

[OmenIV]

Thanks Ricnob,

It is certainly strange that adding a CPU caused a drop in the iperf results!

Ordinary Ping between the two guests while on the same host and vswitch and while on seperate hosts both give <1ms results.

The CPU useage is circa 60% with both guests internal to one host and circa 25% between two different hosts - interesting!

[rickardnobel]

By the way, are you using a standard vSwitch or Distributed vSwitch?

Do you see any dropped packets on the host when doing external host testing?

There is some buffer sizes on the VMXNET3 adapter that could be tried to increase.

[OmenIV]

BTW - we are using a standard VSwitch and we see no dropped packets on a ping. 

How do I enable more offload to the network card?

[OmenIV]

Just checked the config of the vmxnet3 adapter, every Property that has the word Offload in it is infact already enabled.

[rickardnobel]

OmenIV wrote:

BTW - we are using a standard VSwitch and we see no dropped packets on a ping. 

With dropped packet I meant on the vSwitch, while running the network stress tool. This should be visible in the ordinary Performance graphs in the vSphere Client, look for dropped in or outbound packets.

OmenIV wrote:

The CPU useage is circa 60% with both guests internal to one host and circa 25% between two different hosts - interesting!

Is that host CPU or guest CPU usage?

For the VMXNET3 adapter there is some options to increase some buffers, I think they are called RxRing and TxRing or similar. I shall verify, and see if there would be any gain in increasing them.

One other thing to try: disable the VMXNET adapters on both guests and add e1000 adapters instead and see if there is any difference!

[OmenIV]

Thanks ricnob,

Sorry for the miunderstanding. I have now added Transmit packets dropped and Recieve packets dropped for the performance graphs and re-run the stress test. Both hosts show no packets dropped.

With regard to the CPU usuage data I gave you, this was the guest CPU.

Using e1000 adapters on both guests gives bandwith results of approx 450 Mbits/Sec.

[rickardnobel]

Certainly very strange. The explanation you got from VMware support seems somewhat incorrect (that Windows uses the hardware different and that causes the slower transmission) because the guest does only see the virtual adapter and should use this device exactly the same no matter if the packets are passed through the physical VMnic or not. And since you get very good speed internal in the host it seems somewhat specific to the virtualization engine/vSwitch behaviour.

Some other things to might try:

Increase the Recieve and Transive buffers on the VMXnet3 adapter and see if there is any gain. See KB 1010071.

http://kb.vmware.com/kb/1010071

Something else to might think about could be enabling jumbo frames and see if this increases in your throughput. Here is a post I wrote earlier on throughput gains with jumbo frames where in theory you would get a few percent increase.

[OmenIV]

Hi ricnob,

Thanks for your continued support on this issue, we are getting there..

OK, so setting various settings on vmxnet3 for Rx ring #1 Size, Small Rx Buffers and Tx Ring size on both servers with the same values (using the maximum's where possible) didn't really have an impact. However, switching on "Jumbo Packet" to Jumbo 9000 did. I can now attain appox 740 Mbits/sec with the following set up:

1 x vCPU

both servers using vmxnet3

RSS enabled on each server

Jumbo Packet set to Jumbo 9000

Unfortunately we are under pressure here to progress the VMware project, so I may have to draw a line in the sand at this stage. However, we will be installing VMWare 5, so I will post if any differences is seen by this upgrade.

Many thanks again for your help!

Greg

[rickardnobel]

No problem, and at least you got a gain of +200 Mbit/s since start. :smileyhappy:

It would be interesting to know if ESXi 5 behaves different, so update when you have done some testing on the new platfrom!

[OmenIV]

Just to complete this discussion. It turns out that upgrading to VMWare 5 improved things further on our two test Windows 2008 servers giving 840 Mbits/Sec (which is approaching normality). I did a further experiment on the vmxnet3 NIC of turning Jumbo packets down to the standard setting and this significantly hit the performance pretty much halfing it. Therefore, I would recommend to anyone reading this that Jumbo Packets and RSS be enabled and that they run VMWare 5.