Slow VMXNET3 performance on 10gig connection

[billybobSSD]

I have an ESXi server with an Intel X520-DA2 10 gig adapter in it. It has an iSCSI data store connected over one port and VM traffic over the other port. The iSCSI speed just couldn't be better but the problem seems to be that none of my VM's will do over 300 megabits/sec. Their all using VMXNET3 adapters. I have gone so far as to hook the second 10gig port directly up to another standalone windows server to eliminate the network/switch as a bottleneck via an SFP+ cable, but am still limited to the ~300 megabit ceiling. Any clues to what could be causing this? Thanks in advance!

[billybobSSD]

So, nobody else has had this issue?

[infra201110141]

I do have the exact same issue under RHEL6.   I don't have that problem with RHEL5.   No clue so far.

Did you engaged support on the issue ?

[billybobSSD]

I'm assuming that since I am using the free version I cannot open up a support case.

[billybobSSD]

VMware ESXi  5.0 Update 1 does not fix this problem.....

[Heimatplan]

Did you use NetPerf to analyse? You did a crossover connection to another server to test the speed? How about QOS?

[billybobSSD]

QOS uninstalled. Directly connected SFP+ to another server, still hitting a 300 megabit ceiling. Have not used netperf, just using resource mon to look at the speed.

[peterdabr]

Hi,

In your post there's no mention what type of test your runningt. Since you mentioned Resource Mon, I suspect you're simply copying large files to measure network throughput...if that's the case then your disk subsystem could be the culprit...

Iperf is a good tool to measure networking throughput between two servers without doing any i/o.

Peter D

[billybobSSD]

I am copying files. All of the copying is between raid arrays or SSD's. The VMXNET3 adapter is creating the bottleneck.

[Heimatplan]

Please use Iperf like mentioned. Single SSDs are not fast enough. RAID10 with 7k4 disks tend to be around 300MB/s as well. Another bottleneck: Having RAID+NIC on the same PCI-Channel.

Ingo

[billybobSSD]

"Please use Iperf like mentioned.  Single SSDs are not fast enough. RAID10 with 7k4 disks tend to be around  300MB/s as well. Another bottleneck: Having RAID+NIC on the same  PCI-Channel.

Ingo"

Ingo, an Intel 510 series SSD can certainly swallow more than 2 gigabits per second. The VM is only putting out 300 megabits. I'll run Iperf and let you know what I see. Thanks!

[Heimatplan]

You found out yourself why you fail with your tests, a single SSD cannot give a sustained output to occupy a 10G connect. Its all about writing. 300MB/s is the best you can get when the SSD is empty. So you are just measuring the writespeed of your SSD. I used SSDs and EFDs for testing but the bottleneck was always the RAID-Controller or the SAN-Connection.

Ingo

[billybobSSD]

Here are the iperf results between two VM's.

------------------------------------------------------------
Client connecting to 192.168.1.12, UDP port 5123
Sending 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.13 port 62071 connected with 192.168.1.12 port 5123
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec   305 MBytes   512 Mbits/sec
[  3]  5.0-10.0 sec   313 MBytes   525 Mbits/sec
[  3] 10.0-15.0 sec   311 MBytes   523 Mbits/sec
[  3] 15.0-20.0 sec   312 MBytes   524 Mbits/sec
[  3] 20.0-25.0 sec   310 MBytes   521 Mbits/sec
[  3] 25.0-30.0 sec   312 MBytes   524 Mbits/sec
[  3]  0.0-30.0 sec  1.82 GBytes   521 Mbits/sec
[  3] Sent 1329616 datagrams
[  3] Server Report:
[  3]  0.0-30.0 sec  1.79 GBytes   512 Mbits/sec   0.000 ms 22806/1329615 (1.7%)

[  3]  0.0-30.0 sec  1 datagrams received out-of-order

Why would the bandwidth between two VM's internally be limited to 500 megabits?

And here it is connected to a physical box, still limited to 500 megabits:

------------------------------------------------------------
Client connecting to 192.168.1.121, UDP port 5123
Sending 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.13 port 62072 connected with 192.168.1.121 port 5123
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec   302 MBytes   507 Mbits/sec
[  3]  5.0-10.0 sec   313 MBytes   525 Mbits/sec
[  3] 10.0-15.0 sec   311 MBytes   522 Mbits/sec
[  3] 15.0-20.0 sec   313 MBytes   525 Mbits/sec
[  3] 20.0-25.0 sec   311 MBytes   521 Mbits/sec
[  3] 25.0-30.0 sec   313 MBytes   524 Mbits/sec
[  3]  0.0-30.0 sec  1.82 GBytes   521 Mbits/sec
[  3] Sent 1328524 datagrams
[  3] Server Report:
[  3]  0.0-30.0 sec  1.81 GBytes   518 Mbits/sec   0.751 ms 5971/1328523 (0.45%)

[  3]  0.0-30.0 sec  1 datagrams received out-of-order

Any ideas?

[peterdabr]

Guys,

Let's standardize on megabits (Mb) or megabytes (MB) in this posting ;-)

The single SSD performance ranges from 100MB/s to 500MB/s; quickly googling for Intel 510 series SSD tells me read speeds up to 500 megabytes per second (MB/s) and sequential writes up to 315 MB/s. If you intial post talks about megabytes then you were effectively maxing out what SSD can provide.

Now, with iperf getting ~ 500Mb/s is somewhat dissapointing and you should determine if the problem lies within OS or further down the networking chain. With that being said, can you test it with two VMs being on the same host? When you do that, the iperf traffic should not traverse pNIC, but rather stay within the same vswitch two test VMs are connected to and, as such, you should be getting much better results. Can you confirm that?

Also, have you ever tested what you'd get running iperf between two xover'd physical boxes with the same pNICs? My suspicious is that maybe iperf does not work well with 10GB/s ethernet and running such test  could possibly verify that.

Also, although I'm sure you had already verified that, can you confirm that there's no ingress/outgress traffic shaping policies or network i/o control configured?

Peter D.

[billybobSSD]

Thanks Peter, I have been talking only megabits and gigabits since the first post. Ingo has them mixed up.

The first set of results above are VM traffic on the same host, on the same vswitch. I am wondering if maybe there is an artificial bottleneck in ESXi 5?

I have no traffic shaping policies set up, nor network i/o control (I have no vnetwork distributed switch)

This is really frustrating me....

[Josh26]

What OS are you testing under?

If Linux, please review this:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1027511

Although the document states it "does not affect ESXi5" it certainly does in our environment. Try this on both servers as per the guide, and rerun iperf:

ethtool -K eth0 lro off

[billybobSSD]

The operating systems are Windows7 and Server2008 R2.

[peterdabr]

I feel your pain.

Again, can you test between two physical servers having the same network card as in ESX host xover'd and test with iperf. This way you'll be able to tell whether iperf can produce adequate results for 10Gb/s network and whether the issue is only when ESXi is in the equation. If you can get throughput significantly higher than 500Mb/s then compare NIC properties between VM's OS and the physical server. For instance, compare Receive Side Scaling setting; I believe VMXNET3 does not enable it on vNIC by default so compare it with the physical. Other properites to compare: TCP offloading, Receive/Transmit Buffers.

Also: 1) Do you have jumbo frames enabled with your testing? 2) Do you have VMware Tools  on test VM matching ESXi5 host (I'm not sure if vmxnet driver version in 4.x Tools is the same as for 5.0...)

Peter D.

[billybobSSD]

I loaded Win7 on the ESXi hardware and tested with a 2008r2 server and got spectacular speeds. I am absolutely sure the hardware is not a problem. Since I am seeing a traffic bottleneck between VM's on the same host, on the same vswitch, I'm thinking its an ESXi problem. The ESXi host has stock network settings as do the VM's. I will do some experimentation with frame sizes and TCP offloading. I have VMware tools 8.65 on both machines.

[billybobSSD]

I have changed quite a few of the default VMXNET3 adapter settings to see if it would make any difference. It has not. Before anybody gets smart alecky, I realize changing of any of the TCP settings wont affect the Iperf UDP performance.

setting	Value
enable adaptive RX ring sizing	disable
Interrupt moderation	disable
IPV4 checksum offload	disable
IPV4 Giant TSO offload	no change
Large RX buffers	no change
Max TX queues	no change
Offload IP options	disable
Offload TCP options	disable
Recieve throtttle	no change
RSS	disable
Rx Ring #1 size	no change
Rx ring #2 size	no change
TCP checksum offload (IPV4)	disable
Tx ring size	no change
UDP checksum offload (IPV4)	disable
Small Rx buffers	no change

[cdc1]

Indeed, if you are seeing the issue between vmxnet3 VMs in the same port group on the same vswitch, then it certainly appears to be an issue with ESXi.

Curious to see how this one gets resolved. I'll be keeping an eye on this one.

Good luck!

[jhirsh]

I recently came across a similar problem. I kept only being able to hit around 450 Mbps with iperf between two Linux guests (using VMXNET3) operating on two different blades in the same chassis. Enabling jumbo frames in vSphere and upping the MTU on the guests to 9000 allowed the same test to reach around 8 Gbps.

It may not be applicable in your situation, but if you're only able to squeeze out around 500 Mbps, it probably can't hurt to give it a shot.

Cheers

-Joshua

[billybobSSD]

The vswitch is at 9000 MTU and the clients are as well. Thanks for trying!  :smileyhappy:

[rklarr]

B-Bob, similar issue here. We created a separate Vswitch with a 1 gig physical connection, and it's flying at expected rates. Just the 10gb crawls. We have a case open.

Oh, and some of the trolls around here will probably be nicer to you if you change your user name to "UberCosmoEurophile". In the grip of their prejudices, it creates cognitive dissonance for them to imagine anyone named BillyBob using kitchen utensils, let alone technology. They lash out.

[billybobSSD]

I guess I will change my avatar to Billy Bob Thornton holding an SSD so they will "get it".

Do you have a problem with two VM's on the same vswitch being rate limited as well?

[PCookman]



I am currently out of the Office and will be back on Tuesday . If you need me I can be contacted on 07957 168744.

In my absence please contact [e-mail] or [e-mail].

Regards, Paul..

Paul Cookman

Technical Account Manager

[phone]

[phone]

[e-mail]

http://www.selection.co.uk

[cid:imagee44f5a.JPG@0d50c6ae.42be976f]

This e-mail is confidential and is intended for the exclusive use of the addressee only. Selection Services Ltd accepts no liability for personal views expressed. While every effort has been made to ensure the attachments are virus-free, they must be checked before further use, especially those containing encrypted data. If you have any problems with this e-mail, please contact our IT Manager on [e-mail]<mailto:[e-mail]>

Registered in England and Wales Registered Number: 2758710 Registered Office: Provident House, 122 High Street, Bromley, Kent BR1 1EZ

--

This message has been scanned for viruses and

dangerous content by MailScanner,

and is believed to be clean.

[Heimatplan]

Sorry for the not answering for so long, I'm very busy.

When you use your Intel X520-DA2 on Windows, a lot of work gets offloaded to the NIC. My guess is, that ESXi does not use or recognize Intel VT which is mandatory to use offloading. Check out this article, this one perfectly describes your situation: http://maso.dk/2011/10/07/use-10-gigabit-with-intel-vt/

PS: NetQuenue is disabled by default on 4.0 - http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004278

[billybobSSD]

Ingo, VMkernel.Boot.netNetqueueEnabled is enabled by default in ESX5. Furthermore I am seeing the issue between vmxnet3 VMs in the same port group on the same vswitch, which leads me to believe the problem has nothing to do with the 10gig ethernet hardware.

[Heimatplan]

Please use restop to monitor the origin of those latencies. Download and deploy VMware vSphere Management Assistant 5.0 (vMA) from its .ovf template. Connect to your Test-ESXi-Server with putty and start restop by typing "restop --server your.vcenter.server --vihost host.name --username vcenter-user

Stretch the putty window so you can see all the columns. Type n to display the network screen. Can you see any %DRPTX or %DRPRX rise If larger than 0 transmit packets are being dropped,hardware is overworked due to high network utilization.

[billybobSSD]

Ingo,  how can the LOW limit between vmxnet3 VMs in the same port group on the same vswitch even remotely be attributed to "overworked hardware"? Especially since I've shut down every other VM on that server.

[konakcc]

I agree, this makes no sense.  If the case, why would anyone deploy the vmxnet3 driver?  Is there anyway to bind/team multiple E1000 virtual NICs together in a windows virtual machine?  I need about 1.8 Gbps bandwidth per VM.

[Zaigui Wang]

I ran into the same issue as well on ESXi 5.0 U2 + Redhat 6.x. RH 5.x works just fine, but RH6.x just stinks. VMware is aware of this. in the mean time here is the solution (the vmware KB on LRO is slightly off).

Disable LRO - DocWiki

or this:

esxcli system settings advanced set --int-value 0 -o /Net/VmxnetSwLROSL

esxcli system settings advanced set --int-value 0 -o /Net/Vmxnet3SwLRO

esxcli system settings advanced set --int-value 0 -o /Net/Vmxnet3HwLRO

esxcli system settings advanced set --int-value 0 -o /Net/Vmxnet2SwLRO

esxcli system settings advanced set --int-value 0 -o /Net/Vmxnet2HwLRO

Hope this ends the misery for all!

[billybobSSD]

This actually made the traffic worse. Before it would hit around the 500 megabit ceiling and stay there, now the bit rate just jumps around wildly....

[cdc1]

Here is a nifty Intel Case Study paper that explains similar results with their tests done in conjunction with FedEx.

ftp://download.intel.com/support/network/sb/fedexcasestudyfinal.pdf

The gist of it is this:

You may find that you get better performance with single stream file copies in VMs with 1 vCPU rather than a VM with more than 1 vCPU.

However, YMMV, as it depends on how you've got things setup.  They also list some other best practices to keep in mind (see pg. 10).  They specifically refer to vSphere 4.0, but I'm sure the majority of them (if not all) still apply with vSphere 5.

[Anand Lingaraj]

HI

Do this settings in the VMs if windows

RSS enabled  = Yes

Large Rx Value = 2048

Ring #2 Size = 32

If you have enabled Jumbo frames enable Jumbo frames.

Regards

[billybobSSD]

vm4u, I tried those settings and get the same results. Thanks!

[billybobSSD]

I just tried Iperf with these VM's again ON A NEW ESXi INSTALL THAT I JUST BUILT, on a completely different piece of hardware and I get THE SAME BANDWITH CAPS. Is there a built in limitation in ESXi 5 ????

Will somebody from VMware answer?

[peterdabr]

Is the new ESXi install still ESXi5? If so, maybe you should test it on ESXi4.1 with the same 10Gb nic....I realize that it won't solve your problem on ESXi5 but it'd be interesting to see the behavior on the previous version.

Peter D.

[billybobSSD]

Peter, the new hardware I tested it on was ESXi 5 and it did not have 10 gigabit adapters in it, I was testing to see whether the 500 megabit bandwidth limit between vmxnet3 VMs in the same port group on the same  vswitch exists on different hardware, and it does.

I'll probably get annoyed enough any moment now to go and try it on ESXi 4, but then somebody here will complain its on the wrong forum.... :smileylaugh:

[konakcc]

I have a similar problem.  If I install my application on a server with two physical nics, I can see it use about 1.6 Gbps.  It runs on windows 7 or windows 2008.  When I install the OS and application under ESX, it runs fine on the E1000 driver, however, it suffers performance issues.  When I try it with the vmxnet3 interface all kinds of strange things start happening, as if packets are disappering some place.  If in fact, the vmxnet3 interface is limited to 300-450 mbps this would account for my issues as well.  Unfortunately, my account doesnt have support attached to it, so I can not raise an issue with Tech Support.  Has anyone else done so?

[konakcc]

I think only the vmxnet2 driver is available under ESX4.

[billybobSSD]

I just stood up an ESXi 4 server and created two win2008 R2 VM's with VMXNET3 adapters. I'm about to test this all over again.

[trink408]

Also curious to see what comes of this issue... Keep us posted. 

[billybobSSD]

Ok, just loaded up ESXi 4.1 and created two Server 2008R2 VM's and this is what I got between the two vmxnet3 VM's in the same port group on the same vswitch.

------------------------------------------------------------
Client connecting to 192.168.1.133, UDP port 5123
Sending 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.113 port 58518 connected with 192.168.1.133 port 5123
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec   304 MBytes   510 Mbits/sec
[  3]  5.0-10.0 sec   303 MBytes   508 Mbits/sec
[  3] 10.0-15.0 sec   307 MBytes   516 Mbits/sec
[  3] 15.0-20.0 sec   306 MBytes   513 Mbits/sec
[  3] 20.0-25.0 sec   306 MBytes   513 Mbits/sec
[  3] 25.0-30.0 sec   309 MBytes   518 Mbits/sec
[  3]  0.0-30.0 sec  1.79 GBytes   513 Mbits/sec
[  3] Sent 1308932 datagrams

So, once again it hits the 500 megabit limit. I keep seeing pdf's posted here where they get much higher throughput. Apparently there is some secret to getting around the bottleneck. I sure wish I knew what it was......

[konakcc]

Can you run the exact same experiment with the E1000 driver and post the results so that they are here in the thread?

I too hope that somebody from vmware wakes up and takes notice.

Thanks.

-Ken

[billybobSSD]

.

.

No comment from VMware?

[Josh26]

billybobSSD wrote:

.

.

No comment from VMware?

This is a community forum.

If you expect an official VMware answer, log a support ticket.

Given you can't buy the product without SnS, you pretty much can't be in this boat on a new deployment without support.

[goppi]

Anything new on this topic?

I see the same 500 mbis/s limit between VMNETX3 boxes located on the

same vSwitch. I would assume that there is not network hardware involved

at all in this situation.

Wose to mention that I'm on ESXi 4.1u2.

Cheers.

[roeschp]

Ich bin derzeit nicht anwesend und erst ab dem 21.05.2012 wieder erreichbar. Ihre eMail wird nicht weitergeleitet. In dringenden Fällen wenden Sie sich bitte an die Hotline, Tel. [phone] oder per eMail mailto:[e-mail].

I am currently out of office scheduled to return on 21st of May 2012. Your email will not be forwarded. During my absence please address urgent inquiries to our hotline, telephone [phone] or via email mailto:[e-mail]

[trink408]

My results were a little better testing between two vm's on the same vswtich, running on esxi 4.1. Both vm's running vmxnet3 adapters...

Maybe I'm not doing the test right, just used default settings for iperf?

Client connecting to 192.168.xxx.xxx, TCP port 5001

TCP window size: 8.00 KByte (default)

------------------------------------------------------------

[156] local 192.168.xxx.xxx port 55395 connected with 192.168.xx.xxx port 5001

[ ID] Interval                Transfer     Bandwidth

[156]  0.0- 5.0 sec  1.02 GBytes  1.76 Gbits/sec

[156]  5.0-10.0 sec   961 MBytes  1.61 Gbits/sec

[156]  0.0-10.0 sec  1.96 GBytes  1.69 Gbits/sec

[156] MSS and MTU size unknown (TCP_MAXSEG not supported by OS?)

My bad, forgot my hosts for this test have 1gig cards in them, not applicable to your situation... Sorry..

[goppi]

@trink408:

I would assume this doesn't make any difference as two VMs located on the same vSwitch located on the same ESXi host conected via VMXNET3 should be independent of network hardware.

Cheers.

[trink408]

Good point.. That 10 gig hardware he was originally trying to test shouldn't play into the picture testing vm's on the same vswitch and same host...

Maybe esxi4.1 handles the vmxnet3 adapters differently? 

[konakcc]

Does anyone know if you can team multiple E1000 virtual NICs together inside the guest OS to get around this issue?

-Ken

[konakcc]

It's been about a month since the last tests were run.  As far as I can find, there is NO way to team nics in the guest OS.  Has there been any progress on the vmxnet3 performance issue?

-Ken

[roeschp]

Ich bin derzeit nicht anwesend. Ihre eMail wird nicht weitergeleitet. In dringenden Fällen wenden Sie sich bitte an die Hotline, Tel. [phone] oder per eMail mailto:[e-mail].

I am currently out of office. Your email will not be forwarded. During my absence please address urgent inquiries to our hotline, telephone [phone] or via email mailto:[e-mail]

[NMW]

Any one confirm this is still an issue, or has anyone found a solution. After taking over a lab and straightening out the hardware I am seeing the same issue on 6 different hosts with ESXi 4.1, 5.0, 5.0 Update1 (A total of 3 different 5.0 builds). The HW are HP DL380G7 and DL580G7 servers with between 120GB and 256GB RAM. Using the same methology as BillyBob and others I spent several hours with VMware support twith no luck yet. Outstanding ticket will hopefully get resolved. 2 2008R2 VM's with VMXNET3, updated VMware tools, issolated vSwitch, no attached. Using iPerf max was 1.6Gb per second, average was about 1.2Gb/second. Going to confirm with Netio and iometer but it reflects what the performance has been like.

[konakcc]

I ran into a couple of VMware engineers at CiscoLive last month and they said that if the server only has 1 GE nics, the max a VM will get will be 1 GE since the NIC teaming done at the ESX level is not true Active/Active.  As far as VM to VM on the same server, I also am not able to get more than 2 Gbps between them.  Since this is a software switch, I guess it depends on the CPU clock rate.  I wonder if servers with fewer higher speed cores get better performance than those with more lower speed cores.  My guess is that the processes associated with the Virtual switch are NOT multithreaded.  If this is truely the case, then this bodes well for the Cisco VIC.  Maybe time to disable the Vswitch and pump all the NIC traffic upstream to a true physical switch.  You should also be able to do this on other servers such as HP which support virtual NICs and using NIC bypass.

[billybobSSD]

As I said earlier in the thread, the server has a dual port 10 gig connection. One of the ports talks to the iSCSI san at up to 6 gigabits. The VM's dont talk to eachother over 500 megabits, nor do they shoot data out the underutilized secondary 10 gig port at more than 500 megabits. This is a VMware problem and not a hardware limitation.

[billybobSSD]

I'm going to try this on Microsoft's free HyperV offering and see if it has the same limitation.

http://www.microsoft.com/en-us/server-cloud/hyper-v-server/

[Rubeck]

Sounds great, BIllyBob

Could you be so kind to update this thread once you're done testing...?

/Rubeck

[ROE]

I'm having the same issue, but seems to have narrowed it a bit down... and then not really :smileyconfused:

My test setup is 3 VMs, with vmxnet3, alone on a IBM 7141 server running vSphere 4.1.

iperf tests using default setting of 8KB TCP window size, intervals of 5 sec. and running time 60 sec.

Linux LiveCD used: http://www.livecd.ethz.ch/index.html

Running iperf from my physical workstation to a virtual 2008R2, as well as from a virtual 2008R2 to another virtual 2008R2, gave poor performance in the 2-300Mbit/s range.

Running the same test from my physical workstation to a physical 2008R2 server gave results around 950Mbit/s.

I then booted a Linux LiveCD on one of the VMs and ran the iperf server from that.

Then performace from my physical workstation to the Linux VM jumped to 948Mbit/s.

Booting an additional VM on the Live CD and running iperf between those two resultet in speeds of up to 4,5Gbit/s (but averaging 2,97GBit/s). Still a bit from the theoretical 10Gbit limit of the vmxnet3, but a substantial increase.

So, at least, in my setup, the problem seems to be isolated to Windows guests.

I figured it might be a buggy vmxnet3 driver, but switching to the E1000 driver dropped MBit/s througput to the lower 200s.

No solution, but something more to think about. If anyone finds a solution please follow up here :-)

/Robert

(to keep tests consistent I used default iperf settings, but Linux-to-Linux tests with 256K window size averaged speeds at 5Gbit/s)

[ROE]

Problem solved!

Actually a bit embarrasing, since we all know troubleshooting should be on as clean installs as possible :smileyblush:

Symantec Endpoint Protection 12 was causing the problem. Even with Network Threat Protection disabled we still couldn't get more than around 500Mbit/s throughput.

Completely uninstalling SEP jumped the speeds to expected levels.

[ouderblade]

Having a similar issue, however we are experiencing sever performance degredation almost to a point where users cannot do their work as our server is terribly slow.

Our issues started when i upgraded to esxi 4.2 u2 and the virtual machines to version 7 with paravirtual scsi and vmxnet3.

heres a typical output of my scenario

[3932] 87.0-88.0 sec  20.4 MBytes   171 Mbits/sec
[3932] 88.0-89.0 sec  26.6 MBytes   223 Mbits/sec
[3932] 89.0-90.0 sec  8.87 MBytes  74.4 Mbits/sec
[3932] 90.0-91.0 sec  10.9 MBytes  91.8 Mbits/sec
[3932] 91.0-92.0 sec  15.5 MBytes   130 Mbits/sec
[3932] 92.0-93.0 sec  8.10 MBytes  68.0 Mbits/sec
[3932] 93.0-94.0 sec  3.53 MBytes  29.6 Mbits/sec
[3932] 94.0-95.0 sec  3.63 MBytes  30.4 Mbits/sec
[3932] 95.0-96.0 sec  3.64 MBytes  30.5 Mbits/sec
[3932] 96.0-97.0 sec  24.9 MBytes   209 Mbits/sec
[3932] 97.0-98.0 sec  36.2 MBytes   304 Mbits/sec
[3932] 98.0-99.0 sec  41.1 MBytes   345 Mbits/sec
[3932] 99.0-100.0 sec  37.9 MBytes   318 Mbits/sec.

Very erratic and jumping below 30Mbits/sec. Im in the process of removing antivirus software (sophos) to see if that resolves anything.

[HendersonD]

This is a fascinating discussion. We have an IBM H Chasis Bladecenter are in the process of pulling out the Cisco 1Gb switches we have been using and replacing them with BNT 10Gb switches. This goes hand in hand with purchasing new blades with two, 8 core Intel E5-2600 processors. The setup is not complete but I should have VMs migrated in the next week and will test this setup. Like everyone here I am hoping to get throughput at least close to 10Gbps.

One question, are folks just copying a large file between two virtualized Windows 2008 R2 servers while iPerf is running to test throughput? Just looking for a sound methodology so I can run tests and post back my results.

[HendersonD]

Is this of any help?

http://download.intel.com/support/network/sb/fedexcasestudyfinal.pdf

[janusz_w]

Slow VMXNET3 performance on 10gig connection (and 1Gb too)

Nobody else has had this issue?  EVERYBODY HAS :smileyangry:

I do some transfer tests using iperf between two virtual W2008R2 machines.

(no file transfer tests,  iperf tests only LAN - so no disk/raid/datastore… issue)

Virtual machine LAN drivers - VMXNET3 (W2008R2 report  as 10GB drivers)

No any change of the default VMXNET3 adapter settings (default).

First machine on ESXi 4.0 farm (farm 4xIBM x3610  with 1Gb interfaces)

Second machine on ESXi 5.0 (farm 4xCISCO C260 with 1Gb interfaces)

Farms on the same localization (but 3 Cisco 1 Gb switches between)

Results ( in MBytes/s !!! ):

Standard connection (no change in iperf TCP parameters - default)

------------------------------------------------------------

[164]local 172.19.220.xxx port 49353 connected with 172.19.230.xxx port 5001

[ ID]   Interval         Transfer         Bandwidth

[164]  0.0- 1.0 sec  23.1 MBytes  23.1 MBytes/sec

[164]  1.0- 2.0 sec  22.0 MBytes  22.0 MBytes/sec

[164]  2.0- 3.0 sec  23.0 MBytes  23.0 MBytes/sec

[164]  3.0- 4.0 sec  22.0 MBytes  22.0 MBytes/sec

[164]  4.0- 5.0 sec  22.1 MBytes  22.1 MBytes/sec

[164]  5.0- 6.0 sec  22.4 MBytes  22.4 MBytes/sec

[164]  6.0- 7.0 sec  22.4 MBytes  22.4 MBytes/sec

[164]  7.0- 8.0 sec  22.1 MBytes  22.1 MBytes/sec

[164]  8.0- 9.0 sec  21.9 MBytes  21.9 MBytes/sec

[164]  9.0-10.0 sec  22.2 MBytes  22.2 MBytes/sec

[164]  0.0-10.0 sec  223 MBytes  22.3 MBytes/sec 

very bed :smileysad:

Changed in iperf - TCPWindowsSize from def. 8kB to 56kB on Client  side

------------------------------------------------------------

[164]local 172.19.220.xxx port 49356 connected with 172.19.230.xxx port 5001

[ ID]   Interval      Transfer    Bandwidth

[164]  0.0- 1.0 sec  81.1 MBytes  81.1 MBytes/sec

[164]  1.0- 2.0 sec  79.0 MBytes  79.0 MBytes/sec

[164]  2.0- 3.0 sec  76.9 MBytes  76.9 MBytes/sec

[164]  3.0- 4.0 sec  83.7 MBytes  83.7 MBytes/sec

[164]  4.0- 5.0 sec  85.3 MBytes  85.3 MBytes/sec

[164]  5.0- 6.0 sec  79.2 MBytes  79.2 MBytes/sec

[164]  6.0- 7.0 sec  83.5 MBytes  83.5 MBytes/sec

[164]  7.0- 8.0 sec  79.4 MBytes  79.4 MBytes/sec

[164]  8.0- 9.0 sec  81.2 MBytes  81.2 MBytes/sec

[164]  9.0-10.0 sec  77.9 MBytes  77.9 MBytes/sec

[164]  0.0-10.0 sec  807 MBytes  80.5 MBytes/sec 

Much better :smileyhappy:    (up to 4 times ?)

Changed in iperf  TCP Buffer Length to 2MB on Client side

------------------------------------------------------------

[164]local 172.19.220.xxx port 49363 connected with 172.19.230.xxx port 5001

[ ID] Interval      Transfer    Bandwidth

[164]  0.0- 1.0 sec  106 MBytes 106.0 MBytes/sec

[164]  1.0- 2.0 sec  96.0 MBytes  96.0 MBytes/sec

[164]  2.0- 3.0 sec  100 MBytes 100.0 MBytes/sec

[164]  3.0- 4.0 sec  100 MBytes 100.0 MBytes/sec

[164]  4.0- 5.0 sec  96.0 MBytes  96.0 MBytes/sec

[164]  5.0- 6.0 sec  68.0 MBytes  68.0 MBytes/sec

[164]  6.0- 7.0 sec  88.0 MBytes  88.0 MBytes/sec

[164]  7.0- 8.0 sec  92.0 MBytes  92.0 MBytes/sec

[164]  8.0- 9.0 sec  88.0 MBytes  88.0 MBytes/sec

[164]  9.0-10.0 sec  64.0 MBytes  64.0 MBytes/sec

[164]  0.0-10.0 sec  900 MBytes  89.7 MBytes/sec  

90MBytes/s on 1Gb interface –  900Mbytes/s - what else we need :)))

And now all together…

Changed in iperf TCP TCPWindowsSize to 56k and Buffer Length to 2M

------------------------------------------------------------

[164]local 172.19.220.xxx port 49365 connected with 172.19.230.xxx port 5001

[ ID] Interval      Transfer    Bandwidth [164]  0.0- 1.0 sec  92.0 MBytes  92.0 MBytes/sec

[164]  1.0- 2.0 sec  94.0 MBytes  94.0 MBytes/sec

[164]  2.0- 3.0 sec  82.0 MBytes  82.0 MBytes/sec

[164]  3.0- 4.0 sec  104 MBytes 104.0 MBytes/sec

[164]  4.0- 5.0 sec  98.0 MBytes  98.0 MBytes/sec

[164]  5.0- 6.0 sec  96.0 MBytes  96.0 MBytes/sec

[164]  6.0- 7.0 sec  90.0 MBytes  90.0 MBytes/sec

[164]  7.0- 8.0 sec  98.0 MBytes  98.0 MBytes/sec

[164]  8.0- 9.0 sec  94.0 MBytes  94.0 MBytes/sec

[164]  9.0-10.0 sec  104 MBytes 104.0 MBytes/sec

[164]  0.0-10.0 sec  954 MBytes  95.1 MBytes/sec

  :))))))

No ESXi tuning !

Iperf change TCP parameters only ‘insaid‘ virtual machine !!

So now VMware staff !!!

Prepare 'LAN Best Practices' where to put this parameters

on W2008R2/ W2012/ RHEL... for VMXNET3, E1000.... (register, NIC driver param...)

Perhaps it help :smileyhappy:

Janusz

PS: sorry for my English

[Scissor]

Some thoughts after briefly skimming this thread:

• Configure your test Guests with 1 vCPU each to rule out SMP scheduling overhead as the cause of the problem.  Ensure that the ESX Host CPU utilization is ok before starting testing. 

• I would suggest getting rid of Jumbo frames unless you can confirm that every single piece of equipment between your two Guests has been properly configured with the same Jumbo frame parameters (only really applicable when Guests are on different Hosts). 

• Why did you decide to use UDP when testing with iPerf [1]?  Please re-run your tests using TCP.  Also, at higher speeds increasing the iperf TCP Window Size parameter ( -w ) can make huge differences in throughput.
• One thing about iperf that always seemed "backwards" to me:  During a test, it is the iperf Client that sends data to the iperf Server.

Below are stats from testing between two Win2012 Guests on the same ESXi 5.1 Host using different iperf TCP Window Sizes:

===============================================================

server: iperf -w 64k -s

client: iperf -w 64k -c x.x.x.x

C:\Temp\iperf-2.0.5-cygwin> .\iperf.exe -c 192.168.120.20

------------------------------------------------------------

Client connecting to 192.168.120.20, TCP port 5001

TCP window size: 64.0 KByte (default)

------------------------------------------------------------

[  3] local 192.168.120.22 port 65051 connected with 192.168.120.20 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.2 sec   549 MBytes   451 Mbits/sec

===============================================================

===============================================================

server: iperf -w 256k -s

client: iperf -w 256k -c x.x.x.x

PS C:\Temp\iperf-2.0.5-cygwin> .\iperf.exe -c 192.168.120.20 -w 256k

------------------------------------------------------------

Client connecting to 192.168.120.20, TCP port 5001

TCP window size:  256 KByte

------------------------------------------------------------

[  3] local 192.168.120.22 port 18217 connected with 192.168.120.20 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.2 sec  7.96 GBytes  6.72 Gbits/sec

PS C:\Temp\iperf-2.0.5-cygwin>

===============================================================

[1] More info on UDP vs TCP iPerf testing:  http://serverfault.com/questions/354166/iperf-udp-test-show-only-50-of-bandwidth 

[janusz_w]

Generally agree but:

1. yes, 1 vCPU for tested vm each for overhead as the cause of the problem and check ESX Host CPU utilization – ok,

     but for fine tuning, not for results 5-10 times lower than expected.

2. jumbo frames – ok, but my tests show that even without jumbo frames (sorry, now only on 1Gb interface)

    we can get transfers like 900Mb/s

3. yes, agree, I use iperf only with TCP

4. we can use iperf options for dual/trade communication tests ( -d –L 5001  or  -d –r –L 5001 ).

    In my tests I use only one way transfers and change parameters only on clieny side.

Best Regards,

Janusz

[janusz_w]

Tests is only tests, so I decide to modify only BufferSize (for the beginning), on some production Windows2008R2 servers and...

first signal from backup staff:

why our LAN agent backup starts to work 3 times faster :smileyhappy:

(1 and 30 minutes to 30 minutes)

3 times for the beginning...

It works.

So now TCP parameters tuning on Windows2008R2, then on RHEL.

(little problem is, that on 2008R2 register there is no such a parameter,

so you have to localize interface and add that parameter )

Soon tests on 10Gb.

But the question is:

why we can't find such information on VMware paper like

'Best Practices for Windows/Linux vm LAN tuning'  ???

I think it will be a HIT .

Janusz