VMware vSphere

 View Only

  • 1.  Jumbo Frames - mtu size in windows 2k3 guest

    Posted May 13, 2011 09:12 AM

    I'm currently playing around with jumbo frames in windows 2k3 guests on vsphere 4.1 hosts. I have 10gbe network and I'm measuring network performance from one w2k3 guest to antoher w2k3 guest with iperf. Without Jumbo Frames, I was able to get out about 8.5gbit/s. When I enable Jumbo Frames in my dvSwitch (MTU 9000), physical switches and guest NICs (vmxnet3, set "Jumbo Packet" in device manager to "Jumbo 9000") the throughput drops down to ~30mbit/s. After some trial and error adjustment of the MTU-setting in windows (in the registry), I found out that an MTU of 8232 is the biggest MTU from which I will get adequate speed results. If I use 8233, speed drops to ~30mbit/s again.

    Can anyone explain, why MTUs above 8232 are decreasing throughput that much?

    Thanks to anyone who can give me info on this!



  • 2.  RE: Jumbo Frames - mtu size in windows 2k3 guest

    Posted May 13, 2011 01:29 PM

    Which vNIC did you use in the VM's?? For VM to VM traffic, the physical network has little to do with it. That is, unless you're not using shared storage, and the VM's are in different datacenters (so physically seperated from each other). If on the same datastore set, and cluster, ie, same infrastructure, then using a VMXNET3 vNIC should get you the max performance available. I've seen those vNIC's list out 10Gb connection speeds even on Gb LAN's. You'll see the best performance when going VM to VM, with more 'normal' rates going through the physical LAN... If you have the VM's on the same vSwitch (or distributed switch) then you'll have max performance right out of the gate...

    BTW, if you configured the vSwitch for jumbo frames, you really shouldn't need to configure the VM's vNIC too. What you're probably encountering, when trying to push the VM's into the same area for MTU, is contension for resources that is better handled by the hypervisor.

    Maybe we can get one of the VMware tech's to post up more technical details as to why this is how it is... Or one of the other 'big brains' could post up why this is... I've not dived too deep into the code under the networking items so far. Just not had a need to...



  • 3.  RE: Jumbo Frames - mtu size in windows 2k3 guest

    Posted May 13, 2011 02:00 PM

    I am using VMXNET3 in both vms. I've tried multiple scenarios for measuring vm-to-vm network performance: same host, different hosts in same datacenter, different hosts in different datacenters. So, among others, I ensured that I was utilizing the physical network. But in every scenario I tried, I was running into the same issue: As soon as the MTU is bigger than 8232, the performance goes down.

    Actually, that's not a problem for me, as I now know that I have to set the MTU to 8232 to benefit from jumbo frames inside a W2k3-VM. But I am just interessted in the cause of that. So I would definitely welcome any VMWare technician who can provide some information about this issue.



  • 4.  RE: Jumbo Frames - mtu size in windows 2k3 guest

    Posted May 13, 2011 02:03 PM

    Have you tried testing with a Server 2008 VM? My home LAN is only Gb, so I won't be able to compare with a 10Gb LAN there. I could test out over the weekend to see what I get for results. If I manage to find the time. Got a busy couple of weekends ahead of me...



  • 5.  RE: Jumbo Frames - mtu size in windows 2k3 guest

    Posted May 13, 2011 02:08 PM

    No, not yet. But I'll give it a try next week! I'll post the results here...



  • 6.  RE: Jumbo Frames - mtu size in windows 2k3 guest

    Posted May 17, 2011 01:05 PM

    I just tried the same performance tests on win2k8 r2 64 and ran into the same issue:

    1. iperf with normal setup without modifications: ~8.8 Gbit/s
    2. iperf with Jumbo frames enabled in vmxnet3-driver (windows device manager): ~10 Mbit/s
    3. iperf with Jumbo frames enabled in vmxnet3-driver (windows device manager) and a mtu set to 9,000 in the windows registry: ~10 Mbit/s
    4. iperf with Jumbo frames enabled in vmxnet3-driver (windows device  manager) and a mtu set to 8,232 in the windows registry: ~10 Mbit/s: ~9.8 Gbit/s

    What I've additionally found out is, that Windows ist sending out ethernet frames > 1,500 Bytes, even if jumbos are not enabled neither on the vmxnet nor in the registry. I've monitored some iperf-sessions with wireshark and saw that windows was sending frames at about 8,000 bytes. On the receiving machine, I could only see frames ~1,500 bytes arriving. I'm just wondering where the frames are split into 1.5k-sizes. Are the vmxnet-driver or the dvs doing such tasks?

    Maybe someone has a bit more knowledge of this behaviour and can tell me something about it...

    Thanks!