VMware vSphere

 View Only

  • 1.  Guests sending packets that exceed the MTU of their interfaces

    Posted May 12, 2011 10:37 PM

    I have an issue which is proving ellusive and very disruptive.  I have multiple guests on a pair of ESXi 4.0 U2 hosts that are sending packets that exceed the MTU of 1500 on their interfaces.  I have tried all the various NICs flexible, e1000 and vmnet3, all exhibit the same issue.  I am seeing it on multiple fedora servers running various kernels.  I am also seeing it on a Win2k3 server.  Having open-vm-tools installed or not on Linux doesn't seem to make any difference. The only thing in common left is the fact they are all running as VM guests on ESXi

    The ICMP frag messgaes coming back from the router seem to be ignored by the guests.

    The packet size has been as high as 20k !! but is typically 2x normal i.e. 1480 -> 2960. When a frag message is obeyed the count seems to go down by exactly 40 ie. 2960 -> 2920 which is still too big.

    The vSwitches are set with an MTU of 1500 but I do believe that at some point in the past they were set to 9000 and returned to 1500 when we decided not to use jumbos.

    Any help would be very much appreciated.



  • 2.  RE: Guests sending packets that exceed the MTU of their interfaces

    Posted May 13, 2011 07:31 AM

    Hi ,

    can post the packet cap using wireshark ? it will be muoch helpfull to know what is going on your interfaces



  • 3.  RE: Guests sending packets that exceed the MTU of their interfaces

    Posted May 13, 2011 08:32 AM

    See traces attached to https://bugzilla.redhat.com/show_bug.cgi?id=702560 as I initially thought it was a Linux kernel issue.



  • 4.  RE: Guests sending packets that exceed the MTU of their interfaces

    Posted Jun 15, 2011 09:27 PM

    The issue turned out to be the TCP offload of segment re-assembly with the e1000 and vmxnet3 drivers.

    If the guest forwards packets, i.e. a linux firewall or router, then having the incoming TCP segments combined by the driver can result in quite large packets ( depending on the arrival rate ) being presented further up the stack.  When such packets are passed through the upper layers and the OS attempts to transmit the now oversize packets out another interface, they exceed the MTU of the out bound interface.  An ICMP fragmentation required response is sent back to the sender.  Trouble is, it was already sending packets with the correct MTU. So communication is now broken.

    In my case, setting the NICs on my firewall/router back Flexible forces the guest to use the pcnet32 driver which does not support the offloading functionality.

    A comment found on another forum indicated that using ethtool to turn off the TSO functionality with the vmxnet3 driver no longer worked for them at ESXi4.1U1 so I didn't bother trying that.



  • 5.  RE: Guests sending packets that exceed the MTU of their interfaces

    Posted Jun 15, 2011 09:28 PM

    See my final reply