Intermittent UDP Performance / Dropped packets

[spetty7326]

I have been struggling with a UDP performance issue for some time, and am just about out of ideas so I'm looking for some help.  Our application relies heavily on UDP broadcast packets - specifically clients in other subnets elect one client in each subnet to individually send their messages, and then that one client in the different subnet re-broadcasts it to the entire subnet.  Everything works fine in the physical world, never a problem.  Once we've started to virtualize, the VMs that are acting as the broadcast clients are dropping UDP packets intermittently.

I've tried a number of things:

• Switched all of the clients to e1000 instead of vmxnet3 - we saw a lot of dropped UDP packets with vmxnet3
• Changed all esx hosts (DL380G6) to high performance power profile, eliminating any CPU throttling for power saving
• Increased e1000 tx and rx descriptors to max - 2048
• Added a second vCPU - again, lot's of dropped packets with one vCPU, disappeared with 2.

In the physical world, we use single-core Pentium 4s, 2.0 GHz, and there's never a problem.  The DL380G6s have quad core, dual CPUs, and we're having problems constantly - missed UDP packets at the VM clients.

The switches aren't the issue either.  We have physical clients connected to the same switches in the same sub-net, and they perform fine.

These are very, very low usage machines - typically running <100 MHz with dual CPUs.  I suspect that when they get multiple UDP packets when they are idle, the first couple get dropped.  The problem is that just dropping 1 or 2 packets can create a major issue in our application.  Changing the application isn't an option, either.

The only thing I've been able to find that I haven't tried is this setting in the vmx file:   monitor.idleLoopSpinUS     I don't know that it will help much, so I'm reluctant to try it yet.

Can anyone suggest anything that might improve the performance of our bursty,, UDP application, to allow it to keep up with the intermittent UDP packets?

Any help or questions would be very appreciated!

[marcelo_soares]

I would check:

- The driver you are using on your pnics, check on www.vmware.com/go/hcl for compatibility and driver updates. I can help if you provide me the escat bios/model of the pnics.

- If the VMs communicate between themselves inside a single ESX (or grouped), I would try enabling VMCI, as per KB http://kb.vmware.com/kb/1010806. This will avoid packects between VMs on the same hosts to be sent to the physical nic and will be communicating only inside the ESX.

Good luck,

Marcelo Soares

[spetty7326]

Marcelo,

As I understand it, the application has to be written to utilize the VMCI API.  Unfortunately, ours isn't, and we aren't the developer so we have no access to modify the product.  I tried enabling VMCI to see if it would help anyway, but saw no change.

I looked at the NICs with ethtool, and here's how they break down:

(1) Broadcom NC382i quad port card

driver:  bnx2

version:  2.0.7d-4vmw

firmware version on 2 of the ports is 4.6.4 NCSI 1.0.3, the other two are just 4.6.4

(2) Broadcom NC382T dual port cards

driver: bnx2

version:  2.0.7d-4vmw

firmware version: 4.4.14 on all of them

When I look at the HCL, it says the 2.0.7 is "inbox" but there is a 2.0.15g.9.v40.1 listed as "async."  I don't know if that would help, or if I should just stick with what I have.  The odd firmware version on the first 2 ports of the NC382i is a little odd to me as well.  What would NCSI 1.0.3 be, and should it be on all 4 ports?

I also pulled up esxtop, and watched for dropped packets as I ran a 100 MB/s iperf test.  I didn't see any evidence of dropped packets at all, so again I'm at a loss.