Broadcast packets on standby adapter

[tarkus72]

Hi, I have on my datacenter running vSphere 5 one virtual switch with two adapters, each one connected to one different physical switch (e.g. vmnic0 to switch 0, vmnic4 to switch 1). I setup the vswitch with 'use explicit failover order', vmnic0 as active, vmnic4 as standby.

What I'm expecting on vm connected to such vswitch is just one broadcast packet incoming. But actually the vm receive two packets for each broadcast: this is right behaviour (imho) if I have teaming, e.g. both vmnic0 and vmnic4 as active adapters; moving vmnic4 to unused adapter give me same result.

The only way to get one broadcast is physically disconnecting the cable from vmnic4.

The problem is that on one vm connected to such vswitch there's one network application that is not capable to handle duplicated arp broadcast, then in such desired configuration the application mess around.

Any idea? Is it right that a standby adapter should not forward any packet to vswitch?

Thanx.

[ranjitcool]

If I remember right, the standby adapter will send a health check packet (my own term) to the switch so they know the link is up.

But i dont think vm should see that.

R

[tarkus72]

I don't think too... but actually it is (tested with tcpdump).

[SteveFuller2011]

In which direction are the broadcast - from the physical switch to the vSwitch?

If a pNIC is set as a standby adapter that's only standby from a point of view of sending traffic from your ESX host to the network. As far as the physical switch is concerned the interface to which the standby NIC is connected is up and so broadcasts will be sent on the link.

[tarkus72]

Yes, from physical to vswitch.

So receiving packets from standby adapter it's by design, right? Is there some possibility to avoid that?

[SteveFuller2011]

I've not seen it documented exactly how the vSwitch would handle broadcast traffic it receives on an adapter configured in standby, but I'm not sure why it would just be thrown away. If that were to happen then e.g., the CDP information received on the standby NIC would be lost.

The duplicate ARP broadcast you mentioned earlier, are they duplicate ARP requests seen by the server for the servers IP address?

Regards

[tarkus72]

I think a separation should be done between traffic for vswitch management (e.g. CDP) and traffic sent to the connected virtual machines: in the first case is it right, everything should be delivered to the vswitch, in the second case should not.

The ARP are for every connected host on the network, just arp broadcast.

[MKguy]

No, it doesn't make sense and I'm certain it's not supposed to work like  that. Any vNIC has only one dedicated physical Uplink assigned at any given point in time (which can be viewed in (r)esxtop); that is unless you're use IP-hashing (which you aren't). So only through that one Uplink can it receive/send traffic, including broadcasts.

I just tested your setup with on one of our lab hosts and I don't see any duplicate broadcasts.

1. Are you running ESXi5 with U1? Pre-U1 there were some serious issues when using standby/unused teaming configuration like in your case, see:

http://vmtoday.com/2012/02/vsphere-5-networking-bug-2-affects-management-network-connectivity/

http://kb.vmware.com/kb/2008144

2. Are you sure those are duplicates of the same packet? Ping the broadcast address and check the sequence number for real dupes. It's not uncommon for some networking stacks to send multiple identical ARPs during a short amount of time. I also highly doubt multiple identical ARPs should be able to mess up some application.

3. Do you really only see those packets when using a standby link? What about setting one to unused? What if your active and standby pNIC change places or when both are active? Also try setting the load balancing policy to port-ID based keeping the standby settings (which effectively won't change anything in your setup).

[tarkus72]

1. ESXi5 U1 (build 623860)

2. sure, checked many times: with one adapter, pinging an un-existant IP (just to be sure) usually are sent broadcast arp every one second, connecting the standby adapter the same result in six, every couple has very close timestamp:

One active adapter:

tcpdump -n -i vlan2 -eee "arp and broadcast and ether host d0:67:e5:35:d2:f2"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan2, link-type EN10MB (Ethernet), capture size 96 bytes
15:39:37.994403 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:39:38.772359 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:39:39.772311 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:39:40.774394 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:39:41.772343 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:39:42.772413 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179

One active adapter + one standby (or unused, same result):

tcpdump -n -i vlan2 -eee "arp and broadcast and ether host d0:67:e5:35:d2:f2"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan2, link-type EN10MB (Ethernet), capture size 96 bytes
15:41:26.226789 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:41:26.226827 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:41:26.768375 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:41:26.768394 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:41:27.768465 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:41:27.768489 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:41:28.769545 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:41:28.769571 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:41:29.767538 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179
15:41:29.767557 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.61 tell 10.0.0.179

Of course, pinging an existant IP address with one adapter:

tcpdump -n -i vlan2 -eee "arp and broadcast and ether host d0:67:e5:35:d2:f2"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan2, link-type EN10MB (Ethernet), capture size 96 bytes
15:45:20.858072 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.215 tell 10.0.0.179

Existant IP with one adapter + one standby:

tcpdump -n -i vlan2 -eee "arp and broadcast and ether host d0:67:e5:35:d2:f2"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan2, link-type EN10MB (Ethernet), capture size 96 bytes
15:42:42.807105 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.4 tell 10.0.0.179
15:42:42.807161 d0:67:e5:35:d2:f2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.0.4 tell 10.0.0.179

3. as before, standby or unused give back same result. Both active still duplicated packets (but this is right behaviour), swapping standby and active still duplicated, load balancing to port-ID same as well.

[MKguy]

>Both active still duplicated packets (but this is right behavior)

No, it's actually still wrong. As mentioned there is only one physical Uplink used by each vNIC. Frames arriving at any other Uplink are simply not forwarded (or shall I say should not?). See this post for example which demonstrates how to find the currently used one: http://v-front.blogspot.de/2011/05/network-troubleshooting-part-i-what.html

I would still be interested in checking a capture of the actual ICMP packets when pinging the broadcast address from outside the VM (ping -c1 10.0.0.255 -b or whatever your broadcast address is).

But this really seems odd. I have no idea why you would see duplicate broadcasts like this in your case.

[tarkus72]

Finally, the problem was the vswitch that allows the promiscuos mode for a VM that need it, then the default settings of hosts allows duplicated incoming packets; the solution is setting this parameter in the host Advanced Settings (from Configuration/Software):

Net.ReversePathFwdCheckPromisc = 1

Or with the CLI command:

esxcli system settings advanced set -o /Net/ReversePathFwdCheckPromisc -i 1