Broadcast traffic from unknow mac address on ESX host

[yves]

Hello,

I have a question and I am hoping someone can help me.

The networkteam has asked me to investigate some broadcast traffic from a 00:50 mac address via a vmnic of a esx host to address ff:ff:ff:ff:ff:ff.

The problem is that I can't find this virtual mac address to trouble shoot.

I want to know what is causing this broadcast traffic and why.

It isn't the address of any of the vswif, the vmk ports or vm's

I was woundering if there was any layer 2 traffic in a vsphere environment between hidden mac addressen.

I am using a full blown vsphere 4.1 cluster (ha, drs, srm, nic teaming, ...)

Thanks

[rickardnobel]

What amounts of broadcast traffic do you see? Is it causing trouble on the network or is it of curiosity?

[yves]

Yes, it is quite some traffic.  It is not problematic but switches are running at 70%.

[rickardnobel]

yves wrote:

Yes, it is quite some traffic.  It is not problematic but switches are running at 70%.

You get 70% broadcasts from the hosts - that is really a lot and something that should be fixed. Are you able to see if this traffic comes from one ESX hosts or several?

Is it just one unknown MAC address or different?

[kristofdegraen]

Yves' colleague here:

I've investigated the problem further, but I don't seem to be able to trace back this MAC address.

Our networking team traced back the MAC address to a specific physical NIC.

This NIC is attached to a dvSwitch used only by VM's (so no management or vmkernel interfaces used).

On this dvSwitch I cannot find the MAC address we are looking for.

The networking team is certain that this MAC address is broadcasting trafic, and that it is coming from that ESX host.

When inspecting the packets we see that it is indeed vmware related. (for example, we can read the text vmnic2 in the packet).

I  was wondering... could it be possible this MAC address is used in any  way by the dvSwitch (or the internally created vSwitch on the host)?.

[rickardnobel]

Do you know the setting of Beaconing on your vSwitch? That could cause some broadcast packets to be sent, but not this amount.

Another type of packet that should contain the term "vmnic2" is CDP or LLDP, do you know if either of these are enabled?

[kristofdegraen]

On our dvSwitch we have activated Beacon probing for every portgroup.

CDP is also enabled on the vSwitch in listening mode.

What's bothering me however is the MAC address that's untraceable. Any ideas on why we see this specific MAC address sending broadcast traffic?

Is it an internal MAC address of the vSwitch port? Is this possible?

Actually... when thinking about it... if it would be beaconing that is doing the broadcast, we would see the MAC of the pnic on the network switches...

The MAC address we are seeing is definitely one in the 00:50:56 range... so a virtual one...

For CDP i don't know...

[Shakaal]

Hi,

data is coming from some virtual machine  or could be a vmkernel or service console.

do one thing to drill down further

as packets are showing vmnic2.

go to each host, select "Network Adapters" under "Configuration" tab, in the right hand pane you will see all the vmnics. check the vswitch to which vmnic2 is connected. once you know the vSwitch, now go to "Networking" under "Configuration" tab. check for the vSwitch which was diagnosed in above step

now look for all the VM's conencted to this vSwitch, check their Mac address and see if matches the one that you are getting

This has to be done on all the ESX hosts

Regards

[orthohin]

To trace the traffic where come from you have to use a third party tool. You can install wireshark then you can find out the traffic source and destination mac address and IP address. Try this may help.

Regards,
Milton

[kristofdegraen]

We allready traced the trafic on the physical switches.

We know the source and destination MAC addresses. However I cannot find the source MAC address on the ESX host that the traffic should be originating from...

[kristofdegraen]

Hi,

We allready verified these steps...

We know what ESX host the traffic is coming from...

We also know the pnic... thus we know the vSwitch it is connected to.

It's a distributed vSwitch... so we can easily search the MAC address, and it cannot be found.

On this distributed vSwitch we don't have any management portgroups defined (so no service console, vmkernel...).

I've also double checked each VM on this host, and the MAC address we see on the physical switches cannot be found.

[kristofdegraen]

We're also researching an alternative path.

It could be possible the MAC address was changed within the guest OS. Would you then still see the original MAC in ESX, the one that's contained in the vmx file?

That might explain our problem...

[kristofdegraen]

The alternative path didn't work out.

No VM's had their MAC addresses changed within the guest OS on that ESX host.

[J1mbo]

Forgive me for suggesting the obvious, but I'd vMotion the VMs off the host one-by-one and see with which it follows (that is, if it follows).  Once no VMs are left obviously that only leaves the host itself.

[kristofdegraen]

FYI

This question has been answered by vmware support.

It turned out to be the distributed vSwith internal MAC address. In the esx.conf file we where able to retrieve the internal MAC address of the dvSwitch.

It was indeed some Beacon Probing packets we've been seeing.