vmk0 MAC active on two switch ports

[pirx2020]

I'm having teaming issues with route based on physical NIC load mode. ESXi hosts are connected to two independent Cisco C9300 switches (IOS, not ACI!).

First question, is it expected that vmk0 (mgmt) has the MAC of the physical interface vmnic0? Other vmk interfaces for vmotion have 00:50:56 MAC's.

When I take down vmnic0 and bring it up again the connection to the server is interrupted. This only happens in LBT mode, with originating on virtual port mode it does not happen.

When this happens network team sees the vmk0/vmnic0 MAC on the connected server ports of both switches. They say this can't work and that it is a problem of ESXi.

I did not find a documentation what exactly happens in LBT mode in case of a link down event. I read somewhere that in case a VM is moved to other vmnic due to load balancing, RARP packages are sent out.

I don't see any obvious error in switch config. There is also not much that can be done in vDS config. So either my test with taking down a vmnic with esxcli is not valid, or something else must be wrong.

[root@xxxx:~] esxcli network nic list |grep -i 10000
vmnic0 0000:85:00.0 bnxtnet Up Up 10000 Full 00:62:0b:1f:29:30 1500 Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
vmnic2 0000:a2:00.0 bnxtnet Up Up 10000 Full 00:62:0b:1e:f7:30 1500 Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller


[root@xxxxx:~] esxcfg-vmknic -l |grep IPv4
vmk0 10 IPv4 172.16.83.103 255.255.255.224 172.16.83.127 00:62:0b:1f:29:30 1500 65535 true STATIC defaultTcpipStack
vmk1 18 IPv4 172.16.83.75 255.255.255.224 172.16.83.95 00:50:56:6e:d9:60 1500 65535 true STATIC defaultTcpipStack
vmk2 26 IPv4 172.16.83.76 255.255.255.224 172.16.83.95 00:50:56:61:82:23 1500 65535 true STATIC defaultTcpipStack

interface TenGigabitEthernet1/1/1
description ESXi
switchport trunk native vlan 153
switchport trunk allowed vlan 1-800
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
ip dhcp snooping trust
end

 

[memaad]

I dont see any problem in this configuration,  in my lab too it is same with no issue, as  I am using Route based on physical NIC load

[pirx2020]

That's the thing, I know it should work and I used LBT for years without thinking about it. Then we had some issues and discussion started what is supported and not. Our network guys say (after consulting Cisco)  that we should migrate to LACP. And with this new location I did now some tests and try to understand why its not working as expected.

[pirx2020]

In the end it seems to be an issue of the Broadcom NIC. There is the setting "LLDP nearest bridge" in NIC UEFI menu. When this is enabled this results in MAC address of the interface is seen on two different switch access ports and it breaks LBT.

UEFI HII Menu for Ethernet Network Adapters (broadcom.com)

I've disabled this feature and could not reproduce the issue. VMware support pointed this out and it seems I'm not the first one that has such kind of issues.

That's kind of unexpected..