I working on deploying Nexus 1000 on our ESX hosts while migrating to Emulex 10Gig Cards. Its been interseting to say the least but we almost have it working after firmware and dirver issues. Currently i'm seeing a odd behavior that I've been able to recreate. Normally I have two 10 gig ports added to the ESX and added to the distributed virtual switch (DVS) and a virtual port-channel created between the Nexus 5000's they are connected. As as part of my trouble shooting i've removed one of the nics from the DVS uplink. this creates a behavior i can recreate. While the remaining nic is the only nic in the port-channel, and the only nic connected to the DVS.
This causes the nexus 1000 to reset its link with the nexus 5000 every 1minute like clockwork. it will be down for 30 secs, work for 30 secs, then repeat.
heres an example of the log from the nexus 1000.
2010 Dec 7 12:12:07 ac02-ns1-01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet6/3 is down (Initializing)
2010 Dec 7 12:12:07 ac02-ns1-01 %ETHPORT-5-SPEED: Interface port-channel4, operational speed changed to 10 Gbps
2010 Dec 7 12:12:07 ac02-ns1-01 %ETHPORT-5-IF_DUPLEX: Interface port-channel4, operational duplex mode changed to Full
2010 Dec 7 12:12:07 ac02-ns1-01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel4, operational Receive Flow Contol state changed to on
2010 Dec 7 12:12:07 ac02-ns1-01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel4, operational Transmit Flow Contol state changed to on
2010 Dec 7 12:12:40 ac02-ns1-01 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet6/3 is operationally individual
2010 Dec 7 12:12:40 ac02-ns1-01 %ETHPORT-5-IF_UP: Interface Ethernet6/3 is up in mode trunk
2010 Dec 7 12:13:07 ac02-ns1-01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet6/3 is down (Initializing)
2010 Dec 7 12:13:08 ac02-ns1-01 %ETHPORT-5-SPEED: Interface port-channel4, operational speed changed to 10 Gbps
2010 Dec 7 12:13:08 ac02-ns1-01 %ETHPORT-5-IF_DUPLEX: Interface port-channel4, operational duplex mode changed to Full
2010 Dec 7 12:13:08 ac02-ns1-01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel4, operational Receive Flow Contol state changed to on
2010 Dec 7 12:13:08 ac02-ns1-01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel4, operational Transmit Flow Contol state changed to on
2010 Dec 7 12:13:44 ac02-ns1-01 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet6/3 is operationally individual
2010 Dec 7 12:13:44 ac02-ns1-01 %ETHPORT-5-IF_UP: Interface Ethernet6/3 is up in mode trunk
2010 Dec 7 12:14:08 ac02-ns1-01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet6/3 is down (Initializing)
2010 Dec 7 12:14:08 ac02-ns1-01 %ETHPORT-5-SPEED: Interface port-channel4, operational speed changed to 10 Gbps
2010 Dec 7 12:14:08 ac02-ns1-01 %ETHPORT-5-IF_DUPLEX: Interface port-channel4, operational duplex mode changed to Full
2010 Dec 7 12:14:08 ac02-ns1-01 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel4, operational Receive Flow Contol state changed to on
2010 Dec 7 12:14:08 ac02-ns1-01 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel4, operational Transmit Flow Contol state changed to on
2010 Dec 7 12:14:41 ac02-ns1-01 %ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet6/3 is operationally individual
2010 Dec 7 12:14:41 ac02-ns1-01 %ETHPORT-5-IF_UP: Interface Ethernet6/3 is up in mode trunk
and the following log from esx host.
Dec 7 11:43:02 nkuvmhost9 vmkernel:
Dec 7 11:43:30 nkuvmhost9 vmkernel: 3:21:34:09.172 cpu8:4531)Need to send MAC Move for Inband Port
Dec 7 11:43:30 nkuvmhost9 vmkernel:
Dec 7 11:44:02 nkuvmhost9 vmkernel: 3:21:34:41.376 cpu3:4319)Not removing sys vlan 60 from the ltl 18
Dec 7 11:44:02 nkuvmhost9 vmkernel:
Dec 7 11:44:02 nkuvmhost9 vmkernel: 3:21:34:41.376 cpu3:4319)Not removing sys vlan 70 from the ltl 18
Dec 7 11:44:02 nkuvmhost9 vmkernel:
Dec 7 11:44:02 nkuvmhost9 vmkernel: 3:21:34:41.376 cpu3:4319)Not removing sys vlan 200 from the ltl 18
Dec 7 11:44:02 nkuvmhost9 vmkernel:
Dec 7 11:44:02 nkuvmhost9 vmkernel: 3:21:34:41.376 cpu3:4319)Not removing sys vlan 268 from the ltl 18
Dec 7 11:44:02 nkuvmhost9 vmkernel:
Dec 7 11:44:02 nkuvmhost9 vmkernel: 3:21:34:41.376 cpu3:4319)Not removing sys vlan 274 from the ltl 18
Dec 7 11:44:02 nkuvmhost9 vmkernel:
Dec 7 11:44:02 nkuvmhost9 vmkernel: 3:21:34:41.376 cpu3:4319)Not removing sys vlan 275 from the ltl 18
Dec 7 11:44:02 nkuvmhost9 vmkernel:
Dec 7 11:44:31 nkuvmhost9 vmkernel: 3:21:35:10.172 cpu8:4104)Need to send MAC Move for Inband Port
Dec 7 11:44:31 nkuvmhost9 vmkernel:
Dec 7 11:45:02 nkuvmhost9 vmkernel: 3:21:35:41.376 cpu8:4319)Not removing sys vlan 60 from the ltl 18
Dec 7 11:45:02 nkuvmhost9 vmkernel:
Dec 7 11:45:02 nkuvmhost9 vmkernel: 3:21:35:41.376 cpu8:4319)Not removing sys vlan 70 from the ltl 18
Dec 7 11:45:02 nkuvmhost9 vmkernel:
Dec 7 11:45:02 nkuvmhost9 vmkernel: 3:21:35:41.376 cpu8:4319)Not removing sys vlan 200 from the ltl 18
Dec 7 11:45:02 nkuvmhost9 vmkernel:
Dec 7 11:45:02 nkuvmhost9 vmkernel: 3:21:35:41.376 cpu8:4319)Not removing sys vlan 268 from the ltl 18
Dec 7 11:45:02 nkuvmhost9 vmkernel:
Dec 7 11:45:02 nkuvmhost9 vmkernel: 3:21:35:41.377 cpu8:4319)Not removing sys vlan 274 from the ltl 18
Dec 7 11:45:02 nkuvmhost9 vmkernel:
Dec 7 11:45:02 nkuvmhost9 vmkernel: 3:21:35:41.377 cpu8:4319)Not removing sys vlan 275 from the ltl 18
Dec 7 11:45:02 nkuvmhost9 vmkernel:
Dec 7 11:45:32 nkuvmhost9 vmkernel: 3:21:36:11.172 cpu8:4531)Need to send MAC Move for Inband Port
Check my Blog for other Nexus issues I've been solving along the way.
www.ChrisTowles.com