ESXi hosts showing disconnected but switch shows port as up

[chuckado1]

I have an interesting issue, I have 2 esxi hosts with connect x 3 dual sfp cards in them connected via twinax to a ubiquiti 16-xg. After a few days of the hosts being up one of the ports will show as disconnected in vcenter for the hosts but will show was up on the switch. I have installed the latest supported driver vib for the card based on the HCL.  I have tried configuring aggregating the ports on the unifi side and doing lacp in vmware but the behavior came back.  I have removed this configuration since.  After rebooting a host, both ports will show as up and then after a day or two one of the ports will go down on the ESXi side. Any ideas?

[SureshKumarMuth]

  doing lacp in vmware  -- Are you creating a LAG in VDS ?

[chuckado1]

yes that is what I did but the result was the same after a few days one of the ports went down so I removed the ports from the lacp uplinks and I am just using them as regular uplinks in the VDS and the bonding has been removed from the switch config.

[SureshKumarMuth]

So the issue persists with or without LACP so it is not a config issue due to LACP.

Is the same port(vmnicX) going down everytime or it is random, what are you seeing in vmkernel logs during the time of issue, what makes the vmnic to go down ?

[chuckado1]

attached is a snippet from the vmkernel from before and after vmnic3 goes down.  I dont see anything conclusive but I am not super familiar with going through this log.

[chuckado1]

it happened again to the host so i grabbed a new snippet from the vmkernel.log.  The offending interfaces are vmnic3 and vmnic1000302 which are the two nics for the connect-x 3 card.

[SureshKumarMuth]

Before the nic went down , I can see similar stack for both the vmnics

2018-09-26T15:09:19.114Z cpu2:65589)<NMLX_INF> nmlx4_en: vmnic1000302: nmlx4_en_RxQAlloc - (partners/mlnx/nmlx4/nmlx4_en/nmlx4_en_multiq.c:628) RX queue 1 is allocated

2018-09-26T15:09:19.115Z cpu2:65589)<NMLX_INF> nmlx4_en: vmnic1000302: nmlx4_en_QueueApplyFilter - (partners/mlnx/nmlx4/nmlx4_en/nmlx4_en_multiq.c:2114) MAC RX filter (class 1) at index 0 is applied on

2018-09-26T15:09:19.115Z cpu2:65589)<NMLX_INF> nmlx4_en: vmnic1000302: nmlx4_en_QueueApplyFilter - (partners/mlnx/nmlx4/nmlx4_en/nmlx4_en_multiq.c:2121) RX ring 1, QP[0x51], Mac address 00:50:56:6c:2e:f5

2018-09-26T15:17:04.108Z cpu1:65589)<NMLX_INF> nmlx4_en: vmnic1000302: nmlx4_en_QueueRemoveFilter - (partners/mlnx/nmlx4/nmlx4_en/nmlx4_en_multiq.c:2294) MAC RX filter (class 1) at index 0 is removed from

2018-09-26T15:17:04.108Z cpu1:65589)<NMLX_INF> nmlx4_en: vmnic1000302: nmlx4_en_QueueRemoveFilter - (partners/mlnx/nmlx4/nmlx4_en/nmlx4_en_multiq.c:2301) RX ring 1, QP[0x51], Mac address 00:50:56:6c:2e:f5

2018-09-26T15:17:04.108Z cpu1:65589)<NMLX_INF> nmlx4_en: vmnic1000302: nmlx4_en_RxQFree - (partners/mlnx/nmlx4/nmlx4_en/nmlx4_en_multiq.c:789) RX queue 1 is freed

I also found another thread for the same issue but for VSAN environment , looks like it is a known issue where vendor has to be involved and check why this disruption happens

Mellanox ConnectX-3 Pro strange random network disruption with vSAN

[SureshKumarMuth]

Just noticed, you have already added a comment on the post which I referred :smileyhappy:  We have to wait and watch what vmware say about this issue.

[yoeri]

Are you using Zerto?