I've been installing a number of the 100Gbit Mellanox Cards, they're on the supported list, proper firmware, I've done about 60 in the last two weeks but I've been running into the below situation:
Ports are up, lights showing however no "vmnics" shown
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 15492: vmnic1: clear flags 0x51822 DEVICE_OPEN_FAILED
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 15493: vmnic1: set flags 0x51826 DEVICE_OPENED
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 12471: vmnic1: set flags 0x51a26 DEVICE_SCHED_CONNECTED
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 13003: The default queue id for vmnic1 is 0x3c4000.
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)VMKAPIMOD: 110: failed to query port alias for Not found:
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 13015: enabled port 0x82000005 with mac b8:ce:f4:21:22:33
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 3122: vmnic1, fast switch on uplink(0x82000005) connected to PPS is not supported
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 15683: vmnic1: clear flags 0x51a0e DEVICE_OPENING
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_INF> nmlx5_core: vmnic1: nmlx5_en_UplinkStateSet - (nmlx5_core_en_uplink.c:4806) change state <NONE> to <ENABLED,PROMISC,BROADCAST,MULTICAST>
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_INF> nmlx5_core: vmnic1: nmlx5_en_UplinkStateSet - (nmlx5_core_en_uplink.c:4857) current state is <ENABLED,PROMISC,BROADCAST,MULTICAST>
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_INF> nmlx5_core: vmnic1: nmlx5_en_UplinkStartIOLocked - (nmlx5_core_en_main.c:2487) called
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_INF> nmlx5_core: vmnic1: nmlx5_en_SetDevPortMtu - (nmlx5_core_en_main.c:2018) Port MTU (-22) is different than configured MTU (1500)
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_INF> nmlx5_core: vmnic1: nmlx5_en_OpenChannel - (nmlx5_core_en_channel.c:1977) called for channel idx 0
2024-04-17T20:00:46.370Z Wa(180) vmkwarning: cpu0:2097533)WARNING: <NMLX_WRN> nmlx5_CoreCreateCq:110 command failed: IO was aborted
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_ERR> nmlx5_core: vmnic1: nmlx5_en_OpenCq - (nmlx5_core_en_channel.c:1903) nmlx5_en_EnableCq failed: IO was aborted
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_ERR> nmlx5_core: vmnic1: nmlx5_en_OpenChannel - (nmlx5_core_en_channel.c:2052) done, status: IO was aborted
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_ERR> nmlx5_core: vmnic1: nmlx5_en_UplinkStartIOLocked - (nmlx5_core_en_main.c:2512) nmlx5_en_UplinkOpenChannels failed: IO was aborted
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_ERR> nmlx5_core: vmnic1: nmlx5_en_UplinkStartIOLocked - (nmlx5_core_en_main.c:2620) done, status: IO was aborted
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_ERR> nmlx5_core: vmnic1: nmlx5_en_UplinkStartIO - (nmlx5_core_en_main.c:2639) done, status: IO was aborted
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 14858: vmnic1: set flags 0x51a4e DEVICE_CLOSING
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)NetPort: 1879: disabled port 0x82000005
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 12382: vmnic1: clear flags 0x5184e DEVICE_SCHED_CONNECTED
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu67:2098689)NetSched: 724: vmnic1-0-tx: worldID = 2098689 exits
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 14717: vmnic1: clear flags 0x51842 DEVICE_OPENED
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_INF> nmlx5_core: vmnic1: nmlx5_en_UplinkQuiesceIOLocked - (nmlx5_core_en_main.c:2653) called
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 14862: vmnic1: clear flags 0x51802 DEVICE_CLOSING
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 14874: vmnic1: clear flags 0x51800 DEVICE_PRESENT
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Device: 637: uplink_drv:driver->ops.startDevice:0 ms
2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Device: 642: Device 0x4b9a430be3e257d3 start from driver uplink_drv failed with bad00e0