ESXi 8.0U2 Wont Work with Mellanox Connectx-5

[TBB1]

Recently installed ESXi 8.0U2 on to a SuperMicro server.  The server has a Connectx-5 (MCX556A-ECAT ConnectX-5 VPI Adapter Card EDR-100GbE), this NIC is only shown when the command `lspci` is ran, but no where else. 

As shown here https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=45950 this card should be compatible out of the box.  The nmlx5_core drivers are all shown when checking the software VIB's installed.  

I did attempt to install the Mellanox firmware tools (MFT) onto the ESXi server, install works fine, starts fine, but when running `mst status` The output below is shown:

No MST devices were found or MST modules are not loaded.
You may need to run 'mst start' to load MST module.

Looking for some input on this.

Thanks. 

 

 

 

[pcgeek2009]

You do not mention the type of server the card is installed in. I read a few post on other sites where Dell servers were involved and the solution was to upgrade all of the Dell Server firmware to the most recent. Then the card appeared  on version 8.0U2. Have you tried updating all of your hosts firmware?

[TBB1]

Card is installed in a SuperMicro server.  Should be running the latest firmware already.  Planned on seeing if I could burn new firmware to the card via ESXi using the Mellanox MST tool, but no luck finding the device. 

[TBB1]

Updated SuperMicro firmware as well, no luck.

[pcgeek2009]

This is on the support page for 7, but wonder if you could be seeing this in 8? Might not hurt to check. 

Description: Adapter cards that come with a pre-configured link type as InfiniBand cannot be detected by the driver and cannot be seen by MFT tools. Thus its link type cannot be changed.

Workaround: Unload the driver.   unload nmlx5_core module   Make the device visible to MFT by loading the driver in a recovery mode.   vmkload_mod nmlx5_core mst_recovery=1 kill the devmgr   Check the device available on your machine.   /opt/mellanox/bin/mst status   Change the link type to Ethernet using MFT.   opt/mellanox/bin/mlxconfig -d mt4115_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2   Power Cycle the host.

Keywords: Link type, InfiniBand, MFT

Adapter Cards / DPU: ConnectX-4 Onwards HCAs

[TBB1]

That seems to work, steps are slightly different for ESXi 8.  But does not solve the problem. 

I am trying to use the infiniband.  I found this document which shows how to set up a connectx-5 with infiniband on ESXi 7.  This also did not work.  The NIC is not shown and SR-IOV shows "enabled/needs reboot" no matter how many reboots there are. 

[Raudi]

Today i had the same with a "Mellanox Technologies ConnectX-4 VPI NIC; EDR/100GbE; single-port QSFP28; (MCX455A-ECA)"

It wasn't shown as vmnic and and physical adapter in the GUI. Aafter doing this, all was working fine:

First download and install the NVIDIA Firmware Tools (MFT) on the host, here it is importat that both parts must be installed (MFT-Tools & NATIVE-NMST).

Then after a reboot the card can be configured, but first use that commands:

esxcfg-module -u nmlx5_core
vmkload_mod nmlx5_core mst_recovery=1
/opt/mellanox/bin/mst stop
/opt/mellanox/bin/mst start

Now the card should be listed with this command:

opt/mellanox/bin/mst status

This command shows the current config:

opt/mellanox/bin/mlxconfig -d mt4115_pciconf1 query

"LINK_TYPE_P1" was in my case set to "IB(1)"

This command changes from Infiniband to Ethernet:

opt/mellanox/bin/mlxconfig -d mt4115_pciconf1 set LINK_TYPE_P1=2

And after a power cycle the NIC was shown in the GUI and was normal useable and is working.