Driver at VMware site and Firmware at Dell site ;)

[lucasitteam]

Hallo ,

I'm curious know this. I was bit hesitant to ask this but few colleagues is office thinks I'm did not follow the right procedure and blame gaming is starting.

We have Dell R730 Server. It has both 10 Gbps and 25 Gbps NICs.

Initially only 10 Gbps card was used but then we decided to move the migrate to 25 Gbps. So our network team patched the cables and I inserted in all ESXi hosts.

But the links showed down. After some troubleshooting we though that Network card drivers must be updated.

I went to dell site and searched for the driver. Driver was not there. There is only firmware. So I download the firmware, update the firmware and the Link was up.

So I repeated it on all 10 servers.

After one week, 4 ESXi servers lost network connectivity, vSphere HA got triggered and shutdown the VMs. Issue got escalated big time. We asked VMware Support. GSS  concluded the Network card was a problem. We went to dell. Dell asked me, why did you not upgraded the driver. I asked where is the driver. He said driver is owned by VMware. VMware says they do not own neither firmware or driver.

My question - When the card is owned by Dell, am I wrong when I went to Dell site to update the firmware or you think I should first go to Vmware site , then update the driver and then firmware.

[a_p_]

What I usually do is to verify that a hardware component (e.g. a network adapter) is supported by VMware.  The VMware HCL also lists the required firmware, and driver version (combination).

The required firmware can usually be downloaded from the vendor's web site, as it's not specific to vSphere. The vSphere drivers - no matter whether these are VMware's native inbox drivers, or vendor "async" drivers, can usually be downloaded from a common place on VMware's web site.

André

[lucasitteam]

Andre, you mentioned it right. When I upgrade the firmware. The firmware and driver combination was not matching as per the HCL. But then are you saying driver is owned by VMware.

It is only when I upgrade my firmware, 25 Gbps NICs are seen and I did not felt the need to upgrade the driver because driver was already on HCL

What is does vendor async driver means?

[IRIX201110141]

I have deployed a couple of 25G NIC based Dell Hosts in the past.  Based on my experience with hundreds of different Intel/Broadcom Hosts  i can say today its more complicated than in it was in the past.  You have to check the certified driver in combination with the right Firmware.  Speaking for Dell the FW comes always from dell.com/support and the driver always from vmware.com. But it doesnt mean that the latest FW is the one which works best or is certified.

So you need to find out the PCIID/Vendor/SubID for your given device and than go the the VMware HCL and search for the device ID (the search fields in the upper right). I always picks the IDs from the TSR which is a report i create trough the iDRAC -> Troubleshooting/SupportAssist. The TSR contains a HTML based report with detailed information.

The HCL than returns a list of all testet combinations of FW and Driver.  A async driver means that this driver is updated and provided not trough the vanilla VMware ESXi iso. It needs to manually download from vmware.com. It can be included in the $Vendor ESXi ISO which Dell and others provide.

Regards,
Joerg

[vGuy]

It's complicated for Dell HW. Dell does not provide a central location for the compatible driver and firmware combination. HPE does a good job in publishing software recipe document for each new vSphere update (major and minor) and HPE SPP released. Your best bet for Dell would be to use the latest Firmware repository and latest Dell supplied ESXi image available at that time.

[IRIX201110141]

Your best bet for Dell would be to use the latest Firmware repository and latest Dell supplied ESXi image available at that time

No. Because exactly that cause problems. The FW release cycle have nothing to do with the the Custom ISO and the latest FW  is often not certified.

Regards,
Joerg

[lucasitteam]

Joerg,

you are right. I have installed ESXi6.7 from Dell ISO. And now Driver and firmware combination is not matching.

In fact the firmware number do not even match with the firmware reported by Dell. This is worst thing.

It gives me a strong feeling driver is owned by VMware. This also explains why Driver is not there on Dell site irrespective it is async or not.

NOTE: For recommendations on latest combination of supported driver and firmware refer to OEM vendor matrix.

VMware Knowledge Base

But the driver is not here on Dell site. This sound funny. VMware Support suggests we are not responsible for Driver and Dell says we only offer firmware. This is chicken and egg problem but at the cost of customer money.

[IRIX201110141]

In fact the firmware number do not even match with the firmware reported by Dell. This is worst thing.

Well, i understand but its not knew (at least for me because i have to deal with it on a daily basis).  The repository (catalog) is only updated every few weeks/month but new FW releaes comes every "day" and are visible under the Server Model or better when using your Service Tag of your device for search on dell.com/support.

This always bother me because we use OMIVV (OpenManage Integration for VMware vCenter) for FW upgrades.

Please tell me what type of NIC you have or better pull out the  IDs as describe aboved.

Regards,
Joerg

[lucasitteam]

I spend my time i finding 214.4.32.0. My GSS engineer was trying to give super explanation how they have moved the dots and took no responsibility about this data. Even though it is on their website. This firmware version is not there on Vmware website. This is on dell website and you see only on iDRAC on via esxcli.

To me . Dell says "Hello my name is Firmware" and VMware says "Hallo, ich bin Ware"

As an engineer where should I read this and how do i make sense of this. Should I always burn fingers and create resume generating events because someone choose to play blame games.

[IRIX201110141]

Please provide the PCIID/VID/SSID of your NIC.

Regards,
Joerg

[lucasitteam]

Sorry Joerg, I though I gave you above. I was wrong.

0000:b4:00.0 14e4:16d7 14e4:4141  vmnic4

0000:b4:00.1 14e4:16d7 14e4:4141  vmnic5

By the way I do not see a place to "iDRAC -> Troubleshooting/SupportAssist." to get this information. Though it is not important.

This thread only tell me - I did nothing wrong. It is how the world is structure. I have to be careful in future.

Thanks Joerg and vGuy

Hope others learn lesson from this and do not end up in Resume generating events.

[IRIX201110141]

About the TSR.

iDRAC -> Maintanence->Support Assist.  Cancel the Registration and create a collection. You will get a progress bar at the top and after a couple of  minutes a download.

About the IDs.. VMWare HCL asks for 4 IDs and iam unsure whats your 14e4:16d7 14e4:4141 is.

[IRIX201110141]

Ok the famous 57414

VMware Compatibility Guide - I/O Device Search

Edit: With your current FW you should use https://my.vmware.com/en/web/vmware/details?downloadGroup=DT-ESXI67-BROADCOM-BNXTNET-2160424&productId=742

Regards,
Joerg

[vGuy]

I doubt VMware is testing all the driver and firmware version combinations for each IO device. Even the note on HCL states "Firmware versions listed are the minimum supported versions". Moreover, the HCL does not always list the FW version of lot of ethernet controllers (refer the attachment).

By relying on Dell FW ISO, we are not completely in uncharted waters. As the expectation is Dell has tested and certified the Firmwares for the operating systems listed in their support release document.

Again this is not an ideal way and Dell should seriously consider publishing a supported driver and firmware matrix with each (point) update to ESXi or Firmware repository.

[IRIX201110141]

That "Firmware versions listed are the minimum supported versions". is a standard disclaimer.

Yes its true that often drivers are listed without the FW version. Keep also notice than 2 FW columns exists. But as we can see that in the terms of a "async driver" the HCL provides the URL for downloading the right driver from vmware.com. There is also the Driver&Tools section within the ESXi download area which often provides driver packages.

What drives me nuts is that Broadcom sold some models to QLogic and now there is a name mismatch in ordering documents/systeminventory which ends that the software packages sometime have different brand names as the HW.

Regards,
Joerg

[lucasitteam]

Just for others to take lessons from this.

Joreg you also mentioned this.

I have built the ESXi 6.7 host using Dell ISO

Green line -Driver version, Yellow line Firmware version.

By default/using ISO CD combination is not matching.

Joreg, I have already have to report. Thanks a lot. Hope others learn from this thread.