qlnativefc driver shows offline after update

[jeffj2000]

Without posting a myriad of logs we have looked it, basically the issue is our Qlogic FC cards show "offline" after a firmware/driver/esxi update. 

Running Dell hardware, with ESXi v7 U3s. We upgraded all Dell firmware following the Dell/ESXi catalog which at this time included iDrac/BIOS/Marvel firmware. We used lifecycle to patch the hypervisor to U3v and the Dell add-on from A25 to A26.

All ran successfully and following this the Qlogic cards all show offline. We could not pinpoint much in the logs other than the lifecycle mentioned vmkapi several times having to upgrade or something, I do not remember.

We reverted the Qlogic and BIOS firmware on the box, and booted the hypervisor Shift-R to recover the last one, which seemed successful. Still offline. 

I am perplexed why a rollback would still not be able to get our FC cards online. Did a pci map change or something? The cards were not re-enumated as they kept their names they had, vmhba5, etc.

Running Dell 760xs and Marvel 2772 which is 32Gb but on a 16Gb Cisco switch. 

Thanks in advance for any ideas we may want to try or look at. 

[ninjabrum]

You'll want to open support tickets with both Dell and Broadcom.  Dell to look at the hardware/firmware, and Broadcom to look at the hypervisor.  Firmware updates can break things if the fw version and driver version aren't compatible.  A roll back should have brought to back to where you started.  Unfortunately, without a deep dive into the hardware and hypervisor, the reasons are many.

Now you know why you should test one system before upgrading them all.

[Casseau33]

Been through similar issues in years gone past. Typically, always has to do with the driver/firmware combination that you are using. Try looking up your device on the Broadcom Compatibility IO Devices page and ensure your running the proper firmware/driver versions.

Broadcom Compatibility Guide

I know in one case I had to revert the qlnativefc driver to the native one that comes in the first edition of vSphere 7. Had to extra the image just to get the driver as it wasn't available as a standalone download

Best of luck

[jeffj2000]

This was our test system. We are not that stupid. 

We were using the matched Dell firmware/driver catalog in Lifecycle which is why this is so odd. The driver actually did not change upon further inspection between the catalogs, only the firmware which was rolled back. I will look to see if it changed anything in the qlogic bios later.

[Alexandru Capras]

Support tickets sounds good.

In the meantime I'd check the current version of the driver installed on the ESXi and see if you also have its compatible firmware installed. 
We struggled with something similar in the past after upgrading the ESXi from 6.7 to version 7 we lost our FC adapters. If I recall well it was because the ESXi version 7 driver was incompatible with the physical adapter model, so we had to install an older driver on the hosts.

[ninjabrum]

Exactly. There is a possibility of hardware failure due to firmware upgrade, or a mismatch. 

[Hlynur]

I'm posting a possible solution here for anyone else who stumbles upon this thread.

We had an identical issue with QLE2772s that shipped with firmware 16.20.10. In testing we found that every single one of them went offline when upgraded. Older identical HBAs did not have this issue.

Reverting the firmware made no difference.

The solution in our case was to downgrade the firmware to version 15.30.10. That brought the HBAs online again and allowed us to fully upgrade them to the latest firmware version.

Downgrading a working/online HBA to 15.30.10 and then to the desired version also works and does not cause the HBAs to go offline.

[Franck THOMAS]

Dear Hlynur,
I had the same issue on 8 ESXi DELL R760.
One ticket was opened to DELL support team which was not able to help me.
I confirmed that the issue is also solved on my 8 ESXi by downgrading QLogic QLE2772 firmware to version 15.30.10 then reapplied the last FW version 17.05.10 in my case.
It is not very simple to find the old firmware 15.30.10. Here is 2 links thaht can help :

https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=3mygv&oscode=ws19l&productcode=poweredge-r750

https://dl.dell.com/FOLDER08053642M/1/Fibre-Channel_Firmware_3MYGV_WN64_15.30.10_A00-00.EXE?uid=4fc7e1ae-4600-48cb-558a-adba05dbe5a0&fn=Fibre-Channel_Firmware_3MYGV_WN64_15.30.10_A00-00.EXE

Thanks again for this workaround which solve the issue for me.