HPE DL380Gen10 + FC adapter SN1200E (lpfc) = PSOD

[pirx2020]

Hi,

we got a bunch of new DL380Gen10 that are directly connected to IBM V3700 storage (vSphere 6.7 U3 + latest updates, HPE image). The servers are equipped with StoreFabric SN1200E adapter (OEM of Broadcom LPe31000-M6). First we ran into a problem where no LUN's were visible, I could find out that this was due to a driver version before lpfc 12.8.542.25. We are now using FW 12.8.542.32 and lpfc driver 12.8.614.2 (from Broadcom downloads, as HPE told us to use upstream versions, not HPE SPP etc version). Now LUN's / datastores are accessible. 

But I still get a PSOD when rebooting/shutting down the server. There is something about lpfc driver in the trace, but as this happens at the very end of a reboot, no dump is written to local scratch location or netdump. Sometimes the server only hangs with "Shutting down device drivers...." forever. If either of the two happens monitoring is sending out an alert that one or both adapter ports are down (NicAllLinksDown Event from ILO). This does not happen during reboot when there is no PSOD. I tested different firmware / driver versions, it seems to make no difference. It also happend with no storage connected at all - disconnected FC cables.

My VMware case was closed after some time now, as VMware is pointing at HPE as the manufacturer. HPE is pointing at IBM and IBM's SSIC matrix (which contains no Proliant server for V3700 storage + vSphere). As well as Emulex as the manufacturer of the adapter.

 

Does anyone have an idea from looking at the PSOD what the problems might be? Any lpfc driver parameter to try out?

 

 

[E_Espinel]

Hello.
The first thing is that in the IBM compatibility matrix, there are 23 HBAs, but none is the LPe31000-M6 or equivalent in P/N IBM or Lenovo.
Attached link

https://www-03.ibm.com/systems/support/storage/ssic/interoperability.wss#

The minimum recommended firmware level for the V3700 is 7.8.X.

What Firmware version do you have on the IBM V3700?
You could try to raise the firmware level of the V3700 one level at a time to see how it behaves if it is too low. Maybe you will get lucky and it will work out.

Ideally you would use any of the 23 HBAs that are certified for an X86 Server connected directly to an IBM V3700 and supported by your HP DL380Gen10.

The  LPe31000-M6 HBA is supported by VMware for versions 7.0 and 6.7.

Another option to test would be to set the connection to 8Gb on both the HBA and the IBM V3700 ports.

 

 

[E_Espinel]

Hello.
The Qlogic QLE2660 and QLE2662 HBAs that are supported on the IBM V3700 nad VMware version 6.7, have their equivalents on HP attached details.

http://sup.xenya.si/sup/info/qlogic/HP_QLogicProductPortfolio.pdf

Now you have to check with your HPE supplier if the indicated HBAs (HP P/N QW971A/QW972A)  are supported or can work on the HPE DL380 Gen 10 Server.

 

[pirx2020]

We are on 7.8.1.14. According to Broadcoms Interoperability Matrix 12395428 (broadcom.com) the Emulex Gen 6 16GFC LPe31000-series (same as OEM adapter SN1200 from HPE) is supported for V3700. And as this PSOD also happens without any storage connected, I don't see any reason for changing the HBA brand.

[E_Espinel]

Hello.
Please perform the following tests:
1. Remove the HBA from the server, check that the server is working properly by performing several reboots and server startups. Does the PSOD occur ?

2. Update all the firmware of the HPE DL380 Gen 10 server (must have the HBA installed) with the latest Service Pack for ProLiant (SPP) available, publish the Firmware listing. Verify that the server is working properly by performing several reboots and server startups. Does the PSOD occur ?

3. Reinstall from scratch the VMware vpshere version 6.7 on the server only with the HPE custom image for DL380 Gen 10. Verify that the server works correctly by performing several reboots and server startups. Does the PSOD occur ?

If you perform the requested tests, please post your results in this post.

About the Broadcom (Emulex) compatibility matrix that indicates that the HBA is compatible (working) with IBM V3700. In the IBM online matrix this HBA does not appear in the list of supported HBAs. Therefore there should be a case with the HBA manufacturer.