Broken FCoE FIP Solicitation?

[Richard May]

Got a strange one here.

• Blades running ESXi 8.0U2 with QLogic/Marvell FastLinq CNA cards in a Dell MX7000 chassis.
• Chassis has FC uplinks to a group of Brocades.
• Chassis interconnects are Dell MX9116n in SmartFabric mode.
• These Fastlinq CNAs are in NPar mode with the default four partitions per port. The second partition is in its FCoE mode. No iSCSI, no iSER, no RoCE, no iWarp, no SR-IOV. Pretty basic setup here.
• qedf: version:2.74.1.0
• 8.65 firmware in the CNA.

Some blades can connect to storage; others show their vmhbas as "offline."  Note that incorrect zoning on the Brocades does not prevent a vmhba from coming online -- it just won't see anything. Regardless the zoning is fine -- we've checked twice.

Set qedf_debug_level and ql_fcoe_debug_level to 1 and rebooted. Found some interesting lines in the kernel log:

• vmkernel: cpu11:2098143)ql_fcoe:vmhba5:SendFCoEVlanSolicitation:1477:Info: Sending FCoEVlanSolicitation request (0x1)
• vmkernel: cpu11:2098143)ql_fcoe:vmhba5:SendFCoEVlanSolicitation:1477:Info: Sending FCoEVlanSolicitation request (0x2)
• vmkernel: cpu11:2098143)ql_fcoe:vmhba5:SendFCoEVlanSolicitation:1477:Info: Sending FCoEVlanSolicitation request (0x3)
• vmkwarning: cpu6:2098143)WARNING: ql_fcoe:vmhba5:FipVlanTimeoutWork:254: FIP VLAN Max Retries reached, cur vlan and pri: 1002

Our storage VLANs in the chassis fabric are 3900/3901.  99% sure VLAN 1002 is a fallback hard-coded into qedf.

Later we see:

• vmkernel: cpu6:2098143)ql_fcoe:vmhba5:SendFCoEDiscoverySolicitation:1204:Info: Sending FIP discovery for vlan_id = 0x3ea (0x4315eb593ac0

0x3ea is 1002, fyi. This repeats several times then:

• vmkwarning: cpu6:2098143)WARNING: ql_fcoe:vmhba5:FipDiscoveryTimeoutWork:125: Max retry exhausted for Fabric = 0x4315eb593ac0 vlan_id = 0x3ea
• vmkernel: cpu6:2098143)ql_fcoe:vmhba5:StartPortLogout:1236:Info: Sess 0x4315eb594c50 port_id fffffe

So FCoE VLAN Solicitation isn't working. Until it does:

• vmkernel: cpu71:2097919)ql_fcoe:vmhba4:ql_fcoe_parse_vlan_response:1319:Info: Received VLAN response: VLAN ID: 3900
• vmkernel: cpu71:2097919)ql_fcoe:vmhba4:CreateFabric:112:Info: Fabric created: 0x4315bd76ca80
• vmkernel: cpu71:2097919)ql_fcoe:vmhba4:SendFCoEDiscoverySolicitation:1204:Info: Sending FIP discovery for vlan_id = 0xf3c (0x4315bd76ca80
• vmkernel: cpu71:2097919)ql_fcoe:vmhba5:ql_fcoe_parse_disc_resp:743:Info: Discovered FCF 20:04:0f:6d:07:8a Fabric 0x4315bd76bd60
• vmkernel: cpu71:2097919)ql_fcoe:vmhba4:ChangeSessionStateInternal:920:Info: PRLI done, FcpSess = 0x4315bd7870a0
• vmkernel: cpu71:2097919)ql_fcoe:vmhba5:FabricLoginCompletion:1378:Info: FabricLoginCompletion: Fabric 0x4315bd76bd60 FLOGI/FDISC status Success

Or maybe it doesn't:

• vmkernel: cpu10:2098164)ql_fcoe:vmhba5:FailNpivOwnerProc:4865:Info: FailNpivOwnerProc: NPIV owning Fabric 0x4315bd76bd60 has failed

Usually it's the first case (SendFCoEVlanSolicitation fails).  Every once in a while it's the second (SendFCoEVlanSolicitation succeeds, PLOGI/FLOGI succeeds, then the connection drops less than a minute later).

Anyone seen this before?

 

 

 

[Jason Holubiw]

Having the same issue with same hardware, only we're running vsphere 7.0u3

Did you find any resolution?  Not even sure which vendor (Dell/VMware) to point the finger at...

[Richard May]

No fix. I've been waiting for new hardware to arrive so I can shift away the workload (from the blades that are still working fine) and dig into this. The affected Dell hardware is out of warranty so I haven't bothered calling them. VMware support has been so universally bad that I have no interest in fighting with them.

I'd like to start by rebooting the switches but I'm afraid the working blades will stop working then I'll have a mess on my hands. What troubleshooting steps have been taken on your side?

[Jason Holubiw]

In our case, all the blades in the dell chassis have the issue, so we have rebooted everything to no avail.
Focusing on the FW versions right  now and trying to compare to other working dell chassis we have.

 

[Richard May]

Keep me posted, please. I'm hoping to find a solution in the next two weeks or so.

[Jason Holubiw]

We found the issue was because we had LLDP enabled on our distributed switch.  The Dell MX fabric is depending on LLDP and having it enabled on the dswitch was interfering with the FCOE.

https://www.dell.com/support/manuals/en-ae/dell-emc-smartfabric-os10/smartfabric-os-user-guide-10-5-0/dcbx-configuration-notes?guid=guid-fdfd8033-1e6c-4f65-b896-9cf17e9adc13&lang=en-us

• DCBX uses LLDP to advertise and automatically negotiate the administrative state and PFC or ETS configuration with directly connected DCB peers. DCBX cannot run if LLDP is disabled on an interface. Enable LLDP on all the DCBX port. 

[Ahsan Fareed]

Hye Richard,, Thanks for sharing your detailed post. I'm curious-were you able to resolve this FCoE FIP Solicitation issue eventually? If yes, could you share what fixed it and any configuration changes that were required?

 

 

For context, here's my environment and the issue I'm seeing:

 

ESXi Version: 8.0 U3

CNA Cards: QLogic/Marvell FastLinQ 

Chassis: Dell MX7000 with MX9116n interconnects in SmartFabric mode

FC Uplinks: Brocade switches

Observed Issue: vmhba4 go offline, FCoE VLAN solicitation repeatedly fails

there ESXI logs are saying this .. 

 

 

vmkernel: cpu77:2098461) ql_fcoe: vmhba4: CreateFabric: 112: Info: Fabric created: 0x431436d71d60

vmkernel: cpu77:2098461) ql_fcoe: vmhba4: SendFCoEDiscoverySolicitation: 1204: Info: Sending FIP discovery for vlan_id = 0x3ea (0x431436d71d60)

vmkwarning: cpu77:2098461) WARNING: ql_fcoe: vmhba4: FipDiscoveryTimeoutWork:125: Max retry exhausted for Fabric = 0x431436d71d60 vlan id = 0x3ea

vmkernel: cpu77:2098461) ql_fcoe: vmhba4: StartPortLogout: 1236: Info: Sess 0x431436d7e1d0 port id fffffe

vmkernel: cpu76:2098463) ql_fcoe: vmhba4: DeleteFabric:123: Info: Fabric 0x431436d71d60 destroyed

vmkernel: cpu76:2098463) ql_fcoe: vmhba4: SendFCoEVlanSolicitation: 1477: Info: Sending FCoEVlanSolicitation request (0x1)

vmkernel: cpu77:2098461) ql_fcoe: vmhba4: SendFCoEVlanSolicitation: 1477: Info: Sending FCoEVlanSolicitation request (0x2)

vmkernel: cpu77:2098461) ql_fcoe: vmhba4: SendFCoEVlanSolicitation: 1477: Info: Sending FCoEVlanSolicitation request (0x3)

vmkwarning: cpu77:2098461) WARNING: ql_fcoe: vmhba4: FipVlanTimeoutWork: 254: FIP VLAN Max Retries reached, cur vlan and pri: 1002

-------------------------------------------