Fiber targets missing after reboot

[gmitch64]

We're having an issue with a couple of our hosts with fiber targets going missing after a reboot. If we wait for 24 hours or so, 2 or 3 of the targets will reappear. If we leave it for a day or so, then, then maybe 5 or 6 targets will reappear. A day or so later, all 11 missing targets will appear. Now, on one of the hosts, we have an additional HBA installed, and running in pass through mode to a Windows 2008R2 guest. The Windows guest can see all of the targets all of the time - even when the ESXi host that it is running on cannot.

We started off at the Cisco end, since we're fairly new to Cisco, and we had TACS remote in and look at our setup. They were happy that all the LUNs were being presented to the fiber switch that the host was connected to, and that there was full connectivity, and that we could ping them over the fabric.

Since the cards are QLogic, we installed the Qlogic snapin on the vCenter Server, along with the CIMs on the hosts. Once that was done, we could see each of the hosts being presented and visible to the QLogic card - even the ones that the ESXi host was currently unable to see. At this point we thought that it was either a driver issue, or an ESXi issue.

So, we opened a call with vmware about this. And here's the issue. The missing targets are fiber tape drives, and vmware doesnt support tape drives, so they weren't interested in helping. They did say that we should talk to the partner responsible for the drivers (QLogic - which we are), but they wouldn't be able to do anything. The fact that I was not looking for support on connecting up a drive, and that I just wanted to be able to see all the targets being presented to the host, fell on deaf ears.

So... Has anyone seen an issue like this, and/or have any ideas?

Graham

[SG1234]

interesting -- can you share relevant logs from the hosts - especially when the targets disapper and also when they reappear

[gmitch64]

They disappear every time I reboot the hosts, so no real logs there. I was trying to work out what made the tape drives show back up, and, to make this more bizzare than ever, if I load a tape into one of the drives, the LUN shows backup in ESXi. If I load tapes into the 10 different drives, all 10 LUNs show up. Now, once the LUNs have shown up, they stay there, until the host is rebooted.

Now, if we reboot the server, and do a

     esxcfg-mpath -l

We get this output for one of the tape drive, when it's not being seen by ESXi

fc.20000024ff4894ae:21000024ff4894ae-fc.500308c098674010:500308c098674011-
   UID: fc.20000024ff4894ae:21000024ff4894ae-fc.500308c098674010:500308c098674011-
   Runtime Name: vmhba1:C0:T12:L0
   Device: No associated device
   Device Display Name: No associated device
   Adapter: vmhba1
   Channel: 0
   Target: 12
   LUN: 0
   Plugin: NMP
   State: dead
   Transport: fc
   Adapter Identifier: fc.20000024ff4894ae:21000024ff4894ae
   Target Identifier: fc.500308c098674010:500308c098674011
   Adapter Transport Details: Unavailable or path is unclaimed
   Target Transport Details: Unavailable or path is unclaimed

One we load a tape into the drive, and do another esxcfg-mpath, the output changes to

fc.20000024ff4894ae:21000024ff4894ae-fc.500308c098674010:500308c098674011-naa.500308c098674010
   UID: fc.20000024ff4894ae:21000024ff4894ae-fc.500308c098674010:500308c098674011-naa.500308c098674010
   Runtime Name: vmhba1:C0:T12:L0
   Device: naa.500308c098674010
   Device Display Name: IBM Fibre Channel Tape (naa.500308c098674010)
   Adapter: vmhba1
   Channel: 0
   Target: 12
   LUN: 0
   Plugin: NMP
   State: active
   Transport: fc
   Adapter Identifier: fc.20000024ff4894ae:21000024ff4894ae
   Target Identifier: fc.500308c098674010:500308c098674011
   Adapter Transport Details: WWNN: 20:00:00:24:ff:48:94:ae WWPN: 21:00:00:24:ff:48:94:ae
   Target Transport Details: WWNN: 50:03:08:c0:98:67:40:10 WWPN: 50:03:08:c0:98:67:40:11

So it does know about the tape drive, it's just not making the connection for some reason. Running a rescan gives the following

Dead path vmhba1:C0:T12:L0 for device  not removed.
Device is in use by worlds:

<repeated for each device>

I was wondering if the device isn't being claimed for some reason, but adding a new claimrule

MP            114  runtime  vendor     NMP        vendor=IBM model=ULTRIUM-TD3

and doing a rescan hasn't helped either.

Graham