Problem with missing LUNs and high iowait

[gufete]

Hello

We have 3 ESX 4.0 host (Dell R710, 2xquad core, 72 GB ram, all patches applied) connected with software iSCSI to two pairs of HP LeftHand Networks iscsi SANS (4 nodes, 1 pair of nodes with SAS 8x300GB_15K_RPM@raid5 and the other pair with SAS 8x450GB_15K_RPM@raid10). This has been running flawlessly hosting 30+ VM (RHEL 3 & RHEL 5).Switches are gigabit ethernet Cisco Catalyst. All vsphere host with the latest patches available (and I have tried serveral times "rescanning LUNs" and "rebooting hosts)

This system should give electronic email (postfixdovecotAV+antispam) to 75K users. But one (unlucky) day I made a mistake: I deleted two emty LUNs "in the wrong order" and the ESX hosts still try to connect to the old LUNs.

This is what I get if I launch "top" in one of the RHEL 5 VM:

Tasks: 120 total, 1 running, 119 sleeping, 0 stopped, 0 zombie

Cpu(s): 0.8%us, 0.8%sy, 0.0%ni, 40.0%id, 57.6%wa, 0.3%hi, 0.5%si, 0.0%st

You can see 57,6%, a very high iowait. In the ESX hosts I can see the following:

esxcfg-rescan vmhba33 # I rescan the iscsi software adapter

tail -30 /var/log/vmkernel # Let´s see the logs (in bold what I think that is "bogus")

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.507 cpu9:4118)ScsiScan: 844: Path 'vmhba33:C0:T2:L0': Vendor: 'LEFTHAND' Model: 'iSCSIDisk ' Rev: '8000'

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.507 cpu9:4118)ScsiScan: 847: Path 'vmhba33:C0:T2:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.555 cpu12:4118)ScsiScan: 844: Path 'vmhba33:C0:T7:L0': Vendor: 'LEFTHAND' Model: 'iSCSIDisk ' Rev: '8500'

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.555 cpu12:4118)ScsiScan: 847: Path 'vmhba33:C0:T7:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.556 cpu12:4118)ScsiScan: 844: Path 'vmhba33:C0:T1:L0': Vendor: 'LEFTHAND' Model: 'iSCSIDisk ' Rev: '8000'

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.556 cpu12:4118)ScsiScan: 847: Path 'vmhba33:C0:T1:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.557 cpu12:4118)ScsiScan: 844: Path 'vmhba33:C0:T6:L0': Vendor: 'LEFTHAND' Model: 'iSCSIDisk ' Rev: '8500'

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.557 cpu12:4118)ScsiScan: 847: Path 'vmhba33:C0:T6:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.564 cpu12:4118)ScsiScan: 844: Path 'vmhba33:C0:T0:L0': Vendor: 'LEFTHAND' Model: 'iSCSIDisk ' Rev: '8000'

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.564 cpu12:4118)ScsiScan: 847: Path 'vmhba33:C0:T0:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.573 cpu12:4118)ScsiScan: 844: Path 'vmhba33:C0:T5:L0': Vendor: 'LEFTHAND' Model: 'iSCSIDisk ' Rev: '8500'

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.573 cpu12:4118)ScsiScan: 847: Path 'vmhba33:C0:T5:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3: 1561: Could not open device '4be3aa32-1602d741-4477-0024e8330661' for probing: No such target on adapter

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3: 615: Could not open device '4be3aa32-1602d741-4477-0024e8330661' for volume open: No such target on adapter

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)FSS: 3702: No FS driver claimed device '4be3aa32-1602d741-4477-0024e8330661': Not supported

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3: 1561: Could not open device '4ab32747-7d85650d-15e0-0024e8330661' for probing: No such target on adapter

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3: 615: Could not open device '4ab32747-7d85650d-15e0-0024e8330661' for volume open: No such target on adapter

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)FSS: 3702: No FS driver claimed device '4ab32747-7d85650d-15e0-0024e8330661': Not supported

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3: 1561: Could not open device '4ab326d3-d9e22b4a-d076-0024e8330661' for probing: No such target on adapter

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3: 615: Could not open device '4ab326d3-d9e22b4a-d076-0024e8330661' for volume open: No such target on adapter

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)FSS: 3702: No FS driver claimed device '4ab326d3-d9e22b4a-d076-0024e8330661': Not supported

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3: 1561: Could not open device '4a8d3eb1-b428a951-d1eb-0024e8336099' for probing: No such target on adapter

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3: 615: Could not open device '4a8d3eb1-b428a951-d1eb-0024e8336099' for volume open: No such target on adapter

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)FSS: 3702: No FS driver claimed device '4a8d3eb1-b428a951-d1eb-0024e8336099': Not supported

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3: 1561: Could not open device '4be15215-f89bc974-60fb-0024e833609b' for probing: No such target on adapter

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3: 615: Could not open device '4be15215-f89bc974-60fb-0024e833609b' for volume open: No such target on adapter

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)FSS: 3702: No FS driver claimed device '4be15215-f89bc974-60fb-0024e833609b': Not supported

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.886 cpu15:4117)Vol3: 1561: Could not open device '4c1c6b7f-d3a2652c-6b84-0024e833609b' for probing: No such target on adapter

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.886 cpu15:4117)Vol3: 615: Could not open device '4c1c6b7f-d3a2652c-6b84-0024e833609b' for volume open: No such target on adapter

Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.886 cpu15:4117)FSS: 3702: No FS driver claimed device '4c1c6b7f-d3a2652c-6b84-0024e833609b': Not supported

I have read a lotin the net about this problem (ex: http://virtualkenneth.com/2009/12/02/vsphere-freezing-vms-after-deleting-a-volume-from-the-san/ and here http://virtualgeek.typepad.com/virtual_geek/2009/12/an-important-vsphere-4-storage-bug-and-workaround.html) but I am still not able to solve the problem (and vmware support only blames de storage vendor, and the storage vendor blames vmware).

I want to mask out the "problematic" luns, but I "get lost" with the exact command I have to write. May somebody help me? I can provide additional info if required.

Regards

Javier

[mittim12]

Maybe this KB on unpresenting a LUN will help

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1015084

If you found this or any other post helpful please consider the use of the Helpful/Correct buttons to award points

[DSTAVERT]

I would file an SR with VMware. http://www.vmware.com/support/policies/howto.html

[gufete]

I have already filled a SR with vmware; I have spoken (& webex) with 3 different people in vmware; still no luck; I only want to mask those luns, but I am not completly

sure about the exact command I should write.

[DSTAVERT]

What did VMware have you do? Did you follow the Knowledge-base article posted by mittim12

[gufete]

I want vmware to help me to unmask the lun. But this output is not enough for them:

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3:

1561: Could not open device '4be3aa32-1602d741-4477-0024e8330661' for

probing: No such target on adapter*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3:

615: Could not open device '4be3aa32-1602d741-4477-0024e8330661' for

volume open: No such target on adapter*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)FSS:

3702: No FS driver claimed device '4be3aa32-1602d741-4477-0024e8330661':

Not supported*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3:

1561: Could not open device '4ab32747-7d85650d-15e0-0024e8330661' for

probing: No such target on adapter*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3:

615: Could not open device '4ab32747-7d85650d-15e0-0024e8330661' for

volume open: No such target on adapter*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)FSS:

3702: No FS driver claimed device '4ab32747-7d85650d-15e0-0024e8330661':

Not supported*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3:

1561: Could not open device '4ab326d3-d9e22b4a-d076-0024e8330661' for

probing: No such target on adapter*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3:

615: Could not open device '4ab326d3-d9e22b4a-d076-0024e8330661' for

volume open: No such target on adapter*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)FSS:

3702: No FS driver claimed device '4ab326d3-d9e22b4a-d076-0024e8330661':

Not supported*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3:

1561: Could not open device '4a8d3eb1-b428a951-d1eb-0024e8336099' for

probing: No such target on adapter*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3:

615: Could not open device '4a8d3eb1-b428a951-d1eb-0024e8336099' for

volume open: No such target on adapter*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)FSS:

3702: No FS driver claimed device '4a8d3eb1-b428a951-d1eb-0024e8336099':

Not supported*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3:

1561: Could not open device '4be15215-f89bc974-60fb-0024e833609b' for

probing: No such target on adapter*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)Vol3:

615: Could not open device '4be15215-f89bc974-60fb-0024e833609b' for

volume open: No such target on adapter*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.881 cpu10:4117)FSS:

3702: No FS driver claimed device '4be15215-f89bc974-60fb-0024e833609b':

Not supported*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.886 cpu15:4117)Vol3:

1561: Could not open device '4c1c6b7f-d3a2652c-6b84-0024e833609b' for

probing: No such target on adapter*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.886 cpu15:4117)Vol3:

615: Could not open device '4c1c6b7f-d3a2652c-6b84-0024e833609b' for

volume open: No such target on adapter*

*Jul 6 17:17:27 vmware1 vmkernel: 8:02:23:45.886 cpu15:4117)FSS:

3702: No FS driver claimed device '4c1c6b7f-d3a2652c-6b84-0024e833609b':

Not supported*

They tell me that "the badly-removed LUNs do NOT show when you do esxcfg-scsidevs -m"

naa.60024e8076f5a40011f12b5b06f9f01b:5 /dev/sda5 4a8d3eb2-c916d9b3-9777-0024e8336099 0 Interno

naa.6000eb39b2cd84ca0000000000000019:1 /dev/sdc1 4be15218-b00aa75a-1fe6-0024e833609b 0 VolumenA

naa.6000eb3d22b2edcb0000000000000012:1 /dev/sdd1 4ab326e0-5d0fa885-179f-0024e8330661 0 Analog, listas y consigna

naa.6000eb3d22b2edcb0000000000000036:1 /dev/sdf1 4ab3274b-e6403ec8-ba04-0024e8330661 0 Volumen 2

naa.6000eb39b2cd84ca000000000000001d:1 /dev/sdg1 4be3aa32-b6c75e21-3039-0024e8330661 0 Volumen B

naa.6000eb39b2cd84ca0000000000000038:1 /dev/sdi1 4c1c6b80-6e698ca7-b0a3-0024e833609b 0 VolumenE

naa.6000eb3d22b2edcb0000000000000073:1 /dev/sdn1 4aea05dc-fa5c6d3d-c7c4-0024e8330661 0 Volumen3

naa.6000eb39b2cd84ca0000000000000033:1 /dev/sdm1 4c19e8ea-4f196567-4cd4-0024e8330661 0 Volumen D

naa.6000eb39b2cd84ca000000000000003e:1 /dev/sdl1 4c21b620-6307cb5d-7ed7-0024e8330661 0 VolumenF

naa.6000eb3d22b2edcb000000000000001a:1 /dev/sdk1 4ab32704-a48f60f0-3c65-0024e8330661 0 Desarrollo y pruebas

naa.6000eb39b2cd84ca0000000000000025:1 /dev/sdj1 4be9889e-bf21655e-6be9-0024e833609b 0 Desarrolo y pruebas en RAID 10

naa.6000eb3d22b2edcb0000000000000033:1 /dev/sdh1 4ab3269b-bca365e3-687c-0024e8330661 0 Volumen 1

naa.6000eb39b2cd84ca0000000000000020:1 /dev/sde1 4be8f76e-899de08f-651f-0024e8330661 0 Volumen C

And it is true, they are not there. But ESX host still keep trying to connect them. I have read this advirosy from HP (), not related with my storage (I use hp lefthand, the advisory is about HP EVA), but HP says that are "open issues with esx 4.0 (specifically with All Paths Down states of LUN)"

There are several websites of people with the same problem as me

One of the comments of the last link says +" http://virtualgeek.typepad.com/virtual_geek/2009/12/an-important-vsphere-4-storage-bug-and-workaround/comments/page/2/#comments ESXi 4 host connected to an IBM DS4700 array and did not resolve. VMware support says they are still working with certain storage vendors on the issue"

I will try to escalate the problem. The first-level-engineer continues telling me that he can not umask a LUN not listed. Maybe un ESX 4.1 they correct this, but i do not know the ETA of ESX 4.1