Adding iSCSI luns, can only see 2 out of 4

[bwallis21]

I have an Essentials license with a cluster of three machines and two iSCSI servers. This is a development/test cluster done somewhat on the cheap but has worked very well for us so far.

We are running vSphere Version 4.0.0 (Build 162856) and ESXi Version 4.0.0 (Build 261974).

The iSCSI servers are Debian linux running a 2.6.26 kernel with the SCST iSCSI target package installed (iSCSI SCST Target - version 2.0.0/0.4.17r214-procfs) and using 3ware 9650 8 port raid cards. The first of these two servers has two Raid5 arrays of 500G disks exported as two 1.36TB luns (LUN 0 and LUN 1) and formatted/mounted as two VMFS3 filesystems on the the cluster ESXi servers.

The second server has a stripped pair of 1TB disks exported as a single 1.82TB lun (LUN 0) and also formatted/mounted as a VMFS3 filesystem. This is used for volume data in load testing so needs to be fast but not necessarily reliable :smileyhappy:

This has been running reliably and performing well for more than 6 months now.

We have recently added five new 2TB disks in a Raid 6 array to the second server and have configured this at the server as three more 1.82TB luns (LUN 1, LUN 2 and LUN 3). It was setup this way after I discovered the 2TB limit on LUN size.

Now here is the problem.

From my mac with the Global San iSCSI Initiator I can see the three new luns (and all the old ones) and use them with no problems.

From the ESXi servers I can only see the first of the three new LUNs (LUN 1). It is configured and formated with VMFS3 and works with no problems. But I cannot seem to convince any of the ESXi servers to see the other two LUNs (LUN2 and LUN3).

Is there something special I have to do or some configuration that I am missing.

The first two luns appear in the Storage Adapters tab as vmhba37:C0:T1:L0 and vmhba37:C0:T1:L1. I expected to see two new Storage Adapters for the other two luns called vmhba37:C0:T1:L2 and vmhba37:C0:T1:L3 when I do a scan for new storage devices but I don't.

Any help would be most appreciated.

[AndreTheGiant]

Welcome to the community.

Sure that the new LUN are assigned to the ESXi host?

Have you force a storage adapter rescan?

Andre

[bwallis21]

Hi and thanks for the response.

I'm not sure what you mean by

Sure that the new LUN are assigned to the ESXi host?

as far as

Have you force a storage adapter rescan?

yes I have done that. It detected the first of the three new LUNs (1) but not the other two. Repeated scans do not find the others.

[AndreTheGiant]

On the Debian side you can assign permission to each LUNs.

Those permission are the same of the first LUN?

Check on the software docs (I suppose Openfiler?), and have a look if there is some bugs with ESX/ESXi environment (google is your best friend in this case :smileyhappy: )

Andre

[bwallis21]

Been googling quite a bit about this but nothing found so far.

We are not using openfiler, we tried it about 9 months ago but it was using the IET iscsi implementation and that does not work reliably with ESXi due to some missing protocol support for reservations. It was ok until you get two ESX servers talking to it then goes horribly wrong

So we are using the scst implementation on top of a base debian install and that seems to work fine so far even under heavy loads.

Part of our configuration is shown below. The two devices that work are Stripped2G and LargeRaid6-1, the other two are not seen by ESXi but they are seen by my mac so the server is exporting them correctly.

I cannot see any configuration difference between the ones that work and the ones that don't. It is on an isolated network so we have no security setup so I don't think that is it either.

I suspect that there is some ESX setting that is stopping access to the other two devices or maybe some incompatibility between ESX and the SCST iscsi target but no idea what.

im-san2:/etc# cat /proc/scsi_tgt/vdisk/vdisk 
Name              Size(MB)    Block size  Options         File name                                     T10 device id   
LargeRaid6-1      1907334     512         NV              /dev/sdd1                                     LargeRaid6-1 236a890f
LargeRaid6-2      1907334     512         NV              /dev/sdd2                                     LargeRaid6-2 697a1ceb
LargeRaid6-3      1907342     512         NV              /dev/sdd3                                     LargeRaid6-3 ab8fdd72
Stripped2G        1907328     512         NV              /dev/sdc                                      Stripped2G cf2f2bb3

im-san2:/etc# cat /proc/scsi_tgt/groups/Default/devices 
Device (host:ch:id:lun or name)                             LUN          Options
LargeRaid6-2                                                2            
LargeRaid6-3                                                3            
Stripped2G                                                  0            
LargeRaid6-1                                                1      

[rickardnobel]

There is no "access list" or similar that controls which IP addresses can see which LUNs?

[bwallis21]

Yes, there are access lists possible.

You create initiators.allow and initiators.deny files to control this but I have neither file and the default is all access allowed.

The identity of the mac that can access all 4 LUNs has not been configured into the system in anyway and all systems are on the same subnet.

[bwallis21]

I just had a colleague who has suse 11.3 on his workstation connect to the iSCSI server and see what disks are available and he can see all 4 luns as well. This is with no prior setup or configuration on the iSCSI server. I think this pretty much eliminates the concern of access control lists or other iSCSI server related configuration things.

This does seem to be an ESXi issue in that it cannot see the other two luns for some reason or other.

[bwallis42]

No one has had this problem or any answers?

I can see all 4 luns over iSCSI within a virtual machine (SUSE 11.1) on the cluster and am currently formatting one of them as a 1.82TB ext3 filesystem.

ESX can still only see 2 of the 4 luns. I wonder if it is worth looking at updating ESX to 4.1?

[BJSmith20111014]

I am having a similiar problem, I have set up a ubuntu 10.04 LTS doing HA across identical machines and it all works windows/linux can see the 3 targets on the 1 machine.

however esxi 3.5 sees the first target 3 times and does not see the other 2.

I am at a loss and do not have the extra storage to migrate to a different volume before an upgrade and i do not trust the upgrade to not wipe out my current virtual machines.

what a catch 22  I need to add storage to upgrade but I have to upgrade to add storage?

Hopefully you have found the answer and can help me out!

BJ Smith

[bwallis42]

Unfortunately not, the problem remains. As I am using the other two "invisible" luns directly from VMs it is not a high priority for me. I am hoping an upgrade to ESX 4.1 will resolve the issue but really have no idea if it will or not.