Solaris 10 iSCSI CHAP with VSphere

[GuloGulo]

Currently I am trying to build a test environment for a temporary SAN solution, as my primary SAN has ran out of disk space due to a poor LUN design.

So far I have been able to get the temp SAN solution running on Solaris 10 using ZFS and iSCSI, but when ever I try to implement CHAP ESXi will lose the target and I can't get it back until I disable CHAP.

At first I thought the problem was with my implementation of CHAP on Solaris, that was until I was able to get the Microsoft Software Initiator on Windows XP to see my targets without issue. So at this point I am stumped as to where to go or what to try.

So for my setup I am running ESXi 4.0.0, 208167 with one NIC configured for the VM Port Group and the VMKernel (vSwitch0).

10.1.11.114 255.255.0.0

I then have another NIC configured as a VMKernel (vSwitch1) specifically for iSCSI (vmhba33 iqn.1998-01.com.vmware:localhost-4d5e6611)

10.1.2.125 255.255.255.0

Here is my setup for my Solaris iSCSI box:

iscsitadm list initiator -v

Initiator: ESXi1vmhba33

iSCSI Name: iqn.1998-01.com.vmware:localhost-4d5e6611

CHAP Name: ESXi1vmhba33

CHAP Secret: Set

iscsitadm list target -v

Target: san2

iSCSI Name: iqn.1986-03.com.sun:02:9f52fdba-163b-44c3-a400-92d9647db0e6.san2

Connections: 0

ACL list:

Initiator: ESXi1vmhba33

TPGT list:

TPGT: 1

LUN information:

LUN: 0

GUID: 600144f04b66e2100000000000000000

VID: SUN

PID: SOLARIS

Type: disk

Size: 1010G

Backing store: /dev/zvol/rdsk/vmfs/san2lun0

Status: online

LUN: 1

GUID: 600144f04b66e2110000000000000000

VID: SUN

PID: SOLARIS

Type: disk

Size: 1010G

Backing store: /dev/zvol/rdsk/vmfs/san2lun1

Status: online

iscsitadm list tpgt -v

TPGT: 1

IP Address: 10.1.2.126

As I said when I disable the initiator and CHAP I am able to connect without issue, but as soon as I enable CHAP ESXi will no longer see the iSCSI targets.

Now I set my CHAP password to test123456789 to make sure that I did not have any typos, and I still run into the problem.

I currently have ESXi setup for dynamic discovery of 10.1.2.126:3260 and my CHAP settings are to:

Use CHAP (target authenticates host)

Name: ESXi1vmhba33

Secret: test123456789

Do not use CHAP (host authenticates target)

I have tried rebooting the ESXi host, clearing out the Dynamic Discovery and readding it, I have resetup the initiator on Solaris a dozen times but still no luck. I also tried setting up Static Discovery but still no luck.

I tried doing an export of the logs on ESXi but there was no mention of any of the iSCSI stuff, just a bunch of ha-property-collector.

Can anyone shed some light on this issue? I will be placing the iSCSI environment on seperate subnets, VLAN's and physical switches, but I would prefer to have a little more protection than that.

Once I get this part up and running, I will try expanding my TPGT for more redundancy, and then practice growing my zpool's and LUNs.

If anyone else is having problems getting iSCSI up and running on Solaris for VMWare drop me a line, I written up step-by-step instructions for everything but the CHAP :smileyhappy: I can also give some explanations of why things are done too, which I found lacking in a lot of the instruction sets that I was reading online.

[Simon Zhang]

can you post the vmkernel messages /var/log/messages when ESXi dynamic discovery of 10.1.2.126:3260? thanks

binoche, VMware VCP, Cisco CCNA

[GuloGulo]

Here is the output of the /var/log/messages after a rescan is performed:

Feb 3 14:22:45 Hostd: ReconcileVMFSDatastores called: refresh = true, rescan = true

Feb 3 14:22:45 Hostd: RefreshVMFSVolumes called

Feb 3 14:22:45 Hostd: RescanVmfs called

Feb 3 14:22:45 Hostd: FetchDVPortgroups: added 0 items

Feb 3 14:22:45 vmkernel: 1:00:16:29.953 cpu0:4969)Vol3: 1488: Could not open device 'mpx.vmhba1:C0:T0:L0:6' for probing: Permission denied

Feb 3 14:22:45 vmkernel: 1:00:16:29.955 cpu1:4969)Vol3: 608: Could not open device 'mpx.vmhba1:C0:T0:L0:6' for volume open: Permission denied

Feb 3 14:22:45 vmkernel: 1:00:16:29.963 cpu1:4969)Vol3: 1488: Could not open device 'mpx.vmhba1:C0:T0:L0:5' for probing: Permission denied

Feb 3 14:22:45 vmkernel: 1:00:16:29.964 cpu1:4969)Vol3: 608: Could not open device 'mpx.vmhba1:C0:T0:L0:5' for volume open: Permission denied

Feb 3 14:22:45 vmkernel: 1:00:16:29.972 cpu1:4969)Vol3: 1488: Could not open device 'mpx.vmhba1:C0:T0:L0:2' for probing: Permission denied

Feb 3 14:22:45 vmkernel: 1:00:16:29.973 cpu1:4969)Vol3: 608: Could not open device 'mpx.vmhba1:C0:T0:L0:2' for volume open: Permission denied

Feb 3 14:22:45 vmkernel: 1:00:16:29.975 cpu1:4969)FSS: 3702: No FS driver claimed device 'mpx.vmhba1:C0:T0:L0:2': Not supported

Feb 3 14:22:45 vmkernel: 1:00:16:29.976 cpu1:4969)Vol3: 1488: Could not open device 'mpx.vmhba1:C0:T0:L0:8' for probing: Permission denied

Feb 3 14:22:45 vmkernel: 1:00:16:29.977 cpu1:4969)Vol3: 608: Could not open device 'mpx.vmhba1:C0:T0:L0:8' for volume open: Permission denied

Feb 3 14:22:45 Hostd: VmfsUpdate: got VMFS message timestamp=1265206965322362 specific=0 name= label=

Feb 3 14:22:45 Hostd: RefreshVMFSVolumes: refreshed volume, id 4b5d5250-a0bdf374-ca82-0017a48e39dd, name datastore1

Feb 3 14:22:45 Hostd: SetVolume: Datastore 4b5d5250-a0bdf374-ca82-0017a48e39dd has changed provider volume pointer

Feb 3 14:22:45 Hostd: ReconcileVMFSDatastores: Done discovering new filesystem volumes.

Feb 3 14:22:45 Hostd: ReconcileNASDatastores: Discovering new filesystem volumes.

Feb 3 14:22:45 Hostd: RefreshNASVolumes called

Feb 3 14:22:45 Hostd: ReconcileNASDatastores: Done discovering new filesystem volumes.

Feb 3 14:22:45 Hostd: SendStorageInfoEvent() called

Feb 3 14:22:45 Hostd: Task Completed : haTask-ha-host-vim.host.StorageSystem.refresh-184 Status success

Feb 3 14:22:46 Hostd: Unrecognized HBA status FC HBA: vmhba2 - 3

Feb 3 14:22:46 Hostd: Unrecognized HBA status FC HBA: vmhba3 - 3

Feb 3 14:22:46 Hostd: CreateISCSIHBA

Feb 3 14:22:46 Hostd: IP properties discovery failure on iSCSI HBA: iqn.1998-01.com.vmware:localhost-4d5e6611: Check vmkernel tcp/ip settings for network properties for sw iscsi

Feb 3 14:22:46 vmkernel: 1:00:16:31.190 cpu0:4103)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x4100050a1640) to NMP device "mpx.vmhba0:C0:T0:L0" failed on physical path "vmhba0:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

Feb 3 14:22:46 Hostd: CreateDurableName: Header length field exceeds actual payload

Feb 3 14:22:46 vmkernel: 1:00:16:31.190 cpu0:4103)ScsiDeviceIO: 747: Command 0x12 to device "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

Feb 3 14:22:46 vmkernel: 1:00:16:31.215 cpu3:26405)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x4100050a1640) to NMP device "mpx.vmhba1:C0:T0:L0" failed on physical path "vmhba1:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

Feb 3 14:22:46 vmkernel: 1:00:16:31.215 cpu3:26405)ScsiDeviceIO: 747: Command 0x12 to device "mpx.vmhba1:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

Feb 3 14:22:46 Hostd: CreateDurableName: Header length field exceeds actual payload

Feb 3 14:22:46 Hostd: FetchDVPortgroups: added 0 items

Feb 3 14:23:06 Hostd: Task Created : haTask-ha-host-vim.host.StorageSystem.rescanAllHba-185

Feb 3 14:23:06 vmkernel: 1:00:16:50.736 cpu1:5316)ScsiNpiv: 1304: GetInfo for adapter vmhba2, , max_vports=0, vports_inuse=0, linktype=0, state=0, failreason=0, rv=-2, sts=bad0020

Feb 3 14:23:06 vmkernel: 1:00:16:50.737 cpu1:5316)ScsiNpiv: 1304: GetInfo for adapter vmhba2, , max_vports=0, vports_inuse=0, linktype=0, state=0, failreason=0, rv=-2, sts=bad0020

Feb 3 14:23:06 vmkernel: 1:00:16:50.737 cpu1:5316)ScsiNpiv: 1304: GetInfo for adapter vmhba3, , max_vports=0, vports_inuse=0, linktype=0, state=0, failreason=0, rv=-2, sts=bad0020

Feb 3 14:23:06 vmkernel: 1:00:16:50.738 cpu1:5316)ScsiNpiv: 1304: GetInfo for adapter vmhba3, , max_vports=0, vports_inuse=0, linktype=0, state=0, failreason=0, rv=-2, sts=bad0020

Feb 3 14:23:06 Hostd: FetchDVPortgroups: added 0 items

Feb 3 14:23:11 vmkernel: 1:00:16:55.743 cpu2:4759)Tcpip_Socket: 2886: msleep returned 4

Feb 3 14:23:11 iscsid: connection failed for discovery (err = Interrupted system call)!

Feb 3 14:23:11 iscsid: connection to discovery address 10.1.2.126 failed

It appears that I might have a software problem with the iSCSI setup, so I am going to go play with that and see if that is the cause.