VMware site recovery manager doesn't detect datastores on recovery site

[v_potnis2001]

In the protection setup, the MirrrorView SRA is able to detect the local (protection) as well as remote(recovery) arrays.

But when you rescan the arrays, the datastores are not detected although the replicated luns are detected.

When I promote the secondary mirror using Navisphere (outside SRM) and then run rescan hba and for vmfs on the new primary using VI client, the ESX box is able to see the datastore.

This means that the replication is set up correctly.

Questions:

1. How do I enable verbose logging to debug this SRM issue? (see the log excerpts below)

2. What am I missing here? Do I need to do any setup on the recovery site? Right now, I don't have anything configured on secondary except the Connection under Protection setup.

SRM logs:

....

Found 3 replicated LUN pairs

Found replicated lun:

(dr.san.Lun) {

dynamicType = <unset>,

info = (dr.san.Lun.Info) {

dynamicType = <unset>,

arrayId = "50060160806006A50000000000000000",

id = "6006016074B10D0044341A4F0712DB11",

wwn = <unset>,

number = (dr.san.Lun.Number) [

(dr.san.Lun.Number) {

dynamicType = <unset>,

value = 6,

initiatorGroupId = "52341A4F0712DB11805EF3E6AB449028",

targetId = <unset>,

}

],

consistencyGroupId = <unset>,

},

peerInfo = (dr.san.Lun.PeerInfo) {

dynamicType = <unset>,

arrayKey = "500601609060249A0000000000000000",

lunKey = "60060160EDA21000FC4B0BBF9E42DD11",

},

}

Found replicated lun:

(dr.san.Lun) {

dynamicType = <unset>,

info = (dr.san.Lun.Info) {

dynamicType = <unset>,

arrayId = "50060160806006A50000000000000000",

id = "60060160813F09001680F259867EDB11",

wwn = <unset>,

number = (dr.san.Lun.Number) [

(dr.san.Lun.Number) {

dynamicType = <unset>,

value = 0,

initiatorGroupId = "1F80F259867EDB11806487D149880481",

targetId = <unset>,

}

],

consistencyGroupId = <unset>,

},

peerInfo = (dr.san.Lun.PeerInfo) {

dynamicType = <unset>,

arrayKey = "500601609060249A0000000000000000",

lunKey = "6006016078B4100022AFB6226185DB11",

},

}

Found replicated lun:

(dr.san.Lun) {

dynamicType = <unset>,

info = (dr.san.Lun.Info) {

dynamicType = <unset>,

arrayId = "50060160806006A50000000000000000",

id = "6006016074B10D00C0E5AE6A4086DB11",

wwn = <unset>,

number = (dr.san.Lun.Number) [

(dr.san.Lun.Number) {

dynamicType = <unset>,

value = 2,

initiatorGroupId = "1F80F259867EDB11806487D149880481",

targetId = <unset>,

}

],

consistencyGroupId = <unset>,

},

peerInfo = (dr.san.Lun.PeerInfo) {

dynamicType = <unset>,

arrayKey = "500601609060249A0000000000000000",

lunKey = "60060160EDA210000651524A52C8DB11",

},

}

No lun groups created since there are no replicated datastores

'Computed LUN groups for array 'array-441' with ID '50060160806006A50000000000000000'' took 0.625 seconds

Calculating a diff between two sets of VM groups

0 VM groups on the left:

...

[Michelle_Laveri]

I don't know much about MirrorView. But i've heard two things. Firstly only asycfhronious replication is supported - and you have to be careful how you structure your groups...

Regards

Mike

[v_potnis2001]

Tried configuring clariion mirrors in the async mode. Same issue.

SRM can detection both protection and recovery side arrays, it can detect that luns are replicated but its unable to create protectiong groups because it says it cannot find replicated datastores (please see the error message in the earlier posting).

Double checked the SRM configuration. Configured a SQL database for SRM, with sql authenticated login and so on...seems nothing wrong with the SRM setup.

Checked the storage array configuration too. If I promote the clarion mirror outside SRM (reverse the direction of replication) and run ESX hba rescan, the datastores are visible on the recovery site.

Is there anything else that needs to be checked?

[admin]

Can you make sure that the datastore contains at least one VM? If this doesn't help, please attach the SRM logs.

Thanks,

-Alvin

[v_potnis2001]

Didn't work with VM, already tried that. I'll post the logs soon.

[v_potnis2001]

Attaching SRM logs:

Received properties for an object of unexpected type: 'vim.Datacenter:datacenter-2'

Adding datastore 'test_srm_101' with MoId 'datastore-40' and VMFS volume UUID '48ab43c5-812708d7-f24e-000e0cb60b25' spanning 1 LUNs

Found 2 replicated LUN pairs

Found replicated lun:

(dr.san.Lun) {

dynamicType = <unset>,

info = (dr.san.Lun.Info) {

dynamicType = <unset>,

arrayId = "50060160806006A50000000000000000",

id = "6006016074B10D00C0E5AE6A4086DB11",

wwn = <unset>,

number = (dr.san.Lun.Number) [

(dr.san.Lun.Number) {

dynamicType = <unset>,

value = 2,

initiatorGroupId = "1F80F259867EDB11806487D149880481",

targetId = <unset>,

}

],

consistencyGroupId = <unset>,

},

peerInfo = (dr.san.Lun.PeerInfo) {

dynamicType = <unset>,

arrayKey = "500601609060249A0000000000000000",

lunKey = "60060160EDA210000651524A52C8DB11",

},

}

Found replicated lun:

(dr.san.Lun) {

dynamicType = <unset>,

info = (dr.san.Lun.Info) {

dynamicType = <unset>,

arrayId = "50060160806006A50000000000000000",

id = "60060160813F09001680F259867EDB11",

wwn = <unset>,

number = (dr.san.Lun.Number) [

(dr.san.Lun.Number) {

dynamicType = <unset>,

value = 0,

initiatorGroupId = "1F80F259867EDB11806487D149880481",

targetId = <unset>,

}

],

consistencyGroupId = <unset>,

},

peerInfo = (dr.san.Lun.PeerInfo) {

dynamicType = <unset>,

arrayKey = "500601609060249A0000000000000000",

lunKey = "6006016078B4100022AFB6226185DB11",

},

}

No lun groups created since there are no replicated datastores

'Computed LUN groups for array 'array-729' with ID '50060160806006A50000000000000000'' took 0.609 seconds

Progress advanced to 100

State set to success

No pending requests for lun groups computation

Scanning for tasks to reap

Task Array.Synchronize-9 marked as complete

Task com.vmware.vcDr.San.RecomputeLunGroups-10 marked as complete

Free disk space: 11866 Mb

CPU usage: 6 %

Available memory: 369 Mb

Server Available.

The feature 'PROD_SRM' is not present on the license server

The feature 'SRM_PROTECTED_HOST' is not present on the license server

Task destroyed

Task destroyed

Scanning for tasks to reap

Free disk space: 11866 Mb

CPU usage: 4 %

Available memory: 369 Mb

Free disk space: 11866 Mb

CPU usage: 3 %

Available memory: 369 Mb

[admin]

Does the forum allow you to post the complete log (or to send it to me privately)? It would be good to set the SRM SAN log level to trivia:

<Config>
   ...
   <level id="PrimarySanProvider">
     <logName>PrimarySanProvider</logName>
     <logLevel>trivia</logLevel>
   </level>

   <level id="SanConfigManager">
     <logName>SanConfigManager</logName>
     <logLevel>trivia</logLevel>
   </level>
   ...
</Config>

Things to check:

1) Make sure that the primary VC inventory shows complete datastores residing on the replicated LUN or LUNs. (Note that all LUNs underlying multi-extent datastores must be replicated.)

2) Make sure that the replicated datastores contain VMs.

3) Make sure that you refresh the storage on all hosts that have access to the replicated datastores.

[v_potnis2001]

Complete log is attached (pls see the attachment).

The log level I believe is trivia.

An excerpt is givem below:

==========================

Added LUN '21:00:00:e0:8b:87:9b:be;0;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:1:0' and 'host-8;02000000006006016074b10d0053341a4f0712db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;1;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:1' and 'host-8;02000100006006016074b10d0054341a4f0712db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;2;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:2' and 'host-8;02000200006006016074b10d0055341a4f0712db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;3;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:3' and 'host-8;02000300006006016074b10d0056341a4f0712db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;4;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:4' and 'host-8;02000400006006016074b10d0057341a4f0712db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;5;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:5' and 'host-8;02000500006006016074b10d00caf9483f4588da11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;6;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:6' and 'host-8;02000600006006016074b10d0044341a4f0712db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;7;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:7' and 'host-8;02000700006006016074b10d0062eda53fa458db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;8;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:8' and 'host-8;02000800006006016074b10d00501518b6b649db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;9;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:9' and 'host-8;02000900006006016074b10d0063eda53fa458db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;10;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:10' and 'host-8;02000a00006006016074b10d006aeda53fa458db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;11;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:11' and 'host-8;02000b00006006016074b10d0049341a4f0712db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;12;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:12' and 'host-8;02000c00006006016074b10d0085eda53fa458db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;13;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:13' and 'host-8;02000d00006006016074b10d0049f9483f4588da11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;14;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:14' and 'host-8;02000e00006006016074b10d003b341a4f0712db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;15;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:15' and 'host-8;02000f00006006016074b10d004d341a4f0712db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;16;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:16' and 'host-8;02001000006006016074b10d0047f9483f4588da11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;17;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:17' and 'host-8;02001100006006016074b10d00807c4de39726dd11524149442031'

Added LUN '21:00:00:e0:8b:87:9b:be;18;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:18' and 'host-8;02001200006006016074b10d0059341a4f0712db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;19;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:19' and 'host-8;02001300006006016074b10d00817c4de39726dd11524149442031'

Added LUN '21:00:00:e0:8b:87:9b:be;20;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:20' and 'host-8;02001400006006016074b10d00c1e5ae6a4086db11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;21;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:21' and 'host-8;020015000060060160813f09001a80f259867edb11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;22;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:22' and 'host-8;02001600006006016074b10d0046f9483f4588da11524149442035'

Added LUN '21:00:00:e0:8b:87:9b:be;26;50:06:01:62:00:60:06:a5' with keys 'host-8;vmhba1:0:26' and 'host-8;02001a00006006016074b10d003c341a4f0712db11524149442035'

Added vmfs extent 'host-8;vmhba1:0:1' with key 'host-8;48ab43c5-812708d7-f24e-000e0cb60b25;0'

Found 1 properties for object 'vim.Datacenter:datacenter-2'

Found 3 properties for object 'vim.Datastore:datastore-11'

Found 3 properties for object 'vim.Datastore:datastore-40'

Found 4 properties for object 'vim.VirtualMachine:vm-42'

Received properties for an object of unexpected type: 'vim.Datacenter:datacenter-2'

Adding datastore 'test_srm_101' with MoId 'datastore-40' and VMFS volume UUID '48ab43c5-812708d7-f24e-000e0cb60b25' spanning 1 LUNs

Found 2 replicated LUN pairs

Found replicated lun:

(dr.san.Lun) {

dynamicType = ,

arrayKey = "500601609060249A0000000000000000",

lunKey = "60060160EDA210000651524A52C8DB11",

},

}

Found replicated lun:

(dr.san.Lun) {

dynamicType = ,

arrayKey = "500601609060249A0000000000000000",

lunKey = "6006016078B4100022AFB6226185DB11",

},

}

No lun groups created since there are no replicated datastores

'Computed LUN groups for array 'array-844' with ID '50060160806006A50000000000000000'' took 0.609 seconds

Progress advanced to 100

State set to success

No pending requests for lun groups computation

Persisting array-manager-835 to database

Saving Array Manager 'array-manager-835' to DB object

Stored credentials, key='array:array-manager-835:CLARiiON', username='admin'

Saved Array Manager 'array-manager-835' to DB object

Result set to 'dr.san.Array:array-844'

State set to success

Work function completed successfully

Task destroyed

Task destroyed

Scanning for tasks to reap

Task com.vmware.vcDr.San.ArrayManager.AddArrayTask-24 marked as complete

Task com.vmware.vcDr.San.RecomputeLunGroups-25 marked as complete

Free disk space: 11865 Mb

CPU usage: 6 %

Available memory: 363 Mb

Task destroyed

[admin]

Thank you for the log!

We see that the VC inventory contains one datastore, named "test_srm_101". We also see that the Clariion SRA is reporting two replicated LUNs. However, it appears that the test_srm_101 datastore does not live on either of these replicated LUNs. Furthermore, it appears that these replicated LUNs are not visible to any of the hosts in the VC inventory. The replicated LUNs are reported by the SRA as being part of an initiator group that is not seen by any of the ESX hosts.

This analysis suggests that replication needs to be enabled for the LUN underlying the test_srm_101 datastore.

Thanks,

-Alvin

[v_potnis2001]

I tried promoting the remote mirror outside SRM, using Navisphere gui and then rescannining hba on the secondary (recovery site) ESX host.

It was able to detect VMFS datastore test_srm_101 without issues.

Don't think there is an issue with replication although yes, I haven't configured consistency group and snapview snapshots.

[bwhitman1]

It definitely won't work without the consistency groups, page 18: "Before you install SRM, MirrorView mirror states must be synchronized or consistent and be part of a consistency group."

The snapshots are used when you do an isolated test of your recovery plan. The SRA talks to Mirrorview and creates writable snaps for the VMs that it will bring up in the isolated network. This is done so you can test your DR plan without impacting your prod or DR volumes.

Can you setup the consistency groups and snaps and then test and let us know?

[v_potnis2001]

Yes, you are correct!

Configured it inside a consistency group with snapshot luns and replicated datastores could be discovered!

Thank you, I probably should have figured it out myself but I didn't exactly see a connection between consistency group, snapshot luns and datastores not being discovered as replicated.

[cjt]

hello,

pls verify your config against the SRM compatability matrix - page 9 for EMC.

Most of the time these annoying little errors come down this.

Mirrorview/S is only supported - Mirrorview /A support is coming very soon.

[v_potnis2001]

I am using CX600 and CX700. It isn't listed in the compat matrix, I had already checked that. I'm going to try with CX3-80 which is supported.

But if that is the issue, how are the storage arrays being discovered by the SRA? The SRA logs also say that replicated luns are discovered but say that no replicated datastores are discovered. Will post the complete logs soon, an excerpt of the logs is already present in one of the previous posts.

Btw, I tried with both MirrorView/S and MirrorView/A. Same error message.

All I am trying to do right now is evaluate SRM, what is the simolest way of doing that? Thanks for the all the nice responses.

[v_potnis2001]

http://www.lefthandnetworks.com/vsa_eval.aspx

Maybe, should get started with this.

[Lee Dilworth]

are your luns/device groups defined in more than one storage group? the current version of the mirrorview SRA does not support having the lun/device groups defined in more than one storage group. I saw this with a customer last week. once we changed the configuration to have all the ESX host initiators / luns contained within a single storage group the datastores then appeared in the array config wizard within SRM. EMC are working to alter the SRA for mirrorview to permit multi-storage group configs. this should be available sept 2008. if you speak to EMC directly you may be able to test an early beta copy of the modifed SRA.

Also ensure you check your CX config as per the release notes (that ship with the SRA) and best practice guide:

cheers

Lee Dilworth

[Connectr]

Some comments / thoughts:

• Have you tried contacting VMware and EMC customer support to get the issue addressed / tracked with their engineering teams? Could possibly provide some quick answers.

• Check out the VMware support tab on EMC's Powerlink customer portal, it contains some SRM config guides that may also help

• As mentioned erlier in the thread, CX300's are not supported. CX 3's / 4's are. Asych mode to be supported soon.

- Cheers

[bwhitman1]

In regards to Lee Dilworth's reply (Smoggy), pay special attention to page on creating snapshots. The snapshots have to be in a specific naming convention:

The SRA requires the string to exist somewhere in the snapshot name: VMWARE_SRM_SNAP_LUNID

[v_potnis2001]

Thanks for the comment.

Can you explain this in details? Specifically, what do I need to do...

[bwhitman1]

Here's the install doc reference in case you can't get to Powerlink. Page 17 in particular talks about setting up the snapshots. These are only necessary for the isolated test but it may still be causing your issue. Regardless, if you haven't seen this guide yet I would go through it and make sure you follow the setup steps, especially pages 13-18 for the array.

[v_potnis2001]

Thanks for this doc!

One thing I was missing out on was not configuring consistency group, the other was not configuring snapview snapshots (wonder why this is necessary for replication).

Anyways, I'll run through the steps and let you know.