Automation

 View Only
Expand all | Collapse all

Replicated LUNs and RDMs

  • 1.  Replicated LUNs and RDMs

    Posted Apr 15, 2013 06:40 PM

    Hello there,

    we have 2 nodes and 2 storages in same location. storage 1 is replicated to storage 2. initially on storage 1 we have created 4 LUNs and added them as RDMs to 2 nodes. if first storage goes off or disconnected, we need to add replicated 2 LUNs to 2 nodes from storage 2. if system works on storage 1 then LUN numbers 100 and 101, if on storage 2 then LUN numbers 110 and 111. we need to make this using powerCLI.

    Any help would be greatly appreciated.

    LUCD, I have read your amazing solutions/answers, would you please help me? Thank you.



  • 2.  RE: Replicated LUNs and RDMs

    Posted Apr 15, 2013 06:52 PM

    How, and where, will you detect that storage1 becomes unavailable ?

    Because that would be the place to trigger the script that will connect the LUNs on storage2 as RDMs.



  • 3.  RE: Replicated LUNs and RDMs

    Posted Apr 15, 2013 06:56 PM

    Thank you for your quick responce.

    we will do this manually by a script that will "unexport" the LUNs that are from storage 1 and "export" the LUNS that are from storage 2. so this script trigger the powercli script.



  • 4.  RE: Replicated LUNs and RDMs

    Posted Apr 15, 2013 07:47 PM

    Hey Luc,

    I am planning to use your solution below, do you think it will cover what I need ?

    http://communities.vmware.com/message/2057164



  • 5.  RE: Replicated LUNs and RDMs

    Posted Apr 15, 2013 07:57 PM

    Or answers must have crossed each other :smileygrin:

    That should work, but the code above does more or less the same thing.

    And the problem I mentioned in that other thread might have been solved in the latest PowerCLI build.

    You'll have to try that out in your environment.



  • 6.  RE: Replicated LUNs and RDMs

    Posted Apr 15, 2013 07:53 PM

    If I understood the question correctly, the following shows the way to find the backup LUNs and replace the harddisks connected to both nodes by RDM to the backup LUNs.

    $luns = Get-VMHost | Get-ScsiLun -LunType disk
    $lun1 = $luns | where {$_.LunID -eq 110}
    $lun2 = $luns | where {$_.LunID -eq 111}
    
    $node1 = Get-VM -Name node1 
    $node2
    = Get-VM -Name node2
    Get-HardDisk
    -VM $node1 -Name "Hard disk 2" | Remove-HardDisk -Confirm:$false
    New-HardDisk -VM $node1 -DeviceName $lun1.ConsoleDeviceName -DiskType RawPhysical Get-HardDisk -VM $node2 -Name "Hard disk 2" | Remove-HardDisk -Confirm:$false
    New-HardDisk -VM $node2 -DeviceName $lun2.ConsoleDeviceName -DiskType RawPhysical

    But like I said I'm not sure I understood the question correctly.



  • 7.  RE: Replicated LUNs and RDMs

    Posted Apr 15, 2013 08:11 PM

    in your script it's supposed "Hard disk 2" is already added from storage 1. so I think the script should be:

    $luns = Get-VMHost | Get-ScsiLun -LunType disk
    $lun1 = $luns | where {$_.LunID -eq 110}
    $lun2 = $luns | where {$_.LunID -eq 111}
    
    $node1 = Get-VM -Name node1
    $node2 = Get-VM -Name node2
    Get-HardDisk -VM $node1 -Name "Hard disk 2" | Remove-HardDisk -Confirm:$false
    Get-HardDisk -VM $node2 -Name "Hard disk 2" | Remove-HardDisk -Confirm:$false

    at this point, LUNs from storage 1 must be unexported and LUNs from storage 2 must be exported.
    Then scan for new LUNs and add the disks.

    Get-VMHost | Get-VMHostStorage -RescanAllHBA

    New-HardDisk -VM $node1 -DeviceName $lun1.ConsoleDeviceName -DiskType RawPhysical
    New-HardDisk -VM $node2 -DeviceName $lun2.ConsoleDeviceName -DiskType RawPhysical

    this looks correct. I only don't know when I add (replicated) 2 disks that are from storage 2 by using this script then I will lost the data on these disk or not. what do you think about this ?

    Thank you.



  • 8.  RE: Replicated LUNs and RDMs

    Posted Apr 15, 2013 08:24 PM

    The unexport and export of the LUNs is from the point of view of the storage controllers, right ?

    And that is done via the management interface to these controllers ?

    In that case you will indeed need to do a scan to "see"  the new LUNs.

    When storage1 becomes unavailable, is that done gracefully ? In other words is all IO finished before the underlying LUNs are unexported ?

    And I assume that the synchronisation to the LUNs on storage2 continues ?

    In that case there shouldn't be any data lost, provided the application(s) can cope with a removal of a disk.



  • 9.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 05:49 AM

    yes, the unexport and export of the LUNs is from the point of view of the storage controllers. say a database works on nodes, after we shutdown the database, we unexport the LUNs, export replicated LUNs from storage 2, and start the database. yes there is no I/O traffic to LUNs at this time.

    when we start to use storage 1, synchronisation continous. replication works reverse, from storage 2 to storage 1.

    I will try the script and let you know if it works or not :smileyhappy:

    Thank you.



  • 10.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 06:59 AM

    are you sure Get-ScsiLun has "LunID"property ? becase it gives nothing



  • 11.  RE: Replicated LUNs and RDMs
    Best Answer

    Posted Apr 16, 2013 07:23 AM

    My mistake, I'm using the VIProperties for ScsiLun, adn LunID is in there.

    You can use the formula instead

    New-VIProperty -Name lunID -ObjectType ScsiLun -Value {
        param($lun)

        [int](Select-String ":L(?\d+)$" `
            -InputObject $lun.RuntimeName).Matches[0].Groups['lunID'].Value
    } -Force



  • 12.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 07:40 AM

    as you said I need to add/define the LunID property. I also found it at your web page: http://www.lucd.info/2011/07/26/the-making-of-a-new-viproperty-called-lunid/

    :smileyhappy:

    then final script is:

    New-VIProperty -Name lunID -ObjectType ScsiLun -Value {
      param($lun)
      [int](Select-String ":L(?<lunID>\d+)$" `
        -InputObject $lun.RuntimeName).Matches[0].Groups['lunID'].Value
    } -Force | Out-

    $esxName = "XXXX"  #used, becase there 5 hosts in cluster and 5 lines goes to LUN1
    $luns = Get-VMHost -Name $esxName | Get-ScsiLun -LunType disk
    $lun1 = $luns | where {$_.LunID -eq 110}
    $lun2 = $luns | where {$_.LunID -eq 111}

    $node1 = Get-VM -Name "NodeA"
    $node2 = Get-VM -Name "NodeB"

    Get-HardDisk -VM $node1 -Name "Hard disk 2" | Remove-HardDisk -Confirm:$false
    Get-HardDisk -VM $node2 -Name "Hard disk 2" | Remove-HardDisk -Confirm:$false

    Get-VMHost | Get-VMHostStorage -RescanAllHBA

    New-HardDisk -VM $node1 -DeviceName $lun1.ConsoleDeviceName -DiskType RawPhysical
    New-HardDisk -VM $node2 -DeviceName $lun2.ConsoleDeviceName -DiskType RawPhysical

    Thank you very much dude, you are great !



  • 13.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 08:08 AM

    in the script, we need to add same disk to two nodes. I mean after adding RDM to nodeA, I must add same RDM to nodeB. do you have any short solution?



  • 14.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 09:30 AM

    Do you mean that you need to share the RDM between 2 VMs ?

    If yes, then you will also have to create a new SCSI controller with one of the 2 SCSI Bus sharing modes



  • 15.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 09:42 AM

    yes I need to share RDM betwen two nodes. it says nodes must be "powered-off"

    so in my scenario, after removing disk, I dont need to add/remove scsi controller to add new disk. so I need to use existing scsi controller.

    New-VIProperty -Name lunID -ObjectType ScsiLun -Value {
      param($lun)
      [int](Select-String ":L(?<lunID>\d+)$" `
        -InputObject $lun.RuntimeName).Matches[0].Groups['lunID'].Value
    } -Force | Out-Null


    $esxName = "XXXX"
    $luns = Get-VMHost -Name $esxName | Get-ScsiLun -LunType disk
    $lun1 = $luns | where {$_.LunID -eq 100}

    $vm1 = Get-VM -Name "NodeA"
    $vm2 = Get-VM -Name "NodeB"

    $disk = Get-VM $vm1 | Get-HardDisk | Select -First 5
    $ctrl1=Get-ScsiController -HardDisk $disk | where {$_.BusSharingMode -eq "Virtual"}

    $disk = Get-VM $vm2 | Get-HardDisk | Select -First 5
    $ctrl2=Get-ScsiController -HardDisk $disk | where {$_.BusSharingMode -eq "Virtual"}

    Get-HardDisk -VM $vm1 -Name "Hard disk 2" | Remove-HardDisk -Confirm:$false
    Get-HardDisk -VM $vm2 -Name "Hard disk 2" | Remove-HardDisk -Confirm:$false

    $DeviceName=$lun1.ConsoleDeviceName

    $hd1 = New-HardDisk -VM $vm1 -DeviceName $DeviceName -DiskType RawVirtual
    $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
    $spec.DeviceChange += New-Object VMware.Vim.VirtualDeviceConfigSpec
    ....

    ..... Here what to do :smileyhappy:

    .....

    $vm2.ExtensionData.ReconfigVM($spec)



  • 16.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 09:55 AM

    Afaik you have to start with a SCSI controller that allows bus sharing.

    See Sharing a .vmdk between two VMs for an example script.



  • 17.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 11:02 AM

    what I mean is;

    when two nodes are powered-off, I added scsi controllers and shared RDMs.

    then two nodes are powered-on, then removes disks but now scsi controllers. then I need to add replicated disk using existing scsi controllers.



  • 18.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 11:30 AM

    well,

    $crtl1 is on vm1 and $crtl2 on vm2, when I try

    $hd1 = New-HardDisk -VM $vm1 -DeviceName $DeviceName -DiskType RawVirtual -controller $ctrl1

    New-HardDisk -VM $vm2 -DiskPath $hd1.Filename -controller $ctrl2

    it add the disk on vm1 but it gives following error on vm2

    New-HardDisk : 16.04.2013 14:25:47    New-HardDisk        Incompatible device backing specified for device '0'.   
    At line:31 char:13
    + New-HardDisk <<<<  -VM $vm2 -DiskPath $hd1.Filename -controller $ctrl2
        + CategoryInfo          : NotSpecified: (:) [New-HardDisk], InvalidDeviceBacking
        + FullyQualifiedErrorId : Client20_VirtualDeviceServiceImpl_AttachHardDisk_ViError,VMware.VimAutomation.ViCore.Cmdlets.Commands.VirtualDevice.NewHardDisk

    :smileysad:



  • 19.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 11:34 AM

    I suspect this might be because the SCSI controller is not configured for sharing.

    If you do a "Get-ScsiController", is the BuSharingMode showing physical or virtual ?



  • 20.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 11:53 AM

    busSharing mode is virtual.

    when nodes powered-off I already added scsi controllers and disk, and power on them, and remove the disk. I just want to add same disk to nodes using existing scsi controllers on nodes. when I look I see controllers exist on nodes.

    $disk = Get-VM "NodeS" | Get-HardDisk | Select -First 5
    $ctrl1=Get-ScsiController -HardDisk $disk | where {$_.BusSharingMode -eq "Virtual"}
    $disk = Get-VM "NodeB" | Get-HardDisk | Select -First 5
    $ctrl2=Get-ScsiController -HardDisk $disk | where {$_.BusSharingMode -eq "Virtual"}

    $esxName = "vmserver01.ibb.gov.tr"
    $luns = Get-VMHost -Name $esxName | Get-ScsiLun -LunType disk
    $lun1 = $luns | where {$_.LunID -eq 200}

    $vm1 = Get-VM -Name "NodeA"
    $vm2 = Get-VM -Name "NodeB"

    $DeviceName=$lun1.ConsoleDeviceName

    $hd1 = New-HardDisk -VM $vm1 -DeviceName $DeviceName -DiskType RawVirtual -controller $ctrl1

    until now script works, now I need to add same disk to nodeB, $hd1.Filenam ise [XXX_VMFS01] NodeA/NodeA_1.vmdk

    then I run the command in the script:

    New-HardDisk -VM $vm2 -DiskPath $hd1.Filename -controller $ctrl2

    here it give the error



  • 21.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 12:25 PM

    Is there anything in the vpxd logs on the vCenter.

    Sometimes you more info about the error in there.



  • 22.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 01:57 PM

    no way. when I try the following script it gives same error:

    $esxName = "esxhostXX"
    $luns = Get-VMHost -Name $esxName | Get-ScsiLun -LunType disk
    $lun1 = $luns | where {$_.LunID -eq 200}

    $DeviceName=$lun1.ConsoleDeviceName
    $vm1 = Get-VM -Name "NodeA"
    $vm2 = Get-VM -Name "NodeB
    $hd1 = New-HardDisk -VM $vm1 -DeviceName $DeviceName -DiskType RawVirtual
    $ctrl1 = New-ScsiController -HardDisk  $hd1 -BusSharingMode Virtual -Type VirtualLsiLogicSAS
    $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
    $spec.DeviceChange += New-Object VMware.Vim.VirtualDeviceConfigSpec
    $spec.deviceChange[0].device += $ctrl1.ExtensionData
    $spec.deviceChange[0].device.Key = -101
    $spec.deviceChange[0].operation = "add"
    $spec.DeviceChange += New-Object VMware.Vim.VirtualDeviceConfigSpec
    $spec.deviceChange[1].device += $hd1.ExtensionData
    $spec.deviceChange[1].device.Key = -102
    $spec.deviceChange[1].device.ControllerKey = -101
    $spec.deviceChange[1].operation = "add"
    $vm2.ExtensionData.ReconfigVM($spec)

    it adds scsi controller and RDM to nodeA but not to NodeB, the error is same:

    Incompatible device backing specified for device '1'.

    + $vm2.ExtensionData.ReconfigVM <<<< ($spec)
        + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
        + FullyQualifiedErrorId : DotNetMethodException



  • 23.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 02:08 PM

    With "no way" you mean there is nothing in the vpxd logs on the vCenter ?



  • 24.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 02:54 PM

    I undertood the problem. there are 5 vmhosts in cluster. NodeA was working vmhost2 and NodeB was working on vmhost3.

    so, if I migrate nodeB to vmhost2, I dont ger error.

    or if I change -BusSharingMode "Virtual" to "Physical" then again I dont get the error. :smileyhappy:

    it's about if you focus on an error more than enough you may lose some easy things :smileyhappy:

    Thank you very much. I really appreciate.



  • 25.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 04:02 PM

    Indeed, for cross-Esxi sharing you need physical.

    Glad you found the solution



  • 26.  RE: Replicated LUNs and RDMs

    Posted Apr 16, 2013 04:17 PM

    I couldn't do without your help. thank you very much.