VMware vSphere

 View Only
  • 1.  vCenter Data Protection | Collection of Problems

    Posted Feb 25, 2013 01:17 AM

    All,

    I have recently started using the vCenter Data Protection (2TB VDP Appliance) and thought i'd share with you some of the problems and fixes that i've come across. It's documented articles explaining it's capabilities of variable block length de-duplication is great, and across all backup jobs over all time... just awesome. they claim it's capable of 98% data de-duplication on standard file shares... we shall wait and see.

    Setup:

    • Ensure that the Single Sign On capabilities are made AND create yourself a Backup User (giving them full administration privilidges inside your VCM)
    • Dozens of Live VMs running on centralised SAN storage (EMC VNX 5300 over iSCSI)
    • Deploy the OVF file to dedicated storage (I used a Freenas host with a 12x 450GB SAS Disk, Configured with a raid 6 data set, shared over multi-pathed iSCSI networking)
    • Storage vMotioned the Configuration File and Boot Drive of the VDP appliance back to SAN storage
    • Configuring the startup wizard using https://<IP of VDP>:8543/vdp-configure and rebooted the VM using the vSphere Client (right click > power > shutdown guest)
    • Before powering on the VM; giving it 8GB of RAM and 16 processors (4sockets x 4cores) - nice performance gain.  Powed it on now.
    • After waiting for 20 minutes, the vSphere Web-Client has the modules installed for managing VM backups: goto: https://<IP of VCM>:9443/vsphere-client/ and login

    This got me going..

    Issues:

    1. Backing up my first VM failed with Error Code: E10056.  Restore failed due to existing snapshot.

    Fixed with a Consolidate Disks command

    See here: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2036109

    2. Backing up the same VM failed again with Error Code: E10055.  Failed to attach disk

    Fixed by shutting down the VM; Editing Settings > Options > Advanced > General > Configuration Parameters

    Adding the following:  disk.EnableUUID = false

    See here: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2035736

    3.  Backup Jobs take a long time - sometimes taking in excess of 24 hours!

    Consider the first job as being 100% of the data being copied

    Subsequent jobs are DELTAs; however a good overhead of reads take place to compare the previous blocks.. so this too can take time on a slow backup store.

        • Check the Multi-pathing options
        • Give more cache to the iSCSI target
        • Enable battery backed write cache on the raid set
        • Migrate the VDP's Host OS disk and Configuration file away from the VDP's data disks
        • Enable Jumbo Frames for your iSCSI targets (specifically MTU=9000)

    4.  Still taking a long time to backup... 8+ hours..

    The only thing I could do here is the following:

    5.  Backup Job Fails with no error.. but I wanted to read the logs.. SSH to the VDP appliance and read the job log files:

         ssh root@vdp.localdomain

         <login>

         tail -f /usr/local/avamarclient/var-proxy-1/<JOBNAME>-<JOBID>-<GUID>-vmimagew.log

           <a few lines are cut out>

    -------------------------------------------------------------------------------------------------------------
    END avvcbimage log 2013-02-23 17:55:46 GMT  (0 warnings, 0 errors, 1 fatal error)
    --------------------------------------------------------------------------------------------------------

    Running a CAT of the log and hunting for the errors I found:

    2013-02-23 17:48:41 avvcbimage Error <0000>: vSphere Task failed (snapshot error=96): 'The operation is not allowed in the current state.'.

    2013-02-23 17:48:41 avvcbimage FATAL <16018>: The datastore information from VMX '[SAN|R6-C] SRV38/SRV38.vmx' will not permit a restore or backup.

    After a quick google: it's failed because the VM is either:

    • Mid-Booting (not quite loaded all services yet)
    • Missing VMware Tools

    I will post some more events and errors shortly..  Anyone else got stuff to share on this one?



  • 2.  RE: vCenter Data Protection | Collection of Problems

    Posted Mar 01, 2013 09:43 AM

    Another problem i've found.. actually on a seemingly successful backup..

    root@vdp:/usr/local/avamarclient/#: tail -f ./var-proxy-3/SRVMANFS02-1362037693052-0ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd-3016-vmimagew.log
    2013-02-28 23:36:14 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVimLoadVMCb
    2013-02-28 23:36:14 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVimLogout
    2013-02-28 23:36:14 avvcbimage Info <0000>: Re-enabled migrate and relocate operations on VM
    --------------------------------------------------------------------------------------------------------
    ----- END avvcbimage log 2013-02-28 23:36:17 GMT  (10 warnings, 0 errors, 0 fatal errors)
    --------------------------------------------------------------------------------------------------------
    root@vdp:/usr/local/avamarclient/#: cat ./var-proxy-3/SRVMANFS02-1362037693052-0ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd-3016-vmimagew.log | grep Warning
    2013-02-28 07:49:14 avvcbimage Warning <16004>: Soap fault detected, Get In-Use Block problem, Msg:'SOAP 1.1 fault: "":ServerFaultCode [no subcode]
    2013-02-28 07:49:14 avvcbimage Warning <14654>: The in-use blocks (pass 1) could not be found for 'VDP-13620377040ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd', using disk extents.
    2013-02-28 08:03:23 avvcbimage Warning <16004>: Soap fault detected, Get In-Use Block problem, Msg:'SOAP 1.1 fault: "":ServerFaultCode [no subcode]
    2013-02-28 08:03:23 avvcbimage Warning <14654>: The in-use blocks (pass 1) could not be found for 'VDP-13620377040ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd', using disk extents.
    2013-02-28 09:02:23 avvcbimage Warning <16004>: Soap fault detected, Get In-Use Block problem, Msg:'SOAP 1.1 fault: "":ServerFaultCode [no subcode]
    2013-02-28 09:02:23 avvcbimage Warning <14654>: The in-use blocks (pass 1) could not be found for 'VDP-13620377040ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd', using disk extents.
    2013-02-28 11:43:38 avvcbimage Warning <16004>: Soap fault detected, Get In-Use Block problem, Msg:'SOAP 1.1 fault: "":ServerFaultCode [no subcode]
    2013-02-28 11:43:38 avvcbimage Warning <14654>: The in-use blocks (pass 1) could not be found for 'VDP-13620377040ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd', using disk extents.
    2013-02-28 16:44:13 avvcbimage Warning <16004>: Soap fault detected, Get In-Use Block problem, Msg:'SOAP 1.1 fault: "":ServerFaultCode [no subcode]
    2013-02-28 16:44:13 avvcbimage Warning <14654>: The in-use blocks (pass 1) could not be found for 'VDP-13620377040ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd', using disk extents.
    root@vdp:/usr/local/avamarclient/#:

    Is this an actual problem??

    My assumptions are: "The VM is currently running and it's disks are in use; backing up at the file level instead of block level"..



  • 3.  RE: vCenter Data Protection | Collection of Problems

    Posted Jun 20, 2013 07:44 AM

    Another fundamental problem..

    After weeks of using this product.. backing up all my production fileserver VMs daily... I hit my first reason to do a file restore..

    Naturally I remote desktop'd to the server where the file was.. opened the VDP's File Level Restore page:

    https://vdp.appliance.web.page:9443/flr

    Logged in..

    I could see a list of mount points..  after mounting ANY of them, I cannot browse the file structure.. 

    Weeks pass after raising the issue with VMware.. their support literally took weeks until I kicked off that the retention policy will be clearing the content out shortly.. 5 remote sessions later and a rather large set of log files exchanged..

    VMware Data Proteciton and Data Protection Advanced does NOT support Windows 2008+ GPT partitions!!!

    Normally I would have used MBR partitions, but these are concert fileservers and I didn't even think about GPT being an issue anymore.  Evidently i'm wrong.



  • 4.  RE: vCenter Data Protection | Collection of Problems

    Posted Jun 20, 2013 08:14 AM

    none010101 wrote:

    Another problem i've found.. actually on a seemingly successful backup..

    root@vdp:/usr/local/avamarclient/#: tail -f ./var-proxy-3/SRVMANFS02-1362037693052-0ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd-3016-vmimagew.log
    2013-02-28 23:36:14 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVimLoadVMCb
    2013-02-28 23:36:14 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVimLogout
    2013-02-28 23:36:14 avvcbimage Info <0000>: Re-enabled migrate and relocate operations on VM
    --------------------------------------------------------------------------------------------------------
    ----- END avvcbimage log 2013-02-28 23:36:17 GMT  (10 warnings, 0 errors, 0 fatal errors)
    --------------------------------------------------------------------------------------------------------
    root@vdp:/usr/local/avamarclient/#: cat ./var-proxy-3/SRVMANFS02-1362037693052-0ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd-3016-vmimagew.log | grep Warning
    2013-02-28 07:49:14 avvcbimage Warning <16004>: Soap fault detected, Get In-Use Block problem, Msg:'SOAP 1.1 fault: "":ServerFaultCode [no subcode]
    2013-02-28 07:49:14 avvcbimage Warning <14654>: The in-use blocks (pass 1) could not be found for 'VDP-13620377040ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd', using disk extents.
    2013-02-28 08:03:23 avvcbimage Warning <16004>: Soap fault detected, Get In-Use Block problem, Msg:'SOAP 1.1 fault: "":ServerFaultCode [no subcode]
    2013-02-28 08:03:23 avvcbimage Warning <14654>: The in-use blocks (pass 1) could not be found for 'VDP-13620377040ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd', using disk extents.
    2013-02-28 09:02:23 avvcbimage Warning <16004>: Soap fault detected, Get In-Use Block problem, Msg:'SOAP 1.1 fault: "":ServerFaultCode [no subcode]
    2013-02-28 09:02:23 avvcbimage Warning <14654>: The in-use blocks (pass 1) could not be found for 'VDP-13620377040ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd', using disk extents.
    2013-02-28 11:43:38 avvcbimage Warning <16004>: Soap fault detected, Get In-Use Block problem, Msg:'SOAP 1.1 fault: "":ServerFaultCode [no subcode]
    2013-02-28 11:43:38 avvcbimage Warning <14654>: The in-use blocks (pass 1) could not be found for 'VDP-13620377040ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd', using disk extents.
    2013-02-28 16:44:13 avvcbimage Warning <16004>: Soap fault detected, Get In-Use Block problem, Msg:'SOAP 1.1 fault: "":ServerFaultCode [no subcode]
    2013-02-28 16:44:13 avvcbimage Warning <14654>: The in-use blocks (pass 1) could not be found for 'VDP-13620377040ee7d1dc29796b22dbfd5fb0a3fc3a90f23ae9cd', using disk extents.
    root@vdp:/usr/local/avamarclient/#:

    Is this an actual problem??

    My assumptions are: "The VM is currently running and it's disks are in use; backing up at the file level instead of block level"..

    Here is the explanation about the below error:

    2013-06-10 19:47:11 avvcbimage Warning <14654>: The in-use blocks (pass 1) could not be found for 'VDP-1370905261eb59958813df89e55ebd23a80818e3f1d002d209', using disk extents

    This is the period when VDDP API uses ctk to query the blocks that have changed since last backup. The querying is done using a hash of QueryChangedDiskAreas method  and an array of VirtualMachine.DiskChangeInfo.DiskChangeExtent.

    QueryChangedDiskAreas method is a VirtualMachine.DiskChangeInfo data object containing a description of the area of the vmdk disk covered (start offset and length), and VirtualMachine.DiskChangeInfo.DiskChangeExtent describes the starting offset and length of various disk areas that have changed.

    So, the disk extent referred here is the method used to query changed blocks and has nothing to do with extents. vmdk has no concept of extents.

    The above error is encountered when the concerned method is not able to query the in-use blocks which is taken during full backup of VM due to SOAP communication error. So instead of doing an incremental backup, it goes for a full backup again.



  • 5.  RE: vCenter Data Protection | Collection of Problems

    Posted Jun 25, 2013 12:47 PM

    Nice explanation - Thank you.