VMware vSphere

 View Only

  • 1.  orphan snapshots or something else wrong?

    Posted Jun 14, 2012 10:30 PM

    Hello all,

    I have 1 vm that seems to have a bunch of orphan snapshots (10 of them) that was created by VDR all within a 6 hrs period.  Odd thing is that I don't see any snapshot in the snapshot manager and when I tried to manually create a snapshot it fails (I've tried with all combination of 'snapshot the virtual machine's memory' and 'quiesce quest file system') and VDR fails everytime as well with creating snapshot.

    I tried to svmotion to another datastore and get same error.  I tried rebooting the vm, rebooted the esxi host, and rebooted the vCenter server.  I was able to vmotion the VM to another host but I'm still unable to svmotion nor do snapshots.

    The error I'm getting is

    ------------------------

    "The migration has exceeded the maximum switchover time of 100 seconds(s). ESX has preemptively failed the migration to allow the VM to continue running on the source.  To avoid this failure, either encrease the maximum allowable switchover time or wait until the VM is performing a less intensive workload."

    The VM failed to resume on the destination during early power on.

    Reason: the system cannot find the file specified.

    Cannot open the disk 'xxxxxxxxxxx_1-000010.vmdk' or one of the snapshot disks it depends on.

    VMware ESX cannot find the virtual disk 'xxxxxxxxxx_1-000010.vmdk'. Verify the path is valid and try again.

    ------------------------

    I can tell you that this VM is not doing anything intensive and as a matter of fact it is idle 99% of the time.

    I can see the xxxxxxxxxx_1-000010.vmdk file in the datastore where the VM resides.

    At this point I am open to any and all suggestion.

    Thanks.



  • 2.  RE: orphan snapshots or something else wrong?

    Posted Jun 14, 2012 11:03 PM

    Take a look at the VM's vmware.log to see whether this file contains any hints. If changed block tracking is enabled, it may be worth a try deleting all the ...-ctk.vmdk files from the datastore.

    André



  • 3.  RE: orphan snapshots or something else wrong?

    Posted Jun 14, 2012 11:53 PM

    Andre,

    In the vmware.log I see the following error so either the change tracking file is corrupt or the datastore is corrupt as I just encounter 2 more VMs on the same datastore that I can not svmotion off BUT I am able to create snapshots on those 2 other VMs.

    ---------------------

    2012-06-14T20:49:32.159Z| Worker#0| DISKLIB-CTK   : Could not open change tracking file "/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-ctk.vmdk": Change tracking invalid or disk in use.
    2012-06-14T20:49:32.160Z| Worker#0| DISKLIB-CTK   : Change tracking info is unclean. Queries will fail.
    2012-06-14T20:49:32.186Z| Worker#0| DISKLIB-VMFS  : "/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk" : closed.
    2012-06-14T20:49:32.188Z| Worker#0| DISKLIB-LIB   : Cannot remove extent `/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk': Device or resource busy
    2012-06-14T20:49:32.188Z| Worker#0| DISKLIB-LIB   : Failed to delete disk '/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010.vmdk' or one of its components: Device or resource busy
    2012-06-14T20:49:32.866Z| vmx| Migrate: Attempting to continue running on the source.
    2012-06-14T20:49:33.221Z| Worker#0| DISKLIB-VMFS  : "/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk" : open successful (1041) size = 2181124096, hd = 0. Type 8
    2012-06-14T20:49:33.221Z| Worker#0| DISKLIB-VMFS  : "/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk" : closed.
    2012-06-14T20:49:33.237Z| Worker#0| DISKLIB-LIB   : Cannot remove extent `/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk': Device or resource busy
    2012-06-14T20:49:33.237Z| Worker#0| DISKLIB-LIB   : Failed to delete disk '/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010.vmdk' or one of its components: Device or resource busy
    2012-06-14T20:49:35.327Z| Worker#0| DISKLIB-VMFS  : "/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk" : open successful (1041) size = 2181124096, hd = 0. Type 8
    2012-06-14T20:49:35.327Z| Worker#0| DISKLIB-VMFS  : "/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk" : closed.
    2012-06-14T20:49:35.334Z| Worker#0| DISKLIB-LIB   : Cannot remove extent `/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk': Device or resource busy
    2012-06-14T20:49:35.334Z| Worker#0| DISKLIB-LIB   : Failed to delete disk '/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010.vmdk' or one of its components: Device or resource busy
    2012-06-14T20:49:37.869Z| vmx| Migrate: Attempting to continue running on the source.
    2012-06-14T20:49:38.525Z| Worker#0| DISKLIB-VMFS  : "/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk" : open successful (1041) size = 2181124096, hd = 0. Type 8
    2012-06-14T20:49:38.526Z| Worker#0| DISKLIB-VMFS  : "/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk" : closed.
    2012-06-14T20:49:38.527Z| Worker#0| DISKLIB-LIB   : Cannot remove extent `/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk': Device or resource busy
    2012-06-14T20:49:38.527Z| Worker#0| DISKLIB-LIB   : Failed to delete disk '/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010.vmdk' or one of its components: Device or resource busy
    2012-06-14T20:49:42.592Z| Worker#0| DISKLIB-VMFS  : "/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk" : open successful (1041) size = 2181124096, hd = 0. Type 8
    2012-06-14T20:49:42.592Z| Worker#0| DISKLIB-VMFS  : "/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk" : closed.
    2012-06-14T20:49:42.632Z| Worker#0| DISKLIB-LIB   : Cannot remove extent `/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010-delta.vmdk': Device or resource busy
    2012-06-14T20:49:42.632Z| Worker#0| DISKLIB-LIB   : Failed to delete disk '/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010.vmdk' or one of its components: Device or resource busy
    2012-06-14T20:49:42.632Z| Worker#0| SVMotion: scsi0:0: SVMotionCleanupDisks: failed to delete the destination disk '/vmfs/volumes/4d51da9c-1b345953-0960-001e0b61d802/xxxxxx/xxxxxx-000010.vmdk' after failed migration: Device or resource busy.  Ignoring...
    2012-06-14T20:49:42.634Z| Worker#0| SVMotion: Enter Phase 0

    -------------------

    As for the -ctk.vmdk files do I just delete them and leave the correpsonding .vmdk files alone? ie.

    xxxxxxxx_1-000001.vmdk

    xxxxxxxx_1-000001-ctk.vmdk

    .

    .

    .

    xxxxxxxx_1-000010.vmdk

    xxxxxxxx_1-000010-ctk.vmdk

    My other question is none of these 10 snapshots shows up in the snapshot manager to delete so how do I merge/consolidate/delete them without losing any data that is not in the main .vmdk file.

    Below is what I have in the .vmx file and as you can see the 000010.vmdk file is reference there but none of the other 9 snapshots are.

    -----------------------------

    .encoding = "UTF-8"
    config.version = "8"
    virtualHW.version = "7"
    pciBridge0.present = "true"
    pciBridge4.present = "true"
    pciBridge4.virtualDev = "pcieRootPort"
    pciBridge4.functions = "8"
    pciBridge5.present = "true"
    pciBridge5.virtualDev = "pcieRootPort"
    pciBridge5.functions = "8"
    pciBridge6.present = "true"
    pciBridge6.virtualDev = "pcieRootPort"
    pciBridge6.functions = "8"
    pciBridge7.present = "true"
    pciBridge7.virtualDev = "pcieRootPort"
    pciBridge7.functions = "8"
    vmci0.present = "true"
    nvram = "xxxxxx.nvram"
    virtualHW.productCompatibility = "hosted"
    powerType.powerOff = "soft"
    powerType.powerOn = "hard"
    powerType.suspend = "hard"
    powerType.reset = "soft"
    displayName = "xxxxxx"
    extendedConfigFile = "xxxxxx.vmxf"
    scsi0.present = "true"
    scsi0.sharedBus = "none"
    scsi0.virtualDev = "lsisas1068"
    memsize = "4096"
    scsi0:0.present = "true"
    scsi0:0.fileName = "xxxxxx-000010.vmdk"
    scsi0:0.deviceType = "scsi-hardDisk"
    sched.scsi0:0.shares = "normal"
    sched.scsi0:0.throughputCap = "off"
    scsi0:1.present = "true"
    scsi0:1.fileName = "xxxxxx_1-000010.vmdk"
    scsi0:1.deviceType = "scsi-hardDisk"
    sched.scsi0:1.shares = "normal"
    sched.scsi0:1.throughputCap = "off"
    ide1:0.present = "true"
    ide1:0.clientDevice = "TRUE"
    ide1:0.fileName = "/usr/lib/vmware/isoimages/windows.iso"
    ide1:0.deviceType = "cdrom-raw"
    ide1:0.startConnected = "FALSE"
    ethernet0.present = "true"
    ethernet0.virtualDev = "e1000"
    ethernet0.networkName = "SERVERS"
    ethernet0.addressType = "vpx"
    ethernet0.generatedAddress = "00:50:56:a3:00:14"
    disk.EnableUUID = "true"
    guestOS = "longhorn"
    uuid.bios = "42 23 08 dc bb 2b e8 f6-88 a5 dd 29 fd 7b ae 95"
    vc.uuid = "50 23 be 3b 93 d1 10 ff-7c 6c c3 5d 26 00 29 7f"
    log.fileName = "vmware.log"
    snapshot.action = "keep"
    sched.cpu.min = "0"
    sched.cpu.units = "mhz"
    sched.cpu.shares = "normal"
    sched.mem.minsize = "0"
    sched.mem.shares = "normal"
    tools.upgrade.policy = "manual"
    replay.supported = "TRUE"
    unity.wasCapable = "FALSE"
    replay.filename = ""
    scsi0:0.redo = ""
    pciBridge0.pciSlotNumber = "17"
    pciBridge4.pciSlotNumber = "21"
    pciBridge5.pciSlotNumber = "22"
    pciBridge6.pciSlotNumber = "23"
    pciBridge7.pciSlotNumber = "24"
    scsi0.pciSlotNumber = "160"
    ethernet0.pciSlotNumber = "32"
    vmci0.pciSlotNumber = "33"
    scsi0.sasWWID = "50 05 05 6c bb 2b e8 f0"
    vmotion.checkpointFBSize = "4194304"
    hostCPUID.0 = "0000000a756e65476c65746e49656e69"
    hostCPUID.1 = "0001067600040800000ce3bdbfebfbff"
    hostCPUID.80000001 = "00000000000000000000000120100800"
    guestCPUID.0 = "0000000a756e65476c65746e49656e69"
    guestCPUID.1 = "0001067600010800800822010febfbff"
    guestCPUID.80000001 = "00000000000000000000000120100800"
    userCPUID.0 = "0000000a756e65476c65746e49656e69"
    userCPUID.1 = "0001067600040800000822010febfbff"
    userCPUID.80000001 = "00000000000000000000000120100800"
    evcCompatibilityMode = "TRUE"
    mem.hotadd = "true"
    scsi0:1.ctkEnabled = "TRUE"
    scsi0:1.redo = ""
    scsi0:0.ctkEnabled = "TRUE"
    vmci0.id = "-42226027"
    tools.syncTime = "FALSE"
    uuid.location = "56 4d 14 73 c4 bf 51 d1-9f 39 39 2a 20 c8 07 38"
    cleanShutdown = "FALSE"
    migrate.hostlog = "./xxxxxx-7955f14f.hlog"
    ctkEnabled = "TRUE"
    sched.mem.max = "unlimited"
    sched.swap.derivedName = "/vmfs/volumes/4e52a727-92de126d-dbbf-002264983bf0/xxxxxx/xxxxxx-7955f14f.vswp"
    scsi0:2.deviceType = "scsi-hardDisk"
    scsi0:3.deviceType = "scsi-hardDisk"
    floppy0.present = "FALSE"
    config.readOnly = "FALSE"

    -----------------------



  • 4.  RE: orphan snapshots or something else wrong?
    Best Answer

    Posted Jun 15, 2012 12:09 AM
    As for the -ctk.vmdk files do I just delete them and leave the correpsonding .vmdk files alone?

    Yes, only delete the ...-ctk.vmdk files. These files contain the CBT information and will automatically be recreated with the next snapshot.

    My other question is none of these 10 snapshots shows up in the snapshot  manager to delete so how do I merge/consolidate/delete them without  losing any data that is not in the main .vmdk file.

    Most backup applications are lazy and don't update the VM's .vmsd file which contains the information presented in the Snapshot Manager. To delete all snapshots you may (once it works) create another snapshot and then click "Delete All". If you are running a version of ESXi 4.0 prior to Update 2 (which I don't assume, since you posted in the ESXi 5.0 forum) and/or the virtual disk is thin provisioned, you may need additional disk space to consolidate the snaphots.

    Below is what I have in the .vmx file and as you can see the 000010.vmdk  file is reference there but none of the other 9 snapshots are.

    Snapsots in VMware products work as a chain (see http://kb.vmware.com/kb/1015180). Only the latest snapshot is mentioned in the .vmx file, but all files are in use.

    André



  • 5.  RE: orphan snapshots or something else wrong?

    Posted Jun 15, 2012 08:39 PM

    Andre,

    That worked!  After I deleted all of the -ctk.vmdk files I was able to created a new snapshot and then did a 'delete all' which cleaned up everything.

    Thanks.