ESXi

 View Only
  • 1.  Veeam created 256 snapshots VM unresponsive, needs consolidation

    Posted Apr 12, 2022 08:59 AM

    Hi

    We are running an old esxi 6.0 and unfortuantely something has tricked us

    We now have a VM running on a snapshot with the number 256

    In the vmware log, it opens all the snapshots and tries to boot, but it now has been running for over an hour and has not completed the boot yet

    2022-04-12T07:36:44.887Z| Worker#1| I120: DISKLIB-LINK : Opened '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000256.vmdk' (0xa): vmfsSparse, 251658240 sectors / 120 GB.
    2022-04-12T07:36:44.891Z| vmx| I120: DISKLIB-CBT : Initializing ESX kernel change tracking for fid 532625.
    2022-04-12T07:36:44.891Z| vmx| I120: DISKLIB-CBT : Successfuly created cbt node 82091-cbt.
    2022-04-12T07:36:44.891Z| vmx| I120: DISKLIB-CBT : Opening cbt node /vmfs/devices/cbt/82091-cbt
    2022-04-12T07:36:44.891Z| vmx| I120: DISKLIB-LIB : Opened "/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB_1.vmdk" (flags 0xa, type vmfs).
    2022-04-12T07:36:44.892Z| vmx| I120: DISK: Disk '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB_1.vmdk' has UUID '60 00 c2 96 38 74 9e ce-15 ac c9 2d ab f4 f1 27'
    2022-04-12T07:36:44.892Z| vmx| I120: DISK: OPEN '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB_1.vmdk' Geo (13054/255/63) BIOS Geo (0/0/0)
    2022-04-12T07:36:44.892Z| vmx| I120: UTIL: Change file descriptor limit from soft 4096,hard 4096 to soft 4211,hard 4211.
    2022-04-12T07:36:44.925Z| Worker#1| I120: DISKLIB-VMFS : "/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000255-delta.vmdk" : open successful (14) size = 302239744, hd = 598165. Type 8
    2022-04-12T07:36:44.925Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "DDB-000255-delta.vmdk" (0xe)
    2022-04-12T07:36:44.925Z| Worker#1| I120: DISKLIB-LINK : Opened '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000255.vmdk' (0xe): vmfsSparse, 251658240 sectors / 120 GB.
    2022-04-12T07:36:44.941Z| Worker#1| I120: DISKLIB-VMFS : "/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000254-delta.vmdk" : open successful (14) size = 1510199296, hd = 663702. Type 8
    2022-04-12T07:36:44.941Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "DDB-000254-delta.vmdk" (0xe)
    2022-04-12T07:36:44.941Z| Worker#1| I120: DISKLIB-LINK : Opened '/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000254.vmdk' (0xe): vmfsSparse, 251658240 sectors / 120 GB.
    2022-04-12T07:36:44.951Z| Worker#1| I120: DISKLIB-VMFS : "/vmfs/volumes/4fe079f5-ab353d1f-3b4b-e41f13eb4db2/DDB/DDB-000253-delta.vmdk" : open successful (14) size = 17027072, hd = 663703. Type 8
    2022-04-12T07:36:44.951Z| Worker#1| I120: DISKLIB-DSCPTR: Opened [0]: "DDB-000253-delta.vmdk" (0xe)

    ..... and so on....

    Is there any way I can consolidate that many snapshots?

    They are not visible in the snapshot manager

    Can something be done via SSH?

    The backup is not even complete so we can not restore from it

    Thanks



  • 2.  RE: Veeam created 256 snapshots VM unresponsive, needs consolidation

    Posted Apr 12, 2022 09:51 AM

    Suggested procedure has been tested hundred of times and is safe. ...
    Power off VM.
    Consolidate first 100 snapshots via vmkfstools into a new directory.
    Consolidate next 100 snapshots via vmkfstools ....
    Consolidate remaining snapshots ....

    Option A: do it on the fly with a few very short downtimes
    Option B: do it offline: this is the faster option

    Dont do this if you are unexperienced with editing vmdk-descriptorfiles !!!

    Ulli



  • 3.  RE: Veeam created 256 snapshots VM unresponsive, needs consolidation

    Posted Apr 12, 2022 10:00 AM

    Hi

    I have no option as to try and do it anyways

    What is required to edit the descriptor files?



  • 4.  RE: Veeam created 256 snapshots VM unresponsive, needs consolidation

    Posted Apr 12, 2022 10:03 AM

    I highly recommend that we do it together - feel free to call me via skype then we should be back in save waters in a few hours.
    We will need enough free space to store the basedisk / or basedisks in a new directory. Expect thick provisioned requirements ....

    Ulli - skype: sanbarrow
    If you are in germany - send a PM with your phonenumber

    About editing descriptorfiles: I recommend to strictly use the embedded editor of WinSCP - it has the lowest failure-rate especially for newbies.

     



  • 5.  RE: Veeam created 256 snapshots VM unresponsive, needs consolidation

    Posted Apr 12, 2022 10:14 AM

    Here is the plan:
    - power off VM
    - vmkfstools -i snapshot000090.vmdk newdir/basedisk.vmdk
    - when done edit snapshot000091.vmdk so that it points to newdir/basedisk.vmdk
    - vmkfstools -i snapshot000182.vmdk newdir/basedisk2.vmdk
    - when done edit snapshot000183.vmdk so that it points to newdir/basedisk2.vmdk
    - vmkfstools -i snapshot000255.vmdk newdir/basedisk3.vmdk
    - clean up and rename back to standard nameing conventions



  • 6.  RE: Veeam created 256 snapshots VM unresponsive, needs consolidation

    Posted Apr 12, 2022 10:16 AM

    Can I call you by phone I only have access to teams

    I´m in Denmark.. no problem

    \Lars



  • 7.  RE: Veeam created 256 snapshots VM unresponsive, needs consolidation

    Posted Apr 12, 2022 10:27 AM

    check your PMs
    I expect you have Teamviewer, Putty and WinSCP installed on your adminhost
    Please dont use a Mac-adminhost - thats too painful ...

     

    Ulli



  • 8.  RE: Veeam created 256 snapshots VM unresponsive, needs consolidation

    Posted Apr 12, 2022 10:45 AM

    we just started the consolidation at snap000090.

    Ulli



  • 9.  RE: Veeam created 256 snapshots VM unresponsive, needs consolidation

    Posted Apr 12, 2022 12:09 PM

    started consolidation with snapshot000180

     

    Ulli



  • 10.  RE: Veeam created 256 snapshots VM unresponsive, needs consolidation

    Posted Apr 12, 2022 03:03 PM

    all done successfully

     

    Ulli



  • 11.  RE: Veeam created 256 snapshots VM unresponsive, needs consolidation

    Posted Apr 13, 2022 03:29 PM

    Everything is back to normal

    Thanks to Ullis effort, and superb guidance in helping me with this matter.