Need to consolidate months old snapshots on 2TB VM

[manofbronze]

Well, they say the most lasting lessons are those learned the hard way and it looks like this is going to be one of those lessons.

I have a VM running Windows 2008 Server 64bit with (2) virtual hard disks,a 100GB "OS/APPS only disk" with a single logical drive and a 2TB "data only" disk with a single logical drive. Both disks are virtual "thin provisioned" disks and reside on the same host storage volume. The (2) TB disk has over 1TB of data on it, most of which has been added since the last snapshot was taken. This is a high traffic system with significant data change on the 2TB volume. The host environment is VSphere ESX 4.0.1.

The problem? (3) snapshots left for over (2) months time that I want (need?) to consolidate. I simply forgot these snapshots existed and now I need to clean up after myself. I have read horror stories relating to the time it takes to consolidate long standing snapshots of production systems and have personally experienced a 4 hour consolidation on a much smaller system with far less activity (yes, I forgot to do house cleaning on more than one system ... arrgh!).

I am seeking any and all recommendations on how to go about consolidating these snapshots in such away that:

1. Minimizes risk of data corruption.

2. Can be performed while system is live. This is a requirement if the consolidation takes longer than 8 hours.

3. If possible, provides some measure of "roll back" protection in case of disaster during the consolidation.

I know I can simply perform a "delete all" on the snapshots, cross my fingers, toes and heart and hope for the best, but I would like to know if there are other steps I can, or should take before doing this.

Thanks in advance.

[maishsk]

That is a lot of data.

There is the option of actually converting the VM to a new VM using converter which will consolidate it all into one disk

You will need the additional space though in order to create the VM

If I am not mistaken it answers all of your 3 criteria.
Maish - VCP - vExpert 2010

VMware Communities User Moderator

Virtualization Architect & Systems Administrator

Twitter

[arturka]

Hi

Man, you are in big trouble :smileysad: A friend of mine had similar problem but snapshot had "only 25GB" - he had lost all data - vmdk file get corrupted

My advice:

1. Full backup of VM before snapshot removing

2. Prepare users for system slowness - if possible stop all activities on server to minimise risk of data corruption - you have really "big chance" to screw your system

3. Do snapshot removing outside working hours and less VM activity

4. BEFORE snapshot removing - create NEW snapshot - it will release old snapshot file from workload coming from system and redirect to new snapshot

5 Start deleting Snapshot starting from the oldest to the newest - at the end delete the newest snap (I don't know how long it can takes to remove all of them and how fast data is growing on your system but just to make sure create another one snapshot )

6 Be PATIENT - it will take a time to remove such a big snapshot - in case of Timeout on VI client check activity on Storage

Good Luck

visit my blog

if my answer it's useful click some stars :smileyhappy:

[Troy_Clavell]

I fully agree, VMware Converter is probably your best option in this case.

[RParker]

1. Minimizes risk of data corruption.

I don't agree with the other users. Snapshots just need TIME. IO will be a little high for while while the disk thrashes about, but I have yet to see a snapshot that did NOT commit. It just seems like they get hung. I would first try to commit it. You are on ESX 4, so the IO and stability is much better than previous ESX versions.

I say GO FOR IT! What do you have to lose? Oh that's right the VM.. Ah, where is your sense of adventure? It will work.. do it when IO is at the lowest point.

Also before you start, just make sure you have enough space to cover the snapshot AND the changes. So if it's a 2TB vmdk, 1TB snapshot you need at LEAST 1 TB of temp storage to commit those changes.. plus the time it takes to complete, so another 300GB (if there is a lot of traffic).

So you need 4.5TB (temporarily) then it will revert to the 2TB when its done, and the snapshot SHOULD delete. If you have that much free space.. then no reason not to try and commit first. Make a backup!

I know I can simply perform a "delete all" on the snapshots, cross my fingers, toes and heart and hope for the best,

That's the spirit! You can waste all that time an energy doing a convert, rebuilding the VM, deleting the original.. or do JUST that, hit the delete all, it will be fine. Sometimes the best course of action is to TRUST the technology... otherwise why do we have these $20K servers, $10K ESX license, and multi million dollar SANs, if they don't work right? That makes no sense.

[Borja_Mari]

Hi,

maybe upgrading to U2 is a good idea. U2 consolidates the snaphoots in a better way than U1.

Regards/Saludos,
Pablo

Please consider awarding

any helpful or corrrect answer. Thanks!! -

Por favor considera premiar

cualquier respuesta útil o correcta. ¡¡Muchas gracias!!

Virtually noob blog

[DSTAVERT]

I agree with Richard Parker. If you want additional comfort it might be time to use your VMware support.

[manofbronze]

I would like to thank everyone for your eclectic feedback; it was truly helpful.

Ultimately, I opted to follow Richard Parker's advice to "have faith" in the technology and infrastructure and just run the "delete all"; waiting the job out with crossed fingers and too many bitter memories of faith misplaced in technologies past. Time constraints made this seem to be the logical choice. Still, I will confess that I had a tense 42 hrs while the consolidation ran. It was both fascinating and worrisome to watch as the snapshots slowly merged into each other, consuming more and more disk space with each merge. Ultimately, the consolidate took an additional 700GB off the available disk space while it processed. When all was said an done it took 42hrs and freed up 1.75TB of disk space.

For those who might be interested, here is what I observed about the process:

1. The 95% completion indicator was WILDLY off target(as we already knew).

2. In addition to the existing (3), the consolidation generated a 4th snapshot for each of the (2) hard drives assigned to the VM. Since we were unable to quiescent the system, it seems this 4th snapshot was the ultimate repository of any data changes/additions that occurred during consolidation. This file grew to around 50Gb over the 42 hour period.

3, As pointed out in the artical referenced by Borja_Mari, the consolidate merged snapshot 3 with snapshot 2, then snapshot 2 with snapshot1. Ultimately, snapshot1 was committed to the original vmdk. Once Snapshot2 was merged with snapshot1, no significant data increase was noticed; just the slight growth in snapshot 4. While the worst case scenario of total free space consumed was not realized, it was pretty darn close, as snapshot1 ended up being 90% of the greatest possible size.

4. Apparently, the consolidation process switched the disk type of the original 2TB vmdk (VMAsigra1_1.vmdk) to "thick provisioned" because that is what it now shows as despite having been initially created as a "thin provisioned" disk. Can someone verify this for me?

I am a well chastened and will now pay much closer attention to snapshot cleanup.

Once again, my thanks to everyone for taking the time to submit your suggestions.

Best,