Manually cleaning up CTK files

[adrianmarshubi]

Hi,

So my question is how to clean up CTK files after a Veeam backup fails to remove a snapshot.

I've an ESX 4.1 Vsphere 4.1 setup.  One of my VMs, our SQL server has been erroring in Veeam the past few runs complaining that it couldn't clean up the snapshots complaining that a file lock was in place.

I wasn't able to clone the VM for the same reason whilst running live.

After powering the VM down, I've been able clone the VM ok, and the fles all look ok in the clone.  The VM has one 80gb thick disk, and one 100gb thick disk.

In the original VM though I'm trying to understand whats happened with the disks.

Below's a file listing from the host.  I know the CTK files and delta files are to do with Change blocking. I don't know how to clean them up though now.

There are no snapshots remaining on the VM.

I've been able to run Veeam successfully with the VM powered off, hoping it would clean up, but it hasn't.

I'll try it again with the VM powered but I expect I'll hit the same issue again after a while.

I tried running a vmfstools:

[root@UBIQ-SERV21 UBIQ-SERV15.ubiquisys.local]# vmkfstools -x check UBIQ-SERV15_1.vmdk -v 9
DISKLIB-VMFS  : "./UBIQ-SERV15_1-flat.vmdk" : open successful (285) size = 107374182400, hd = 0. Type 3
DISKLIB-VMFS  : "./UBIQ-SERV15_1-flat.vmdk" : closed.
Disk is error free
AIOMGR-S : stat o=1 r=3 w=0 i=0 br=49152 bw=0

I think I can move the cloned machine in as a substitute failing everything else, but I do want to know how to deal with these files.

Heres the file list:

-rw------- 1 root root      5243392 Jun 30 22:36 UBIQ-SERV15-000001-ctk.vmdk
-rw------- 1 root root   4932669440 Jul  1 22:30 UBIQ-SERV15-000001-delta.vmdk
-rw------- 1 root root          419 Jun 25 22:35 UBIQ-SERV15-000001.vmdk
-rw------- 1 root root      5243392 Jul  1 22:31 UBIQ-SERV15-000002-ctk.vmdk
-rw------- 1 root root     16945152 Jul  1 22:35 UBIQ-SERV15-000002-delta.vmdk
-rw------- 1 root root          426 Jul  5 10:34 UBIQ-SERV15-000002.vmdk
-rw------- 1 root root      5243392 Jul  1 22:35 UBIQ-SERV15-000003-ctk.vmdk
-rw------- 1 root root    184717312 Jul  2 22:30 UBIQ-SERV15-000003-delta.vmdk
-rw------- 1 root root          426 Jul  2 22:35 UBIQ-SERV15-000003.vmdk
-rw------- 1 root root      5243392 Jul  2 22:31 UBIQ-SERV15-000004-ctk.vmdk
-rw------- 1 root root    201494528 Jul  3 22:31 UBIQ-SERV15-000004-delta.vmdk
-rw------- 1 root root          426 Jul  3 23:33 UBIQ-SERV15-000004.vmdk
-rw------- 1 root root      5243392 Jul  3 22:31 UBIQ-SERV15-000005-ctk.vmdk
-rw------- 1 root root    218271744 Jul  4 22:31 UBIQ-SERV15-000005-delta.vmdk
-rw------- 1 root root          426 Jul  4 22:36 UBIQ-SERV15-000005.vmdk
-rw------- 1 root root      5243392 Jul  4 22:31 UBIQ-SERV15-000006-ctk.vmdk
-rw------- 1 root root    218271744 Jul  5 10:34 UBIQ-SERV15-000006-delta.vmdk
-rw------- 1 root root          426 Jul  5 12:33 UBIQ-SERV15-000006.vmdk
-rw------- 1 root root      5243392 Jul  5 10:44 UBIQ-SERV15-000007-ctk.vmdk
-rw------- 1 root root     16945152 Jul  5 10:44 UBIQ-SERV15-000007-delta.vmdk
-rw------- 1 root root          426 Jul  5 11:31 UBIQ-SERV15-000007.vmdk
-rw------- 1 root root      5243392 Jul  5 10:44 UBIQ-SERV15-000008-ctk.vmdk
-rw------- 1 root root     84054016 Jul  5 12:33 UBIQ-SERV15-000008-delta.vmdk
-rw------- 1 root root          426 Jul  5 19:30 UBIQ-SERV15-000008.vmdk
-rw------- 1 root root      5243392 Jul  5 12:33 UBIQ-SERV15-000009-ctk.vmdk
-rw------- 1 root root    134385664 Jul  5 19:26 UBIQ-SERV15-000009-delta.vmdk
-rw------- 1 root root          426 Jul  5 20:50 UBIQ-SERV15-000009.vmdk
-rw------- 1 root root      5243392 Jul  5 20:50 UBIQ-SERV15-000010-ctk.vmdk
-rw------- 1 root root       167936 Jul  5 20:50 UBIQ-SERV15-000010-delta.vmdk
-rw------- 1 root root          371 Jul  5 22:01 UBIQ-SERV15-000010.vmdk
-rw------- 1 root root      5243392 Jul  5 20:52 UBIQ-SERV15-000011-ctk.vmdk
-rw------- 1 root root       167936 Jul  5 20:52 UBIQ-SERV15-000011-delta.vmdk
-rw------- 1 root root          348 Jul  5 20:52 UBIQ-SERV15-000011.vmdk
-rw------- 1 root root      6554112 Jun 30 22:36 UBIQ-SERV15_1-000001-ctk.vmdk
-rw------- 1 root root   1929588736 Jul  1 22:31 UBIQ-SERV15_1-000001-delta.vmdk
-rw------- 1 root root          425 May 24 22:38 UBIQ-SERV15_1-000001.vmdk
-rw------- 1 root root      6554112 Jul  1 22:31 UBIQ-SERV15_1-000002-ctk.vmdk
-rw------- 1 root root     16986112 Jul  1 22:35 UBIQ-SERV15_1-000002-delta.vmdk
-rw------- 1 root root          432 Jul  5 10:34 UBIQ-SERV15_1-000002.vmdk
-rw------- 1 root root      6554112 Jul  1 22:35 UBIQ-SERV15_1-000003-ctk.vmdk
-rw------- 1 root root   1241722880 Jul  2 22:30 UBIQ-SERV15_1-000003-delta.vmdk
-rw------- 1 root root          432 Jul  2 22:35 UBIQ-SERV15_1-000003.vmdk
-rw------- 1 root root      6554112 Jul  2 22:31 UBIQ-SERV15_1-000004-ctk.vmdk
-rw------- 1 root root   1241722880 Jul  3 22:31 UBIQ-SERV15_1-000004-delta.vmdk
-rw------- 1 root root          432 Jul  3 23:33 UBIQ-SERV15_1-000004.vmdk
-rw------- 1 root root      6554112 Jul  3 22:31 UBIQ-SERV15_1-000005-ctk.vmdk
-rw------- 1 root root   1292054528 Jul  4 22:31 UBIQ-SERV15_1-000005-delta.vmdk
-rw------- 1 root root          432 Jul  4 22:36 UBIQ-SERV15_1-000005.vmdk
-rw------- 1 root root      6554112 Jul  4 22:31 UBIQ-SERV15_1-000006-ctk.vmdk
-rw------- 1 root root   1224945664 Jul  5 10:33 UBIQ-SERV15_1-000006-delta.vmdk
-rw------- 1 root root          432 Jul  5 12:33 UBIQ-SERV15_1-000006.vmdk
-rw------- 1 root root      6554112 Jul  5 10:44 UBIQ-SERV15_1-000007-ctk.vmdk
-rw------- 1 root root     16986112 Jul  5 10:44 UBIQ-SERV15_1-000007-delta.vmdk
-rw------- 1 root root          432 Jul  5 11:31 UBIQ-SERV15_1-000007.vmdk
-rw------- 1 root root      6554112 Jul  5 10:44 UBIQ-SERV15_1-000008-ctk.vmdk
-rw------- 1 root root     16986112 Jul  5 12:33 UBIQ-SERV15_1-000008-delta.vmdk
-rw------- 1 root root          432 Jul  5 19:30 UBIQ-SERV15_1-000008.vmdk
-rw------- 1 root root      6554112 Jul  5 19:26 UBIQ-SERV15_1-000009-ctk.vmdk
-rw------- 1 root root     50540544 Jul  5 19:25 UBIQ-SERV15_1-000009-delta.vmdk
-rw------- 1 root root          432 Jul  5 20:50 UBIQ-SERV15_1-000009.vmdk
-rw------- 1 root root      6554112 Jul  5 20:50 UBIQ-SERV15_1-000010-ctk.vmdk
-rw------- 1 root root       208896 Jul  5 20:50 UBIQ-SERV15_1-000010-delta.vmdk
-rw------- 1 root root          377 Jul  5 22:01 UBIQ-SERV15_1-000010.vmdk
-rw------- 1 root root      6554112 Jul  5 20:52 UBIQ-SERV15_1-000011-ctk.vmdk
-rw------- 1 root root       208896 Jul  5 20:52 UBIQ-SERV15_1-000011-delta.vmdk
-rw------- 1 root root          354 Jul  5 20:52 UBIQ-SERV15_1-000011.vmdk
-rw------- 1 root root      6554112 May 23 22:34 UBIQ-SERV15_1-ctk.vmdk
-rw------- 1 root root 107374182400 May 24 17:30 UBIQ-SERV15_1-flat.vmdk
-rw------- 1 root root          590 Apr 18 22:33 UBIQ-SERV15_1.vmdk
-rw------- 1 root root      5243392 May 23 22:34 UBIQ-SERV15-ctk.vmdk
-rw------- 1 root root  85899345920 May 24 17:30 UBIQ-SERV15-flat.vmdk
-rw------- 1 root root     17350656 Nov  1  2011 UBIQ-SERV15.ubiquisys.local-000001-delta.vmdk
-rw------- 1 root root          381 Nov  1  2011 UBIQ-SERV15.ubiquisys.local-000001.vmdk
-rw------- 1 root root          612 Apr 18 22:33 UBIQ-SERV15.vmdk

[a_p_]

Welcome to the Community,

There are no snapshots remaining on the VM.

You may not see any snapshots in the Snapshot Manager, but the "00000x" files are snapshots. Most backup applications don't maintain the VM's .vmsd file which contains the information for the Snapshot Manager.

... has been erroring in Veeam the past few runs complaining that it  couldn't clean up the snapshots complaining that a file lock was in  place.

What you need to do is to check the log file to find out which file is locked. Once the file lock is removed you could - assuming you use thick provisioned virtual disks - open the Snapshot Manager, create a new snapshot and then hit the "Delete All" button. This should commit all deltas into the base (flat) disks and afterwards remove the "00000x" files from the datastore.

André

[adrianmarshubi]

Hi Andre,

Thanks for the info... When I shutdown the VM, I also migrated it to another Host, as I figured that process would have to free the lock up.

I just created a new snapshot, and in the manager it says theres one called 'Consolidate Helper-0' above it.

When I get the error, it always says 'Unable to access file <unspecified filename> since it is locked' when complaining about the lock, so I've no idea which is the locked file.

----

Ok, so I tried again, but it failed with the same error again.

So now I've powered the VM down, created another snapshot, and run the Delete all (selectiing Consolidate Helper-0.  This time, powered off, theres been no error, but the file listing still shows the deltas so I don't think it worked.

[a_p_]

With the snapshots you have, the "Delete All" may run some time. Do you see the Delete Snapshots task in the task list?

Btw. the Consolidate-Helper snapshot is a snapshot which is created automatically by ESXi if you delete the snapshots with the VM powered on.

Regarding the file lock, please check whether http://www.veeam.com/KB1213 helps.

If this also does not help, you may delete all the "ctk" files from the VM's folder (be very careful, to not delete any other files!). I've seen discussions where the ctk files caused issues.

André

[adrianmarshubi]

Hmmm.

ok thanks for that.

I decided to switch over to using the Cloned machine, that seemed to have files that made sense. But then I thought I'd try a manual snapshot and s/shot delete.  I think its worked ok as the clone only has one set of CTK files (list below)

Yup I see the snapshot messages in the Tasks list.

I'm not sure what you meant about the Consolodate s/shots.  You're saying it creates a snapshot when I delete snapshots ??

I'll try a full veeam backup on the clone now to see if that has an issue.

-rw------- 1 root root      6554112 Jul  5 23:11 copy-serv15_1-ctk.vmdk
-rw------- 1 root root 107374182400 Jul  5 23:12 copy-serv15_1-flat.vmdk
-rw------- 1 root root          590 Jul  5 20:18 copy-serv15_1.vmdk
-rw-r--r-- 1 root root           13 Jul  5 23:10 copy-serv15-aux.xml
-rw------- 1 root root      5243392 Jul  5 23:11 copy-serv15-ctk.vmdk
-rw------- 1 root root   8489271296 Jul  5 22:41 copy-serv15-d60e3160.vswp
-rw------- 1 root root  85899345920 Jul  5 23:12 copy-serv15-flat.vmdk
-rw------- 1 root root         8684 Jul  5 22:56 copy-serv15.nvram
-rw------- 1 root root          586 Jul  5 20:18 copy-serv15.vmdk
-rw-r--r-- 1 root root           43 Jul  5 23:10 copy-serv15.vmsd
-rwxr-xr-x 1 root root         3461 Jul  5 23:10 copy-serv15.vmx
-rw-r--r-- 1 root root          266 Jul  5 22:54 copy-serv15.vmxf
-rw-r--r-- 1 root root        83980 Jul  5 19:31 vmware-16.log
-rw-r--r-- 1 root root      1359399 Jul  5 19:31 vmware-17.log
-rw-r--r-- 1 root root        85634 Jul  5 19:31 vmware-18.log
-rw-r--r-- 1 root root       514700 Jul  5 19:31 vmware-19.log
-rw-r--r-- 1 root root      1165066 Jul  5 19:31 vmware-20.log
-rw-r--r-- 1 root root       156988 Jul  5 20:48 vmware-21.log
-rw-r--r-- 1 root root       331536 Jul  5 23:10 vmware.log

[a_p_]

I think what you did - switching over to the clone - was the best solution in this case.

I'm not sure what you meant about the Consolodate s/shots.  You're saying it creates a snapshot when I delete snapshots ??

Yes, the helper snapshot is created temporarily. This way all changes to the VM are written to the helper snapshot while all the other existing snapshots are committed/merged into the base disk. Once this is done, the helper snapshot gets committed too.

André

[adrianmarshubi]

Ok,  well thanks for your help anyway.

This could be something thats been fixed in v5 or in a 4.1 Service Pack I guess.  Although we only installed last year we're already starting to get behind.

For now the new VM seems to be working ok.

[adrianmarshubi]

Its not clear what the issue was, or what the fix is - but I've marked it answered as at least I was doing the right thing