Snapshot won’t delete

[coastalitsuppor]

Hi all,

wandering if someone can please assist.

The backup program creates a snap shot as per usual, however if we consolidate says complete but nothing happens. If you try remove a snapshot manually single, all, children etc keeps failing. I have tried through powercli also and same see error below. We have had to stop backups as our data store is about to run out of space. It’s creating 400GB snapshots and we have about 14 we need removed. Even if we remove every snap shot and start fresh don’t mind.

 thank you

[NathanosBlightc]

First, check the detailed information of the *.vmsd file in this VM's directory and compare the snapshots that you couldn't remove, are they mentioned there or not? ...

Then try consolidation operation again while you directly connected to the ESXi host (not via the Center server or executing Power-CLI) then give back the result of this, please.

[Ashil Krishnan]

Hi ,

If Veeam is a VM within vSphere environment, go to edit settings of it and ensure it does not have the disk of the problematic VM attached to it. If yes, detach it.

Create a test snapshot within vSphere and initiate delete all to consolidate the snapshots

[continuum]

Discussing snapshots problem without details is a bit risky .
Send all descriptor-vmdks, the vmsd and latest 2 vmware.logs.

Do the snpshot files appear in /dev/disks/deltadisks/ ?

When you use automatic backup-tools you MUST also use daily monitoring because if you run out of space with a sesparse snapshot it probably gets corrupted.

[coastalitsuppor]

Hi all,

Appreciate the feedback. Please see attached latest log and vsrm file as only allows me to upload the ZIP. The screen shot below you can see in the data store we actually have about 24 snapshots attached.

 

[continuum]

Rule number one when troubleshooting snapshots: DONT USE DATASTORE-BROWSER
When troubleshooting snapshots you need to see the content of the directory as it really is - not the dumbed down oversimplified version that you see in the datastorebrowser.
Easiest tool is WinSCP
With WinSCP you will also see the vmdk-descriptorfiles that I asked for ...

from the vmware.log:

SNAPSHOT: SnapshotGenerateDeleteDisks Failed to fetch disk tree: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)
SNAPSHOT: SnapshotDeleteNode failed: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)
SNAPSHOT: Snapshot_Delete failed: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)

from the vmsd:
snapshot0.disk0.fileName = "SERVER 2019 DC-000004.vmdk"
snapshot2.disk0.fileName = "SERVER 2019 DC-000004.vmdk" !!! vmdk appears twice = vmsd is bad

This means you have a bad vmsd - and the result is Veeam can not clean up after each backup = you will run out of free space soon.
Because of the bad vmsd you also cant use the Consolidation function.

Suggested fix : created empty sub-directory "deleteme-later" and move the vmsd-file into it.

Ulli

 

[coastalitsuppor]

Hi Ulli,

 

Thanks for the updated and noted for the DS browser. Appreciate the advice.

 

So from understanding below, if just create a blank folder in the DS and move the vmdd file into it, that should allow me to consolidate or delete the snap shots? Guessing the VM would have to be shutdown or restarted after moving the file to rebuild a new one?

Thank you appreciate you advice and time!

Mark

[continuum]

short answer via phone

the suggest fix is a complete list, dont do anything else.

No need to reboot  - a running vm does not need a vmsd.

If winscp does not allow to move the vmsd - rename it to invalid.vmsd , wait half a minute and then move to deleteme-later dir.

Next Veeam backup should be a full one. 

snaphotmanager in the gui will be confused and display nothing. you should not use  it when veeam jobs are running so thats no problem either.

When you want to consolidate just use create new and delete all.

Ulli

[coastalitsuppor]

Ok thank you. Have renamed and can see its cleared the previous snapshot names. Then can go back and manually delete all those old ones? Tried pressing consolidate but nothing happens and the 26 existing snapshots are still there

[a_p_]

Did you already check whether one of the VM's .vmdk files (virtual disks) is still mounted to one of the backup servers, or proxies?
I've seen issues like this a few times when backup jobs failed unexpectedly, and the backup server was not able to do the cleanup (unmont the virtual disk).

André

[continuum]

> Tried pressing consolidate but nothing happens and the 26 existing snapshots are still there

I told you that snapshotmanager will be confused and that you should not use it.
So instead create a new snapshot and then use the delete-all function.

 
Andre - if I remember right the error message would be different and would complain about locked vmdks - not about "already in use"

It would help to see the content of /dev/deltadisks ...

[coastalitsuppor]

Anyone able to give me some help? The DC wont turn on now. Anyone on Skype even able to assist???

[coastalitsuppor]

Failed - Unable to enumerate all disks.

Errors

•Unable to enumerate all disks.
•A required file was not found

[Scott Vessey]

 

Moderator: Please note the following:

1. Moved this thread to the Backup & Recovery Discussions area

2. Please do not create duplicate threads for the same issue - the second thread you created has been archived

[continuum]

Sorry - last night I was too tired to help via skype ...

Please try to provide more details - we dont know anything about your DC - is that the same VM that we discussed already ?
Quoting error messages without details does not help either.
If you receive an error message - it will appear in the vmware.log.
In the vmware.log you will see details that are necessary to deal with the problem.

If you read your last post - the obvious question is "which file ?"
Please answer all obvious questions when you post a question - that will help to find a solution faster.

Ulli

[continuum]

> Discussing snapshots problem without details is a bit risky .

When the details are available it is so much easier ...
We just needed less than an hour via Teamviewer to fix the problem.

Ulli

[coastalitsuppor]

Hi,

This is what's listed in the dev/disks as you asked also.

 

[DizZy_64]

Hi!

I got almost the same problem (see attachments):

from the VMWare log after vim-cmd vmsvc/snapshot.removeall 37 :

SNAPSHOT: SnapshotDiskTreeAddFromSnapshot: Trying to add snapshot SRV-E3-Snapshot574.vmsn to disk /vmfs/volumes/58beb247-340b1d1a-fc73-1402ec905020/SRV-E3/SRV-E3_2.vmdk which already has snapshot SRV-E3-Snapshot573.vmsn.
SNAPSHOT: SnapshotGenerateDeleteDisks Failed to fetch disk tree: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)
SNAPSHOT: SnapshotDeleteNode failed: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)
SNAPSHOT: Snapshot_Delete failed: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot (21)

from the vmsd:

snapshot0.uid = "573"
snapshot0.disk0.fileName = "SRV-E3_2.vmdk"
snapshot0.disk1.fileName = "SRV-E3_3.vmdk"
snapshot0.disk2.fileName = "SRV-E3_4.vmdk"

snapshot1.uid = "574"
snapshot1.disk0.fileName = "SRV-E3_2.vmdk"
snapshot1.disk1.fileName = "SRV-E3_3.vmdk"
snapshot1.disk2.fileName = "SRV-E3_4.vmdk"

I suppose the snapshot1.diskx.filename should be SRV-E3_x-000001.vmdk.

I see two options:

1) Should I try to manually correct the vmsd file and retry the vim-cmd vmsvc/snapshot.removeall 37?

2) Should I, as suggested in your solution,move the vmsd file?

In case of option 2, if I understand correctly the next steps are:

1) Create a new snapshot (vim-cmd vmsvc/snapshot.create which will create a new vmsd

2) Attempt a new vim-cmd vmsvc/snapshot.removeall 37

Is this correct? Do I have to shutdown the server? (It's my production Exchange server...)

Attachment(s)

SRV-E3.vmsd.txt   3 KB 1 version

vmware-log-extract.log   8 KB 1 version

content of dir.txt   7 KB 1 version

vicmd snapshot get result.txt   1 KB 1 version

[coastalitsuppor]

Thank you to Sanbarrow (Ulli) who managed to get this fixed. Was very prompt, very helpful and really knows his stuff was very impressed!!

[hamidja]

Hi

one of advantages of cloning is to consolidate all snapshots.sometimes old snapshots get stuck when we need to consolidate them.in this situation you need to create a clone from VM .your clone is consolidated.but parent stay in untouched.now you can use clone VM and remove parent.

hope this help you