VM storage usage skyrocketed after consolidating snapshots

[Josh Spern]

We were getting an error message regarding "too many levels of redo logs" when trying to take backups of one of our VMs, so we deleted all snapshots of the VM, but the error still happened. We then ran a snapshot consolidation on the VM, which also did not fix the issue and actually dramatically increased the amount of space the VM is taking up on the datastore. Prior to running the "consolidate disks" command, the VM was using only 795 GB of space, however it is now using 1.65 TB and we now have only 160 GB of free space on the datastore whereas we previously had over 1 TB. There are no snapshots showing for the VM in the snapshot manager, and there are no snapshots located in the folder for the VM, however there are two "vmdk-sesparse.redo" files for the VM located in /vmfs/devices/deltadisks, which total 1.1 TB. Are these files the source of the issue and if so, how do we safely consolidate/delete them?

Thanks in advance.

[continuum]

> There are no snapshots showing for the VM in the snapshot manager,
This simply means nothing more than that the vmsd-file is damaged. Period.

> there are no snapshots located in the folder for the VM, 
This very likely means that you are not aware of the full pathes where your snapshots are stored.

It would help if you show use the last 2 or 3 vmware.logs

Ulli

 

[Josh Spern]

Hello Ulli, thank you for the response. Here are the three most recent vmware.log files.

Attachment(s)

vmware-4.log   3.88 MB 1 version

vmware.log   33.37 MB 1 version

vmware-5.log   84.99 MB 1 version

[continuum]

Please show a filelisting or a screenshot of WinSCP of the directory /vmfs/volumes/5d2748d7-d9c95262-78b2-8030e03a0658/Server01/
Please include full names and filesize.

Ulli

[Josh Spern]

Here you go

[continuum]

The Server_1.vmdk appears twice - one time with a prefix 1561c757.
The version with the "obscure"prefix is configured in undoable mode - which is quite odd for a production VM.
The REDO-log for this mode is 1TB in size !
Can you tell us something about the background story ??

Ulli

[Josh Spern]

Hello Ulli, pardon my ignorance, but are you referring to the two "vmdk-sesparse.REDO" files located in /vmfs/devices/deltadisks? If so, I believe these are each related to two different vmdk files for the VM; the VM has three virtual disks attached to it, Server01.vmdk, Server01_1.vmdk, and Server01_2.vmdk. I believe the file in the deltadisks folder that is prefixed with "1561c757" is related to Server01_1.vmdk.

Some additional context; the VM is being replicated to a second datastore by Veeam, and we are also taking daily backups of the VM using Acronis. The Acronis backups had been running fine until the other day when we began getting the "too many levels of redo logs" error, which was when we tried to delete all of the snapshots for the VM and consolidate. After consolidation, we were left with the VM size in the datastore almost doubled, presumably because of these redo log files.

[continuum]

Oops - sorry did indeed misread your filelists.

> when we began getting the "too many levels of redo logs" error, 
Can you show the log file that mentions that error ?

 

 

[Josh Spern]

This is the error that we are seeing in Acronis:

MESSAGE

VMware returned the following error while creating quiesced snapshot: 'An error occurred while taking a snapshot: Too many levels of redo logs. '.


Additional info:

------------------------
Error code: 5439590
Fields: {"$module":"esx_srv_glxa64_28610"}
Message: Creating a crash-consistent snapshot of virtual machine 'Server01_replica' because the creation of its application-consistent snapshot has failed.
------------------------
Error code: 5439784
Fields: {"$module":"esx_srv_glxa64_28610"}
Message: Failed to create a snapshot.
------------------------
Error code: 5439784
Fields: {"$module":"esx_srv_glxa64_28610","fault":"MethodFault"}
Message: An error occurred while taking a snapshot: Too many levels of redo logs.
------------------------
Error code: 5439784
Fields: {"$module":"esx_srv_glxa64_28610","fault":"GenericVmConfigFault"}
Message: An error occurred while taking a snapshot: Too many levels of redo logs.

------------------------

I can try to find where this error might be present on the ESXi host logs, but I am not sure where to look for this.

[continuum]

When ever something about snapshots goes wrong - look at the vmsd-file.
Is your one ok ?

[Josh Spern]

I am not sure how to tell if it is ok necessarily; here is the content of the Server01.vmsd file in the folder of the original VM:

.encoding = "UTF-8"
snapshot.lastUID = "1639"

And here is the content of the Server01.vmsd file in the folder of the replica VM:

.encoding = "UTF-8"
snapshot.lastUID = "1924"
snapshot.needConsolidate = "TRUE"

 

[continuum]

LOL - second one is obviously invalid and should be deleted.

[Josh Spern]

So just to confirm, the vmsd file in the replica VM folder can be completely deleted safely? Will deleting it affect the large sesparse.REDO files that are presumably taking up all the space?

[continuum]

No - it will not help with the wasted space.
To find the unused items we are missing details - we still have not seen a comprehensive listing of all involved files.

 

[Josh Spern]

Ok I have deleted the invalid vmsd file. Thank you very much for your help thus far.

Here is a listing of the files in the replica VM folder in case it is needed. Which other file listings can I provide that will be helpful?

Attachment(s)

Server01_replica.txt   124 KB 1 version

[continuum]

To fix this VM with over 250 snapshots you need to use vmkfstools in several steps - last time with a number like that I used 3 steps.
Do you have another datastore lots of free space ?
If necessary call me via skype then I do the consolidation for you - its just a few commands but lot of waiting involved.

What is wrong with your automatic backup-tool - it should throw red alerts since middle of december ???

 

 

[Josh Spern]

The other datastore has 956 GB of free space currently; will that be enough? I have used vmkfstools before, if you provide the commands I can give it a try. Will the VM need to be shut down while running the commands?

The backups have been failing since January 5th, so I am not sure why it did not begin failing until then.

[continuum]

Do I guess right ?
Server01.vmdk = 100gb eager zeroed thick provisioned
Server01_1.vmdk = 1TB eager zeroed thick provisioned
Server01_2.vmdk = 500GB eager zeroed thick provisioned

[Josh Spern]

Not exactly:

 

[continuum]

ok  - to deal with this safely we will need at least one 15 minutes downtime of the VM.
AND to do it safely it requires indepth study of your local environment.
I will stop giving any further suggestions unless I had at least one hour via Teamviewer to study the details.
The chance to fill up a disk is just too high.

I am located in germany and call it a day right now. its 2300 here ....

Ulli