Disk consolidation fails with error Cannot complete the operation due to an incorrect request to the server.

[Vince_Garcia]

Good Day Everyone

I have an ESX6.7 Host with VM on it that will not consolidate disks.  This VM was failing Veeam backups with the "Too many redo logs" error.  The VM is now showing Virtual machine disks consolidation is needed.

When I attempt to consolidate disks it fails with error "An error occurred while saving the snapshot: Too many levels of redo logs."

Snapshot manager shows no snapshots

I SSH into the host and there are 256 (at least) snapshot files named 000002.vmdk etc.

I tried to clone the disk and that fails with error "An error occurred while saving the snapshot: Too many levels of redo logs."

I tried to migrate the storage to another datastore thinking that should delete the snapshots but that fails with error "Cannot complete the operation due to an incorrect request to the server"

Any attempt Ive made to delete the files has run into one of these errors

Any assistance is greatly appreciated :smileyhappy:

[a_p_]

Welcome to the Community

I SSH into the host and there are 256 (at least) snapshot files named 000002.vmdk etc.

Please run ls -lisa > filelist.txt in the VM's folder and attach the filellist.txt to your next reply.

André

[Vince_Garcia]

I have the filelist attached

[a_p_]

Do you have at least 120GB free disk space on a datastore, and for how long is it possible to shut down the VM.

What I'm thinking of is to try, and clone clone the virtual disk from the command line. If downtime is an issue, it should be possible to limit it to a minimum, but this will require manual file editing, and more effort.

André

[Vince_Garcia]

I have about 1TB free on that datastore and can take down the VM for a little while without issue.  an offline clone has failed for me but I have not attempted that from the command line.  Please let me know how to proceed.

[a_p_]

What I'm thinking of is:

1. make sure that non of the virtual disks (snapshots) is still accessed by the Veeam Backup server, or one of the proxies.
2. shutdown the VM
3. clone the virtual disk including all of its snapshots to a new virtual disk (this will be renamed later on) by running the following commend from within the VM's folder
   Caution: Please ensure that the source .vmdk name is the one that's currently used by the VM, i.e. is the one from the VM's .vmx file.
   vmkfstools -i PS-PHONE[phone].vmdk XX-PHONE2.vmdk
4. Create a new sub-directory and move all of the PS-PHONE2*.vmdk files as well as the .vmsd file to it
5. Rename the cloned virtual disk to the original name (please note that command options are case-sensitive)
   vmkfstools -E XX-PHONE2.vmdk PS-PHONE2.vmdk
6. Edit the VM's configuration (.vmx) file, and replace the virtual disk's name (PS-PHONE[phone].vmdk ---> PS-PHONE2.vmdk)
7. Reload the VM (see steps 2 + 3 in https://kb.vmware.com/s/article/1026043​)

Now you can power on the VM, and - once everything works as expected - delete the sub-directory with the old .vmdk files.

If at any point something doesn't work as expected, please do not proceed with the next steps, but let me know, so that we can think of a Plan B.

André

[Vince_Garcia]

Thank You for this info

I was not able to proceed with the disk clone. When I ran the command to make the copy I got this error, "Reached the host's limit for open files (40)".  I have not seen that before but I suspect that it has to do with the large number of snapshots.

How do we get around this?

[a_p_]

In this case we may need to proceed with Plan B, which is to consolidate the virtual disk step-by-step.

For this please compress/zip all the .vmdk descriptor files - i.e. the small ones, with only a few hundred bytes in size - and attach the resulting .zip archive to a reply post.

André

[Vince_Garcia]

I don't have any VMDK files that are a few hundred bytes in size.  The smallest VMDK file is 7MB.

[continuum]

>The smallest VMDK file is 7MB.

NO !!!

You got this wrong display because you inspect that directory with VMware Datastorebrowser.

As soon as you are doing something like this you cant use Datastorebrowser any longer - you either watch via putty - or way more convenient with WinSCP.

Datastorebrowser is not the appropriate tool when you need to do anything like repair or inspect vmdks !!!!

You Must get used to this asap or you will make mistakes that kill your VMs in future.

Datastorebrowser does not display VMDKs correctly ! Period.

[continuum]

Is the VM still operating ?

If yes - the fastest way is to add another new VMDK with the same size as the  original.

Then boot that VM into a linux liveCD and clone disk to disk with ddrescue.

If the VM can no longer be started at all - use vmkfstools -i and split the whole procedure in several steps - that each one do only consolidate up to 39 snapshots.

That is a pain as it requires several modifications of descriptorfiles along the way  - but it will work and make sure you do not exceed the number of allowed open files which appears to be 40 in this case.

simplified ...:

1. vmkfstools -i name-000039.vmdk  newbase.vmdk

edit name-000040.vmdk so that it uses newbase.vmdk as parent

2. vmkfstools -i name-000078.vmdk newbase2.vmdk

edit name-000079.vmdk so that it uses newbase2.vmdk as parent

....

and so on ...

[Vince_Garcia]

The VM is up and running

Do I have to make the new VMDK the same name as the original?

So the procedure here is to add a disk same size as the original

Boot the VM into a Linux live environment

Clone the old disk to the new one

swap VMDKs in the VMs config

Boot from newly cloned VMDK

Is that right?

[continuum]

> Do I have to make the new VMDK the same name as the original?

Nope - use a good new name instead. Such as "no-more-snapshot-worries.vmdk"

Other than that you got it.

Important note:

use ddrescue and use a syntax like this

Assume existing disk is detected as /dev/sda

New one is detected as /dev/sdb

Then boot into Linux LiveCD

Log in as root and then run

ddrescue /dev/sda /dev/sdbb /tmp/logfile

Make sure you create the logfile !!!

And before you start the VM ... - make sure the you disable changeblocktracking.

Edit the vmx-file and power off your Veeam appliance or what ever you use ...

[a_p_]

They are there, you just cannot see them in the GUI.

Please use e.g. WinSCP to connect to the ESXi host, so that you can see, and download these files.

André

[Vince_Garcia]

ok I have those files attached now

[a_p_]

I actually think that it should work in two steps.

The first step is to run vmkfstools -i PS-PHONE[phone].vmdk XX-PHONE2.vmdk, then download XX-PHONE2.vmdk, and attach it to your next reply.

Based on that file, I'll provide you with the details for step 2.

André

[Vince_Garcia]

The vmdk that was created is 120GB in size  There was also a xx-PHONE2-ctk.vmdk file that is about 7MB.

[a_p_]

What's needed is the "xx-PHONE2.vmdk" that you can download using e.g. WinSCP.

André

[Vince_Garcia]

I have that file attached

[a_p_]

Next steps are:

• shut down the VM
• extract, and upload the attached "PS-PHONE[phone].vmdk" to the VM's folder (this will replace/overwrite the existing file)
• open a putty session, and do the following from within the VM's folder
  1.) Backup/rename the original PS-PHONE2.vmdk and PS-PHONE2-flat.vmdk files
  • vmkfstools -E PS-PHONE2.vmdk ORG-PS-PHONE2.vmdk

2.) Rename xx-PHONE2.vmdk and xx-PHONE2-flat.vmdk

• vmkfstools -E xx-PHONE2.vmdk PS-PHONE2.vmdk

3.) reload the VM (see https://kb.vmware.com/s/article/1026043)

• vim-cmd vmsvc/getallvms
• vim-cmd vmsvc/reload Vmid (replace Vmid with the VM's ID from the first command)

Please remember that commands, options, and file names are case-sensitive!

You should now be able to create a new snapshot, and then run "Delete All" from the Snapshot Manager. This can be done either with the VM powered off, or on (in case you want to limit downtime).

If the "Delete All" task completes successfully, there's some cleanup theat should be done. To ensure that the correct things will be done, provide an updated filelist.txt, analso attach the VM's .vmx file to your next reply.

André

[Vince_Garcia]

I was able to upload the file you have attached but was unable to rename files.  When I tired to rename PS-PHONE2.vmdk I got the error "Failed to rename 'PS-PHONE2.vmdk' : Failed to lock the file (16392)" and I was unable to start the VM.  I had the old file from before and copied back replacing the one you attached and was then able to power the VM back on but im still unable to create or delete snapshots.

[a_p_]

"Failed to rename 'PS-PHONE2.vmdk' : Failed to lock the file (16392)"

Please check whether your Veeam Backup server, or one of the Veeam proxies (if used) still have the VM's virtual disks, or one of its snapshots attached (hot-added) as an adittional virtual disk. If so, detach it from the backup server/proxy.

If that's not the case, please reboot your Veeam Backup servers to ensure that there are not still any processes running which may cause the file lock.

If you cannot find out what's causing the lock, here's Plan C:

• shut down the VM
• replace "PS-PHONE[phone].vmdk" with the one from the attached .zip archive
• lookup the VM's currently used .vmdk name in the .vmx file (e.g.: grep vmdk *.vmx)
• create a new clone by running: vmkfstools -i PS-PHONE[phone].vmdk New-PS-PHONE2.vmdk
  (please replace PS-PHONE[phone].vmdk, if it's not the currently used .vmdk in the .vmx file)
• edit the VM's .vmx file, and replace the virtual disk name with "New-PS-PHONE2.vmdk" (you may either use the native vi editor, or. e.g. WinSCP to edit the file)
• reload the VM (see steps 2 + 3 in https://kb.vmware.com/s/article/1026043)

If that works, migrate the VM to another datastore if you have a liceanse which allows this. This will migrate all required files to the new datastore, and leave obsolete files in the old folder. This will make it easier to do a cleanup later.

André