Cannot consolidate disk snapshots due to lack of space

[G3rman42]

Hi experts,

After spending tons of cycles searching I have reached a point where I need community help.

I have a Vsphere 5.5 setup with 3 hosts. Storage is iSCSI on a Synology NAS.

One of the hosts has a VM which consumes 97% of the space on its own datastore.

Apparently snapshots were started at some point and now when I try to consolidate them it fails due to lack of space.

From the CLI I get the following error:

Failed to clone disk: There is not enough space on the file system for the selected operation (13).

I tried moving some of the files to another datastore, but it's been over 12 hours and will likely take another day or two for the outbound move to complete.

I simply don't have that much time since I need to try and get this system back up and running as quickly as possible of course.

I would appreciate any ideas on how to resolve this and please let me know what additional info you need to make more sense of this.

[daphnissov]

There is no replacement for not having enough space to consolidate a snapshot. You can either find the space somewhere, or basically do a conversion on the VM to a new VM (which requires space for the new VM on some data store).

[Scott Vessey]

Moderator: Thread moved to the vSphere area.

[a_p_]

Welcome to the Community,

as a first step, please provide an overview of the files that are in the VM's folder.

For that, please run ls -lisa > filelist.txt from the command line, and attach the resulting filelist.txt to your next reply.

How much free disk space (in GB) do you currently have on the datastore?

You mentioned "I tried moving some of the files to another datastore, ...". This may or may not be helpful. Please explain the current state, i.e. if all of the VM's files are still in the VM's folder, or if some of them have already been moved to another datastore. In case of already moved .vmdk files, it will be necessary to fix the snapshot chain.

André

[G3rman42]

Thanks André,

File listing and df -h are attached.

I was able to stop they move operation since it would have taken days.

The system I am having trouble with is in "Production2"

I would be grateful for any other suggestions.

Filesystem   Size   Used Available Use% Mounted on

NFS         54.3T  44.1T     10.2T  81% /vmfs/volumes/Amoura_share

VMFS-5      14.6T  14.3T    406.4G  97% /vmfs/volumes/Production2

VMFS-5       9.8T 262.7G      9.5T   3% /vmfs/volumes/Production (1)

VMFS-5      14.6T 877.1G     13.8T   6% /vmfs/volumes/Production3

VMFS-5      19.5T   7.9T     11.6T  40% /vmfs/volumes/DevTest

[CRad14]

How about trying to clone the VM to a different datastore? After the clone is complete, do some testing and then you should be able to delete the original VM and all the problem snapshots.

Another option would be to use the free version of Veeam, do a single backup on the VM, then restore the VM to a different location, then you should be able to delete the original VM.

Hope this helps.

[G3rman42]

My biggest problem is that the VM uses 14 TB of storage (across 4 Windows Disks) and the next largest Datastore I have available is 10 TB.

I thought about vMotioning it but that won't work because I don't have a big enough Datastore available at this time.

I am thinking about trying to clean up space on another Datastore and then expanding the existing LUN via the Synology Storage Manager.

Any thoughts on that?

[a_p_]

According to the file listing, your virtual disks have all been thick provisioned, so that they won't grow with deleting all snapshots. Running "Delete All" from the Snapshot Manager with the VM being powered off should not require any additional, temporary disk space on the datastore.

What confuses me, is  "... and now when I try to consolidate them it fails due to lack of space."

When did this message appear, i.e. what exactly did you do?

André

[G3rman42]

When I tried to consolidate disks from the GUI I would get a pretty non-descript error message.

When I tried to consolidate the disks from the CLI I received an error that the consolidation failed due to lack of disk space.

I have since received permission from the customer to remove 4TB of old file storage. This should now get me enough space to consolidate the disks.

Thank you all for you jumping on this and for all the suggestions!

[a_p_]

Apparently snapshots were started at some point and now when I try to consolidate them it fails due to lack of space.

From the CLI I get the following error:

Failed to clone disk: There is not enough space on the file system for the selected operation (13).

Please share the command that you've used.

André