Unable to use VM after datastore got full

[heavenwillburn]

Hello,

today one of my datastores went full and I got a message like this

'There is no more space for virtual disk xxxx.vmdk. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session.'

Because I didn't want to have any issues with other VMs on the datastore and it's a VM that can be shut down without business impact, I clicked Cancel and the VM shut down. Then I saw that it runs on a snapshot, which I deleted and then set the second disk to independent because it's unnecessary to be snapshotted.

But now I can neither start the VM, consolidate or move it to another datastore. When starting, I get the following error

An error was received from the ESX host while powering on VM XX.

Failed to start the virtual machine.

Cannot open the disk '/vmfs/volumes/575d6dd1-27e4cd9e-ab07-2880239d1af0/XX/XX-000003.vmdk' or one of the snapshot disks it depends on.

Failed to lock the file

Module Disk power on failed.

Cannot open the disk '/vmfs/volumes/575d6dd1-27e4cd9e-ab07-2880239d1af0/XX/XX-000001.vmdk' or one of the snapshot disks it depends on.

Failed to lock the file

The vmdk files are existing though and there are NO lock files on the datastore.

After a few tries to boot it up I tried to create another snapshot and deleted it again, but then the error came up that disk consolidation is needed. This doesn't work either:

An error occurred while consolidating disks: msg.snapshot.error-DISKLOCKED.

I already read KB10051 about disk locks, but the owner IDs of the disks are only zeroes instead of an MAC-address.

[root@at0195:/vmfs/volumes/575d6dd1-27e4cd9e-ab07-2880239d1af0/AT1124] vmkfstools -D /vmfs/volumes/575d6dd1-27e4cd9e-ab07-2880239d1af0/AT1124/XX-000003.vmdk

Lock [type 10c00001 offset 228986880 v 123, hb offset 4157440

gen 49, mode 0, owner 00000000-00000000-0000-000000000000 mtime 2666079

num 0 gblnum 0 gblgen 0 gblbrk 0]

Addr <4, 529, 58>, gen 115, links 1, type reg, flags 0, uid 0, gid 0, mode 600

len 388, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 8192

I already tried restarting the management services on all my hosts all well as restarting the vCenter service.

I also cannot migrate the machine to a datastore with more free space. Host migration is working, but on datastore migration, I get the following

Error caused by file '/vmfs/volumes/575d6dd1-27e4cd9e-ab07-2880239d1af0/XX/XX-000001.vmdk'

Cloning the VM also doesn't work because of

Unable to access file '/vmfs/volumes/575d6dd1-27e4cd9e-ab07-2880239d1af0/XX/XX-000003.vmdk' since it is locked

Does anybody have an idea what I can do? I sure do not have a backup of that damn server..

[Devi94]

can you share vmware.log file for this VM?

[heavenwillburn]

Sure, see attachment.

[Devi94]

You need revert the settings you made on hard disk2 and consolidate VM, and try power on.. You said snapshots are deleted, but snapshot files does exist.

29T13:47:44.549Z| Worker#0| I125: DISK: Cannot open disk '/vmfs/volumes/575d6dd1-27e4cd9e-ab07-2880239d1af0/XX/XX-000001.vmdk' : Failed to lock the file.

2018-05-29T13:47:44.552Z| Worker#0| I125: DISKLIB-VMFS  : "/vmfs/volumes/575d6dd1-27e4cd9e-ab07-2880239d1af0/XX/XX-000002-delta.vmdk" : closed.

2018-05-29T13:47:44.552Z| Worker#0| I125: DISK: Cannot open disk '/vmfs/volumes/575d6dd1-27e4cd9e-ab07-2880239d1af0/XX/XX-000002.vmdk' : Failed to lock the file.

[heavenwillburn]

You need revert the settings you made on hard disk2 and consolidate VM, and try power on.. You said snapshots are deleted, but snapshot files does exist.

I know the files are still there but consolidation is not working because of the DISKLOCKED error..

[Devi94]

Did you try reverting changes in hard disk2 ?

[heavenwillburn]

Did you try reverting changes in hard disk2 ?

Yes I did. But earlier I also already tried to fully remove and re-add the disk (with the snapshot .vmdk as source), I don't know if that made it worse..

[continuum]

So how many free space is left ? - please put some real numbers on the table.

[heavenwillburn]

Sorry, its 12GB as the vRAM were 12GB. If I leave it at 12GB, I get the message that the swap file cannot be created, but if I lower it to 2GB all of the locking errors appear, so I think it should be enough space to at least start the VM if there wouldn't be that lock issue.

[continuum]

You will need to free more space - just being able to create the new vswp - file is not enough

[SparkRezaRafiee]

Revert the changes back and then storage migrate the VM to a different datastore if possible.

Buf before moving it, after reverting the changes back, check the lock on the VMDK file again and make sure it's not locked.

If it doesn't resolve the issue, the workaround would be:

-Removing the VM from inventory and re-adding it back

-Removing the ESXi host from inventory and re-adding it back

-Removing flag from vCenter Server Inventory DB manually

[continuum]

Do you have enough free space now ?
Enough free space means at the very least a few multiples of the vRam that is assigned to that VM.
> then set the second disk to independent

Switch that back to normal mode if that VMDK already has a snapshot.
> The vmdk files are existing though and there are NO lock files on the datastore.
ESXi does not  need lock files - the VMFS-filesystem locks files internally.
If you cant remove the locks try to unregister the VM, rename the directory and re-register the VM - sometimes that helps

[heavenwillburn]

Do you have enough free space now ?

Enough free space means at the very least a few multiples of the vRam that is assigned to that VM.

Well, not really.. The only other VM on the datastore is so big that it doesn't fit any other datastore and there are no capacites to expand the datastore. I already reduced the size of the vRAM though but not idea if that helps. That's why it would be great if vMotion would work, I have an empty datastore but can't move it there.

> then set the second disk to independent

Switch that back to normal mode if that VMDK already has a snapshot.

If you cant remove the locks try to unregister the VM, rename the directory and re-register the VM - sometimes that helps

Those steps didn't help unfortunately.

[heavenwillburn]

The issue has been solved with a reboot of the host where the VM last ran on (we weren't able to do this until yesterday). After that it could be powered on, the lack of free space wasn't the problem.