Disk consolidation doesn't work - ESXi 6.5

[Konnan]

Hello everyone!

We had a storage outage and since then we have an issue with a VM on ESXi 6.5. First it wouldn't start, I used the vmkfstools to check then repair the vmdk, which was done successfully.

Now it starts properly, but vSphere 6.5 ask me to consolidate disks, but this fails. I tried to do it with the VM online or the VM offline, it fails.

When I try to consolidate, I get: Object type requires hosted I/O.

After that if I check in the VM logs, it says that a file is locked.

After that I power off the VM and investigate the file lock in CLI, I get errors trying to use vmfsfilelockinfo

vmfsfilelockinfo -p vmname.vmdk
vmfsfilelockinfo Version 2.0
Looking for lock owners on "vmname.vmdk"
Failed to get the lock owners for /vmfs/volumes/xxxxx/vmname/vmname-000001.vmdk
Reason : Could not get the dump information for '/vmfs/volumes/xxxxx/vmname/vmname--000001.vmdk' (rv -1)
Error: Inappropriate ioctl for device
Could not dump metadata for '/vmfs/volumes/xxxxx/vmname/vmname-000001.vmdk': Inappropriate ioctl for device
Total time taken : 0.023990383999262122 seconds.

Do you have any idea how I could fix this? I searched and found several articles but usually the disk won't consolidate and the VM won't start, but this VM does starts so the suggested troubleshooting doesn't work in my specific case.

There's many files that ends in vmdk so maybe there's "ghost" snapshots that are causing this issue?

Here's the file list with ls -al
drwxr-xr-x    2 root     root          4096 Feb 11 20:34 .
drwxrwxrwx   11 root     root          4096 Feb 11 16:49 ..
-rw-------    1 root     root     4294967296 Feb 11 20:34 vmname-74a7e1d5.vswp
-rw-------    1 root     root     4294967296 Jan  7 19:31 vmname-d13e7862.vswp
-rw-------    1 root     root          8684 Feb 11 20:34 vmname.nvram
-rw-r--r--    1 root     root            77 Feb 11 19:56 vmname.vmsd
-rwxr-xr-x    1 root     root          2791 Feb 11 20:34 vmname.vmx
-rw-------    1 root     root             0 Jun  4  2018 vmname.vmx.lck
-rw-------    1 root     root           150 Feb 11 18:15 vmname.vmxf
-rw-------    1 root     root      21889024 Feb 12 13:19 vmname-000001-s001.vmdk
-rw-------    1 root     root       6356992 Feb 12 13:19 vmname-000001-s002.vmdk
-rw-------    1 root     root         65536 Feb 11 19:55 vmname-000001-s003.vmdk
-rw-------    1 root     root           443 Feb 11 20:34 vmname-000001.vmdk
-rw-------    1 root     root       7274496 Feb 11 19:49 vmname-000002-s001.vmdk
-rw-------    1 root     root        917504 Feb 11 19:49 vmname-000002-s002.vmdk
-rw-------    1 root     root         65536 Feb 11 17:37 vmname-000002-s003.vmdk
-rw-------    1 root     root           509 Feb 11 19:48 vmname-000002.vmdk
-rw-r--r--    1 root     root     975962112 Feb 11 18:05 vmname.vmdk
-rw-r--r--    1 root     root         63918 Feb 11 17:59 vmware-16.log
-rw-r--r--    1 root     root         76040 Feb 11 18:00 vmware-17.log
-rw-r--r--    1 root     root        250876 Feb 11 19:46 vmware-18.log
-rw-r--r--    1 root     root        238704 Feb 11 19:49 vmware-19.log
-rw-r--r--    1 root     root        239364 Feb 11 19:55 vmware-20.log
-rw-r--r--    1 root     root        250119 Feb 11 20:21 vmware-21.log
-rw-r--r--    1 root     root        253294 Feb 12 13:14 vmware.log
-rw-------    1 root     root     115343360 Feb 11 20:34 vmx-vmname-1957159381-1.vswp
-rw-------    1 root     root     115343360 Jan  7 19:31 vmx-vmname-3510532194-1.vswp

Thanks!

Konnan

[a_p_]

I have not seen such files on ESXi for a very long time.

The snapshots look like "2gbsparse" files, which is a VMDK format that is not supported on the ESXi version that you are running.

Has this VM been uploaded from e.g. VMware Workstation, or VMware Server at some point in time?

Maybe https://williamlam.com/2012/09/2gbsparse-disk-format-no-longer-working.html will help.

André

[Konnan]

Hello!

Thanks André for that tip! Unfortunately, it didn't work, tried both ways to load the multiextent module without success, I guess they are maybe not anymore in ESXi 6.5

Also I would like to answer your question on where it was uploaded from, but since I'm only maintaining this cluster for a short amount of time, no idea at all.

Any other ideas? I guess that VM might need a fresh build... hehe!

Thanks!

[a_p_]

Only a guess.

Try cloning the virtual disk including its snapshots to a new virtual disk.

Make note of the VM's current virtual disk file (i.e. the active snapshot file) that's in the configuration (.vmx) file. (e.g. grep vmdk *.vmx).
Then try cloning the virtual disk using this information: vmkfstools -i vmname-00000#.vmdk clone.vmdk

If this works, edit the .vmx file, and replace the virtual disk's name to clone.vmdk. Save the file and reload the VM.

Then delete the .vswp and .lck files (or move them to a temporary sub-directory), and see whether the VM works.

André

[Konnan]

Hello again André,

Thanks for that idea, but I'll have to stop troubleshooting for now unfortunately, someone higher than me decided he prefer to redeploy instead of me continuing this troubleshoot. It annoys me because I'm curious to try it but I guess I have to stop here... maybe next time! :-)

Thank you very much André and have a nice weekend!

Konnan