Snapshots, Datastore Expansion and Broken VM

[Clay Anderson]

Hey everyone. I'm troubleshooting a single VM on a solitary ESXi 7 host.

A user called me saying "the VM won't boot" citing the error: "Unable to enumerate all disks". Apparently the datastore was expanded while running on a snapshot. I don't have any more information. 

When trying to add the virtual disks back into the VM, all fields in the disk section appear blank, and I'm not able to choose "ok". I suspected a corrupt/missing descriptor, though I see one is present. I am unable to delete any snapshots, instead receiving the error "Failed to delete all snapshots from virtual machine 'VM1'."

Here is some relevant output from the host:

[root@localhost:/vmfs/volumes/65119fc8-8f1edd98-a818-c45ab1acf059/VM1] ls -lrht *vmdk*
-rw-------    1 root     root         536 Jan  8  2024 VM1.vmdk
-rw-------    1 root     root      360.0G Jan 19  2024 VM1-flat.vmdk
-rw-------    1 root     root      294.3G Nov 14 13:17 VM1-000001-sesparse.vmdk
-rw-------    1 root     root         332 Dec  2 16:32 VM1-000002.vmdk
-rw-------    1 root     root       15.3G Dec  2 19:58 VM1-000002-sesparse.vmdk

[root@localhost:/vmfs/volumes/65119fc8-8f1edd98-a818-c45ab1acf059/VM1] ls -lh
total 485044416
-rw-------    1 root     root      294.3G Nov 14 13:17 VM1-000001-sesparse.vmdk
-rw-------    1 root     root       15.3G Dec  2 19:58 VM1-000002-sesparse.vmdk
-rw-------    1 root     root         332 Dec  2 16:32 VM1-000002.vmdk
-rw-------    1 root     root       24.0G Jan 19  2024 VM1-Snapshot1.vmem
-rw-------    1 root     root        7.5M Jan 19  2024 VM1-Snapshot1.vmsn
-rw-------    1 root     root       24.0G Nov 14 13:18 VM1-Snapshot4.vmem
-rw-------    1 root     root        7.5M Nov 14 13:18 VM1-Snapshot4.vmsn
-rw-------    1 root     root      360.0G Jan 19  2024 VM1-flat.vmdk
-rw-r--r--    1 root     root      264.5K Dec  2 16:33 VM1.nvram
-rw-------    1 root     root         536 Jan  8  2024 VM1.vmdk
-rw-r--r--    1 root     root         765 Dec  2 20:32 VM1.vmsd
-rwxr-xr-x    1 root     root        3.3K Dec  3 16:59 VM1.vmx
-rw-r--r--    1 root     root       28.0M Oct 12 07:01 vmware-15.log
-rw-r--r--    1 root     root       44.3M Nov 14 13:20 vmware-16.log
-rw-r--r--    1 root     root       24.1M Dec  2 16:32 vmware-17.log
-rw-r--r--    1 root     root      161.9K Dec  2 17:01 vmware-18.log
-rw-r--r--    1 root     root       56.8K Dec  2 17:01 vmware-19.log
-rw-r--r--    1 root     root       56.6K Dec  2 17:02 vmware-20.log
-rw-r--r--    1 root     root       56.9K Dec  2 19:58 vmware.log

Any direction would be appreciated.

[a_p_]

One of the descriptor files (VM1-000001.vmdk) is missing. It should be possible to re-create it, from the content of VM1.vmdk, and VM1-000002.vmdk.

Please compress/zip these two files, and attach the .zip archive to your next reply.

André

[Clay Anderson]

That makes sense, this missing descriptor is breaking the chain. Thank you.

I'm eager to see the file you create so I can learn from this. Files attached.

Attachment(s)

vmdk.zip   937 B 1 version

[a_p_]

I've created the missing file (see attachment) from the contents of the ones that you've provided.

André

Attachment(s)

VM1-000001.zip   359 B 1 version

[Clay Anderson]

André,

This has worked, thank you. I have a couple questions, if you'd be so kind as to indulge me. I also created a descriptor of my own using vmkfstools, but it has two notable differences:

1. The disk size I used (obtained from VM1-000001-sesparse.vmdk) resulted in a different extent integer than yours - RW 754974720 (yours) vs RW 617205568 (mine). How did you obtain this?
2. Mine also contains the "ddb.longContentID" string at the end, the value of which I adjusted by referencing the appropriate CID. Yours does not have this, is that normal?

Thanks again. Always grateful to learn.

[a_p_]

The number of blocks (size) for a snapshot is always the same as the one for the base disk.

All I basically did, was to create a copy of VM1-000002.vmdk, and rename it to VM1-000001.vmdk. Then I changed the file names, set the CID to the parentCID from VM1-000002.vmdk, and the parentCID to the CID from VM1.vmdk to fix the chain. Finally deleted the ddb.longContentID.

André