Locked files - VMFS 6

[continuum]

You probably know the error message: operation impossible since file is locked.
If a VMDK is locked you can not start it inside a VM and you can not copy it.
The knowledgebase basically tells you to stop all processes that access the file and reboot.
This is supposed to fix the problem.
This documentation is not good enough.
In VMFS 6 following this advice may not be enough to release the lock.
This means that you basically lost the file as you can not read it anymore.
Trying to access it via Linux is also impossible at the moment.
Recent recovery third party tools like UFSexplorer also failed to read the vmdks in such a case.
Here is a recent case for your reference .... 
Problem with locked virtual machine after esxi host crash
I have seen a couple of this cases recently so I started to investigate the issue,
Now I may have found a way to recover a VMDK that would other wise be a case for Ontrack or VMware Support.
My procedure requires a direct hexedit of the VMFS heartbeat- section so please do not ask for details at the moment.
I will post details once I know this approach is dafe.
Anyway I believe this is a bug in the way ESXi 6.5 handles the heartbeat section of a VMFS-volume.
This issue should never affect single host environements.
Feel free to contact me via skype if you run into this yourself.
Ulli

[a_p_]

Good to know whom to contact in such a case!

André

[yaangvu]

continuum​ I have seen this problem, pls help me.

Other way, what's your skype, I will contact you to ask.

[irony74]

Hi Ulli

I have the same issue. I moved a VM from a 6.0.0 cluster to a standalone ESXi 6.7U2.

Any help appreciated.

BR

Marc

[continuum]

Hi Marc

you were able to move a VM and then have locked files in the new environment ?

thats strange and unexpected ...

please create a header dump and send it myway so that I can look into it.

See Create a VMFS-Header-dump using an ESXi-Host in production | VM-Sickbay

If possible contact me via skype sanbarrow

Ulli

[defmania]

Hello,

Have you created a post with the steps to try and fix this?

I've just had this happening in my homelab. After an issue with the storage array, which led to a disk replacement, I now have 5-6 VMs which are inaccesible/orphaned and their vmx is locked by one of the ESXi hosts. Of course I've pretty much tried everything, host reboots, storage array reboot etc. I will not release the lock. Since it's my homelab I'm pretty much open to try anything.

Thank you!

[continuum]

This is what I do nowadays:

plan A - safest option :

extract the VMs with Linux

plan B -  smartest option but not for the faint at heart:

patch the .vh.sf file

send from iphone

[thiagoarceno]

Hi Ulli,

Can you explain how to extract the VM with Linux?

Best Regards,
Thiago

[continuum]

Easiest option:

connect to the datastore with sshfs in readonly mode  - then use ddrescue against the flat.vmdks

If that does not work - try if you can get the location of the fragments with vmkfstools -p 0 flat.vmdk

If that does not work - try to get the location of the fragments by analysing the VMFS-metadata

If that does not work - find the first fragment with scalpel and hope that the flat.vmdks are allocated in one piece

[Flinerock]

beleive I ran into this recently 

 

my esxi 8.01 host lost power while a some vm’s were still running and now for the life of me they won’t power on they keep hangning at 22% so I tried multiple forums and restarting services and even   unregistering and re registering to the same issue. I powered off the host for a good 15 mins and then turned it on thinking I was good to go as the power on of that vm went up to 65% that time and then stuck there. My other vm loads but the data is corrupt and when I try to login to it as it s my nextcloud server it just says internal error on that so 2 strange issues but I think there the same issue any help would be great as I don’t wanna lose these 2 VMware’s and snap shots aren’t fixing it