Problem with locked virtual machine after esxi host crash

[SchmidA2]

Hi all,

a few days ago my ESXi host crashed because of an unknown reason.

I had to reinstall my ESXi host (on same hardware) from scratch but I was able to restore configuration from a previous backup.

But unfortunately there is one corrupt vm. Can't register / start / stop this vm with new re-installed ESXi host.

I'm also not able to copy the harddisk *.vmdk from that vm to a different datastore.

It seems that this vm still has a lock from the previous installed esxi host.

I can't delete the <vm-name>.vmx.lck file manually because of insuffizient rights. (0600 - rw----..)

With "vmkfstools -D <vm-name>.vmx" I confirmed the still active lock.

Output:

Lock [type 10c00001 offset 40050688 v 3927, hb offset 3244032

gen 1489, mode 1, owner 5b4d60da-c1f01488-2a46-MACADDR mtime 2303

num 0 gblnum 0 gblgen 0 gblbrk 0]

Addr <4, 11, 1>, gen 2, links 1, type reg, flags 0, uid 0, gid 0, mode 100755

len 3410, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 65536

But this information is not really helpful because that ESXi host doesn't exists anymore.

Any ideas how I could get this vm to work again? It would be ok if I could just create a new vm and choose as target harddisk the still available *.vmdk file...

Many thanks in advance for you help.

Regards,

Schmidi

[SupreetK]

Please try the below steps, in order -

1) Right click the invalid VM entry and remove it from inventory. Check if this has released the lock.

2) If only the vmx file is locked, you can create a new VM and attach the existing disk.

3) If you are unable to remove the invalid entry, clear the contents of the vminventory.xml file -

• Take a backup - <cp /etc/vmware/hostd/vmInventory.xml /etc/vmware/hostd/vmInventory.xml.backup>
• Clear the contents - <cat /dev/null > /etc/vmware/hostd/vmInventory.xml>
• Restart the services - </sbin/services.sh restart>
• Register the VM to the inventory

Please consider marking this answer as "correct" or "helpful" if you think your questions have been answered.

Cheers,

Supreet

[SchmidA2]

Thanks for your reply. I already registered and removed the vm from inventory several times. (through UI)

But that doesn't seem to release the lock. I also tried your suggestion to clear the vmInventory.xml but the problem still exists.

The most important problem is that both files (*.vmdk - VMFS6 formatted and *.vmx) have a lock. Otherwise I would just create a new vm and attach the old *.vmdk, but this is also not working because of the existing lock of the *.vmdk file... I'm also not able to copy or move this file.

Any other suggestions? I really need to get the data from this *.vmdk file.

[SupreetK]

Are you sure that the lock is from the host that has been decommissioned already? The decommissioned host is in powered state right?

Cheers,

Supreet

[SchmidA2]

yes, I'm sure. There is only one host in my environment.

Yes, the host is in powered state. Other vm's from before the crash are running without any problems.

Just one from has this problem with locked files...

--------------------------------------------------------------------------

Is there any other way where I could copy or get read access to the *.vmdk from the locked virtual machine / files.

I would like to copy the file to a different datastore or external drive so that I can re-create a new VM with the already existing disk...

Already tried the following:

- using FTP program

- using WinSCP --> Error code: 4

- using ESXi/vSphere web interface

- mount the datastore disk in a different linux system. --> Tried to mount the VMFS6 formated disk in Linux Ubuntu with vmfs-tools. But this is also not working for VMFS6.

All these ways failed because of the still locked *.vmx and *.vmdk files. And I am not able to get rid of that lock :smileysad:

Any ideas?

[continuum]

> Tried to mount the VMFS6 formated disk in Linux Ubuntu with vmfs-tools. But this is also not working for VMFS6.

My next suggestion would be:

use a Linux helper VM while running ESXi as usual.From the Linux VM mount the complete datastore via sshfs in READONLY mode.
Mount another datastore in Writeable mode.
Use ddrescue to copy the flat.vmdk to the datastore that you can write to.
Use normal copy for the small files.
If that also fails try to get the mapping of the locked flat.vmdk via
vmkfstools -p 0 name-flat.vmdk > mapping-file.txt
If that works we can create a dd-script to extract the flat.vmdk in a raw mode that does not check for locks.
Are you still booted into Linux ?
If you dump the first 1536 MB of the VMFSvolume in question to a file with a command like
dd if=/dev/sdXY of=/tmp/schmid.1536 bs=1M count=1536
and provide the dump I can suggest the best next steps.
Feel free to call me via skype - ich spreche deutsch ...

[SchmidA2]

Hi continuum​,

many thanks for your reply.

I followed all of your steps with no success :-(

1. Mounted datastore via sshfs (with and without readonly mode | parameter: -o ro)

Error message:

There was an error copying the file into ...

Error opening file: Operation not permitted

2. Tried to get the mapping. But this was also failing.

Error message:

Could not open file...

3. Uploaded the VMFS-Header-Dump here: <removed wrong link>

Many thanks again for your time. Hope you get some more details from that dump...

[continuum]

Hi
just checked the dump - the file is good but I need to know which flat.vmdk you need.
Please specify the name of the directory and the name of the flat.vmdk.
If possible call me via skype - I probably have follow-up questions.

Ulli

[SchmidA2]

Tried to register for a skype account but they like to know a lot of information.

And I need to verify that I am an adult via credit card number. But I don't like to give them that information...

I need this *.vmdk:

/vmfs/volumes/5866e0f5-376e9c9e-5768-d05099c0f4ac/HO-APP-SRV/HO-APP-SRV-flat.vmdk

Thanks in advance.

[continuum]

Are you sure you created the dump for the correct datastore ?
I dont see the directory HO-APP-SRV
I see for example HO-VCSA.

Check your private messages.
Ulli

[continuum]

Got  the correct dump now.
Can you explain the file HO-APPSRV-flat.vmdk you created or last modified yesterday.
There is no descriptor-vmdk for this one.
Is this the result of an accident ?

[SchmidA2]

Hi continuum​,

yes, this was an result of an accident during mounting, copying, etc...

This file can be safely ignored.

I'm not at home at the moment. I will try to call you in about 2 hours.

I already prepared a bootable usb-stick with the iso image provided via pm.

Many thanks in advance for your time and assistant.

Regards

[SchmidA2]

The datastore is back and running after two live remote sessions together with continuum​​.

It was a very strange and deep problem. But continuum​ was able to find a for me magic solution.

He solved the problem very fast. Amazing guy with a great knowledge about VMWARE products!

I can absolut suggest to contact him if you have any problems with restoring files from your datastore.

Thanks again for your assistance.

Regards

[continuum]

For anybody else with a similar problem please also read
Locked files - VMFS 6
Ulli

[InSPa]

I wanted to thank you for how tremendously useful this thread was!!

A sudden crash left every vmx and vmdk on one datastore locked, and hours of searching only to find that none of the current tools seem to actually fix stale locks - just let you know the mode and who holds the lock, I came back to this thread.

Pulling the first 1.5G from the datastore as you mentioned, and working with a hex editor, I was able to use that information to clear the mode and lock entity in a single find and replace. You were so tremendously helpful pointing the right direction with this thread!!

[jmisgo]

HI inSPa, 

please can you help on how to proceed to resolv the issue

[jcrozier]

Hi

Just reading your thread and I have a bunch of locked files from several vm's on a shared san storage!

vmx, vmdk, log, vswap all locked, restarted all 3 hosts, and the san, removed from inventory, but files are still locked on the san!

I've tried releasing the locks and breaking them on the device, and they're still locked!

Can you help me with this and get these files unlocked? I'd be eternally grateful if you could?

Cheers

[continuum]

Which ESXi version do you use ?
The latest 6.* and 7.* come with quite good options to handle locked files.
If you tried all supported options already and nothing worked I can patch the heartbeat section - which handles locks - for you.

If that is necessary call me via skype. Patching is a dangerous task and I only do it if I have access to all details I need to know and if I can do it myself ...So do not expect that I send patch instructions to a forum post ...

 

Ulli

[rishibp]

Hi Continuum, 

I have been facing the same issue. We have one ESXi host ( 6.5 update 3) connected to the QNAP NAS ISCSI datastore. A few days ago both host and nas were down suddenly due to power failure. After the power was back, the nas started resynchronizing itself automatically. After that, existing VMs were shown as invalid and I could not get the VMs to power on and lock file exists on the existing VM folder. 

While I could create new VMs on the same datastore and power on, the existing VM cannot be registered back. For now, I have mounted the datastore to another temp ESXI host (7.0 Update 2) and tried accessing them but could not access the locked VMs folder. 

Kindly Please suggest. 

[continuum]

Try options I listed in the last post of this thread:
https://communities.vmware.com/t5/vSphere-Storage-Discussions/Locked-files-VMFS-6/m-p/2752286

[rishibp]

Hi Continuum, 

Do you need support with a recovery problem ? - send a message via skype "sanbarrow"

- I have sent you msg on skype from my skype ID (rishi35), rishi bhagat pradhananga. 

[freestyle31225]

Hi, we are having very same issue after MSA storage went down. We have ESXi 6.7 U3. And now we can't browse vcenter VM's folder on all hosts but the one VM resides on. 
ESXi hosts' hostd.log says: vCenter Server Appliance 6.7_12-ctk.vmdk" : failed to open (Failed to lock the file)
Any help is appreciated.

[Twitched_Out]

I know this is an old thread, but I have been going through the steps, and I can mount the volume as you state in read-only mode. Do I have to write the data to another datastore, or can I use a folder to copy the vmdk? Also, what commands would I need to use in DDRescure?

 

Thank you,

[LucaMCT]

I also had problem with locked file.

ESX is a single host, local datastore, the VM has 4 manual snapshot, no space problem on datastore, need consolidation=true.

The backup software VM started backup, I received alarm, went on vsphere console and found VM was suddenly switched off.

The backup failed (also all 4 retry), inside the vm folder new delta file were present but no snapshot point (not present in vmsd file).

The file locked is the last delta vmdk from manual snapshot, in exclusive mode, from the local esx.

No process are using this file, file is not mounted in any other VM.

Cannot copy or delete the file, can rename it.

Tried consolidation, failed with locked problem.

Did new snapshot and deleted all the old snapshot without error, but the process was too fast, in fact all delta files still there, but not anymore in the snapshot database.

Tried new consolidation and failed with error "file not found".

I will reboot host ASAP but it seems strange that if no one is using the file it still locked. In any case it seems I need to manually rebuild the chain of snapshot.