can't register/add to inventory a vm because of locked file

[swspjcd]

I have several ESXi 5.1 hosts in a cluster. There is one vm that I can't add to inventory. I have followed several kb articles and even removed the ESXi host from the cluster completely after verifying that it was the ESXi host causing the lock by finding the MAC address. I am unable to view the vmware.log file for this virtual machine. I get an "invalid argument" error when trying to cat the vmware.log or the .vmx file. The lock file is "vmname.vmx.lck". I've rebooted and restarted the management agents several times. I'm just not sure where to proceed from here as I've been reading about how to resolve this for about 3 hours and have yet to find anything that works. The contents of the directory contain the following files if this helps at all:

-rw-r--r--	1 root	root	162078 Sep 21 16:06 vmware.log
-rw-r--r--	1 root	root	73 Sep  9 18:33 vmname-63cf5ada.hlog
-rw-------	1 root	root	21474836480 Sep 21 16:51 vmname-flat.vmdk
-rw-------	1 root	root	8684 Sep  9 18:34 vmname.nvram
-rw-------	1 root	root	517 Sep  9 18:33 vmname.vmdk
-rw-r--r--	1 root	root	0 Sep  7 17:50 vmname.vmsd
-rwxr-xr-x	1 root	root	3342 Sep 13 18:18 vmname.vmx
-rw-------	1 root	root	0 Sep  9 18:33 vmname.vmx.lck
-rw-r--r--	1 root	root	262 Sep  7 17:50 vmname.vmxf
-rwxr-xr-x	1 root	root	3341 Sep 13 18:18 vmname.vmx~
-rw-------	1 root	root	37580963840 Sep 21 16:51 vmname_1-flat.vmdk
-rw-------	1 root	root	519 Sep  9 18:33 vmname_1.vmdk

Suggestions?

[Troy_Clavell]

not sure if you've seen the below KB, but just in case.

http://kb.vmware.com/kb/10051

[swspjcd]

That was one of the first things I tried. The ESXi host is in maintenance mode so there are no virtual machines running on it and therefore nothing to kill. I have no clue how to resolve this as all of the suggestions I have found don't seem to work.

[admin]

Hi,

The vmname.vmx.lck seems to be a  NFS lock..? are you having NFS storage if so

1. browse to the folder where the VM is located and run ls -la

It should list the hidden files.

2. find the files with extension .lck and delete them. and rename the vmname.vmx.lck

using the command mv vmname.vmx.lck vmname.vmx.backup.

3. Add the VM back to Inventory and Power ON the VM, it should work.

Thanks,
Avinash

[swspjcd]

We are not using NFS at all. There is only one .lck file and I can't delete it although I can rename it. Even after renaming it, I still can't register the vm. It is still grayed out.

[admin]

can you run the command vmkfstools -D vmname.vmx and  ls -la

and paste the output...?

Thanks,

Avinash

[swspjcd]

Here is the output.

Lock [type 10c00001 offset 4405248 v 132, hb offset 3424256

gen 25, mode 1, owner 51ffc163-c2f3c8e4-8cff-001d092b0694 mtime 126477

num 0 gblnum 0 gblgen 0 gblbrk 0]

Addr <4, 0, 39>, gen 71, links 1, type reg, flags 0, uid 0, gid 0, mode 100755

len 3342, nb 1 tbz 0, cow 0, newSinceEpoch 1, zla 2, bs 8192

[admin]

Hi,

seems still there is a lock on the file from the MAC address - 001d092b0694.

try registering the VM to the Host who is the owner of the  MAC address and power on if still fails.

cd into the vm folder and try running the command rm -rf  *.lck and then power ON

if still a problem then

Power off the host and then run the command rm rm -rf  *.lck

Thanks,

Avinash

[swspjcd]

I've tried all of those. I thoroughly followed the kb article describing how to do all of this. The option to register the vm on the ESXi host that has that mac address, is grayed out. I have tried deleting the lck file. When trying to delete it, I get "invalid argument". Even when powering off the ESXi host that has that MAC address and trying to delete the file on oanother ESXi host, I still get "invalid argument" when trying to delete the file.

[admin]

what does the VMkernel.log and Hostd.log show up..? do you see any corruption messages..?

[swspjcd]

I don't see anything in either log that looks suspicious. We replicate our LUNs off site every night for DR, and even after presenting the clone of the LUN where this server lives, to an ESXi host at a remote site, I still can't delete the lck file or even add the virtual server to inventory.

[hodo]

Was this ever resolved?  I'm running into this exact issue as well.

[tomtom901]

Have you tried the steps outlined above?

[hodo]

Just like swspjcd I followed everything in VMware KB: Investigating virtual machine file locks on ESXi/ESX and still cannot remove the lock file.  It is a VMFS datastore, not NFS.  I've restarted the host that has the MAC address shown when running vmkfstools -D lockfile.  I've also restarted every other host in the cluster for good measure. 

[swspjcd]

We were never able to get it resolved and from it looks like, the problem is due to a bug in equallogic firmware which can on "rare" circumstances, cause corruption in the metadata of a LUN. So far we have had 3 virtual servers with the exact same problem, all of which were on the same LUN. One we recovered from our backup software, the other two were able to be recovered from the replicated LUN of where they lived. The bad firmware, according to Dell is 6.06 and it is fixed by 6.06-H2.

[hodo]

Interesting.  Thanks for replying.

We are running Equallogic as well but still on firmware 6.0.5.  Hopefully I can salvage the VM's from our replicated volumes or snapshots.

[tomtom901]

Severe. Has Dell acknowledged this?

[hodo]

From the latest, 6.0.6-H2, Equallogic firmware update:

Issue Corrected in Version 6.0.6-H2

[CRITICAL]: In rare circumstances, an error handling routine was not properly executed. Currently, this has only been observed in VMware environments, where a VMFS datastore might experience heartbeat metadata corruption, impacting the ability to perform operations on virtual machines (VMs). However, there is a small risk that a similar issue could be observed in non-VMware environments as well, so Dell recommends that all customers upgrade to this release.

[swspjcd]

If you read the release notes for 6.06, it says can cause "metadata heartbeat corruption" under rare circumstances, or something very similar. We were running 6.06 for roughly 2 weeks and so far, have had 3 virtual servers that were on the same LUN, become corrupt. They all had the exact same symptoms as in my original post in this thread. It's possible that the LUN was corrupted from something else but we have nevr had any problems like this before, and not all of the servers that were on that LUN, were corrupted. We have since moved all of the servers to a different LUN and deleted the corrupted/faulty LUN.

[hodo]

We had 3 guests out of 10 that were affected.

I was just able to mount a snapshot of this datastore from earlier today before the problem started and got those guests back. My next step is to clear this datastore out and re-create this LUN. And obviously try to get the firmware upgraded.

Thanks again for replying swspjcd.   It has been a frustrating morning trying to troubleshoot this problem.

[Willy Bong]

Hi,

How do you know the size of .lck files??  There is any calculation about the size of .lck files generated from VMWARE Workstation 15.0 ???

20Gb free it's enough??
Thanks in advance,

Willy

[rclaveau]

I success to repair something similar with voma :

[root@localhost:~] voma -m vmfs -f fix -d /vmfs/devices/disks/mpx.vmhba39\:C0\:T0\:L0\:2

Checking if device is actively used by other hosts

Running VMFS Checker version 1.2 in fix mode

Initializing LVM metadata, Basic Checks will be done

Phase 1: Checking VMFS header and resource files

   Detected VMFS file system (labeled:'USB_ESXi_MASTER') with UUID:55240f29-a13f992b-3aca-f079595b2257, Version 5:61

   Found stale lock [type 10c00002 offset 6360064 v 124, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 158522

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

Phase 2: Checking VMFS heartbeat region

ON-DISK ERROR: Invalid HB address <65536>[Fixed]

Phase 3: Checking all file descriptors.

   Found stale lock [type 10c00001 offset 30605312 v 12, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 119

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30607360 v 13, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 128

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30609408 v 14, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 137

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30611456 v 15, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 146

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30613504 v 16, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 155

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30615552 v 17, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 164

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30617600 v 18, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 173

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30619648 v 19, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 182

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30621696 v 20, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 191

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30623744 v 25, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 105768

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30625792 v 22, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 209

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30627840 v 53, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 151471

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30629888 v 24, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 227

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30631936 v 25, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 236

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30633984 v 26, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 245

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30636032 v 27, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 254

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30638080 v 28, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 263

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30640128 v 29, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 288

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30642176 v 30, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 297

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30644224 v 31, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 306

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30646272 v 34, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 146756

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30648320 v 33, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 324

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30650368 v 34, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 333

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30652416 v 35, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 342

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30654464 v 36, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 351

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30656512 v 37, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 360

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30658560 v 38, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 369

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30660608 v 39, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 378

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30662656 v 40, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 387

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30664704 v 41, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 396

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30666752 v 42, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 405

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30668800 v 43, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 414

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30670848 v 62, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 149421

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30672896 v 45, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 432

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30674944 v 46, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 441

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30681088 v 275, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47703

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30695424 v 99, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47767

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30697472 v 98, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47760

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30838784 v 192, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47689

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30840832 v 193, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47701

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30842880 v 194, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47712

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30844928 v 195, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47736

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

   Found stale lock [type 10c00001 offset 30846976 v 196, hb offset 3207168

         gen 27, mode 1, owner 55241293-ff2e34de-acbf-f079595b2257 mtime 47751

         num 0 gblnum 0 gblgen 0 gblbrk 0][Cleared]

Phase 4: Checking pathname and connectivity.

Phase 5: Checking resource reference counts.

ON-DISK ERROR: FB inconsistency found: (25,0) allocated in bitmap, but never used

Total Errors Found:           2

Total Errors Fixed:           1

Total Partially Fixed errors: 0

[root@localhost:~] voma -m vmfs -f check -d /vmfs/devices/disks/mpx.vmhba39\:C0\:T0\:L0\:2

Checking if device is actively used by other hosts

Running VMFS Checker version 1.2 in check mode

Initializing LVM metadata, Basic Checks will be done

Phase 1: Checking VMFS header and resource files

   Detected VMFS file system (labeled:'USB_ESXi_MASTER') with UUID:55240f29-a13f992b-3aca-f079595b2257, Version 5:61

Phase 2: Checking VMFS heartbeat region

Phase 3: Checking all file descriptors.

Phase 4: Checking pathname and connectivity.

Phase 5: Checking resource reference counts.

ON-DISK ERROR: FB inconsistency found: (25,0) allocated in bitmap, but never used

Total Errors Found:           1

[root@localhost:~]

Thx Everybody :smileyhappy:

[porantes]

We ran into similar issue. We resolve it by renaming the vmx.lck file then simply added the VM to inventory. Let me give scenario we ran into,

Power outage occurred all hosts and Dell Equalogic SANs attached to 2 UPS's in same rack. UPS 1 failed, 1 SAN went offline and a switch. started shutting down VM's and the other UPS died. Needles to say the VM's were in process of vmotion I think because of one host was also shut down.

Power returned and exchange would not start. VM said orphaned.  Removed from inventory and tried to add back in and vmx was greyed out. went to 2nd host and tried, vmx was also greyed out. 3rd host i was able to add to inventory but the VM would never load, imported as orphaned no file name. I removed it from inventory on host #3

Browsed the data store on host #3 where exchange was located and renamed the vmx.lck file and then right click the vmx file and added it back into inventory. The VM appeared in inventory with correct naming convention and powered on in safe mode.A couple reboots later and we were back in business.

[2nia]

Set maintenance Mode and reboot host, and then register vm again, no need to delete locked file