Can't Remove Stale .lck File from VMFS

[BDMcGrewVM]

Hello,

 

I have a 3 node cluster.  I have a single VM that indicates Consolidation is Needed however, consolidation fails with the following error:

 

Unable to access file since it is locked An error occurred while consolidating disks: One or more disks are busy.

 

I notice in the datastore there is a couple of stale .lck files and I can not remove them. 

 

[root@vmw12:/vmfs/volumes/ds01] ls -la .*.lck

-rw-------    1 root     root             0 Jan 15 15:28 .214fa565-16d5-5a54-663d-3cecef57364e.lck

-rw-------    1 root     root             0 Jan 15 19:37 .6f89a565-ec03-925b-9dd5-3cecef57364e.lck

 

When I try to remove the lck files, I get a busy error:

 

[root@vmw12:/vmfs/volumes/ds01] /bin/rm -f .*.lck

rm: can't remove '.214fa565-16d5-5a54-663d-3cecef57364e.lck': Device or resource busy

rm: can't remove '.6f89a565-ec03-925b-9dd5-3cecef57364e.lck': Device or resource busy

 

Checking the lock status appears to be an invalid owner – notice all zero’s.

 

[root@vmw12:/vmfs/volumes/ds01] vmkfstools -D mvplhcns03.vmdk

Lock [type 10c00001 offset 137969664 v 53, hb offset 3276800

gen 7, mode 0, owner 00000000-00000000-0000-000000000000 mtime 13841

num 0 gblnum 0 gblgen 0 gblbrk 0]

Addr <4, 59, 2>, gen 3, links 1, type reg, flags 0, uid 0, gid 0, mode 600

len 674, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 65536

affinityFD <4,8,8>, parentFD <4,0,0>, tbzGranularityShift 20, numLFB 0

lastSFBClusterNum 0, numPreAllocBlocks 0, numPointerBlocks 0

 

I have rebooted all 3 nodes in the cluster.

I have turned off DRS and HA.

I have backed up and shutdown the VM.

 

I am stuck, all help is greatly appreciated.

 

Thank you,

 

-brian

[a_p_]

Just a quick guess.

Does your backup server, or any backup proxy (if applicable) still has the VM's virtual disk mapped? E.g. due to a previously incomplete backup job.

André

[BDMcGrewVM]

Thanks, Andre...

 

No, it doesn't.  I just rebooted the backup server and client and still same behavior.  I'll admit, you got me because that's one thing I didn't think of but a reboot of those devices makes no difference.

 

-b

[a_p_]

A reboot may not help in case the backup uses the hot-add method.
Do you have a virtual backup server, or proxy? In this case check those VM's settings and ensure they do not have the trouble VM's virtual disk mapped.

In doubt, use e.g. RVTools to find out whether the virtual disk is in use by another VM.

André

[BDMcGrewVM]

Thanks, that was exactly the problem.  Backup proxy server had the disk mounted.  All is better now.