Cannot open configuration file: File already exists

[flawaetz]

I've run into an odd problem with a VMWare host that underwent several hard reboots yesterday.  Most all of the VMs are booting up fine but a few can not be powered on.   The host is running ESXi 5.1 799733 The error that pops from the vsphere client is:

An error was received from the ESX host while powering on VM black.
Cannot open the configuration file /vmfs/volumes/50a27883-8c4cae8e-a3fc-002590524262/black/black.vmx.
An error occurred while creating temporary file for /vmfs/volumes/50a27883-8c4cae8e-a3fc-002590524262/black/black.vmx: The file already exists

The file does of course exist.  what's unique about the dirs that have these problematic VMs is that the vmx file name with the tilde extension is in an odd state.

/vmfs/volumes/50a27883-8c4cae8e-a3fc-002590524262/black # ls
ls: ./black.vmx~: No such file or directory
black-3ead7995.vswp  black.nvram          black.vmsd           black.vmxf           vmware.log
black-flat.vmdk      black.vmdk           black.vmx            vmware-1.log

I've searched around and have not seen this exact issue mentioned elsewhere.   Any idea as to what I could try?

Thanks!

[spravtek]

Is this VM on shared storage? Is it possible another host has a lock on the VM in question?

To check for locked files check this KB:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10051

[flawaetz]

Sorry, no, I should have specified.. it's on dedicated internal storage.

[spravtek]

Can you do anything with the VM? like unregister/reregister ...

[flawaetz]

I tried re-registering it but it didn't help.. that file not found error on the ls strikes me as very odd.

[spravtek]

You could try connecting the vmdk files to a newly created VM (create a new VM with the same settings as the current one and connect the existing vmdk(s) ) ...

Maybe there's some problem with the vmx file (config file), you can try download/edit it to see if something isn't right?

[flawaetz]

That's a possible route but I feel like this is a bug -- possibly VMFS corruption -- that I'd like to isolate.  The vmx file looks ok, what's not present, that ESXi seems to expect, is the file with the tilde extension.  All my working VMs have it:  green.vmx~

[spravtek]

Hey, sorry didn't come back to you earlier... How's it going at the moment with the 1 VM that was having issues... There's another thread going on with VM's having those "tildes" at the end of the vmx (vmx~) ...

Could you, if you still see these files, perform a command on them and let us know the outcome?

command: less vm.vmx~

The thread in question: http://communities.vmware.com/thread/428067

[flawaetz]

Hi -

No, I ended up rebuilding those VMs that were unable to boot.  But in the preserved directory under the volume the original vmx file is still there and the same odd behaviour is present.  I did a ls -lR of the "bad" vm and here is the output.

./50a27883-8c4cae8e-a3fc-002590524262/black:
ls: ./50a27883-8c4cae8e-a3fc-002590524262/black/black.vmx~: No such file or directory
-rwxr-xr-x    1 root     root          3156 Dec  5 16:51 black.vmx

if I try to stat or less that file it simply says it doesn't exist.  I don't know much about how VMWare does things under the hood but this behavior of 'ls' strikes me as being something similar to an orphan inode.  'ls' traverses the inodes in the directory tree (in typical unix land) and calls a stat on every file if it needs more information.  I think the readdir() call claims to have a file black.vmx~ file but when the stat() takes place there is nothing present.  At least that's my wild guess hypothesis.

Unfortunately, at least to my understanding, VMFS does not have a fsck equivalent.  Since I don't have a support contract with VMWare I don't have much recourse but to try here and rebuild the VMs.

[continuum]

I have seen the same at a customers site this weekend - if I were in your shoes I would move everything off that datastore and recreate the VMFS volume.
Before recreating the VMFS I would wipe the volume with dd - overwriting the whole volume with zeroes

[flawaetz]

I'm inclined to agree although this strikes me as a not-insignificant issue with VMFS.  In my case the controller was on their HCL but the backplane malfunctioned.  Hardware will fault, the filesystem should survive.

Thanks for your advice.

[nshetty]

I would be able to tell you more if you had the vmware.log files uploaded.

The issue could be with the vmx being corrupted or sometimes a corruption of the vmfs itself.

I can fix the vmfs corruption if at all we see that in your case.

However, at the moment i woudl suggest you to recreate the .vmx file and point the virtual disk and then power on the VM and update the findings.

recreating the vmx is simple.

Create a new virtual machine.

Choose custom.

Specify the exact size of memory, CPU and the OS type.

When choosing the Hard disk, specify " choose an existing disk" point to the .vmdk for the affected VM.

Power on the VM.

CHeers,

Nitin

[continuum]

@ nshetty

with that approach to create a new vmx you would give your VM new MAC addresses and a new UUID - which can have really uglyu results - such as MS Reactivation, Domain-issues and so on.

The good way to recreate a lost vmx is to extract it from the latest vmware.logs

[nshetty]

Hello,

vmx file contains the config info for the virtual machine, so recreating the vmx file does not harm in any way keeping in mind the requirement of the Applications that is running within the guest OS, if the guest requires the same mac address as the previous nic then it could statically added by referring the logs. It is a easy fix for the problem  and faster to get the VM running.

As far as the domain is considered there should not be any problems. Overall, understand the requirements and then recreate the vmx file.

I am pretty sure it would fix the problem, whenever we recreate any file it is key to make sure that we make a note or copy the required infor from the previous config and also understand the requirements :smileyhappy: :smileyhappy:

I call it a 10 min fix :smileyhappy: :smileyhappy:

If there is corruption of VMFS then it is something completely different, if you are experiencing issues with multiple vms on the same datastore, probable corruption.

You can check the vmkernel or message logs and it would indicate corruption.

To fix corruption, i would require a dump from the datastore, 1gb dump from the datastore.

Need to identify the corruption in that dump and then fix the same, the fix file needs to be applied to the datastore which should fix the issue.

Cheers,

NS

[continuum]

> It is a easy fix for the problem  and faster to get the VM running.ith

Faster ? I doubt that - even for a very simple VM with for example a Centos that uses a single vmdk and a single nic creating a new VM from scratch is slower than using a tool like log2vmx.exe against the latest vmware.log

Now consider a different VM with 5 vmdks with 5 snapshots on the 5th vmdk and ten snapshots on the other vmdks - to add to that it uses 4 nics where the first nic is used to licence the CAD application from the vendor that no longer exists.
The approach to recreate the vmx from scratch has zero chance to get it right without time consuming trial and error.
If you now assume that the snapshot withj the highest number is the last one and that using the correct MAC address does not matter that much you may be very surprised when you see how long it takes to fix the snapshot chain after you started it in the wrong order.
You may also have to listen to a rant from your customer who now has no chance to reactivate his CAD app.

Ok - the last one is an extreme example - but I think you get the idea.

Extract the vmx from the log - either manually or scripted always takes the same time - with log2vmx.exe it always takes 3 minutes to connect to vCenter and find the directory and download the log. Then you need one more minute to extract the vmx from the log.

Without vmware.logs - using the "lets guess"  trial and error procedure can take minutes for a simple VM and half a day for a complex one.

If you are interested - log2vmx.exe can be downloaded from my homepage http://sanbarrow.com

About VMFS recovery ... nice to see one more guy who can help our users. At the moment we are just a handful of users that can give good advice.

Maybe you have a tip for me ...what is the best way to fix a volume if the partitiontable is correct and ESXi says the volume is VMFS but no data is visible. vmfs-fuse then typically says "invalid magic number ...

[ArkLaTexIT]

Seeing that this is one of the first results when Googling for "ESXi Cannot Open Configuration file," I wanted to post the solution that worked for me.  Before you take the steps in the solution, make a note of the configuration options for the offending VM.  If you can back up the VMDK files either by browsing the storage pool and downloading or some other way, I would recommend it.  After that, follow the instructions found here http://techhead.co/vmware-esx-how-to-easily-recreate-a-missing-or-corrupt-vmx-file/.

This solution worked for me and saved my butt.  It was a last ditch effort, and I wish I had tried it earlier.

Good Luck,

Stephen

[flawaetz]

It is not the vmx file that's corrupt it's the odd orphan state of the .vmx~ file that seems to neither exist or not.

How would you fix VMFS corruption seeing as there is no fsck type tool?

[giovannimyles]

I've run into this issue before when a failure occured, hopefully I can help. Basically a datastore ran out of space and the VM's were migrated without being visible to me as my vCenter VM was on the datastore that failed. The problem for me was that the vmx file was resident on the original host they were on when the failure occured and it was also migrated to another host and restarted with HA. The vmx was in a locked state because it was present on two hosts in the cluster simultaneously. I had to find both hosts that had the problem and power down the VM, remove from inventory on both and add back to inventory on one. From there it powered up without issue. The key was to find the host, based on the MAC address in the command line output, that had the lock so I could free it up. I'll see if I can dig up the command line for you to find out the MAC address of the host holding the vmx hostage.

[Boblu]

Hello,

same issue exactly occurred, but when I use the cmd to locate the owner, it shows all zero:

/vmfs/volumes/558d6fe9-064a2515-f08a-2c600c3cc341/VDI-B-095 # vmkfstools -D VDI-B-095.vmx

Lock [type 10c00001 offset 115202048 v 2583, hb offset 3997696

gen 51, mode 0, owner 00000000-00000000-0000-000000000000 mtime 1982

num 0 gblnum 0 gblgen 0 gblbrk 0]

Addr <4, 239, 99>, gen 2397, links 1, type reg, flags 0, uid 0, gid 0, mode 100755

len 5129, nb 1 tbz 0, cow 0, newSinceEpoch 1, zla 2, bs 8192

any idears?

[OliAtJsi]

I just encountered this same problem due to a power failure to my host.  As you noted, the directory contains one or two broken files that cannot be deleted.  My solution was, using the datastore browser (and after unregistering the VM), to create a new folder alongside the old one and move all the files to that folder (and re-add the VM to the inventory).  I can't delete the old folder, but at least I can power on my VM (I was prompted whether I moved or copied it - definitely moved) and get it working again.  The corruption is still there; I can't delete the old folder using either the datastore browser nor `rm -rf` but it's fine for now, I'm in the process of moving to a more stable host anyway.

HTH,

- Oli

[Hemant1851983]

Migrate your VM to another datastore the power on it. It worked for me. :smileyhappy:

[Sergey_eXPert]

Just do this:

1) Re-register a VM

2) Move to another host

3) Power On

I wasn't able to power on the VM and noticed presence of .lck file on the datastore. So guess it confused vCenter, when host failed VM was on one host, but I re-registered it on another one. So to get VM location info in sync, simply move the VM to any available host after registering and Power it On.