ESXi 4 invalid argument

[misiak]

Hi

I'm new here and really hope, you can help me.

I have a ESXi 4.0.0 installed on IBM blade server HS22. Virtual machines are stored on IBM storage.

Some time ago, power on the storage system was lost while one virtual machine was running. From that point, I can't access this virtual machine.

In vSphere, it was liste as Unknown. After following one Community discusion, I removed it from Virtuals Inventory and wanted to include it again throught browsing Datastore. I see all files, so I located the .vmx file and wanted to Add it to inventory, but the option is grayed out.

So after some search, I logged throught ssh to server and wanted to copy the Virtual Machine files to another storage, just to have backup. During copy, I got an error on multiple files including .vmx, -flat.vmdk and all -delta.vmdk files...all files that are locked during virtual machine running. The error message was:

cp: can't open 'xxx': Invalid argument

I'm getting the same errors when I try to copy the files throught scp client.

I searched and tried a lot of things, but still no change.

I'm just thinking, that the files still have some "locked" flag set up and that the file system doesn't allow access to them.

Can anyone point me to some direction?

Thanks

[VeyronMick]

If the files are locked then rebooting the ESX host the VM was last running on would release any locks on the files.

You would expect to see a file "Busy" type message if the file was locked or in use.

"Invalid Argument" can point to file system corruption as we are attempting to run a command on a file and the file cannot be found at the expected location.

If you do a 'hexdump -C <name-of-flat>.vmdk | less' do you see hexcode for your file or do you get an "Invalid Argument/Descriptor" type message?

If you can see hexcode then the file may be okay, if you see the above message then I'd say you're in trouble and you need to restore from backup.

[misiak]

Thanks for the reply. You got me out of my mistake.

After using the hexdump, I've got a Invalid artument, Bad file descriptor message.

The really big problem is that my backup machine disks died about week before this Virtual, all during my vacation. This Virtual is the only possibility to save the data I have on the machine.

Is there a possibility to recover the data?

Thanks

[VeyronMick]

There isn't any VMware type tool that can help in this scenario.

When it comes to this type of file corruption you are looking to your backup process to get you back up and runnning.

There are 3rd party recovery specialists such as Kroll OnTrack which may be able to help but it can be expensive so your backups would normally be the first point on call.

Do you have an idea on what may have caused this?

Was there any outages, upgrades, or other changes in your environment recently?

File system corruption does occur, I'd recover from backups, and then see if you can get down to the bottom of the cause of the corruption.

Some times its inconclusive but I would check to see if the cause can be identified.

[misiak]

That is the stupid bad luck...backup server disks died about week before the virtual machine...electric spike fried the whole pc with all the disk electronics...

I'm making clones of the disk and going to distribute it to some people that think they can help and also companing specialized on data recovery.

Anyway, if there would be some vmfs data recovery software awaylable on the market, I would try to buy it also, it it would be in normal price range.

I think, the problem was caused by electric power drop. Until this time, the UPS for storage did last longer then the one for servers...so the virtual machine went offline before storage. This time, the storage power went down before the disk, so the disks where the virtual machine went offline during the virtual machine run...an maybie something was writen to the disk that time.

[misiak]

Ok, so maybie this will help someone in the future.

I made a copy of the disk using linux and dd. After this, I worked just with the copy. In Debian (and also other Linux distribuition), there is a tool called vmfs-tools. With usage of this, I managed to copy all the data to another drive. There was no "Invalid Argument" error and the data were all ok.

After that, I uploaded the virtual machine files to another data storage (the original damaged was unpluged) and it booted without any problem.

Before uploding the recovered files to esxi data storage, I tried to run them in VMWare Workstation - the Workstadion was showing an error in last snapshot, It was not able to fix it and was not able to boot. So if you also try this, just try to run the virtual machine in original esxi, where it has been made.

Hope, This helps someone to not so spend 4 weeks as I did.

[akha666]

How did you do that ? or what is the step you mad to solve your issue ?