vmdk file disappeared suddenly

[GaryDank]

Doing some investigate why the Windows 2008 R2 Server was not responding I discovered the vmdk file had disappeared.   This happened while I was at work.  The file was located in a directory on a FREENAS iSCSI volume which I am able to browse to using the Datastore Browser.  All the other files seem to be there. I've attached the last log file.

Any help would be greatly appreciated.

[vJJSosa]

Hi Gary,

Please check this KB:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004232

You need to browse Datastore from Command line into the Host, should be able to see the flat vmdk, please let me know if you can see it.

Regards

[GaryDank]

No it doesn't exist.  What is in the directory is a .vmss, a .vmx, a .nram, a .vmxf, a .vmsd and 7 log files

[vJJSosa]

Can you explain me with more details what happened? how did you found the incident? some previous task? please, if you can, share your .vmx file

on the other hand, this was a test with another disk?

OPEN scsi0:1 '/vmfs/volumes/50a47359-c3e7d514-3f5d-001e68c5c1d8/New Virtual Machine/New Virtual Machine_1.vmdk' persistent

Regards

[GaryDank]

No clue how it happened, I was at work and when I came home I server wasn't available.

[vJJSosa]

Do you have a vCenter? could you export events from host or vCenter?

[GaryDank]

Here you go

[vJJSosa]

I need the Host Events, or vCenter Events. And Logs (message file) from the Host, not the VM.

[GaryDank]

my bad

Is this what you are after?

[GaryDank]

one more time

[continuum]

The missing flat.vmdk was stored on a VMFS formatted iSCSI LUN ?

Can you create a new VM that can attach that LUN - either via RDM or with a iSCSI-client running in a guest ?

If yes - I have several procedures for that scenarion but no time to explain details here.

If the vmdk is important give me teamviewer access and I can check myself.

[GaryDank]

Yes the vmdk file was stored on a VMFS formatted iSCSI LUN.

I can attempted to create a VM but not for a couple of hours if you are still available to help?

[aaronwsmith]

Does anyone else other than you have access to delete files on a datastore?

I believe you cannot delete a VMDK file that's in use.  We experimented with this a while back when looking at detecting and removing orphaned VMDK files and couldn't delete a VMDK if it was registered to a VM and powered on.

So if the VMDK file was deleted, to me suggests the VM was powered down, the file deleted, and the VM powered back on.  Events in vCenter may not reflect this if someone logged in directly to an ESX/ESXi host and performed these operations via the command line.

But I believe the events should show up on the ESX/ESXi host itself if you login directly to it with a vSphere Client and check its local set of events/tasks?

[GaryDank]

No one has access to the datastore ... this is on a private network.  

[aaronwsmith]

Does anyone else have access to the iSCSI NAS server hosting your VM storage?  Maybe something went wrong on that side and forced a deletion while the VM was still running?

[GaryDank]

No one ... not really concerned with the why but how to recover the vmdk.

[continuum]

> I believe you cannot delete a VMDK file that's in use.

ESXi itself can loose vmdks all by itself - even when ESXi can not see the files they may still be visible to vmfs-fuse

[GaryDank]

How do you want to proceed?

[chin2reddy]

Create a new VM with same OS (you need not install the os here, just create) as the vm for which the vmdk is missing with exactly same hard disk size. Now a new vmdk will be created. Copy this vmdk to the missing place (datastore). power on now.

Thanks,

CHIN2

[GaryDank]

CHIN2,

Will this procedure potentially hinder the recovery of the original vmdk file?  The original contains all the users data.

Gary

[continuum]

:> Will this procedure potentially hinder the recovery of the original vmdk file?

yes - so do not use that "tip"

check your PMs

[GaryDank]

PMs ?

[vJJSosa]

Could you share with us the complete list of files into the VM folder? Please use "ls" from command line.

[GaryDank]

yes in a couple of hrs

[GaryDank]

/vmfs/volumes/50a47359-c3e7d514-3f5d-001e68c5c1d8/AEISVR02 # ls
AEISVR02-69a222fb.vmss

AEISVR02.vmx          

vmware-5.log          

vmware-8.log
AEISVR02.nvram        

AEISVR02.vmxf         

vmware-6.log          

vmware-9.log
AEISVR02.vmsd         

vmware-10.log         

vmware-7.log          

vmware.log

[aaronwsmith]

What do you get when you run this command to show all files (including hidden): ls -lah

[GaryDank]

/vmfs/volumes/50a47359-c3e7d514-3f5d-001e68c5c1d8/AEISVR02 # ls -lah
drwxr-xr-x    1 root     root        1.8K Mar  3 03:39 .
drwxr-xr-t    1 root     root        1.4K Jan 10 00:52 ..
-rw-------    1 root     root        8.5K Mar  3 03:39 AEISVR02.nvram
-rw-r--r--    1 root     root           0 Jan 10 00:52 AEISVR02.vmsd
-rwxr-xr-x    1 root     root        3.2K Mar  3 03:39 AEISVR02.vmx
-rw-r--r--    1 root     root        3.0K Mar  3 02:46 AEISVR02.vmxf
-rw-r--r--    1 root     root       47.0K Feb 28 17:47 vmware-10.log
-rw-r--r--    1 root     root       47.0K Feb 28 17:59 vmware-11.log
-rw-r--r--    1 root     root      240.9K Mar  2 03:21 vmware-12.log
-rw-r--r--    1 root     root       46.7K Mar  2 03:22 vmware-13.log
-rw-r--r--    1 root     root      201.6K Mar  3 03:39 vmware-14.log
-rw-r--r--    1 root     root       47.0K Feb 28 17:42 vmware-9.log
-rw-r--r--    1 root     root       46.7K Mar  3 03:39 vmware.log

[tuanvq84]

It happend to my company vmware system too.

When I  was going to clone a vm from template ,and got an error "cannot clone blah blah..." and I went to datastore ,vmdk file was disappered, luckly  this is only template vm.

Any one got this situation ?

[continuum]

> Any one got this situation ?

Yes - when you run into this again and the data inside the missing vmdk is valueable then try to stop all activity on that datastore as soon as possible.
Power off running VMs so that they do not create additional files on the datastore - so use poweroff rather than suspend.
Then read the datastore with a ESXi-LiveCD or - better - with a LinuxLiveCD using vmfs-fuse.

To improve your chances for recovery do not use thin provisioned vmdks larger than 250 GB

[crazyware88]

I am the same problems. The vmdk disappeared

[a_p_]

Welcome to the Community,

please open a new discussion instead of replying to a 7 years old thread.

Provides details, such as your ESXi version/build, storage type (local, or shared), VMFS version, a complete list of files in the VM's folder (i.e. the output of ls -lisa), a .zip archive which includes the VM's .vmx file as well as the vmware*.log files).

Also explain what happened that may have caused the issue.

André