VM Swap File Corrupt - VM will not power on

[bowulf]

I was resetting a new Windows XP Virtual machine when it got stuck. I tried using a vmware-cmd vm.vmx stop or even "... stop trysoft," but each time it failed saying there was "Operation failed since another task is in progress." At this point, I found out the PID for the VM and killed it "kill -9 ###".

Now the VM will not start. It has an error message:

"Could not power on VM: No swap file."

I am guessing I corrupted the swap file or locked it open. Where should I progress from here? Thanks.

[admin]

Could you post the log file for the virtual machine? It should be present in the virtual machine directory under the name "vmware.log".

Thanks

[bowulf]

Here is the log:

Aug 03 11:04:33.086: vmx| Log for VMware ESX Server pid=101354 version=3.0.0 build=build-27701 option=Release

Aug 03 11:04:33.089: vmx| Hostname=thovmware01.fbfs.com

Aug 03 11:04:33.090: vmx| Command line: "/usr/lib/vmware/bin/vmware-vmx" "-ssched.group=host/user" "-@" "pipe=/tmp/vmhsdaemon-0/vmxb19252f99df44f6b;vm=b19252f99df44f6b" "/vmfs/volumes/44bd0a4d-90ccf2b2-0ecc-001125414324/thpxp03-1/thpxp03-1.vmx"

Aug 03 11:04:33.145: vmx| UI Connecting to pipe '/tmp/vmhsdaemon-0/vmxb19252f99df44f6b' with user '(null)'

Aug 03 11:04:33.174: vmx| pcpu #0 CPUID numEntries=5 GenuntelineI

Aug 03 11:04:33.174: vmx| pcpu #0 CPUID version=0xf43 id1.edx=0xbfebfbff id1.ecx=0x659d id1.ebx=0x20800

Aug 03 11:04:33.174: vmx| pcpu #0 CPUID id80.eax=80000008 id81.edx=0x20000000 id81.ecx=0x0

Aug 03 11:04:33.174: vmx| pcpu #1 CPUID numEntries=5 GenuntelineI

Aug 03 11:04:33.175: vmx| pcpu #1 CPUID version=0xf43 id1.edx=0xbfebfbff id1.ecx=0x659d id1.ebx=0x1020800

Aug 03 11:04:33.175: vmx| pcpu #1 CPUID id80.eax=80000008 id81.edx=0x20000000 id81.ecx=0x0

Aug 03 11:04:33.175: vmx| pcpu #2 CPUID numEntries=5 GenuntelineI

Aug 03 11:04:33.175: vmx| pcpu #2 CPUID version=0xf43 id1.edx=0xbfebfbff id1.ecx=0x659d id1.ebx=0x6020800

Aug 03 11:04:33.175: vmx| pcpu #2 CPUID id80.eax=80000008 id81.edx=0x20000000 id81.ecx=0x0

Aug 03 11:04:33.175: vmx| pcpu #3 CPUID numEntries=5 GenuntelineI

Aug 03 11:04:33.176: vmx| pcpu #3 CPUID version=0xf43 id1.edx=0xbfebfbff id1.ecx=0x659d id1.ebx=0x7020800

Aug 03 11:04:33.176: vmx| pcpu #3 CPUID id80.eax=80000008 id81.edx=0x20000000 id81.ecx=0x0

Aug 03 11:04:33.176: vmx| CPUID id1.edx: 0xbfebfbff id1.ecx: 0x659d id81.edx: 0x20000000 id81.ecx: 0

Aug 03 11:04:33.176: vmx| CPUID id88.ecx: 0 id88.edx: 0

Aug 03 11:04:33.218: vmx| Setup symlink /var/run/vmware/%2fvmfs%2fvolumes%2f44bd0a4d%2d90ccf2b2%2d0ecc%2d001125414324%2fthpxp03%2d1%2fthpxp03%2d1%2evmx -> /var/run/vmware/root_0/1154621073092349_101354

Aug 03 11:04:33.220: vmx| ACL_InitCapabilities: current IPC thread

Aug 03 11:04:33.220: vmx| ACL_InitCapabilities: done

Aug 03 11:04:33.221: vmx| changing directory to /vmfs/volumes/44bd0a4d-90ccf2b2-0ecc-001125414324/thpxp03-1/.

Aug 03 11:04:33.221: vmx| Config file: /vmfs/volumes/44bd0a4d-90ccf2b2-0ecc-001125414324/thpxp03-1/thpxp03-1.vmx

Aug 03 11:04:33.311: vmx| DISKLIB-VMFS : "/vmfs/volumes/44bd0a4d-90ccf2b2-0ecc-001125414324/thpxp03-1/thpxp03-1-flat.vmdk" : open successful (23) size = 4294967296, hd = -1. Type 3

Aug 03 11:04:33.314: vmx| DISKLIB-VMFS : "/vmfs/volumes/44bd0a4d-90ccf2b2-0ecc-001125414324/thpxp03-1/thpxp03-1-flat.vmdk" : closed.

Aug 03 11:04:33.628: vmx| VMXVmdb_LoadRawConfig: Loading raw config

Aug 03 11:04:33.722: vmx| VMXVmdbCbVmVmxExecState: Exec state change requested to state poweredOn without reset

Aug 03 11:04:33.722: vmx| TOOLS delaying state change request to state 3

Aug 03 11:04:33.722: vmx| PowerOn

Aug 03 11:04:33.728: vmx| VMXVmdb_LoadRawConfig: Loading raw config

Aug 03 11:04:33.794: vmx| VMMon_CreateVM: vmmon.numVCPUs=1

Aug 03 11:04:33.877: vmx| Msg_Post: Error

Aug 03 11:04:33.877: vmx| \[msg.vmmonVMK.creatVMFailed] Could not power on VM : No swap file.

Aug 03 11:04:33.877: vmx| \[msg.monitorLoop.createVMFailed] Failed to power on VM----

---

Aug 03 11:04:33.960: vmx| Module MonitorLoop power on failed.

Aug 03 11:04:33.960: vmx| VMX_PowerOn: ModuleTable_PowerOn = 0

Aug 03 11:04:34.554: vmx| vmdbPipe_Streams Couldn't read: OVL_STATUS_EOF

Aug 03 11:04:34.555: vmx| VMX idle exit

Aug 03 11:04:34.584: vmx| Flushing VMX VMDB connections

Aug 03 11:04:34.585: vmx| IPC_exit: disconnecting all threads

Aug 03 11:04:34.585: vmx| VMX exit.

[geirin]

I have the exact same problem after a reset of a Vista Virtual machine got stuck. In the Virtual machine folder i have a .vswp-file which cannot be removed.

My .log looks like this when trying to restart the machine:

Aug 04 09:45:53.327: vmx| Log for VMware ESX Server pid=101178 version=3.0.0 build=build-27701 option=Release

Aug 04 09:45:53.334: vmx| Hostname=xxxx.net

Aug 04 09:45:53.334: vmx| Command line: "/usr/lib/vmware/bin/vmware-vmx" "-ssched.group=host/user" "-@" "pipe=/tmp/vmhsdaemon-0/vmxf2afe72882701c59;vm=f2afe72882701c59" "/vmfs/volumes/44d1d1c2-9b65d88c-75c8-0002a5e7a83d/Vistatest/Vistatest.vmx"

Aug 04 09:45:53.402: vmx| UI Connecting to pipe '/tmp/vmhsdaemon-0/vmxf2afe72882701c59' with user '(null)'

Aug 04 09:45:53.437: vmx| pcpu #0 CPUID numEntries=2 GenuntelineI

Aug 04 09:45:53.438: vmx| pcpu #0 CPUID version=0xf11 id1.edx=0x3febfbff id1.ecx=0x0 id1.ebx=0x2080e

Aug 04 09:45:53.438: vmx| pcpu #0 CPUID id80.eax=80000004 id81.edx=0x0 id81.ecx=0x0

Aug 04 09:45:53.438: vmx| pcpu #1 CPUID numEntries=2 GenuntelineI

Aug 04 09:45:53.438: vmx| pcpu #1 CPUID version=0xf11 id1.edx=0x3febfbff id1.ecx=0x0 id1.ebx=0x102080e

Aug 04 09:45:53.438: vmx| pcpu #1 CPUID id80.eax=80000004 id81.edx=0x0 id81.ecx=0x0

Aug 04 09:45:53.438: vmx| pcpu #2 CPUID numEntries=2 GenuntelineI

Aug 04 09:45:53.438: vmx| pcpu #2 CPUID version=0xf11 id1.edx=0x3febfbff id1.ecx=0x0 id1.ebx=0x202080e

Aug 04 09:45:53.439: vmx| pcpu #2 CPUID id80.eax=80000004 id81.edx=0x0 id81.ecx=0x0

Aug 04 09:45:53.439: vmx| pcpu #3 CPUID numEntries=2 GenuntelineI

Aug 04 09:45:53.439: vmx| pcpu #3 CPUID version=0xf11 id1.edx=0x3febfbff id1.ecx=0x0 id1.ebx=0x302080e

Aug 04 09:45:53.439: vmx| pcpu #3 CPUID id80.eax=80000004 id81.edx=0x0 id81.ecx=0x0

Aug 04 09:45:53.439: vmx| pcpu #4 CPUID numEntries=2 GenuntelineI

Aug 04 09:45:53.439: vmx| pcpu #4 CPUID version=0xf11 id1.edx=0x3febfbff id1.ecx=0x0 id1.ebx=0x402080e

Aug 04 09:45:53.440: vmx| pcpu #4 CPUID id80.eax=80000004 id81.edx=0x0 id81.ecx=0x0

Aug 04 09:45:53.440: vmx| pcpu #5 CPUID numEntries=2 GenuntelineI

Aug 04 09:45:53.440: vmx| pcpu #5 CPUID version=0xf11 id1.edx=0x3febfbff id1.ecx=0x0 id1.ebx=0x502080e

Aug 04 09:45:53.440: vmx| pcpu #5 CPUID id80.eax=80000004 id81.edx=0x0 id81.ecx=0x0

Aug 04 09:45:53.440: vmx| pcpu #6 CPUID numEntries=2 GenuntelineI

Aug 04 09:45:53.440: vmx| pcpu #6 CPUID version=0xf11 id1.edx=0x3febfbff id1.ecx=0x0 id1.ebx=0x602080e

Aug 04 09:45:53.441: vmx| pcpu #6 CPUID id80.eax=80000004 id81.edx=0x0 id81.ecx=0x0

Aug 04 09:45:53.441: vmx| pcpu #7 CPUID numEntries=2 GenuntelineI

Aug 04 09:45:53.441: vmx| pcpu #7 CPUID version=0xf11 id1.edx=0x3febfbff id1.ecx=0x0 id1.ebx=0x702080e

Aug 04 09:45:53.441: vmx| pcpu #7 CPUID id80.eax=80000004 id81.edx=0x0 id81.ecx=0x0

Aug 04 09:45:53.441: vmx| CPUID id1.edx: 0x3febfbff id1.ecx: 0 id81.edx: 0 id81.ecx: 0

Aug 04 09:45:53.442: vmx| CPUID id88.ecx: 0 id88.edx: 0x7a7023

Aug 04 09:45:53.462: vmx| Setup symlink /var/run/vmware/%2fvmfs%2fvolumes%2f44d1d1c2%2d9b65d88c%2d75c8%2d0002a5e7a83d%2fVistatest%2fVistatest%2evmx -> /var/run/vmware/root_0/1154677553337794_101178

Aug 04 09:45:53.463: vmx| ACL_InitCapabilities: current IPC thread

Aug 04 09:45:53.463: vmx| ACL_InitCapabilities: done

Aug 04 09:45:53.464: vmx| changing directory to /vmfs/volumes/44d1d1c2-9b65d88c-75c8-0002a5e7a83d/Vistatest/.

Aug 04 09:45:53.464: vmx| Config file: /vmfs/volumes/44d1d1c2-9b65d88c-75c8-0002a5e7a83d/Vistatest/Vistatest.vmx

Aug 04 09:45:53.482: vmx| DISKLIB-VMFS : "/vmfs/volumes/44d1d1c2-9b65d88c-75c8-0002a5e7a83d/Vistatest/Vistatest-flat.vmdk" : open successful (23) size = 10737418240, hd = -1. Type 3

Aug 04 09:45:53.484: vmx| DISKLIB-VMFS : "/vmfs/volumes/44d1d1c2-9b65d88c-75c8-0002a5e7a83d/Vistatest/Vistatest-flat.vmdk" : closed.

Aug 04 09:45:53.566: vmx| VMXVmdb_LoadRawConfig: Loading raw config

Aug 04 09:45:53.800: vmx| VMXVmdbCbVmVmxExecState: Exec state change requested to state poweredOn without reset

Aug 04 09:45:53.801: vmx| TOOLS delaying state change request to state 3

Aug 04 09:45:53.801: vmx| PowerOn

Aug 04 09:45:53.811: vmx| VMXVmdb_LoadRawConfig: Loading raw config

Aug 04 09:45:53.838: vmx| VMMon_CreateVM: vmmon.numVCPUs=1

Aug 04 09:45:53.843: vmx| Msg_Post: Error

Aug 04 09:45:53.843: vmx| \[msg.vmmonVMK.creatVMFailed] Could not power on VM : No swap file.

Aug 04 09:45:53.843: vmx| \[msg.monitorLoop.createVMFailed] Failed to power on VM----

---

Aug 04 09:45:53.936: vmx| Module MonitorLoop power on failed.

Aug 04 09:45:53.937: vmx| VMX_PowerOn: ModuleTable_PowerOn = 0

Aug 04 09:45:54.312: vmx| vmdbPipe_Streams Couldn't read: OVL_STATUS_EOF

Aug 04 09:45:54.313: vmx| VMX idle exit

Aug 04 09:45:54.333: vmx| Flushing VMX VMDB connections

Aug 04 09:45:54.333: vmx| IPC_exit: disconnecting all threads

Aug 04 09:45:54.333: vmx| VMX exit.

[admin]

Could you file an SR with vm-support output attached? It looks like we'll need to get more information to debug this problem.

Thanks

[jbsengineer]

I was resetting a new Windows XP Virtual machine when

it got stuck. I tried using a vmware-cmd vm.vmx stop

or even "... stop trysoft," but each time it failed

saying there was "Operation failed since another task

is in progress." At this point, I found out the PID

for the VM and killed it "kill -9 ###".

Now the VM will not start. It has an error message:

"Could not power on VM: No swap file."

I am guessing I corrupted the swap file or locked it

open. Where should I progress from here? Thanks.

FYI, Had the same problem. Rebooted the ESX server and that did not help. However a cold migration did work. Wish I knew what caused the SWAP file to do that.

[Kit Colbert]

Can you post the vmkernel log for the period when you attempted to power on the VM but it failed because of the swap file issue? I think doing a 'grep -i swap' should help you find it... Thanks.

[tatung70]

Just boot to linux mode and delete the swap file. Then reboot to where the kernel starts and re-create the swap file.

Then start your VM's.

[Melih85c]

i have the same problem, but i dont understand what i should to do.? to boot into linux mode, ok . but where can i delete the swap file?

i cant start a VM, which i kill by using the kill -9 command in the service console.

Message was edited by:

Melih85c

[stichnip]

how do you boot to linux mode?

[LeaV97]

Did anyone every find a solution for this? I'm running into the same problem.

[Wimo]

We've been having this problem too, and get that "another task is in progress" error when we try to stop or recycle the VM.

Just yesterday I came up with this same solution (I had actually thought of it a long time ago, but was getting the PID from the VM's log file, which is not the same as what you get with a ps -ef, I don't know why). I did just plain kill, not kill -9, and that worked just fine. Don't know why that would make a difference, and I've only done it once so far. Now I'm afraid to do it again next time!

We do have a support case open, for months now.

[oreeh]

Don't know why that would make a difference

the difference is the following

A kill -9 terminates the running process without giving the process a chance to clean up (close filehandles,...)

A "normal" kill sends the process a signal that it should stop, in this case the process is able to do the cleanup

[iwsmith]

I found a similar problem while using HA. I found I could resolve the matter by killing the process as oreeh described. This unlocked the VM and allowed me to power on - no need to remove the swap file.

[nonu]

We had the same issue recently,but i tried finding the process using ps commands but was unable to do so.

Eventually had to reboot the host and that removed the locks.

Also when we powered it on a new host it got a new uid.

I am not sure what it indicate,but that's what happened.

[mats82]

putting my host into maintenance mode and allowing the DRS/HA to move it to another host solved it for me. I rebooted the host as well when in maint. mode. Not sure if that was necessary but did it anyway

[Wimo]

This error can occur if one of the other hosts thinks it is running the VM. Log onto each host and do ps -ef |grep vmname. If it shows up, kill it.

[rodupr]

What if the process for the VM doesn't show up on any of the hosts in the cluster and you still can't power it on?

[meredith_shaeba]

Getting rid of the .vswp file worked for me -- I renamed it, powered on the Virtual Machine, and then deleted it.

[ntshane]

I found that in my case, the VM in question was running on one of my ESX hosts and was not showing up as running in virtualcenter for some reason. Once I logged into the ESX host it was running on with the VIC and killed the vm, I was able to start it via virtualcenter just fine.

-shane

[Oczkov]

For those who can't find the PID of the VM please use the following command:

ps auxfww | grep &lt;vmname&gt;</span>

ps -ef is not sufficient since the behaviour of ps command has changed in one of the Service Console versions (coming with ESX 3.5) I guess. ps -ef for sure worked on 3.0.x ESX Servers.

In my case, I had to both:

- kill the VM (I have just used -TERM (or no singnal number), not the -9, however this should be used if the machine cannot be killed)

- remove the .vswp file from the datastore.

I'm sure this is a bug related to ESX3.5 Upadate 2. The latest patches (as of time of writing this replay) do not solve the problem and it is happening on most of our VI3 farms. ESX 3.5 U1 didn't have this error, I think.

Best regards,

Oczkov

[HotDaddy]

Thanks Oczkov about the tip, saved my day :smileyhappy:

[zakd]

I had the same problems when attempting to power on a VM, Could not power on VM : No swap file.

Also got vmotion timeout at 10% only on certain VM's and the hosts would not complete entering maintenance mode.

I checked all hosts in the cluster for a process lock using the ps command and found none.

None of the files were locked on any host, checked using vmkfstools -D, SAN team also confirmed no problems found.

Checked all network settings and even recreated the portgroups from scratch, this has fixed other problems in the past but not this time.

After lot of digging i found some VM's took way too long to list their fVMFS folder contents because they had thousand of vmware-????.log files.

When trying to list the contents of the VMFS folders from the service console it would appear to hang.

So I ran this to get the file count for all datastores: find /vmfs/volumes/ -iname vmware-.log | wc -l and found over 100,000 files and it took around 60 minutes to complete.

Then I ran this to delete these files on all datastores: find /vmfs/volumes/ -iname vmware-.log -exec rm -f {} \;

This resloved the problem, no more vmotion timeouts or swap file errors, and hosts went into maintenance mode.

Below are the contents of one of the log files.

Nov 06 18:14:41.728: vmx| DISKLIB-VMFS : "/vmfs/volumes/489ee445-d2f27495-850e-00215ab0e71e/VLON1/VLON1-flat.vmdk" : closed.

Nov 06 18:14:45.877: vmx| LogBackupByRenumber: failed to remove /vmfs/volumes/489ee445-d2f27495-850e-00215ab0e71e/VLON1/vmware-1.log failed: Device or resource busy

Nov 06 18:14:46.879: vmx| VMXVmdb_LoadRawConfig: Loading raw config

Nov 06 18:14:47.644: vmx| VMXVmdbCbVmVmxExecState: Exec state change requested to state poweredOn without reset

Nov 06 18:14:47.644: vmx| TOOLS delaying state change request to state 3

Nov 06 18:14:47.644: vmx| PowerOn

Nov 06 18:14:47.648: vmx| VMXVmdb_LoadRawConfig: Loading raw config

Nov 06 18:14:47.737: vmx| VMMon_CreateVM: vmmon.numVCPUs=1

Nov 06 18:14:51.789: vmx| Msg_Post: Error

Nov 06 18:14:51.789: vmx| Could not power on VM : No swap file.

Nov 06 18:14:51.789: vmx| Failed to power on VM----

---

[jjgunn]

oczkov nailed it as well as a few others.

First was removing the vswp file (or renaming it). Then.....

The vmdk file was locked but not by process. To figure out which host locked the vmdk, it is located in a log file via MAC address. listed as "owner" with a long string identifying your LUN and the MAC Address.

I have 8 ESX servers in a cluster so we had to search each one using the "ifconfig" command until we found the proper ESX host that was holding the VM hostage.

Then I simply vmotioned all VM's off the esx host and entered maint. mode then rebooted the host. This reboot released the "lock" on the vmdk file & the VM was able to power on again.

Hope this is helpful for someone else.

[DRGilroy]

Thank you Oczkov! Your post was exactly what I needed.

[mcretan]

doh!

I have the same problem (kill -9 VM, now getting no swap message) - sounds like a tough one to solve - has anyone found the solution?

Regards,

Mike Cretan

[fechegaray]

Check the directory where the .vmx file resides.

If there's a .vswp file in it, remove it (Or rename it if you prefer).

Then try to power up again.