VM Failed - The attempted operation cannot be performed in the current state (Powered off)

[DickyChui2021]

Hi,

On my EXSi 7.0.3U, I have created a VM machine with Windows 10 OS and was shutdown yesterday. But when I'm trying to turn on the machine, an error message pop-up  "Failed to power on virtual machine [VM Name]. The attempted operation cannot be performed in the current state (Powered off). Click here for more details." No matter what action I was trying to apply on this VM machine, it will show the message above. I have tried some of the methods from the internet but still doesn't work at all.

1. I have tried "unregister vm" but when the action is applied the fail message appears.

2. I have tried "shh console" to EXSi host and push the command  "vim-cmd vmsvc/power.on" , but still fail with the error message.

3.  I have tried "Delete VM" but still fail with error message.

Does anyone have the same situation as I did? 

[cmercado]

Hello!

I try to perform actions from the ESXi UI?

[DickyChui2021]

Yes, Tried on ESXi  ui webportal, all actions on that vm machine is failed

[__invited__rob_]

I would look at the host directly. See if you can get some things working on the host, not through the VCSA. Can you vmotion other vm's off host? Is the storage working as expected? Whlas there a space issue when creating the VM?

Can you evict all other VM's off host through the esx gui for the effected host and reboot?

 

[DickyChui2021]

There isnt any space issue as the host still have at least few TBs remain, storage is working, I can create/delete other VM anytime.

From the host, other VMs are all functional through ui and I can turn on/off or anyother action to other VMs. 

Only that specific VM have that fail message, other VMs is normal as usual.

[Patrik Kernstock]

Does give the log file at /var/run/log/hostd.log on the ESXi host itself give any clue? Just "tail -f" the file during power-on or search the timestamp. There should be something more detailed why the PowerOn failed.

[cmercado]

You could paste the contents of the vmware.log located in the datastore folder where you own your VM.

[compdigit44]

Does the VM have any snapshots? If you build another Windows 10 VM the same way, does it have the issue? Have you tried to clone the VM to see if you can power on or edit the cloned VM? Have you tried to power on or edit the VM using PowerCli?

[DickyChui2021]

1. This VM dont have any snapshot, it's a test VM

2. There isnt any issue when I build another Windows 10 VM, any new VM is perfectly run.

3. I have tried to export the VM and re-import as a new vm, it works.

   But that old source VM is still exist and I cannot destory/delete/turn on/off on this old vm. Thats the main problem of this case

[jrmunday]

It sounds like something configured in the VMX file is no longer available, can you upload the VMX file?

You could try deleting the existing .vmsd file (re-created on power on) and then powering the VM on to see if you still get this error.

[DickyChui2021]

These are the logs (part of the logs) from the contents in vmware.log.
I cutted out the latest part from the log.

The latest log date is 5/31 and I cannot do any action from this vm which the logs doesnt generate anything since the error occured.

 

2022-05-31T10:38:07.222Z In(05)+ vmx -
2022-05-31T10:38:07.222Z In(05) vmx - OvhdMem: memsize 4096 MB VMK fixed 3349 pages var(mem) 529 pages var(cpu) 22 cbrcOverhead 0 pages total 5511 pages
2022-05-31T10:38:07.222Z In(05) vmx - VMMEM: Maximum Reservation: 575MB (MainMem=4096MB) VMK=21MB
2022-05-31T10:38:07.223Z In(05) vmx - TOOLS using cached value 1 for isoImageExists (refreshCount=1, lastCount=1).
2022-05-31T10:38:07.223Z In(05) vmx - Tools: ToolsRunningStatus_Exit, delayedRequest is 0x6FC6EC1600
2022-05-31T10:38:07.223Z In(05) vmx - SVMotion_PowerOff: Not running Storage vMotion. Nothing to do
2022-05-31T10:38:07.224Z In(05) vmx - Tools: [AppStatus] Last heartbeat value 9 (last received 802s ago)
2022-05-31T10:38:07.224Z In(05) vmx - TOOLS: appName=toolbox, oldStatus=0, status=0, guestInitiated=0.
2022-05-31T10:38:07.228Z In(05) vmx - Destroying virtual dev for scsi0:0 vscsi=9798942116028466
2022-05-31T10:38:07.228Z In(05) vmx - VMMon_VSCSIStopVports: No such target on adapter
2022-05-31T10:38:07.228Z In(05) vmx - Destroying virtual dev for scsi0:1 vscsi=9798942120222771
2022-05-31T10:38:07.228Z In(05) vmx - VMMon_VSCSIStopVports: No such target on adapter
2022-05-31T10:38:07.229Z In(05) vmx - SVMotion_PowerOff: Not running Storage vMotion. Nothing to do
2022-05-31T10:38:07.230Z In(05) mks - MKS-RenderMain: Stopped MKSBasicOps
2022-05-31T10:38:07.230Z In(05) mks - MKS PowerOff
2022-05-31T10:38:07.230Z In(05) svga - SVGA thread is exiting
2022-05-31T10:38:07.230Z In(05) mks - MKS thread is exiting
2022-05-31T10:38:07.230Z Wa(03) vmx -
2022-05-31T10:38:07.230Z In(05) vmx - scsi0:1: numIOs = 0 numMergedIOs = 0 numSplitIOs = 0 ( 0.0%)
2022-05-31T10:38:07.230Z In(05) vmx - Closing disk 'scsi0:1'
2022-05-31T10:38:07.231Z In(05) vmx - DISKLIB-VMFS : "/vmfs/volumes/624a947e-7ffc7296-6288-3868dd6dc948/EDI Server/EDI Server_1-flat.vmdk" : closed.
2022-05-31T10:38:07.238Z In(05) vmx - scsi0:0: numIOs = 0 numMergedIOs = 0 numSplitIOs = 0 ( 0.0%)
2022-05-31T10:38:07.238Z In(05) vmx - Closing disk 'scsi0:0'
2022-05-31T10:38:07.238Z In(05) vmx - DISKLIB-VMFS : "/vmfs/volumes/624a947e-7ffc7296-6288-3868dd6dc948/EDI Server/EDI Server-flat.vmdk" : closed.
2022-05-31T10:38:07.240Z In(05) vmx - Vix: [mainDispatch.c:1164]: VMAutomationPowerOff: Powering off.
2022-05-31T10:38:07.241Z In(05) vmx - WORKER: asyncOps=1256 maxActiveOps=2 maxPending=2 maxCompleted=1
2022-05-31T10:38:07.569Z In(05) vmx - Vix: [mainDispatch.c:4206]: VMAutomation_ReportPowerOpFinished: statevar=1, newAppState=1873, success=1 additionalError=0
2022-05-31T10:38:07.569Z In(05) vmx - Vix: [mainDispatch.c:4224]: VMAutomation: Ignoring ReportPowerOpFinished because the VMX is shutting down.
2022-05-31T10:38:07.570Z No(00) vmx - ConfigDB: Setting cleanShutdown = "TRUE"
2022-05-31T10:38:07.576Z In(05) vmx - Vix: [mainDispatch.c:4206]: VMAutomation_ReportPowerOpFinished: statevar=0, newAppState=1870, success=1 additionalError=0
2022-05-31T10:38:07.576Z In(05) vmx - Vix: [mainDispatch.c:4224]: VMAutomation: Ignoring ReportPowerOpFinished because the VMX is shutting down.
2022-05-31T10:38:07.576Z In(05) vmx - Transitioned vmx/execState/val to poweredOff
2022-05-31T10:38:07.576Z In(05) vmx - Vigor_ClientRequestCb: failed to do op=5 on unregistered device 'ConfigParams' (cmd=(null))
2022-05-31T10:38:07.576Z In(05) vmx - VMX idle exit
2022-05-31T10:38:07.582Z In(05) vmx - Vix: [mainDispatch.c:815]: VMAutomation_LateShutdown()
2022-05-31T10:38:07.582Z In(05) vmx - Vix: [mainDispatch.c:770]: VMAutomationCloseListenerSocket. Closing listener socket.
2022-05-31T10:38:07.582Z In(05) vmx - Flushing VMX VMDB connections
2022-05-31T10:38:07.583Z In(05) vmx - VigorTransport_ServerCloseClient: Closing transport 6FC6BB2970 (err = 0)
2022-05-31T10:38:07.583Z In(05) vmx - VigorTransport_ServerDestroy: server destroyed.
2022-05-31T10:38:07.584Z In(05) vmx - VMX exit (0).
2022-05-31T10:38:07.585Z In(05) vmx - OBJLIB-LIB: ObjLib cleanup done.
2022-05-31T10:38:07.585Z In(05) vmx - AIOMGR-S : stat o=9 r=40 w=116 i=0 br=655868 bw=29884672
2022-05-31T10:38:07.585Z Wa(03) vmx - VMX has left the building: 0.

[jrmunday]

Can you confirm there are no snapshots and check what files the disks are pointing to? If you attach the VMX file and a directory listing from the VM folder we can check.

I've seen similar issues caused by failed consolidation tasks where the VMX is pointing to a delta file even though no snapshots exist. The fix was to remove the VM from inventory, edit the VMX file to point back to the base disks and not the delta/ redo disks, re-register the VM using the updated VMX file and then power it on. But it sounds like you can't remove it from inventory?

It may also be a file lock which you'll need to trace, but in some instances, I've seen restarting the management agents on the host resolve where the VM is being managed from allowing admin tasks such as removal from inventory.

[DickyChui2021]

At last I have to shutdown the Host on Sat and restart the host.

After the restart I can delete that VM and get rid of it.

Tried different method but doesnt work without restarting.

BTW thanks everyone for the advice.

[ik27]

Hi, no need to restart the host. I simply restart the Hostd service and it works for me.
/etc/init.d/hostd restart

[swinster]

Perfect. For me, I could not start, unregister or even delete the VM, but restating `hostd` did the trick, and nice a quick as well. 

[CpuFixrMan]

 Life saver!!!  Thank you!

[JhunA52681]

Hi everyone, my friend got same issue yesterday.  I gave him the link to see your convo here and just ask me if, is it necessary to shutdown all other VMs in the host before issuing hostd restart command? Please advice. 

[Hafez-Nomad]

Thank you very much !!!!!!! most importantly i did not effect any running VM

[Sajjad-Malik]

Hi.

I just unregister and register again and its working fine.