VMware Fusion 12.2.5 on macOS Monterey 12.7.6 Failed to power on when starting Windows Server 2012 vm

[Ivan Chang]

Would like to request urgent help since this is a time sensitive issue for a business that is currently down. We have removed all the .lck files and snapshot files in the VMware bundle and cleared out the snapshots in the Virtual Machine Library for this vm. In Settings, General ran Clean Up Virtual Machine. After performing the above steps we are still getting the Failed to power on when starting the vm. Any assistance would be greatly appreciated.

------------------------------
Ivan Chang
------------------------------

[Technogeezer]

Can you attach the vmware.log file found inthe virtual machine bundle?

------------------------------
- Paul (technogeezer)
------------------------------

[Ivan Chang]

Paul,

Here is the vmware.log file from the virtual machine bundle.

Regards,

Ivan

------------------------------
Ivan Chang
------------------------------

Attachment(s)

vmware.log   511 KB 1 version

[Technogeezer]

One comment:

The logs are indicating a significant amount of activity related to snapshots. You appear to have autoprotect snapshots enabled. That's generally considered a very bad idea - especially for a  production VM. Autoprotect relies on Fusion snapshots, which are brittle - it doesn't take much to break a virtual disk when multiple layers of snapshots are active - which is what Autoprotect can do. Don't rely on autoprotect to provide virtual machine backup.

I'm not seeing error messages in the log that are pointing to anything I can identify. If there are other files in the VM bundle that have names like vmware-*.log they might help us.

Did something happen to this Mac (e.g. crash) before this Fusion failure occurred? 

Is this VM currently in a powered-down or suspended state?

------------------------------
- Paul (technogeezer)
------------------------------

[Technogeezer]

Just a thought to make data collection a bit simipler, Can you run the script I've attached here to collect more information about this VM.

Unzip the script in the Downloads folder, shut down the Fusion application, and run the data collection script as follows in a Terminal session:

cd ~/Downloads

./collect_info.sh "/Volumes/ThunderBay/Virtual Machines/Windows Server 2012 R2.vmwarevm"

It'll collect info about the system and VM then creates a zip file  with that information you can attach to a reply.

------------------------------
- Paul (technogeezer)
------------------------------

Attachment(s)

collect_info.sh.zip   1 KB 1 version

[Ivan Chang]

The client accidentally disconnected the external drive array for a few seconds while the VMs were live. Can you also recommend best practices for backing up a virtual machine.

------------------------------
Ivan Chang
------------------------------

Attachment(s)

fusion_data.202502200856.zip   1.56 MB 1 version

[Technogeezer]

The client accidentally disconnected the external drive array for a few seconds while the VMs were live

Ouch. That's not good news. Especially when snapshots are in play. It's likely you have a corrupt virtual disk, which may have been made worse by you trying to collapse snapshots on a corrupt VMDK.

Here's what I'd do next:

• Shut down Fusion and make a copy of this VM to another folder of disk using the Finder. You need to preserve its current state if you have any hope of preserving any data on it.  This would have been what I would have done the minute that the issue was reported to me before trying any other kind of repair.
• Turn off AutoProtect on the original VM. Now. Leave it off. Never use it again. 
• Try to make a clone of the VM using the Fusion UI. If it succeeds, see if you can power up the VM - don't worry if it needs re-activation, we're trying to see if there's any chance of this virtual disk being intact.
• If the cloned VM doesn't power up or fails to clone, collect information about the virtual disk structure using the script attached to this message. Unzip is, then run the following in the Terminal:
  
  ./Get-VmdkHeader.sh  "/Volumes/ThunderBay/Virtual Machines/Windows Server 2012 R2.vmwarevm"

  This will collect virtual disk metadata that can be used to see if there's something else corrupted in your VM. We might be able to get the attention of one of our community members that's very versed in detecting and possibly repairing virtual disk corruption errors. Even if that works and the disk is repaired, you should plan on backing up data in the VM from within the VM, rebuilding it, and restoring your data.  Running on a repaired virtual disk is asking for problems later on.

As far as best practices for backing up a virtual machine:

• Do not use Time Machine. It's not a great fit for virtual machine backups and it does not have a good track record for being able to recover virtual machines. 
• One method to back up a VM is to make a copy of the Virtual Machine using the Finder or some other disk file copy utility when the virtual machine is shut down. Don't try this when the VM is running - it will almost certainly give you a corrupted backup.
• Another method is to treat the virtual machine like it was a physical machine and run a utility from within the VM that backs up the data to another destination (a disk share, for example).
• You also might want to investigate a backup utility called Vimalin (www.vimalin.com) that was written by one of our community members. It's more aware of virtual machines and may be able to back up an unencrypted VM while it's running. It has a free 60 day license where you can try out all of its features. After the free trial expires, it runs in reduced functionality mode (no scheduled backups, manual backups only) after that unless you purchase a license.

------------------------------
- Paul (technogeezer)
------------------------------

Attachment(s)

Get-VmdkHeader.zip   3 KB 1 version

[Ivan Chang]

I followed your instructions regarding the copy of the original vm but was not able to make a clone because of the following message-You can not make a clone of a virtual machine or snapshot the is powered on or suspended. I have attached the fusion_data and vmdk_metadata zip files that you requested for your review.

------------------------------
Ivan Chang
------------------------------

Attachment(s)

fusion_data.202502202312.zip   834 KB 1 version

Vmdk-Metadata-20250220-233445.zip   11.53 MB 1 version

[Technogeezer]

Interesting. You have said that the VM is powered down, yet the clone operation is failing because Fusion thinks that the VM is powered on or suspended. 

In addition to what André is recommending, can you see what vmware-related processes are running on the system using the following in a Terminal while Fusion is shut down::

ps -ef | grep -i vmw

I'd like to see if there's an orphaned vmware-vmx process.

------------------------------
- Paul (technogeezer)
------------------------------

[Ivan Chang]

The vm is currently in a powered-down state.

------------------------------
Ivan Chang
------------------------------

[a_p_]

I don't see any grain table errors in the .vmdk files, but there are some things which need to be changed and/or explained.
Just the "Unclean Shutdown" bit is set for some of the "000001" snapshot files.

First thing - as mentioned before - disable Auto Protect! This will create new snapshots every time you use the VM, and therefore make it extremely difficult to troubleshoot/fix the issue.

Next, please explain what steps you've already tried. It seems that you reverted the VM manually to a previous snapshot, which pushed the "000001" snapshot out of the active snapshot chain.

The files that you provided are not consistent, i.e. it looks like VMware Fusion had not been closed when you ran the scripts.

Please provide the information I asked for, disable Auto Protect, close VMware Fusion, then provide a new set of files (data collection as well as metadata).

André

[Ivan Chang]

Hi Andre,

We disabled Auto-Protect, closed Vmware Fusion and reran the Fusion data and Vmdk Metadata scripts as you requested.

After the disconnect of the drive array which the vm resided on, the vm was in a suspended state and the client tried to restore to a previous snapshot.

I was mistaken and the vm was not in a powered off state.

------------------------------
Ivan Chang
------------------------------

Attachment(s)

fusion_data.202502210858.zip   832 KB 1 version

Vmdk-Metadata-20250221-090043.zip   11.53 MB 1 version

[a_p_]

After viewing and testing the files that you've provided, I'd suggest that you delete all the files/folders with a .lck extension, to see whether this already solves the issue.

As you can see in the attached file, just some of the ...-000001-... have errors (have not been closed properly - thus "Unclean shutdown"). That's the snapshot that is not in the current chain anymore, due to the restored snapshot that you've mentioned.

André

Attachment(s)

Check-GrainTable.zip   7 KB 1 version

[Ivan Chang]

After following the below instructions from @Technogeezer the VM started up and appears to working fine.

Force the power-off of the VM by:

Highlighting the VM in the Virtual Machine Library.

Clicking on Virtual Machine in the menu bar. There should be a "Resume" item - but don't click on it.

Hold the Option key, and the greyed-out Shut Down menu will become Power Off. Click on Power Off. 

The VM should now be in a powered-off state (similar to what you'd see in a crashed physical machine). You can try to power on the VM at this point and see if things change.

Andre, I would like to thank you for your time and help in reviewing this VM issue.

------------------------------
Ivan Chang
------------------------------

[Technogeezer]

That's good that you have it working. I'd make sure that Autoprotect is turned off and that no snapshots are active.

I also recommend that you have a discussion with the business (sooner rather than later) on a backup strategy for those VMs that meets their business needs. This time they were lucky.  There are plenty of common data loss scenarios that what they're doing today (relying on VM snapshots from what I can see) isn't able to cover. As an example, you said that that they didn't have another copy of the VM -- what if something happened where that VM couldn't be repaired. What would they do?

------------------------------
- Paul (technogeezer)
------------------------------

[Ivan Chang]

Yes, they did run the virtual machine cleanup from the Fusion GUI.

Attached, are the fusion_data and vmdk_metadata files.

------------------------------
Ivan Chang
------------------------------

Attachment(s)

fusion_data.202502260135.zip   201 KB 1 version

Vmdk-Metadata-20250226-013944.zip   12.00 MB 1 version

[Technogeezer]

Ivan, I'm still seeing things I'm not expecting from the data you've provided.. Your VM is indicating that it's running on the 000006 variant of the virtual disk. Tracing the information reveals the VM still has active snapshot chains. I do think that Vimalin will take snapshots to back up a virtual machine, but I would think the snapshots it uses would be deleted once the backup completes.  (I'm checking with Vimalin's author to confirm this).

There are also remnants of auto-protect settings remaining in the VM's .vmx file.

Can you right click on the virtual machine in the Virtual Machine Library and select "Snapshots..." and see if there are any snapshots listed?  

------------------------------
- Paul (technogeezer)
vExpert 2025
------------------------------

[Ivan Chang]

Paul, yes it is showing Snapshots listed even though Auto-Protect is turned off.

------------------------------
Ivan Chang
------------------------------