Unable to take quiesced snapshot

[Shoomy]

After a reboot of a server, it can happen that it is not possible to take a quiesced snapshot.It happened after an update to 6U1 from 6.0. Before the update there were no problems.

We did the following to troubleshoot the problem.

* Reinstall VMWare Tools

* Check VSS Providers for errors (no errors)

What we found out was that it can happen after a reboot of the server and only if the service "VMware Snapshot Provider" is running (service is set to manual). To correct it we need to stop "VMware Snapshot Provider" and "VMware Tools". If all is stable, ie no running services, which should not run, like "Volume Shadow Copy" we start the service "VMware Tools". After this we can take a quiesced snapshot again.

While this ain't a huge problem, it is annoying at best. Any one have a solution or have the same problem.

[Davoud Teimouri]

Hi,

IF your VM's OS is Windows 2008 R2, please read the below KB and do the mentioned instruction:

VMware KB: Cannot take a quiesced snapshot of Windows 2008 R2 virtual machine

[Shoomy]

Thanks for your reply and I have read this article several times when in search for a solution.

Unfortunately this is not the solution as our backup relies on application awareness (SQL, Exchange etc.). Disabling this feature will have impact on our backup process.

All our servers (w2k8 and w2k12) have the same problem. After a reboot, the service "vmware service provider" keeps running and wont stop (not always but it can happen), which blocks quiesced snapshot.

[Odin_13]

Hi Shoomy,

I can confirm your observations.

We also experience the issue with some of our Windows 2012 servers and stopping the 'VMware Snapshot Provider' service resolves the issue.

After stopping the service it is possible again to create a quiesced snapshot of the server.

Thanks!

[hussainbte]

what is the startup type for the service ? Manual or Automatic?

I believe it should be manual.

[Odin_13]

Startup for the 'VMware Snapshot Provider' service is indeed set to manual.

Strange thing is that the issue is completely random and so far only a couple of servers (all Win2K12) are affected.

[Shoomy]

What happens is that after a reboot VMware tools service starts and triggers "VMware Snapshot Provider" and "Volume Shadow Copy" after a while, few minutes at the most, both services should stop running. Both service are on manual as VMware best practice. Exactly the same behaviour as when taking an quiesced snapshot.

In some cases "VMware snapshot provider" just does not stop and this only happens after a reboot.

Thanks Odin for confirming. May I ask you which version of VMware you are running. We are on 6.0U1 (10240) and yes it is completely random. I have it on w2k8 and w2k12, not only on w2k12.

[Odin_13]

Hi Shoomy,

So far i have only encountered the issue with a couple of Win2K12 servers.

Still trying to figure out why this occurs...

Reinstall of the VMware Tools does not solve the issue as you already stated in an earlier post.

Build  : Releasebuild-3380124

Patch  : 26

Update  : 1

Version : 6.0.0

VMware Tools: 10240

[a11112f]

Hey i'm having the same problem with Server 2012 R2 backups (Veeam 9 Update 1, using VeeamZIP full method). I think the problem started around ESXi 6 Update 1 or Update 1b, I'm running ESXi 6 Update 2 now.

I have reinstalled VMWare tools (with setup /C), and upgraded to ESXI 6u2 but it didn't help.

I get this in the vSphere client:

An error occurred while taking a snapshot: msg.snapshot.error-QUIESCINGERROR.

An error occurred while saving the snapshot: msg.snapshot.error-QUIESCINGERROR.

or

A timeout occurred while communicating with VMware Tools in the virtual machine.

The only error I get on the Windows guest is:

Event ID 7011 Service control manager

A timeout (30000 milliseconds) was reached while waiting for a transaction response from the VMTools service.

No VSS or any other errors.

As of now the only fix I have is to stop the VMWare Snapshot Provider service before I run a backup.  This seems to work 100% for me.

The VMWare Snapshot Provider seems to starts and stops with the VMware tools service, it is set to manual (default setting) start but never seems to stop on it's own.

[bwernli]

@a11112f

Same Problem here with ESXi 5.5 U3b!! We have >1000 VM's and a day certainly 2-3 VM's with this problem :smileyconfused:

[a11112f]

I run a Task Scheduled Powershell script targeted at Veeam for my backups so I was able to add some lines to stop the VMware Snapshot Provider service when the backup starts.  For the first time my backups all completed.

$ServiceStat = Get-Service -Name vmvss -ComputerName $VMName

if ($ServiceStat.Status -eq "Running"){

stop-service -inputobject $(get-service -ComputerName $VMName -Name vmvss)

}

I was also thinking of using Group Policy with a WMI filter targeted at Server 2012 R2 installs to stop the services, but I have the feeling that having the service stop during a backup would be...bad.  Once that service is stopped though, quiesced snapshots work.

I attached the full Veeam script if anyone wants it, I didn't write the original but I have made a few of my own changes.  There is more info in the script comments as to the author.

[HendersonD]

I am seeing the exact same thing under Win 2008 R2 and Win 2012 R2. We are running ESXi 6.0 U2. The VMWare Snapshot provider service is set to manual but upon reboot of the VM the service starts and does not stop. At backup time a quiesced snapshot cannot be taken. If I go in and stop this service, then a snapshot can be taken. We are using Commvault for backup but will be jumping to Veeam in the near future so I will test the script

[tadsmith]

We are seeing the same thing on all version of the 10.x tools, even the 10.0.8 version. They say this will be resolved in 10.0.9 released in June, but I am pushing for a faster resolution. I'm also trying to get them to give me some details on what exact tools / OS versions are affected and what is causing the issue.

[Bleeder]

The 10.0.9 tools were released last week and the notes say this issue should be fixed.

VMware Tools 10.0.9 Release Notes

[diegoazevedo]

Can anyone confirm whether this is fixed with VMware Tools 10.0.9?

[Agevis]

We have the same problem here trying to take backup using Acronis.

Several 2012 R2 server with no problem except for just one SQL Server.

The problem started suddenly maybe after a windows update (not sure) but there are other SQL Server which get updated every week.

Disabling vmware snapshot provider service works for us but don't know if there are any drawbacks.

Updated to tools v10.0.9 build 3917699 did NOT resolve the issue.

Tried also to update ESX host to last patch 6.0.0 4192238 and did NOT resolve the issue.

Problem arise only when take snapshot quiesce=yes memory=no with vmware snapshot provider service not set to disabled.

[Supi_du]

I can confirm this for latest ESXi 5.5 4345813 and Server 2003 32bit vm. When i take Snapshot, i get : msg.snapshot.error-QUIESCINGERROR.

But is worked before for some time. And also 2 other 2003 VM's dont show this Problem?! Downgrade to Tools 9.4.15 helped.

[HendersonD]

I have 41 Windows VMs running under ESXI 6 Update 2

Yesterday I applied 2 Windows updates to all of them, they all had to restart

Last night my backup ran and it shows this morning that for 13 of the VMs a quiesced snapshot could not be taken

I restarted all 13 VMs this morning

I logged into each one and found the VMware Snapshot Provider service still running on 6 of them even though it is set for manual. I stopped this service on all 6

My backup should run fine tonight

The big fix for this cannot come soon enough

[Vitaly Tsipris]

Do you by chance run VMTurbo?

[tomsie1000]

Having the same issue here with 5.5 and 6U2 - we have several environments that share a base template, and since the tools have moved to 10.0.x we've been having this issue across all of them. We also use NetBackup which uses the snapshot functionality, so this meant no backups for us too.

I've found two ways around it:

1. Remove the shadow copy (VSS driver) from the VMware Tools. Bare in mind that you are removing the quiescing abilities so choose this option carefully where it works in tandem with the application and how it uses the filesystem.

2. Restart the services just after startup. I've implemented a script via gpo to roll to any affected machines - not ideal but this way it'll be easy to pull it back out later if/when the bug is fixed.

Here's the code if it helps anyone out:

sleep 60 #give the system time to start the services - may need tweaking for slower systems

if ((Get-Service -Name vmvss).Status -eq 'Running'){stop-service -inputobject $(get-service -Name vmvss)} #stop the VMware Snapshot Provider service

if ((Get-Service -Name VMTools).Status -eq 'Running'){stop-service -inputobject $(get-service -Name VMTools)} #stop the VMware Tools service

Sleep 10 #wait for the services to stop - may need tweaking but this works for us - i was too lazy to write a check loop :smileyhappy:

Start-Service -InputObject (Get-Service -name VMTools) #start VMware Tools service

[hadmi]

Hi guys,

This is the solution

You must rename both files and everything working.

C:\Windows\<pre-freeze-script.bat>

C:\Windows\<post-thaw-script.bat>

:-)

[linnik]

We are seeing this to for 6U2 for Windows 2012 R2 and Windows 2008 VM's.
We are using VSC 6 as a backupsolution.

In the Windows VM's eventviewer we can see: A timeout (3000 miliseconds) was reached while waiting for a transacation response from the VMTools service).

So when it trying to quiece the Windows Guest it stops the VMwareTools service and the quiceing fails.

VSC /SMVI backuplogs does not give anything.

One thing that are kind of weird is that it's not for all VM's. It is just for some.
But the problem seems like the one we discussing here.


Have there been any updates regarding this from VMware?

[andrew_mcse]

same here

veeam 9 ur 2

vmware 6.0.0. 382559

I found random behaviour amongst guests (server 2012 R2) with this

I found the vmware snapshot provider running (set to manual) after reboot

I stopped the service

I re-ran the veeam job and it was successful

[jo_boven]

I had this issue with vmware tools 10.1.x. I opened a case with vmware and this is a know issue. See Quiesced snapshots of Windows 2012/2012R2 VMs with Tools 10.1.0 fails (2149640) | VMware KB  for the workarround.

[matthewmagbee]

I was dealing with similar issues with Commvault and VMware snapshots for server 2012r2 on 6.0u2 . Stopping the vmware service prior to backup resolved  the issue. I have a pre-backup script that runs in commvault to resolve this issue.

Another solution if youre using commvault , is to use Intellesnap. This takes the snap a the block level and the Guest is never touched.

[Rubeck]

"Another solution if youre using commvault , is to use Intellesnap. This takes the snap a the block level and the Guest is never touched."

I do not quite agree.... Even when using IntelliSnap a VM snap is still performed prior to the hardware snap... You can ofc still choose to quiesce it or not. 

/Rubeck

[bahsig]

the only way i could fix this on windows 2012 r2 was to disable quiesced snapshots as per KB2146204

[nparas5]

@Shoomy

I have faced the same issue in my Virtual Infra and worked with VMware Technical support the same, below is what they suggests and it resolved the issue:

Make Sure To Follow All the steps:

To resolve this issue, start the Windows Virtual Disk service:

1. Log in to the Windows operating system as an Administrator.
2. Click Start, type services.msc, and press Enter.
3. Right-click the Virtual Disk service and click Start.

If this does not resolve the issue, disable VMware snapshot provider in the affected virtual machine:

1. Disable the VMware Snapshot provider service in the guest operating system.
2. Restart the Volume shadow copy service.

“msg.snapshot.error-QUIESCING-ERROR” in vCenter Server (2069952) | VMware KB

[Shoomy]

Hi nparas5,

We solved the issue by stopping the services (VMware snapshot provider and volume shadow copy) after a reboot.

The problem seems to be resolved after update to VMware 6.5U1 though.

Hope this helps anyone. Thanks all for your reply.

[IBFsupport]

Hi,

What if you are on a vmware 6.0 Update 3 and the issue is not fixed yet ?

[SupreetK]

Hello IBFsupport,

As a first step, upgrade the VMware Tools version to 10.2.5. Below is the download link -

Download VMware vSphere​

If the above does not help, kindly follow the below steps -

- To resolve this issue, start the Windows Virtual Disk service:

1) Log in to the Windows operating system as an Administrator.

2) Click Start, type services.msc, and press Enter.

3) Right-click the Virtual Disk service and click Start.

- If this does not resolve the issue, disable VMware snapshot provider in the affected virtual machine:

1) Disable the VMware Snapshot provider service in the guest operating system.

2) Restart the Volume shadow copy service.

Regards,

Supreet

[erives]

ESXi 6.7U1

vmtools 10.3.10

This is happening to a few of our 2016 MS servers and the only option that seems to allow a quiesced backup is disabling the VMware Snapshot Provider service.

I have read that the best practice is to set this VMware service to manual but that causes the quiesced snapshot to fail every time.

Is there an issue with disabling this service.

Thanks!

[criszapanta01]

Hi Shoomy​,

I got the same error while creating a clone VM. I tried creating a new snapshot of the VM and this error will be fixed. I'm using vSphere 6.0.0 btw.

Regards,

Cris