Problem with Windows Server 2019 snapshot quiescing

[owiking]

Hi

We are running vSphere 6.7.0.2 and ESXi 6.7.0 11675023 in our environment.

We begun to test with Windows Server 2019. We have made a template an deployed a few test machines.

Everythings seams to be working as it should except that we have problem taking snapshots where we want to quiesce the guest file system.

We are running VMware Tools 10.3.5.10430147 on the servers.

We get the following errors:

Warning message on XXXX on XXXXXXX in vDatacenter: The guest OS has reported an error during quiescing. The error code was: 5 The error message was: 'VssSyncStart' operation failed: IDispatch error #8454 (0x80042306)

Volume Shadow Copy Service error: Unexpected error calling routine IOCTL_DISK_SET_SNAPSHOT_INFO(\\.\PHYSICALDRIVE1) fails with winerror 1168.  hr = 0x80070490, Element not found.

Running VSSADMIN LIST WRITERS show the following problem:

Writer name: 'System Writer'

Writer name: 'ASR Writer'

Writer name: 'WMI Writer'

Writer name: 'Registry Writer'

Writer name: 'COM+ REGDB Writer'

Writer name: 'Shadow Copy Optimization Writer'

All with the same state and error:

   State: [7] Failed

   Last error: Timed out

No errors in VSSADMIN before trying to take a snapshot.

Anyone else with this problem?

Help appreciated.

/Olof

[MARTECIT]

Im having the same problem..

VMware has a KB on it.

VMware Knowledge Base

[IvarHome]

Interesting. Seems 2019 is not very usable server to use. I understand in this case VSS is activated by vmware tools, but interesting how about some other backup software, example Veeam installs some temporary agent that activates VSS. Or at all some non-vmware backup system, like BackupExec or Acronis.

[MARTECIT]

Yep, even IBM tivoli has the problem. :smileysad:

[DANAND84]

Virtual Disk service is started and startup type is Automatic.
* VMware snapshot provider service is stopped and disabled.
* that VMware Tools services are running.
*Ensure that Volume Shadow Copy service start up type is Automatic

[SDZ1470]

Good evening,

Thank you for your information.
DANAND84: So with this solution, the backups are consistent and can therefore be used?
I finally received confirmation that Acronis does not work with Microsoft W2K19 and VmWare. Strangely enough, this information was given to me just by phone.
To conclude, the problem is not with us, go your way.

Great technical support. I am aware that the problem is not with them but I think it is justified to wait for support from Acronis from VmWare.

Thank you for your comments.

[ptaszol]

Hello,

The problem is with GPT partition table and hidden partition recovery. When removing this partition snapshot with quiescing work correctly. With MBR partition table work correctly:)

Workaround from me create new VM on vSphere 6.5 and move the new machine to vSphere 6.7 and snapshot with quiescing working correctly.

Support waiting for KB from Microsoft because they told - the problem is in MS Windows 2019 server not to vSphere 6.7:)

Regards

[robertrosit]

The problem is with GPT partition table and hidden partition recovery. When removing this partition snapshot with quiescing work correctly

i can confirm this.

set up a new 2019 server vm, snapshotting with quiescing fails.

removed the recovery partition with diskpart, now snapshotting with quiescing works flawlessly.

who is responsible to fix this?

[owiking]

After contact with VMware support, they helped us.

Conflict between VMware´s and Windows snapshot providers. Disabled VMware´s and set Microsoft´s to automatic.

After that it worked.

Thanks for good support from VMware.

[MARTECIT]

Can you please provide a detailed description on how you  disable the VMware snapshot provider and enable Microsofts? :smileyhappy:

[robertrosit]

i have disabled the service "VMware Snapshot Provider" and left "Volume Shadow Copy" untouched (it is set to manual per default). not sure if this is the way to go, but my test run is now ok.

[anschluss]

I tried out the advice (disable VMware snapshot provider service) and now snapshot can be taken without an error message. However, does this not completely disable app quiescing which could leave a backup or replica of a VM in a inconsistent state? How dependable is this workaround if a failover or restore is needed for a Windows Server 2019-based domain controller or SQL server?

[SDZ1470]

Good evening,

I have read the various discussions about the problems encountered with Windows Server 2019 and VmWare 6.7.U2 and the 10.3.10 utilities.

I have the same problems with a newly installed infrastructure! Impossible to make a "VSS" capture of the server while a simple Snapshot is working.

With a Windows Server 2016 and Windows 10 client, this solution working perfectly.

I have noticed these problems with Acronis which also uses this function of VmWare and Microsoft.

Now is the solution proposed in this article supported by VmWare, Microsoft and Acronis to ensure that backups are consistent?

If I understand correctly, I just need to disable the "VMware snapshot provider service", correct ?

Has anyone tested a restoration with this solution?

Is VmWare currently developing a patch for this problem?

Thank you in advance and good luck for the rest.

[MASAMIK123]

I confirmed that Veritas NetBackup and Backup Exec provides their own VSS provider named Veritas VSS Provider.

Veritas VSS provider works well for Windows2019 Guest OS to make snapshot with quiescing.

Veritas (NetBackup and Backup Exec)  may only provides the consistent VADP backup for Windows2019 Guest OS now.

[NickPenner]

We have exactly the same issue as well. Luckily we only deployed 1 2019 server so far so we'll just be holding off until they fix this... if they fix this.

[big_vern]

can someone break this down so we can understand what we are disabling and any caveats, getting confused by quiesce , microsoft vss, vmware vss , vmware snapshot providers etc.

we have 2019 servers, they were created with GPT (as we went went with uefi default bios which means gpt).

I am getting quiesce error when trying to clone a VM,  not sure if this means file system quiesce or application VSS quiesce

If we follow the KB article and do this vss.disableAppQuiescing = true

what do we loose, I assume it will get rid of the error but then I ask what benifit are we losing as its the default option..

Also disabling the service as others have mentioned - is that exactly the same effect as the setting above.

[big_vern]

This gets worse you know, disabling the service, as suggested by VMware, gives a false positive.

before disabling;

if you try and take a  quiesced a snapshot from within VMware it fails with an error - expected

A non  quiesced snapshot works without error - expected

A snapshot with memory works - expected

A clone fails with an error - expected (as it uses snapshots I suspect which default to  quiesce)

then you disable the service "" vmvss"" within windows (Im GUESSING the hack config file does the same thing ?)

you then specifically do a  quiesced snapshot,, hey presto it works,, no errors or warnings that there was no  quiescing at all..??

[ewy]

Can some from VMware clarify this (non-consistent backups when VMware Snapshot Provided disabled). We have several non-2019 servers too where this "fix" has been applied and the assumption is that the snapshots are app consistent as long as the app "can be quiesced", not that we were removing the possibility of it being quiesced and being fooled into thinking all is good.

Also where in this architecture if so would this service sit.

Thanks

[giox]

Same problem happened to me: windows server 2019, GPT disk with recovery partition (hidden), ESXi 6.7.

Taking a quiesced snapshot of windows server 2019 with ESXi failed with
EventID: 12363
An expected hidden volume arrival did not complete because this LUN was not detected.

My workaround was:

• Boot the VM with gparted live ISO x64
• Remove diag and hidden flag from the recovery partition (it was the 1st partition). msftdata flag will automatically be set.
• Boot on windows, open disk manager and remove drive letter

[johnw230873]

We had the same issue, fix for us was to convert the Main Drive to a Dynamic Disk.

This allowed us to keep EFI and GPT.

[SDZ1470]

Good morning,

It seems that the problem has now been corrected by Microsoft. I've done several tests and everything works for me. Does anything confirm this?

https://kb.vmware.com/s/article/60395 = we will need KB4534321

Greetings to all and good luck for the future.

[J805]

I am still having the same issue. Running Windows Server 2019 Standard 1809 Build 17763.2029, Esxi 6.7 17700523, vm Tools version:11333 (Current). I cannot take quiesced snapshots. Once I disable the "VMware Snapshot Provider" service, I can take snapshots fine. I opened a case with vmware support and they said this is still an ongoing issue between them and MS. It has not been fixed yet I guess?

 

 

[Ronald_Rijerker]

After some brilliant thinking of an collegue of mine we were be able to solve this issue for fresh installed severs.

So: 
If you install Windows Server 2019 - 1809 (OS Version: 10.0.17763.107) you have the issue. Even if you update the OS with the latest Cummulatives, security patches, etc.

If you install Windows Server 2019 - 1809.18 (OS Version: 10.0.17763.2114) the issue has been solved. (I think also earlier versions, but you need to have an version with those patches installed allready by Microsoft)

If someone bring up this issue to Microsoft they are able to fix it.

We tested these two ISO's:

SW_DVD9_Win_Server_STD_CORE_2019_1809.10_64Bit_English_DC_STD_MLF_X22-47318
SW_DVD9_Win_Server_STD_CORE_2019_1809.18_64Bit_English_DC_STD_MLF_X22-74330

Both working.

[Piotr12345]

I have Windows Server 2019 17763.3406 with newest updates, and VMware 6.7U2 and VMware Tools 12320 and error still persist:

Volume Shadow Copy Service error: Unexpected error calling routine IOCTL_DISK_SET_SNAPSHOT_INFO(\\.\PHYSICALDRIVE3) fails with winerror 1168. hr = 0x80070490, Element not found.