Quiesce guest file system fails on 2016 DC

[rroxxor]

Server 2016, latest OS updates

ESXi 6.7.0 Update 1 (Build 10764712)

VMWare Tools 10.3.2 build 9925305

when i run a snapshot with the "Quiesce guest file system" option, the snapshot results in "Failed - An error occurred while quiescing the virtual machine. See the virtual machine's event log for details." (unless i reboot the server first, see below).

in the application log of the server i see nothing helpful. shadow copy is duplicating discs, the lsass service announcing that it's database moved to a new location and is recovering.

DFSRs (2180) \\.\C:\System Volume Information\DFSR\database_E0D0_742D_D074_BD2\dfsr.db: The database engine stopped the instance (0).

lsass (580) Shadow copy instance 7 freeze started.

svchost (2156) Shadow copy instance 7 freeze started.

DFSRs (2180) \\.\C:\System Volume Information\DFSR\database_E0D0_742D_D074_BD2\dfsr.db: The database engine (10.00.14393.0000) is starting a new instance (0).

lsass (580) The database engine is initiating recovery steps.

lsass (580) A database location change was detected from 'C:\Windows\NTDS\ntds.dit' to '\\?\Volume{0943f5ad-14de-11e9-8b96-000c2998569c}\Windows\NTDS\ntds.dit'.

system log shows nothing helpful either. volumes are mounted, and are healthy. what is strange is that one volume gets mounted 12 times (in this case, volume 40)

Volume \\?\Volume{0943f5aa-14de-11e9-8b96-000c2998569c} (\Device\HarddiskVolume36) is healthy.  No action is needed.

Volume \\?\Volume{0943f5ad-14de-11e9-8b96-000c2998569c} (\Device\HarddiskVolume40) is healthy.  No action is needed.

Volume \\?\Volume{0943f5ad-14de-11e9-8b96-000c2998569c} (\Device\HarddiskVolume40) is healthy.  No action is needed.

.... 12 times in total...

Volume \\?\Volume{0943f5ad-14de-11e9-8b96-000c2998569c} (\Device\HarddiskVolume40) is healthy.  No action is needed.

then the message "disk 1 has been surprise removed" but according to a google search this is nothing to worry about. message appears on other vms too, where q-snapshots work fine, so i guess it can be ignored.

the dfsr service gets stuck and doesn't recover, meaning no more sysvol replication between this DC and it's peer. i can create files in sysvol on the partner DC and these files get synced to me, but creating files in sysvol on the affected DC don't get synced back. only a DFSR service restart fixes this.

when i restart the DC, i can do one snapshot with Quiesce option and it works fine, but trying a second time it results in failure again.

apart from searching the net for the entire day, i have tried restarting several services, like the vmware tools service, the VSS service, the DFSR service, but i can not reliably make it work/fail afterwards, and i have so far not the slightest clue why this happens. only a full server reboot can reliably make it work again once.

the only "solution" i found so far was to uninstall the vmware vss helper

this "solved" the problem for good but i am not confident i should leave it like this?

the server is quite a fresh install (5 month), with a fresh domain on it, there is no other software installed on it and it has not been "played with" configuration wise, so this is the most basic OS+AD+Vmwaretools install you can get.

any ideas, anyone?

[daphnissov]

There is a later version of VMware tools (10.3.5) which comes with the most recent built of ESXi. I'd recommend updating so you can get that package, update tools, then try it again. So, for what it's worth, the VSS provider inside VMware Tools has been notoriously flaky for as long as it has existed. If you're using Veeam, it's recommended that you use application-aware image processing instead (which uses their proprietary at-runtime agent injected into the guest). If you're not then, well, good luck :smileyhappy:

[rroxxor]

hy,

yes we're using veeam and that's the reason i even started investigating this.

so you'd recommend me to disable the quiesce option in veeam at all?

br

[daphnissov]

No, I'm saying if you're using Veeam, you shouldn't be using the VMware Tools quiescence option. You should be using application-aware image processing instead which does ​not​ use VMware tools for that.

[rroxxor]

yes, gotcha,

Storage - Advanced Settings - VSphere - Enable VMWare Tools quiescence - OFF

Guest Processing - Enable application-aware processing - ON

however this requires guest os credentials which i was happy to go along without. i'll think about it. thanks!

[daphnissov]

Yes, it requires in-guest credentials because it injects and registers an agent at runtime. That's the only reliable way you're going to get a good and consistent backup of Windows applications without using the shoddy VMware Tools driver.

[rroxxor]

for reference, 10.3.5 vmwaretools do not solve the issue. thanks for the tip though.