Windows 2008R2 snapshot problem on 4.1

[AntonVZhbankov]

I have problem with snapshots on one of my VMs. And what's strange it started only last friday.

One of my DC's on Win2008R2 refused to take quiesced snapshot after clod migration from one cluster to another. Remaining DC, absolutely the same configuration still can take snapshots.

I've made new VM, built from nothing, clean installation, promoted to DC instead of failed on and problem still persists.

What I've found in Windows logs:

1) Volume Shadow Copy Service error: Unexpected error DeviceIoControl(
?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - 000000000000042C,0x00560000,0000000000000000,0,00000000003C7F70,4096,[0]). hr = 0x80070001, Incorrect function.

.

Operation:

Exposing Recovered Volumes

Locating shadow-copy LUNs

PostSnapshot Event

Executing Asynchronous Operation

Context:

Device:
?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}

Examining Detected Volume: Existing -
?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}

Execution Context: Provider

Provider Name: VMware Snapshot Provider

Provider Version: 1.0.0

Provider ID: {564d7761-7265-2056-5353-2050726f7669}

Current State: DoSnapshotSet

2) lsass (460) An attempt to write to the file "
?\Volume{b6c3b2c0-a388-11df-badc-0050568b0019}\Windows\NTDS\edb.log" at offset 7525888 (0x000000000072d600) for 512 (0x00000200) bytes failed after 0 seconds with system error 19 (0x00000013): "The media is write protected. ". The write operation will fail with error -1032 (0xfffffbf8). If this error persists then the file may be damaged and may need to be restored from a previous backup.

3) lsass (460) Unable to write to logfile
?\Volume{b6c3b2c0-a388-11df-badc-0050568b0019}\Windows\NTDS\edb.log. Error -1032 (0xfffffbf8).

---

MCSA, MCTS Hyper-V, VCP 3/4, VMware vExpert

http://blog.vadmin.ru

[kswail]

Hi there,

I'm having the same problem here only with my 2008R2 box with the ADDS role. I get the same errors as you in my event log.

What's even stranger is that all of this was working fine a week ago, when my VMs were on local DAS. I've since SVMotioned my VMs to an NFS store, and my problem seems to have coincided with this relocation. I haven't tried installing a new DC as you have though. The SVMotion worked fine for the DC and a myriad of other Windows 2008 R2 VMs, and all of them backup with a snapshot quiesce just fine. Only the DC is being a b#tch.

After the snapshot failure, if you run "vssadmin list writers" from your DC you should see this for the NTDS writer:

Writer name: 'NTDS'
Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
Writer Instance Id: {14cf4f25-2ae0-467e-9bcd-de24a5e00ef2}
State: Failed
Last error: Non-retryable error

I can rule out vss as the cultprit since I can use Windows Server Backup to get a full system state backup, and when I run "vssadmin list providers" after that there are no errors reported. I can also confirm I've taken the DC offline to run consistenty checks and all is golden in terms of the AD databases.

I've tried literally everything to resolve this, uninstall, clean registry, reinstall, ensure the VMware snapshot provider is registered correctly... all for nothing.

I've also read another chap is having problems with snapping his

vCenter server due to ADLDS (ADAM). Clearly it has something to do with

an incompatibility with VMware vss and the ADDS VSS Writer.

http://communities.vmware.com/thread/277587

The only fix for now is to not quiesce when you take your snaps, and so I'm using Windows Server Backup to backup system state to a local vmdk which get backed up along with the VM.

Anyone else seeing this or know of a fix?

Cheers.

kswail

[KEPete]

The only way I've gotten quiesced snaps to work was to reinstall vmtools while unchecking the VMWare VSS driver option. After that, all snap based backups are working fine. This issue only seems to be affecting Windows 2008 running AD or AD LDS roles.

[kswail]

KEpete, be careful since my undestanding is that by not installing the VSS provider, even though you select to quiese during the snapshot doesn't mean that it is actually doing it. In fact the disk is being snapped, but the application like ADDS or ADLDS are not being quiesced and you could have a problem if you even need to restore from a backup.

[KEPete]

Ugh, I guess I thought it was working since a manual snapshot using quiesce was working fine. Or at least the event message indicated success. I'll take a look at my backups and see what's there. Thanks.

[RobMokkink]

We also got the same issue.

Really anoying problem.

[KEPete]

Well, I got around to checking my backups and found that it did include the ntds.dit file with size and time stamp correct. However, I could not mount it using dsamain to look at the content. I'm using Windows backup for system state now but still curious about the file that did get backed up using VDR. Any other ways I can check validity of that backup? Thanks.

[AntonVZhbankov]

kswail, I've contacted VMware support on this issue.

---

MCSA, MCTS Hyper-V, VCP 3/4, VMware vExpert

http://blog.vadmin.ru

[kswail]

Thanks Anton, please share with us what you find! Much appreciated!

kswail

[Rob_Bohmann1]

Found this post : ** I have not tested this to confirm if this is the solution to the problem described in thsi thread **

Here is the relevant part:

" Application consistent quiescing of Windows 2008 virtual machines is only available when those virtual machines are created in vSphere 4.1.

Virtual machines created in vSphere 4.0 can be updated to enable application consistent quiescing by modifying a virtual machine’s enableUUID attribute. "

Enable Windows 2008 Virtual Machine Application Consistent Quiescing

1 Start the vSphere Client, and log in to a vCenter Server.

2 Select Virtual Machines and Templates and click the Virtual Machines tab.

3 Right‐click the Windows 2008 virtual machine for which you are enabling the disk UUID attribute, and select Power > Power Off. The virtual machine

powers off.

4 Right‐click the virtual machine, and click Edit Settings.

5Click the Options tab, and select the General entry in the settings column.

6Click Configuration Parameters… The Configuration Paramters window appears.

7Click Add Row.

8 In the Name column, enter disk.EnableUUID.

9In the Value column, enter TRUE.

10 Click OK and click Save.

11 Power on the virtual machine.

if this works hopefully there will be a way to script/cli this and let it take affect on next powercycle op as needed.

Message was edited by: Rob.Bohmann

added disclaimer about not testing. You should of course

[Borja_Mari]

Hi,

as OP pointed, i would recommend you to check this carefully:

http://communities.vmware.com/thread/282199

(Especially page 8 from the "vdr admin" vmware pdf doc)

You can check this too:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007696

http://www.vladan.fr/troubleshoot-virtual-machine-vss-snapshot/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:EsxVirtualization(ESX+Virtualization)

I hope maybe any of this links helps you :smileyhappy:

Regards/Saludos,
Pablo

Please consider awarding

any helpful or corrrect answer. Thanks!! -

Por favor considera premiar

cualquier respuesta útil o correcta. ¡¡Muchas gracias!!

Virtually noob blog

[RobMokkink]

I just did a test: disk.enableuuid = false

I can now do a backup with vranger. I will investigate further.

[RobMokkink]

I created a new template in 4.1, but still the problem occurs. In the VM there are no more VSS error, so that is good news.

But it looks like a serious bug!

[kswail]

Hello all and thanks for the input!

Sadly, in our cases, the problem is not related to disk.EnableUUID = True in our VM configs, which is the essence of all the link provided above.

I have this setting in my VMs, and quiescing works accross the board for all Windows 2008 R2 machines EXCEPT for DCs or ADLDS servers.

Thanks for the tips though!

kswail

[mrjlturner]

Gang,

I'm having the same issues on a newly built 2k8 x64 VM built on a 4.1 host. I am trying to backup vCenter server VM and the ADAM writer keeps failing. I eventually added the disk.EnableUUID=false and was able to get a successful snapshot and backup via VDR. Now, I just need to test a restore to see if it actually backed everything up successfully... If anyone else has any input it'd be appreciated. I hope that VMware fixes this issue!

Thanks!

mrjlturner

[lucmorin]

Hi all,

I can confirm the "quiesced snapshot" problem on a Win 2008 R2 VM where vCenter is installed.

This is a VM I created from scratch using Microsoft ISOs.

All along during install, I'd take quiesced snapshots, and it worked fine. The minute I installed vCenter, then quiesced snapshots stopped working. After reverting back to a previous snapshot, quiesced snapshots were again working fine.

As a side note, at first I had installed a custom instance of MS SQL Standard, so I thought maybe this was causing the problem with vCenter. So I reverted even further back, and let vCenter install its own MS SQL Express instance. This didn't change anything as quiesced snapshots were still failing.

I have 2 other Win 2008 R2 VMs, and I can take quiesced snapshots without a problem.

As others have mentionned, the problem seems to originate in the ADAM writer. if you stop the VMWareVCMSDS service, then the snapshot finishes without error. Please be aware that this service is required for vCenter to properly start at boot time, so I'm not advocating this as a solution, just as a fact to help us pinpoint the cause of this annoying problem.

VMWare, are you reading this ?????

Regards

Luc Morin

[TK22]

Just wanted to tell you all that you´re not alone. We´re also having the same problems.

Trying to snapshot or VDRbackup of a vCenter 4.1 VM on 2008R2. It´s a ew VM on a new vSphere4.1 ESX system.

• Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.

• Device:
  ?\fdc#generic_floppy_drive# errors despite removing floppy

• VSS event errors in VM

We got it all :smileyhappy:

UUID key enabled from default

VCBSnapshotProvider.dll is there

And allt that stuff..

Checked a lot of KBs, VSS troubleshooting etc. but no "real" solution.

As lucmorin writes.... if you stop the VMWareVCMSDS service, then the snapshot or VDRbackup finishes without error.

Anyone got a proper solution for this yet or is it just workarounds?

[MKguy]

Have you also looked into the UUID mismatch problem discussed here in this thread:

http://communities.vmware.com/message/1595294#1595294

Might be it if you have the same problems with the vCenter ADAM instance the VMWareVCMSDS service runs..

[TK22]

As soon as i get the time i´ll look into that UUID mismatch thing.

Our workaround right now is just to disable the VMware Snapshot Provider service in the vCenter VM.

Then we can both snapshot and VDRbackup the vCenter 4.1 server on 2008R2

[KEPete]

disk.EnableUUID = false might make it appear to be quiescing the application state, I believe all that does is force the file system state backups as I had done by removing the VSS Support from VMTools.

See this thread: http://communities.vmware.com/thread/284367;jsessionid=AB53FF4C35DB254C3DCF01F07BDD084D?tstart=0

Now I've also tried disk.EnableUUID=true as the article http://www.blueshiftblog.com/?p=473, http://vknowledge.wordpress.com/2010/08/18/application-consistent-quiescing-and-vdr/, and http://www.sys-con.com/node/1544145 suggests and what was posted above, but that did not work for me either.

So far I have not found anyone getting this to work or at least have a real backup/restore work even with the work arounds.

[lucmorin]

I opened an SR with VMWare, and I was talking with the tech support rep yesterday, and he was convinced that the problem was caused by having SQL server installed on the same machine as vCenter.

I told him this was preposterous, as vCenter install SQL Express by default if you don't have an existing DB.

I obliged him anyway, and I installed my SQL server on another VM, created a new DSN pointing to the remote DB, and installed vCenter on a fresh install of Windows 2008 R2 + VMware tools.

Well, the problem is still there. The VMwareVCMSDS writer is always reporting an error 11 ( 'vssadmin list writers' at DOS prompt), and trying to take quiesced snapshots ends with an error message:

"Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine."

I reported this back to the tech rep, and I'm still waiting. I sure hope they will elevate this because in my opinion, it is unthinkable that I should not be able to snapshot my vCenter VM.

Regards

[pciccone]

Locmorin, we are in the exact same situation at the moment. For me, this is fresh "text book" install of a new vSphere cluster, having this issue "out of the box". let me know what they ultimately tell you as well. Without disabling the whole VSS system, I cannot find a way to exclude the ADAM Writer only despite my many attempts.

I can also confirm that the original tech speculation that SQL on the same box could be the cause is not the case, as our install has a dedicated SQL server, similar to your new install. I can't believe that was even suggested - what does an ADAM writer have to do with SQL?

One more thing, it also affects their VMWare View product line as well, another fresh install server that uses the same VmwareVCMSDS service causes the same VSS ADAM writer problems.

Thanks.

-- Phil

[RobMokkink]

I have sql installed inside vm's and i can backup using vRanger without any probs.

As long as i set enblediskuuid to false (see previous reply)

Can you run a chkdsk inside the vm? This can help solve VSS problems, because VSS problems are also a big issue on physical servers and using backup solutions like MS DPM.

[pciccone]

Thats not really an acceptable solution. What you have essentially done via disabling the UUID is turn off VSS snapshotting completely. VSS won't work with UUID disabled, it's well documented in the vDR documentation. You will notice when UUID is disabled and you make a snapshot check the application event viewer and you will see the VSS routines are never started at all.

[BulletByte]

Hi all,

i installed last night the VMWare Data Recovery VM. All VMs are Windows 2008 R2 and i have 3 different domains with 3 Domain Controllers in my farm (Datacenter). i use also the 4.1 version of vCenter and everything is a fresh install with the latest service packs available.The database is a MS SQL 2008 R2 Express on the same machine as the vCenter.

Ok, here we go:

i can backup everything, also 2 of the 3 domain controllers .BUT the 3rd domain controller has all the failures and error events as described above. I tried everything out, double checked the UUID settings...no success.

Any news from VMWare support on this?

Here some eventview entries:

-

---

Volumeschattenkopie-Dienstfehler: Unerwarteter Fehler "DeviceIoControl(
?\fdc#generic_floppy_drive#6&3b4c39bd&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - 0000000000000498,0x00560000,0000000000000000,0,0000000000416790,4096,[0])". hr = 0x80070001, Unzulässige Funktion.

.

Vorgang:

Wiederhergestellte Volumes werden verfügbar gemacht

Schattenkopie-LUNs werden gesucht

PostSnapshot-Ereignis

Asynchroner Vorgang wird ausgeführt

Kontext:

Gerät:
?\fdc#generic_floppy_drive#6&3b4c39bd&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}

Erkanntes Volume wird überprüft: Existing -
?\fdc#generic_floppy_drive#6&3b4c39bd&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}

Ausführungskontext: Provider

Anbietername: VMware Snapshot Provider

Anbieterversion: 1.0.0

Anbieter-ID: {564d7761-7265-2056-5353-2050726f7669}

Aktueller Status: DoSnapshotSet

Protokollname: Application

Quelle: VSS

Datum: 25.09.2010 14:41:19

Ereignis-ID: 12289

Aufgabenkategorie:Keine

Ebene: Fehler

Schlüsselwörter:Klassisch

Benutzer: Nicht zutreffend

Computer: MYVM

Beschreibung:

Volumeschattenkopie-Dienstfehler: Unerwarteter Fehler "DeviceIoControl(
?\fdc#generic_floppy_drive#6&3b4c39bd&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - 0000000000000498,0x00560000,0000000000000000,0,0000000000416790,4096,[0])". hr = 0x80070001, Unzulässige Funktion.

.

Vorgang:

Wiederhergestellte Volumes werden verfügbar gemacht

Schattenkopie-LUNs werden gesucht

PostSnapshot-Ereignis

Asynchroner Vorgang wird ausgeführt

Kontext:

Gerät:
?\fdc#generic_floppy_drive#6&3b4c39bd&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}

Erkanntes Volume wird überprüft: Existing -
?\fdc#generic_floppy_drive#6&3b4c39bd&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}

Ausführungskontext: Provider

Anbietername: VMware Snapshot Provider

Anbieterversion: 1.0.0

Anbieter-ID: {564d7761-7265-2056-5353-2050726f7669}

Aktueller Status: DoSnapshotSet

Ereignis-XML:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

<System>

<Provider Name="VSS" />

<EventID Qualifiers="0">12289</EventID>

<Level>2</Level>

<Task>0</Task>

<Keywords>0x80000000000000</Keywords>

<TimeCreated SystemTime="2010-09-25T12:41:19.000000000Z" />

<EventRecordID>721</EventRecordID>

<Channel>Application</Channel>

<Computer>MYVM</Computer>

<Security />

</System>

<EventData>

<Data>DeviceIoControl(
?\fdc#generic_floppy_drive#6&3b4c39bd&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - 0000000000000498,0x00560000,0000000000000000,0,0000000000416790,4096,[0])</Data>

<Data>0x80070001, Unzulässige Funktion.

</Data>

<Data>

Vorgang:

Wiederhergestellte Volumes werden verfügbar gemacht

Schattenkopie-LUNs werden gesucht

PostSnapshot-Ereignis

Asynchroner Vorgang wird ausgeführt

Kontext:

Gerät:
?\fdc#generic_floppy_drive#6&3b4c39bd&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}

Erkanntes Volume wird überprüft: Existing -
?\fdc#generic_floppy_drive#6&3b4c39bd&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}

Ausführungskontext: Provider

Anbietername: VMware Snapshot Provider

Anbieterversion: 1.0.0

Anbieter-ID: {564d7761-7265-2056-5353-2050726f7669}

Aktueller Status: DoSnapshotSet</Data>

<Binary>2D20436F64653A20494E434943484C4830303030303532312D2043616C6C3A20434F52485755544330303030303131372D205049443A202030303030323633322D205449443A202030303030303238342D20434D443A2020433A5C57696E646F77735C73797374656D33325C76737376632E6578652020202D20557365723A204E616D653A204E542D4155544F524954C4545C53595354454D2C205349443A532D312D352D313820</Binary>

</EventData>

</Event>

-

---

lsass (468) Versuch, in Datei "
?\Volume{d88f5f65-c8a1-11df-8e28-005056b0001e}\Windows\NTDS\edb.log" bei Offset 9932288 (0x0000000000978e00) für 512 (0x00000200) Bytes zu schreiben, ist nach 0 Sekunden mit Systemfehler 19 (0x00000013): "Das Medium ist schreibgeschützt. " fehlgeschlagen. Fehler -1032 (0xfffffbf8) bei Schreiboperation. Wenn dieser Zustand andauert, ist die Datei möglicherweise beschädigt und muss aus einer vorherigen Sicherung wiederhergestellt werden.

Protokollname: Application

Quelle: ESENT

Datum: 25.09.2010 14:41:20

Ereignis-ID: 482

Aufgabenkategorie:Allgemein

Ebene: Fehler

Schlüsselwörter:Klassisch

Benutzer: Nicht zutreffend

Computer: MYVM

Beschreibung:

lsass (468) Versuch, in Datei "
?\Volume{d88f5f65-c8a1-11df-8e28-005056b0001e}\Windows\NTDS\edb.log" bei Offset 9932288 (0x0000000000978e00) für 512 (0x00000200) Bytes zu schreiben, ist nach 0 Sekunden mit Systemfehler 19 (0x00000013): "Das Medium ist schreibgeschützt. " fehlgeschlagen. Fehler -1032 (0xfffffbf8) bei Schreiboperation. Wenn dieser Zustand andauert, ist die Datei möglicherweise beschädigt und muss aus einer vorherigen Sicherung wiederhergestellt werden.

Ereignis-XML:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

<System>

<Provider Name="ESENT" />

<EventID Qualifiers="0">482</EventID>

<Level>2</Level>

<Task>1</Task>

<Keywords>0x80000000000000</Keywords>

<TimeCreated SystemTime="2010-09-25T12:41:20.000000000Z" />

<EventRecordID>723</EventRecordID>

<Channel>Application</Channel>

<Computer>MYVM</Computer>

<Security />

</System>

<EventData>

<Data>lsass</Data>

<Data>468</Data>

<Data>

</Data>

<Data>
?\Volume{d88f5f65-c8a1-11df-8e28-005056b0001e}\Windows\NTDS\edb.log</Data>

<Data>9932288 (0x0000000000978e00)</Data>

<Data>512 (0x00000200)</Data>

<Data>-1032 (0xfffffbf8)</Data>

<Data>19 (0x00000013)</Data>

<Data>Das Medium ist schreibgeschützt. </Data>

<Data>0</Data>

</EventData>

</Event>

-

---

lsass (468) In Protokolldatei '
?\Volume{d88f5f65-c8a1-11df-8e28-005056b0001e}\Windows\NTDS\edb.log' konnte nicht geschrieben werden. Fehler -1032 (0xfffffbf8).

Protokollname: Application

Quelle: ESENT

Datum: 25.09.2010 14:41:20

Ereignis-ID: 408

Aufgabenkategorie:Protokollierung/Wiederherstellung

Ebene: Fehler

Schlüsselwörter:Klassisch

Benutzer: Nicht zutreffend

Computer: MYVM

Beschreibung:

lsass (468) In Protokolldatei '
?\Volume{d88f5f65-c8a1-11df-8e28-005056b0001e}\Windows\NTDS\edb.log' konnte nicht geschrieben werden. Fehler -1032 (0xfffffbf8).

Ereignis-XML:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

<System>

<Provider Name="ESENT" />

<EventID Qualifiers="0">408</EventID>

<Level>2</Level>

<Task>3</Task>

<Keywords>0x80000000000000</Keywords>

<TimeCreated SystemTime="2010-09-25T12:41:20.000000000Z" />

<EventRecordID>724</EventRecordID>

<Channel>Application</Channel>

<Computer>MYVM</Computer>

<Security />

</System>

<EventData>

<Data>lsass</Data>

<Data>468</Data>

<Data>

</Data>

<Data>
?\Volume{d88f5f65-c8a1-11df-8e28-005056b0001e}\Windows\NTDS\edb.log</Data>

<Data>-1032 (0xfffffbf8)</Data>

</EventData>

</Event>

[BulletByte]

... and also this as the last entry from VSS

-

---

Protokollname: Application

Quelle: VSS

Datum: 25.09.2010 14:41:20

Ereignis-ID: 8229

Aufgabenkategorie:Keine

Ebene: Warnung

Schlüsselwörter:Klassisch

Benutzer: Nicht zutreffend

Computer: MYVM

Beschreibung:

Von einem VSS Writer wurde ein Ereignis mit dem Fehler "0x800423f4, Für den Generator wurde ein nicht vorübergehender Fehler festgestellt. Wenn der Sicherungsvorgang wiederholt wird,

tritt der Fehler höchstwahrscheinlich erneut auf.

" zurückgewiesen. Änderungen, die während der Verarbeitung des Ereignisses an den Writerkomponenten vorgenommen wurden, sind für den Anforderer nicht verfügbar. Zugehörige Ereignisse der Hostanwendung des VSS Writer finden Sie im Ereignisprotokoll.

Vorgang:

PostSnapshot-Ereignis

Kontext:

Ausführungskontext: Writer

Generatorklassen-ID: {b2014c9e-8711-4c5c-a5a9-3cf384484757}

Generatorname: NTDS

Generatorinstanz-ID: {7e4e9f06-19c8-438d-8c4c-e90d7e309f8d}

Befehlszeile: C:\Windows\system32\lsass.exe

Prozess-ID: 468

Ereignis-XML:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

<System>

<Provider Name="VSS" />

<EventID Qualifiers="0">8229</EventID>

<Level>3</Level>

<Task>0</Task>

<Keywords>0x80000000000000</Keywords>

<TimeCreated SystemTime="2010-09-25T12:41:20.000000000Z" />

<EventRecordID>725</EventRecordID>

<Channel>Application</Channel>

<Computer>MYVM</Computer>

<Security />

</System>

<EventData>

<Data>0x800423f4, Für den Generator wurde ein nicht vorübergehender Fehler festgestellt. Wenn der Sicherungsvorgang wiederholt wird,

tritt der Fehler höchstwahrscheinlich erneut auf.

</Data>

<Data>

Vorgang:

PostSnapshot-Ereignis

Kontext:

Ausführungskontext: Writer

Generatorklassen-ID: {b2014c9e-8711-4c5c-a5a9-3cf384484757}

Generatorname: NTDS

Generatorinstanz-ID: {7e4e9f06-19c8-438d-8c4c-e90d7e309f8d}

Befehlszeile: C:\Windows\system32\lsass.exe

Prozess-ID: 468</Data>

<Binary>2D20436F64653A20575254575254494330303030353139372D2043616C6C3A20575254575254494330303030333335362D205049443A202030303030303436382D205449443A202030303030303632342D20434D443A2020433A5C57696E646F77735C73797374656D33325C6C736173732E6578652020202D20557365723A204E616D653A204E542D4155544F524954C4545C53595354454D2C205349443A532D312D352D313820</Binary>

</EventData>

</Event>

[pciccone]

You are correct, I just re-read the thread and it appears we have two different VSS writer problems, one that deals with ADAM and one that deals with NTDS. Both with the same fail reason of "System error 19 (0x00000013): "The media is write protected. ". The write operation will fail with error -1032 (0xfffffbf8)."

[lucmorin]

Actually, these errors are pretty consistent with what the OP reported and with what we're getting here.

It's a bit difficult to pinpoint the real source of the problem, as everytime we try to take a quiesced snapshot, we have about 4-5 log entries that are either warnings or errors.

For example, the "floppy" related error message, and also the "edb" file being protected error, then the VSS warning about a transient problem.

I'll try to post the relevant entries monday when I'm back at work, and the we can compare with others.

Regards

[pciccone]

My apologies, I got side tracked with the ADAM VSS writer and lost scope to the initial report of the NTDS writer. I updated my response above.

[lucmorin]

To the OP and others with same problem:

Discussing the issue with VMWare, we have found that on the VMs that are experiencing this problem we're missing file VCBRequestor.dll in folder:

C:\Program Files\VMware\VMware Tools\Drivers\vss

There should be two dll in this folder, the other one being VCBSnapshotProvider.dll

Could you guys check to see if this file is also missing on your systems ?

It seems that for some reason, VMWare tools doesn't install this file (and maybe others).

Regards

[kswail]

Interesting idea sir!

On my Windows 2008 R2 DC which has the backup problems, I do NOT have a copy of "VCBRequestor.dll" installed.

The only file in that directory is "VCBSnapshotProvider.dll".

What's odd is that NONE of my servers have this file, but all of them work correctly for application aware snapshots. It only fails on the DC.

I wonder if perhaps this file is from older versions of VMware Tools and no longer installed?

cheers,

kswail

[jcwuerfl]

Device:
?\fdc#generic_floppy_drive#

so do you have a Floppy connected to the VM? Have you tried removing that?

[OmniTech]

I have the same problem.

Since it seems that VMWare and Windows 2008R2 is not bugfree, I'll install 2003R2...

[lucmorin]

I have an SR opened with VMWare, and it has been elevated, so I'm hoping they will find the cause AND the cure.

It basically seems to be that some files don't get properly installed/registered on Windows 2008 R2. So far the service reps have not been able to reproduce it on their setup, so it might be something to do with ESXi 4.1 interacting funny on some hardware. Who knows.

I'll post whenever there's progress on this.

[BulletByte]

Device:
?\fdc#generic_floppy_drive#

so do you have a Floppy connected to the VM? Have you tried removing that?

U asking me? No, there was never a floppy drive. I run the Backup creation process again (and therefor the snapshot to copy the backup with Data Recovery) with the same results and the same error messages.

No progress!

[jcwuerfl]

No more directed at @AntonVZhbankov I think but even if its defined and connected on the VM that could cause an issue?

[BulletByte]

Nope, no chance... still the same. It would be nice if we could have ANY statement from VMWare about this issue

[mronsman]

Count me in on this one. Trying to do VDR on vCenter itself as a VM. The DB is remote. Fails on the snapshot every time. Please post if anyone has gotten this to work yet......

Thanx,

Matt

[mrjlturner]

*Duplicate post* my apologies

[mrjlturner]

Matt,

If you read through the thread, you'll find that I had luck w/ backing up the vCenter Server (w/ SQL Express on server) using VDR 1.2 on and ESXi 4.1 host by adding the following: (taken from the thread & modified slightly)

Enable Windows 2008 Virtual Machine Application Consistent Quiescing

1 Start the vSphere Client, and log in to a vCenter Server.

2 Select Virtual Machines and Templates and click the Virtual Machines tab.

3 Right‐click the Windows 2008 virtual machine for which you are enabling the disk UUID attribute, and select Power > Power Off. The virtual machine

powers off.

4 Right‐click the virtual machine, and click Edit Settings.

5Click the Options tab, and select the General entry in the settings column.

6Click Configuration Parameters… The Configuration Paramters window appears.

7Click Add Row.

8 In the Name column, enter disk.EnableUUID.

9In the Value column, enter FALSE.

10 Click OK and click Save.

11 Power on the virtual machine.

By setting the option to 'FALSE' I am able to take successful snapshots and backups.

Thanks!

mrjlturner

[pciccone]

UUID as disabled = crash consistent backups (VSS disabled/bypassed).

Here is an article on this fact: http://vknowledge.wordpress.com/2010/08/18/application-consistent-quiescing-and-vdr/

[mrjlturner]

Pciccone,

Dually noted my friend. I have posted my results simply so that folks may be able to get "some" backups vs. no backups whatsoever against the vCenter server. I am having a hard time believing that VMware has yet to have addressed anyone in this thread. I hope that changes soon...

Thanks again!

mrjlturner

[pciccone]

Ahh, gotcha. I just wanted to make sure you knew - just in case of a restore and you might have missed data and pulled some hair out :smileyhappy:

[mrjlturner]

Pciccone,

I appreciate the looking out for a brother! It's nice to know that at least we as a VMware community have each others' backs... now if we can just get VMware on board with that mentality we should be moving in the right direction!

I found this link while researching this today: http://communities.vmware.com/thread/283944?tstart=60

LucD make a script that will add the disk.EnableUUID key to each virtual machine, which will help us all out when this is finally fixed...

-mrjlturner

[BulletByte]

Your post suggests that this problem is fixed by enabling the UUID but thats NOT the truth!

All my vCenter/ESX 4.1 fresh installed datacenter has on really all machines the UUID enabled but anyway, this is not the solution for all out in the field. There must be something more!

[mronsman]

Indeed that did work changing the UUID to false. Thanks for the help! I understand it does not quiesce the Host OS but something is better than nothing!

I would give you points if this were my original post! :smileyhappy:

Thanx,

Matt

[OmniTech]

This solution is only "good" for VMs with no transactionnal systems. But you could have some corrupted files, beware.

I do hope VMWare will solve this soon enought cause otherwise they should change the Compatibility Matrix for the guest OS...

[kswail]

@Omnitech: You are totally correct, but it seems this problem is only happening on transactional systems thus far (AD and ADLDS to be precise).

@All: Since VMware is not forthcoming on this one and no real solution is in sight (reminder, setting the UUID to false is not a solution, and you don't want to find this out during a DR scenario) I've come up with a simple temp solution.

What I've done for now is to create a new VMDK file for my DC and use the Windows Server Backup to backup to that VMDK. At least now I know that I have a consistent backup as long as I backup the VM outside of the Windows Server Backup backup window.

I hope to see some news on a permanent fix soon.

cheers,

kswail

[MKguy]

2008R2 Snapshot issues with AD/ADAM databases?

Sounds like this issue to me:

http://communities.vmware.com/message/1596547

We were able to solve with the VMware support for our virtual vCenter Server with it's ADAM instance as described in this thread.

[lucmorin]

@all,

Just to keep you posted about the status of my SR with VMWare on this subject.

It seems that this porblem is caused by a bad VMWare tools install, as some files that should be installed aren't. One such example is the VCBRequestor.dll file.

At one point, a VMWare tech rep connected remotely and we managed to copy/register this dll (it's more than simply regsvr32ing it, as he also changed some COM+ settings which I don't remember). After properly registering this dll on the guest OS, quiesced snapshots were successfully taken.

I'm still waiting for a "patch", or at least a detailed step by step instructions list on how to register the missing dll, but it seems that those VMWare guys aren't too quick in returning calls/emails. That's very disapointing, and my customer is really starting to question our decision to go with VMWare.

Regards

[OmniTech]

@lucmorin

Where can you get that specific file?

Where do you copy this file in Windows 2008?

I'll make some test myself.

Thanks

[lucmorin]

@OmniTech,

Like I mentioned, it's not only a matter of copying the file to the guest. It involves registering the dll, but also changing some settings in the COM+ Windows subsystem. I don't have an exact proceudre for you to follow at this time, but I'm on the heels of the VMWare tech reps to provide me one.

As soon as I have it, I'll post it here.

Do I understand that on your Windows 2008 R2 guest you are also missing the VCBRequestor.dll file, and experiencing the same snapshots problems ? Please confirm, as this will give me more weight when discussing with vmware.

Thanks

[OmniTech]

@lucmorin

The wierd thing is, I have searched my Windows 2003 VM and I have not found that file... But Snapshots works fine.

[lucmorin]

@OmniTech,

Is your Win 2003 VM a vCenter server ?

The VCBRequestor.dll file, if I understood correctly what the VMWare people told me, is only used when trying to take a quiesced snapshot on a VM that has an ADLDS role, such as a vCenter server VM.

Also, the fact that snapshots seem to complete normaly doesn't mean that they will be application consistent. Under some circumstances the snapshot will simply "ride through" the abnormal situation, and take a crash consistent snapshot. If I'm not mistaken, one such example is if you install VMWare tools without the VSS driver, and then try to take a quiesced snapshot. It will seem to have succeeded, but the filesystem will not have been quiesced during the snapshot.

In any case, I've recevied instructions on how to register the missing dll, I just need to test things through, and if it works, I'll post the instructions here. I'll also try to setup a download link for the missing dll.

Cheers

[OmniTech]

@lucmorin

It is a Win 2003 VM but not a DC

So I guess it's normal this file is not present.

But in a Win2008R2 VM, the snapshots still does not work and they are not DCs either.

[lucmorin]

But is the VM a vCenter servers ?

[OmniTech]

Yes, build upon a template

[lucmorin]

When you install vCenter on a VM, it adds the "Active Directory Lightweight Directory Service" role to the guest.

So, when taking a quiesced snapshot, you need to tell this service to complete its transaction and then wait for the snapshot to complete (I'm simplifying to extremes here). In order to achieve that, the VSS infrastructure is used, and the ADLDS needs a "writer", and in our case it's the ADAM writer installed by VMWare tools.

Unfortunately, on WIndows 2008 R2, it seems that some components are not properly installed and registered, thus causing the problem. I have received a procedure to "patch" the problem, and it seems to work, but I need to make syure my snapshot is valid and I just don't have the time to do a full DR right now.

Once I'm sure it works, I'll post the steps and files on my blog (not too sure if I can attach files here).

regards

[HendersonD]

In a few days I will be upgrading to vSphere 4.1 from 4.0. Any wise words of wisdom in an attempt to avoid these snapshot issues? I do use Netapp's Snapmanager for Virtual Infrastructure for backups. I have it split into two jobs. One job takes a VMWare snap before taking a filer snapshot, this is bulk of my backup. The other job just takes a filer backup and no VMWare snap first. I have Domain Controllers, SQL, Exchange, and Virtual Center server in this job.

I will be installing the newest VMWare Tools on all my VMs once the upgrade is complete but I am nervous after reading all of these posts since many of my VMs are 2008 R2.

[jcwuerfl]

Still currious if a floppy drive is connected, and if that was removed if it would work after.

Since it was trying to write to the floppy drive?

Device:
?\fdc#generic_floppy_drive

[mrjlturner]

Jcwuerfl,

No floppy disks are connected (at least in my case.) It is entirely related to ADAM or other AD components being installed, as I see the same issues on my DC's that are running 2008 R2 as well.

I'm just hoping that the folks in this thread that have tickets open w/ VMware Support actually make some progress on this one...

-mrjlturner

[jcwuerfl]

Ok, what about a floppy drive though? having a floppy drive present and not having something "connected" to it as its trying to write information to it and it can't ? If you have a floppy drive listed in your VM Edit window, try removing it.

[mrjlturner]

Jcwuerfl,

I apologize for not being more clear. There are no floppy drives present (disconnected or otherwise) on the VMs that are having issues. I've never had a need w/ the newer OS's to have floppy drives attached, so I build them w/o them out of the gate. It is an odd error that it spits out a FDC controller during the VSS process...but the other guy was right...it is spitting out those errors, even though no floppy drives are present.

-mrjlturner

[jmerchan]

Same bug!! no solution

[slusamson]

I'm anxiously awaiting a procedure on this... I'm running vRanger for backups and all of my 2008 R2 servers (one running MS SQL and another running Exchange) are failing when trying to make application consistent quiesed backups. vRanger support pointed out that when a snapshot is taken from their software with quiesing enabled, it actually creates 2 snapshots at the same time for the vmdks. I checked for the dll mentioned in this thread and haven't seen it on any of my servers. Hopefully we'll have something soon!

[lucmorin]

@all,

It seems that the workaround proposed by VMWare is working.

I have taken snapshots, and I've been able to restore the VM without any problem, though I'm not still 100% sure that the snapshot is application consistent (I admit lacking the knowledge to verify this, so guys, feel free to suggest).

I've posted the work around and needed files on my blog at www.stlm.ca/newblog

Let me know if this works for you.

Regards.

[slusamson]

are these for x86 or x64?

[lucmorin]

They are for x64

Forgive my ignorance, but is Windows 2008 R2 even available as x86 ?

Regards

[slusamson]

hehe... forgive my ignorance...it isn't... slipped my mind :smileyhappy:

[SewerRat]

The snapshot does indeed complete after I went through the instructions in your blog. Thank you for that. Now if I look in the Event Viewer after a Snapshot there are 5 warnings and errors in there.

Four of them are similar to this one:

Log Name: Application

Source: VSS

Date: 10/25/2010 2:58:35 PM

Event ID: 8194

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: computername

Description:

Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface. hr = 0x80070005, Access is denied.

. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:

Gathering Writer Data

Context:

Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}

Writer Name: System Writer

Writer Instance ID: {2d606c72-11e4-4063-9e94-a57eb4be60b3}

Event Xml:

[Marcj1]

Did this work for anyone else? It did not work for me, athough I do now get error messages similar to SewerRat which I did not get before.

Similar situation, 2008 R2 Domain controller on Vsphere 4.1. Can't get a quiesced snapshot.

[SewerRat]

I never did get my 2008 R2 domain controller to take a quiesced snap.

I forgot to mention before that I think that blog is missing a step 2.5 to put the zipped VCBRequestor.dll into the c:\program files\vmware\vmware tools\drivers\vss folder.

[lucmorin]

Thanks SewerRat, good catch.

I'll update the blog post.

Also, I have informed VMWare about the new messages in the event log. I haven't heard from them yet.

Regards

[KWBS]

tried lucmorins solution, too. Manual Snapshot is working. But new event log errors occur:

Protokollname: Application

Quelle: VSS

Datum: 28.10.2010 08:33:23

Ereignis-ID: 8194

Aufgabenkategorie:Keine

Ebene: Fehler

Schlüsselwörter:Klassisch

Benutzer: Nicht zutreffend

Computer: XXXXXXXXXX

Beschreibung:

Volumeschattenkopie-Dienstfehler: Beim Abfragen nach der Schnittstelle "IVssWriterCallback" ist ein unerwarteter Fehler aufgetreten. hr = 0x80070005, Zugriff verweigert

. Die Ursache hierfür ist oft eine falsche Sicherheitseinstellung im Schreib- oder Anfrageprozess.

Vorgang:

Generatordaten werden gesammelt

Kontext:

Generatorklassen-ID: {e8132975-6f93-4464-a53e-1050253ae220}

Generatorname: System Writer

Generatorinstanz-ID: {ac43dc03-7287-418c-a4fe-70478fee8ede}

</Event>

Will now open a service request.

[RobMokkink]

When will vmware release a new version of vmware tools?

This problem is more then a couple of months old.

[RobMokkink]

When will VMware release a new version of VMware tools?

This problem is a couple of months old.

[KWBS]

Support is pointing me to http://kb.vmware.com/kb/1018194 ... But when we disable vss it's not crash consistent ^^

[mdgunther]

Support told me this will be fixed in 4.1 U1, but they couldn't tell me when that would be.

It was six months from 4.0 to U1, seven from U1 to U2. 4.1 hit on 7/13. Draw your own conclusions.

[HendersonD]

For now I am going to create two backup jobs, one that does the VMWare snapshot before my Netapp filer snapshot and one that only does the Netapp snapshot. I will include all of my WinServer 2008 R2 servers in the latter job. Not ideal that I cannot take VMWare snapshots of R2 boxes but at least I am gettting backup. As soon as ESX 4.1 U1 comes out I can move all of my R2 boxes to the first job.

[mdgunther]

4.1 U1 came out on 2/10 and look what's in the release notes:

* Cannot take quiesced snapshots of Microsoft Windows Server 2008 R2 virtual machine running vCenter Server 4.1

When creating a snapshot of a Microsoft Windows Server 2008 R2 virtual machine that has vCenter Server 4.1 installed, the snapshot operation might fail to complete. This issue occurs on Microsoft Windows Server 2008 R2 virtual machines when the ADAM database is installed. The issue is resolved in this release.

I'll hopefully be updating before the month is over to see that it works.  After the update, I'll have to undo the workaround.  It's been so long, I can't remember what it was!

[randomotten]

I hate to say this. But we did the update and the problem is still there.

We use Acronis Virtual Edition with the virtual Appliance and still get this kind of errors:

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Log Entry Details
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type:          Error
Date and time: 2/14/2011 12:16:04 PM
Backup plan:   Backup All Virtual Machines
Task:          VCENTER01 - Incremental backup
Code:          5,439,591(0x530067)
Module:        83
Owner:         Administrator@BKV-GROEP
Message:      
  Failed to open the virtual machine ([BIGFOOT] VCENTER01/VCENTER01.vmx).
Additional info:
--------------------
Error code: 103
Module: 83
LineInfo: a859dd78cc91dcf6
Fields:
Message: Failed to open the virtual machine ([BIGFOOT] VCENTER01/VCENTER01.vmx).
--------------------
Error code: 138
Module: 83
LineInfo: a859dd78cc91dce8
Fields:
Message: VMware failed to create a quiesced snapshot of the selected virtual machine. To back up the machine, power off the machine before the process or uninstall the Volume Shadow Copy Services Support feature of VMware Tools from the VM.
--------------------
Error code: 138
Module: 83
LineInfo: c7610e0a857bed85
Fields:
Message: Awaiting task 'CreateSnapshot' has failed. Reason: Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine..
--------------------
Acronis Knowledge Base: http://kb.acronis.com/errorcode/

Event code: 0x00530067+0x0053008A+0x0053008A

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[frankLT]

Hello,

we had exactly the same problem on our vCenter server. Upgrading from ESX 4.1 to U1 and installing the appropriate version of VMware Tools in our vCenter VM solved the issue. Cloning and VDR backup using quiesced snapshot now succeed. Unfortunately, there are some errors/warnings left, so I don't exactly know if we get fully consistent snapshots, especially because of NTFS warnings.

Always numerous warnings of this kind:

Protokollname: System
Quelle:        Ntfs
Datum:         23.02.2011 14:14:15
Ereignis-ID:   57
Aufgabenkategorie:(2)
Ebene:         Warnung
Schlüsselwörter:Klassisch
Benutzer:      Nicht zutreffend
Computer:      LTVC1.domain.de
Beschreibung:
Die Daten konnten nicht in das Transaktionsprotokoll verschoben werden. Die Daten sind möglicherweise beschädigt.
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Ntfs" />
    <EventID Qualifiers="32772">57</EventID>
    <Level>3</Level>
    <Task>2</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-02-23T13:14:15.176483200Z" />
    <EventRecordID>20355</EventRecordID>
    <Channel>System</Channel>
    <Computer>LTVC1.domain.de</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Binary>0000000001000000020000003900048000000000100000C000000000000000000000000000000000</Binary>
  </EventData>
</Event>

And always this error:

Protokollname: System
Quelle:        Ntfs
Datum:         23.02.2011 14:14:15
Ereignis-ID:   137
Aufgabenkategorie:(2)
Ebene:         Fehler
Schlüsselwörter:Klassisch
Benutzer:      Nicht zutreffend
Computer:      LTVC1.domain.de
Beschreibung:
Auf dem Volume "\\?\Volume{5f561ae3-3f4d-11e0-8393-0050568115d9}" konnte der Transaktionsressourcen-Manager aufgrund eines nicht wiederholbaren Fehlers nicht gestartet werden. Der Fehlercode ist in den Daten enthalten.
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Ntfs" />
    <EventID Qualifiers="49156">137</EventID>
    <Level>2</Level>
    <Task>2</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-02-23T13:14:15.205769200Z" />
    <EventRecordID>20360</EventRecordID>
    <Channel>System</Channel>
    <Computer>LTVC1.domain.de</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Data>\\?\Volume{5f561ae3-3f4d-11e0-8393-0050568115d9}</Data>
    <Binary>1C0004000200300002000000890004C000000000100000C000000000000000000000000000000000100000C0</Binary>
  </EventData>
</Event>

I just upgraded the OS (W2K8R2) to SP1. Generally running fine, but previous described messages are still there. Anyone else having the same problem?

Best regards,

Frank

[wadood]

frankLT,

I too have the same error messages as you describe. on a windows 2008 r2 64 bit domain controller. have you had any luck with knowing the cause of this or fixing it?

[frankLT]

Hey wadood,

no, not so far. But this problem is obviously not specific to certain server roles or applications. Each of our 2008 R2 VMs produce these same errors when being backed up with vdr. But those event log entries are the only problem I noticed. Backup jobs are running fine, and VMs can be restored without any problem.

Frank

[wadood]

hi frank,

thanks for the answer. yes, i just confirmed they are on all windows 2008 r2 vms. i was not looking in the system logs for the rest.

at least you confirm the vdr backups are fine, thats what was scaring me.

wadood.

[mdgunther]

I updated to 4.1U1 yesterday and it appears to have fixed the problem for me.  I was able to take a snapshot without error and vDR successfully backed up the vCenter Server VM.  I had to set the EnableUUID back to true to enable the VSS snapshots.  Support had advised me to remove the VM from the inventory and re-add it when I set it to false (the workaround until the bug was fixed), so I did the same for the true edit.

To be clear, I confirmed with support that this is an ESX/Vmware-tools level problem and not a vCenter Server problem.  Just updating the vCenter Server isn't going to fix it, you have to do everything.

[kipz]

hello,

i have also upgraded to the latest binary but the problem is still there. and for me, even few Windows 2003 VMs fails with the same error.

i have also noticed that sometimes rebooting VM helps for a short period. right after reboot i can take quiesced snapshot few times. but eventually it starts to fail again.

regards,

kipz

[Borja_Mari]

Hi,

the same workaround (it seems that can happens in w2k3 too) is explained here too.

Regards/Saludos,
Pablo

Please consider awarding

any helpful or corrrect answer. Thanks!! -

Por favor considera premiar

cualquier respuesta útil o correcta. ¡¡Muchas gracias!!

Virtually noob blog

[Marcj1]

Just got the response from support about the fix coming in 4.1U1. They said that it is scheduled to be released in Q1 2011.

[RobMokkink]

Q1 2011. That is not acceptable.

[HendersonD]

I agree. At best this makes the release 8 weeks off, at worst 20 weeks off. You would think this bug would be serioius enough that an interim release just to address it would happen before U1 comes out.

[jb238]

We have 4 Win2008R2 DC VMs and use VDR 1.2. After upgrading to 4.1, only one DC had this problem.

I noticed that the first snapshot after a reboot works. Subsequent snapshots fail until the next reboot.

[ITWheel]

OK Everyone, let me know if this does not work for you.

The VMware Snapshot Provider is the problem. Reviewing and implementing

the fixes in the current version of the following VMware KB does not

resolve the issue. They may fix it soon when they see this.

Cannot create a quiesced snapshot because the snapshot operation

exceeded the time limit for holding off I/O in the frozen virtual

machine

Use this method if you are getting the following error in the system

event log when trying to start the VMware Snapshot Provider Service in

the guest VM:

Error 1053: The service did not respond to the start or control request in a timely fashion VMWare Snapshot Provider

Also, you will probably be getting the following error when trying to

take a snapshot of the VM with Memory unchecked and Quiesce checked or

when trying to do a VDR backup:

Cannot create a quiesced snapshot because the snapshot operation

exceeded the time limit for holding off I/O in the frozen virtual

machine.

To fix the problem Uninstall and Reinstall VMWare Snapshot Provider as follows:

1. Uninstall Volume Shadow Copy Service (using Modify from VMware Tools setup)

2. Delete VSS folder from C:\Program Files\VMWare\VMWare Tools\Drivers

3. Delete VMWare Snapshot Provider Service by deleting the key

HKLM\System\Current Control Set\Services\vmvss (Just delete the vmvss

key)

4. Delete VMware Snapshot Provider

{564d7761-7265-2056-5353-2050726f7669} from HKLM\system\current control

set\services\VSS\Providers

(If you do not have this key, just omit this step.)

5. Reboot

6. Start the two COM+ services mentioned in kb.vmware.com/kb/1018194

7. Install Volume Shadow Copy Service (using Modify from VMWare Tools setup.exe)

8. Reboot

VMWare Snapshot Provider should now be Manual, and you can start it.You

should now be able to take a snapshot and back up with vdr while the VM

is powered on.

( Also see this Symantec article with a full step by step walkthrough of steps 1-4: )

Marcus

IT Wheel

www.itwheel.com

Consider awarding points for "helpful" and/or "correct" answers.

[hugo5113]

i had the same error.

enviroment:

vSphere 4.1

W2K8R2

DC

Backup with VDR

i repaired it:

1.) uninstall only vmware-tools-option VSS-driver...(see attatched file)

2.) backing up registry

3.) regedit: deleting all vmware-related(!) entries with substring "vcbsnap", e.g. CLSIDs (by using /search...)

4.) reboot

5.) reinstall vmware-tools, userdefined/change, VSS-driver

6.) reboot

it works fine :smileywink:

regards Peter

[KEPete]

Hugo, I've also tried reinstalling the VSS drivers and have been running successful VDR backups as far as VDR is concerned. However, did your VSS errors go away in your Windows event logs? I started noticing these issues back in July: http://communities.vmware.com/message/1577208#1577208

So far, nothing seems to really work as far as getting a real backup of AD. I'm currently using Microsoft's built in tool for backing up the System State to protect myself.

[bingofox]

I have the exact same issue.

ESXi 4.1

vCenter 4.1

All VM's (mix of 2008 and 2003) created from scratch on 4.1

All of them work and can be quisced prior to taking a backup except those running A/D, SQL, or any relational Database for that matter.

It also affect Windows 2003 Servers as well - not just 2008.

I'm wondering why there still isn't a hotfix/patch for this?

[KEPete]

FYI, not sure how many people on here use DFS but my Windows 2008 R2 Standard DFS boxes are also showing VSS errors after being backed up by VDR. It does not show errors in VDR but shows on the server's event logs. What's funny is that when I first upgraded my hosts to 4.1, VDR would not backup at all if it had AD or ADLDS installed. but the others were fine. After I removed VSS and reinstalled VMTools, VDR backups started running but backups were no good for AD. I installed VSS again and backups still ran. But VSS errors continue. Has anyone tested restores with these various fixes being posted here?

[tsinfra]

Hey guys,

I'm running ESXi 4.1 build 381591 and still having VMware VSS probs on a W2K8 R2 SP1 virtual machine.

Uisng Symantec Netbackup 7.1.0.1 to do application consistent / file quiesced backups of multiple W2K3 R2 and W2K8 R2 SP1 vm's.

The 2003 vm's snapshot ok, however the 2008 snapshot always failed with the I/O message displayed.

Worked with Symantec who in the end advised me to completely uininstall VMware VSS and use the Symatec VSS provider instead;

Symantec notes:

Install the symantec vss provider in the VM .

Refere the guide.

http://www.symantec.com/business/support/index?page=content&id=DOC3663

Page 167 onwards.

This seems to have worked!

I've logged an SR with VMWare, but I don't think the VMware VSS provider installed as part of a clean vmware tools install on a 4.1 U1 build 381591 host works with W2K8R2!

Thought I'd share what I'm finding.

[TK22]

Jag är borta på semester och kommer troligast inte att läsa e-post förrän jag förhoppningsvis är tillbaka igen, Onsdagen den 7:e September.

För teknisk assistans hänvisar jag till kontoret på 060 - 14 95 00.

Mvh

/ Kenneth

I´m on vacation and will probably not read mail until i´m back, Wednesday September 7:th.

Kind Regards

/ Kenneth

[rb51]

hi all,

we still get this error on our vcentre server.

event ID: 12289, 8193

hosts: 4.1U1

vcentre: 4.1U1

OS: Windows 2008 R2

vmware tools: 8.3.7 build 341836 (U1)

Just wondering what you guys would suggest. We will soonish be applying U2 to our hosts.

Help is much appreciated.

rb51