VMware vSphere

 View Only
Expand all | Collapse all

VMtools error in Application events on server 2016

  • 1.  VMtools error in Application events on server 2016

    Posted Jul 04, 2017 08:19 AM

    Hi,

    we have faulting application errors every ~5-10 minutes on our 2016 servers with vm tools:

    Faulting application name: WmiApSrv.exe, version: 10.0.14393.0, time stamp: 0x57899ac8
    Faulting module name: vmStatsProvider.dll, version: 10.0.9.55972, time stamp: 0x57426eff
    Exception code: 0xc0000005
    Fault offset: 0x0000000000001d9a
    Faulting process id: 0x14a4
    Faulting application start time: 0x01d2f499b7b2dbde
    Faulting application path: C:\WINDOWS\system32\wbem\WmiApSrv.exe
    Faulting module path: C:\Program Files\VMware\VMware Tools\vmStatsProvider\win64\vmStatsProvider.dll
    Report Id: 42c0409f-03cd-4033-bedc-9194b77373b4
    Faulting package full name:
    Faulting package-relative application ID:

    Tried different vmtools versions including newest 10.1.7, still the same problem.

    Server has latest security and feature updates, also .net 4.7

    I've tried to look at procdump, however I don't have any experience with windbg therefore I'm unable to conclude anything from the output. Some info from the dump:

    AULTING_IP:
    vmStatsProvider+1d9a
    00007ffd`1bf41d9a ff5010          call    qword ptr [rax+10h]

    EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
    ExceptionAddress: 00007ffd1bf41d9a (vmStatsProvider+0x0000000000001d9a)
       ExceptionCode: c0000005 (Access violation)
      ExceptionFlags: 00000000
    NumberParameters: 2
       Parameter[0]: 0000000000000000
       Parameter[1]: 00007ffd241fe520
    Attempt to read from address 00007ffd241fe520

    PROCESS_NAME:  WmiApSrv.exe

    ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".

    EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".

    EXCEPTION_PARAMETER1:  0000000000000000

    EXCEPTION_PARAMETER2:  00007ffd241fe520

    READ_ADDRESS:  00007ffd241fe520

    FOLLOWUP_IP:
    vmStatsProvider+1d9a
    00007ffd`1bf41d9a ff5010          call    qword ptr [rax+10h]

    NTGLOBALFLAG:  0

    APPLICATION_VERIFIER_FLAGS:  0

    LAST_CONTROL_TRANSFER:  from 00007ffd1bf428c5 to 00007ffd1bf41d9a

    ADDITIONAL_DEBUG_TEXT:  Followup set based on attribute [UnloadedModule_Arch_AX] from Frame:[0] on thread:[188c] ; Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD] ; Followup set based on attribute [ip_is_call_value_Arch_ax] from Frame:[0] on thread:[188c] ; Followup set based on attribute [Is_OriginalExceptionThread] from Frame:[0] on thread:[188c]

    FAULTING_THREAD:  ffffffffffffffff

    BUGCHECK_STR:  APPLICATION_FAULT_INVALID_POINTER_READ_CALL_LEAK

    PRIMARY_PROBLEM_CLASS:  INVALID_POINTER_READ_CALL_LEAK

    DEFAULT_BUCKET_ID:  INVALID_POINTER_READ_CALL_LEAK

    STACK_TEXT: 
    00000000`00000000 00000000`00000000 fastprox!CWbemInstance::_vftable_+0x0


    STACK_COMMAND:  .ecxr ; ~~[188c] ; .frame 0 ; ** Pseudo Context ** ; kb

    SYMBOL_STACK_INDEX:  0

    SYMBOL_NAME:  fastprox!CWbemInstance::_vftable_

    FOLLOWUP_NAME:  MachineOwner

    MODULE_NAME: fastprox

    IMAGE_NAME:  fastprox.dll

    DEBUG_FLR_IMAGE_TIMESTAMP:  0

    FAILURE_BUCKET_ID:  INVALID_POINTER_READ_CALL_LEAK_c0000005_fastprox.dll!CWbemInstance::_vftable_

    BUCKET_ID:  X64_APPLICATION_FAULT_INVALID_POINTER_READ_CALL_LEAK_fastprox!CWbemInstance::_vftable_

    WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/WmiApSrv_exe/6_2_14393_0/57899ac8/vmStatsProvider_dll/10_0_9_55972/57426eff/c0000005/00001d9a.htm?Retriage=1;

    and some more:

    This dump file has an exception of interest stored in it.
    The stored exception information can be accessed via .ecxr.
    (688.188c): Access violation - code c0000005 (first/second chance not available)
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for vmStatsProvider.dll -
    vmStatsProvider+0x1d9a:
    00007ffd`1bf41d9a ff5010          call    qword ptr [rax+10h] ds:00007ffd`241fe520=????????????????
    0:004> k
    # Child-SP          RetAddr           Call Site
    00 0000001c`d977f500 00007ffd`1bf428c5 vmStatsProvider+0x1d9a
    01 0000001c`d977f540 00007ffd`1bf42403 vmStatsProvider+0x28c5
    02 0000001c`d977f580 00007ffd`1bf56190 vmStatsProvider+0x2403
    03 0000001c`d977f5d0 00007ffd`1bf5171f vmStatsProvider!DllUnregisterServer+0x10830
    04 0000001c`d977f600 00007ffd`1bf51965 vmStatsProvider!DllUnregisterServer+0xbdbf
    05 0000001c`d977f640 00007ffd`378ea35f vmStatsProvider!DllUnregisterServer+0xc005
    06 0000001c`d977f680 00007ffd`378d3d2d ntdll!LdrpCallInitRoutine+0x4b
    07 0000001c`d977f6e0 00007ffd`378d1621 ntdll!LdrpProcessDetachNode+0xf5
    08 0000001c`d977f7b0 00007ffd`3790eefc ntdll!LdrpUnloadNode+0x49
    09 0000001c`d977f800 00007ffd`3790ee24 ntdll!LdrpDecrementModuleLoadCountEx+0xc4
    0a 0000001c`d977f850 00007ffd`347602cd ntdll!LdrUnloadDll+0x94
    0b 0000001c`d977f880 00007ffd`367f5b38 KERNELBASE!FreeLibrary+0x1d
    0c 0000001c`d977f8b0 00007ffd`367f5a56 combase!CClassCache::CDllPathEntry::CFinishObject::Finish+0x28 [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 3368]
    0d 0000001c`d977f8e0 00007ffd`3679f1b4 combase!CClassCache::CFinishComposite::Finish+0x56 [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 3478]
    0e 0000001c`d977f910 00007ffd`3679c4cb combase!CClassCache::CleanUpDllsForProcess+0x124 [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 6969]
    0f (Inline Function) --------`-------- combase!CCCleanUpDllsForProcess+0xe [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 8680]
    10 0000001c`d977fb10 00007ffd`367ded00 combase!ProcessUninitialize+0x1c3 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 2236]
    11 0000001c`d977fb50 00007ffd`367deb23 combase!DecrementProcessInitializeCount+0x44 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 992]
    12 0000001c`d977fb80 00007ffd`367db835 combase!wCoUninitialize+0x87 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 4115]
    13 0000001c`d977fbc0 00007ff7`5bd44351 combase!CoUninitialize+0x85 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 3946]
    14 0000001c`d977fbf0 00007ff7`5bd45472 WmiApSrv!WinRun+0x89
    15 0000001c`d977fc30 00007ffd`35223dd2 WmiApSrv!WmiAdapterService::_ServiceMain+0x52
    16 0000001c`d977fc60 00007ffd`37728364 sechost!ScSvcctrlThreadA+0x22
    17 0000001c`d977fc90 00007ffd`379270d1 kernel32!BaseThreadInitThunk+0x14
    18 0000001c`d977fcc0 00000000`00000000 ntdll!RtlUserThreadStart+0x21



  • 2.  RE: VMtools error in Application events on server 2016

    Posted Jul 04, 2017 10:36 AM

    What is the version and build number of ESXI you are using.



  • 3.  RE: VMtools error in Application events on server 2016

    Posted Jul 04, 2017 10:45 AM

    Hi,

    it is ESXi, 6.0.0, 5224934



  • 4.  RE: VMtools error in Application events on server 2016

    Posted Jul 04, 2017 11:46 AM

    I believe you already had correlated tools version with the installed build number (10.0.9)

    Uninstall and reinstall VMware tools with Custom Install, Uncheck WMI performance logging while choosing feature and monitor for sometime.

    https://packages.vmware.com/tools/esx/6.0u3/windows/x64/VMware-tools-windows-10.0.9-3917699.iso



  • 5.  RE: VMtools error in Application events on server 2016

    Posted Jul 04, 2017 12:08 PM

    What if we want to use WMI performance logging? This does not sound like a solution in that case.



  • 6.  RE: VMtools error in Application events on server 2016

    Posted Jul 04, 2017 12:18 PM

    Of Course this is not a solution, neither there is much documentation for Server 2016 yet so this can help to reproduce the cause.



  • 7.  RE: VMtools error in Application events on server 2016

    Posted Jul 04, 2017 01:10 PM

    I can confirm that the error is not logged when wmi performance logging is not installed.



  • 8.  RE: VMtools error in Application events on server 2016

    Posted Jul 04, 2017 07:12 PM

    Were you always having this issue since the server was deployed and VMware Tools installed or did something change recently? I've been testing with a 2016 server with WMI and haven't been able to reproduce the error.



  • 9.  RE: VMtools error in Application events on server 2016

    Posted Jul 06, 2017 02:44 PM

    We were not using server 2016 up to this point, therefore we never had it working previously.



  • 10.  RE: VMtools error in Application events on server 2016

    Posted Jul 06, 2017 02:48 PM

    Install with VMWareToolsSetup.exe incase if you are installing with VMwareTools64.msi and check.



  • 11.  RE: VMtools error in Application events on server 2016

    Posted Jul 10, 2017 01:49 PM

    I was using Setup64.exe.

    Setup.exe does not run on 64bit OS.



  • 12.  RE: VMtools error in Application events on server 2016

    Posted Jul 11, 2017 01:54 PM

    Hi,

    we have on serveral servers (around 50) the same error.

    Have anyone found a solution?

    It would be great, if there is another option.

    Disabling wmi in the setup dialog of VMware Tools, could only be a workaround, but no solution!

    Thanks + Best regards

    Stefan



  • 13.  RE: VMtools error in Application events on server 2016

    Posted Jul 12, 2017 09:22 AM

    We have few servers that do not have this issue, the only difference I was able to identify between these servers are that VMs with errors are running in BIOS mode, and those without errors are running in UEFI, anyone else having the problem can check if they are running UEFI or BIOS?



  • 14.  RE: VMtools error in Application events on server 2016

    Posted Jul 12, 2017 01:18 PM

    My VM with no errors is running in BIOS.



  • 15.  RE: VMtools error in Application events on server 2016

    Posted Jul 26, 2017 07:20 AM

    It seems this problem affects only the VMs that had OS installed via SCCM (or cloned from template which was isntalled via SCCM). If you install from ISO, this problem does not appear.



  • 16.  RE: VMtools error in Application events on server 2016

    Posted Sep 28, 2017 08:26 AM

    Do anyone have a fix for this ?
    I have tried all the solutions I could find on this, but it keeps going on with WMI appcrash and spams the log with eventcode 1001. I have this on all the win 2016 servers, and yes they are deployed from a template, but I still need a solution :-)



  • 17.  RE: VMtools error in Application events on server 2016

    Broadcom Employee
    Posted Nov 08, 2017 07:23 PM

    This issue is fixed since VMware Tools 10.1.10, please refer to VMware Tools 10.1.10 Release Notes

    • WMI performance adapter service fails on windows guest operating systems
      WMI performance adapter service (wmiapsrv.exe) fails on virtual machines running Windows 10 and Windows Server 2016. This issue is resolved in this release.

    Please download the latest available VMware Tools 10.1.15 to get the fix.



  • 18.  RE: VMtools error in Application events on server 2016

    Posted Sep 28, 2018 02:57 PM

    We have the same issue,

    Server 2016; vmstatsprovider.dll is crashing as shown in WER logs and application log.

    Other OS do not exhibit this issue.

    We updated to tools 10.1.10 from 10.1.7 to no avail - still causing an issue.

    We also tried a full uninstall of 10.1.7, restart and install of 10.1.10 on one Server.

    Oddly both in event log and WER logs vmstatsprovider.dll looks to still be registered as 10.1.7 despite the file version being 10.1.10 on the machine - this is the same on all we have tried to update.

    Sig[0].Name=Application Name

    Sig[0].Value=WmiApSrv.exe

    Sig[1].Name=Application Version

    Sig[1].Value=10.0.14393.0

    Sig[2].Name=Application Timestamp

    Sig[2].Value=57899ac8

    Sig[3].Name=Fault Module Name

    Sig[3].Value=vmStatsProvider.dll

    Sig[4].Name=Fault Module Version

    Sig[4].Value=10.1.7.61298

    Sig[5].Name=Fault Module Timestamp

    Sig[5].Value=59163c3a

    Sig[6].Name=Exception Code

    Sig[6].Value=c0000005

    Sig[7].Name=Exception Offset

    Sig[7].Value=0000000000001d9a

    DynamicSig[1].Name=OS Version

    DynamicSig[1].Value=10.0.14393.2.0.0.272.7

    Should I log with VMware and if so has anyone else done this and had a resolution?