Windows Server 2012 R2 BSOD 0x0000013c

[Roland_Schauer]

Hi,

I've some problems with one of my Windows 2012 R2 Fileservers. Every week the server crashes with a BSOD INVALID_IO_BOOST_STATE (code: 0x0000013c). Some searching told me that it should be a driver with a memory leak.

Guest: Windows Server 2012 R2 up to date, hardware version 10, LSI Logic/Paravirtual Storage Controller (tried both), vmxnet3 NIC

Host: HP Proliant DL380G7 and G8, ESXi 5.5 Update 1 and 2

3 other Fileservers with same configuration have no problems, they are all from the same template.

As I don't know what else to do, do you have any suggestions how to fix?

Thanks in advance!

best regards

Roland Schauer

[Devid Miatello]

Hi, i have the same problem.

Did you resolve them?

[Alistar]

Hi there,

can you please upload the crash dumps (default is %windir%\minidump), I'd give a shot debugging them :smileyhappy:

[Devid Miatello]

this is the minidump

thank you in advance

[Alistar]

Oh wow, didn't see this one for a long time -

as for my analysis it seems that your Trend Micro Security Software is going haywire in your system. Your OS crash was caused by OVERLAPPED_MODULE: Address regions for 'TmXPFlt' and 'TMEBC64.sys' overlap. Where "TMEBC64.sys" belongs to Trend Micro's "Early Boot Clean Driver" and TmXPFlt seems to be some form of core library for the security suite. The threading somehow touches an invalid memory the OS enters a kernel panic. I suggest either finding another security solution or patching up to the latest available version - application and OS.

Roland, can you send us your dumps so we can take a look as well?

[Roland_Schauer]

ok, very interesting, we have installed TM OfficeScan too...

so here we go, but how would you explain other servers don't have this issue?

Thank you very much for your effort!

[COS]

What Trend Suite and version is the OP using?

It might be too old.

Thanks

[jrmunday]

Hi Roland,

Have you analysed the crash dump with WinDbg ... this should give you a good idea of what's going on.

I had a similar issue in the past (also HP hardware) that was resolved by a BIOS update;

Intel microcode issue affecting E5-2600 v2 series processors

Cheers,

Jon

[COS]

In the VM, from the VM tools, remove the VMCI item "Guest Introspection" from each VM then reboot them one by one and try it again.

Let us know if this helps. It helped me.

[Roland_Schauer]

Hi guys!

Thanks for your answers!

I'm on this topic with MS Premium Support actually. Unfortunately they don't have any results until now.

I don't think it's a problem with the Proliants as it's a mixed G7 and G8 environment and my other fileservers don't have this issue but this article is really interesting.

As described in the first post I've tried some hardware configuration changes with no improvement and my last change yesterday has been a vCPU reduction to 1. Previously the server had 4 sockets, which is quite unnecessary.

Removing the VMCI driver didn't help either.

I've also played with the memory dumps, but didn't get any significant information (at least for me - but MS doesn't have any idea too).

So let's see if vCPU reduction is the key.

Do you have any other suggestions?

cheers,

Roland

[jrmunday]

Perhaps you could upload the dump (mini or full kernel dump), so that we can debug it?

Also, is there anything in the guest VM's vmware.log file or any particular pattern as to when this occurs (ie. the same time on a specific dat of the week, or after a specific event)?

[Roland_Schauer]

@Jon

no not really - different day ties, not after any scheduled tasks, ...

here are probably interesting vm log entries (invalid pagetype?)

2014-12-15T10:08:56.333Z| mks| I120: SVGA disabling SVGA

2014-12-15T10:08:56.806Z| vcpu-1| I120: LSI: Invalid PageType [21] pageNo 0 Action 0

2014-12-15T10:09:10.806Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.

2014-12-15T10:09:18.033Z| vmx| I120: Tools: Tools heartbeat timeout.

2014-12-15T10:09:25.806Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.

2014-12-15T10:09:25.806Z| vmx| I120: GuestRpc: app toolbox's second ping timeout; assuming app is down

2014-12-15T10:09:25.814Z| vmx| I120: GuestRpc: Reinitializing Channel 0(toolbox)

2014-12-15T10:09:25.814Z| vmx| I120: GuestMsg: Channel 0, Cannot unpost because the previous post is already completed

2014-12-15T10:09:25.831Z| mks| I120: SOCKET 5 (153) Creating VNC remote connection.

2014-12-15T10:09:30.716Z| vmx| I120: Vix: [163264 vmxCommands.c:680]: VMAutomation_Reset. Trying hard reset

2014-12-15T10:09:30.717Z| vmx| W110:

2014-12-15T10:09:30.717Z| vmx| W110+

2014-12-15T10:09:30.717Z| vmx| W110+ VMXRequestReset

2014-12-15T10:09:30.717Z| vmx| I120: Vigor_Reset: Attaching to reset.

2014-12-15T10:09:30.717Z| vmx| I120: Stopping VCPU threads...

2014-12-15T10:09:30.717Z| vcpu-2| I120: VMMon_WaitForExit: vcpu-2: worldID=163268

2014-12-15T10:09:30.717Z| vcpu-1| I120: VMMon_WaitForExit: vcpu-1: worldID=163267

2014-12-15T10:09:30.717Z| vcpu-0| I120: VMMon_WaitForExit: vcpu-0: worldID=163265

2014-12-15T10:09:30.717Z| vcpu-3| I120: VMMon_WaitForExit: vcpu-3: worldID=163269

2014-12-15T10:09:30.717Z| svga| I120: SVGA thread is exiting

2014-12-15T10:09:31.350Z| vmx| I120:

2014-12-15T10:09:31.350Z| vmx| I120+ OvhdMem: Final (Power Off) Overheads

[jrmunday]

So here is the basic bug check analysis ...

Loading Dump File [C:\Users\Jon\Downloads\120914-20843-01.dmp]

Mini Kernel Dump File: Only registers and stack trace are available

************* Symbol Path validation summary **************

Response                         Time (ms)     Location

Deferred                                       SRV*C:\symbols*http://msdl.microsoft.com/download/symbols

Symbol search path is: SRV*C:\symbols*http://msdl.microsoft.com/download/symbols

Executable search path is:

Windows 8 Kernel Version 9600 MP (4 procs) Free x64

Product: Server, suite: TerminalServer SingleUserTS

Built by: 9600.17238.amd64fre.winblue_gdr.140723-2018

Machine Name:

Kernel base = 0xfffff800`05210000 PsLoadedModuleList = 0xfffff800`054da350

Debug session time: Tue Dec  9 10:58:57.701 2014 (UTC + 0:00)

System Uptime: 6 days 7:58:01.474

Loading Kernel Symbols

.

Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.

Run !sym noisy before .reload to track down problems loading symbols.

..............................................................

................................................................

.........

Loading User Symbols

Loading unloaded module list

..........

*******************************************************************************

*                                                                             *

*                        Bugcheck Analysis                                    *

*                                                                             *

*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 13C, {ffffe000e7b69500, 1, 0, 0}

Probably caused by : ntkrnlmp.exe ( nt! ?? ::NNGAKEGL::`string'+547d3 )

Followup: MachineOwner

---------

3: kd> !analyze -v

*******************************************************************************

*                                                                             *

*                        Bugcheck Analysis                                    *

*                                                                             *

*******************************************************************************

INVALID_IO_BOOST_STATE (13c)

A thread exited with an invalid I/O boost state.  This should be zero when

a thread exits.

Arguments:

Arg1: ffffe000e7b69500, Pointer to the thread which had the invalid boost state.

Arg2: 0000000000000001, Current boost state.

Arg3: 0000000000000000

Arg4: 0000000000000000

Debugging Details:

------------------

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT_SERVER

BUGCHECK_STR:  0x13C

PROCESS_NAME:  System

CURRENT_IRQL:  0

ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) amd64fre

LAST_CONTROL_TRANSFER:  from fffff8000578edb3 to fffff80005363ca0

STACK_TEXT: 

ffffd000`2840db48 fffff800`0578edb3 : 00000000`0000013c ffffe000`e7b69500 00000000`00000001 00000000`00000000 : nt!KeBugCheckEx

ffffd000`2840db50 fffff800`05600ec4 : 00000000`00000000 00000000`00000000 ffffe000`e7b69500 ffffe000`e6a3d710 : nt! ?? ::NNGAKEGL::`string'+0x547d3

ffffd000`2840dbc0 fffff800`0525fa2f : 00000000`00000000 ffffd000`2840dd10 ffffe000`e7b69500 00000000`00000000 : nt!ObpRemoveObjectRoutine+0x64

ffffd000`2840dc20 fffff800`052dc6b1 : ffffe000`e7b69500 ffffe000`e7b69a90 ffffe000`ea5cde10 ffffd000`2840dd10 : nt!ObfDereferenceObjectWithTag+0x8f

ffffd000`2840dc60 fffff800`0524ca2b : fffff800`052dc610 ffffd000`2840dd10 00000000`00000000 ffffe000`eb041880 : nt!PspReaper+0xa1

ffffd000`2840dc90 fffff800`052e9514 : 00000001`00000001 ffffe000`eb041880 ffffe000`eb041880 ffffe000`e6ad7900 : nt!ExpWorkerThread+0x293

ffffd000`2840dd40 fffff800`0536a2c6 : fffff800`054f6180 ffffe000`eb041880 fffff800`0555da00 00000000`0029b61d : nt!PspSystemThreadStartup+0x58

ffffd000`2840dda0 00000000`00000000 : ffffd000`2840e000 ffffd000`28408000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16

STACK_COMMAND:  kb

FOLLOWUP_IP:

nt! ?? ::NNGAKEGL::`string'+547d3

fffff800`0578edb3 cc              int     3

SYMBOL_STACK_INDEX:  1

SYMBOL_NAME:  nt! ?? ::NNGAKEGL::`string'+547d3

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: nt

IMAGE_NAME:  ntkrnlmp.exe

DEBUG_FLR_IMAGE_TIMESTAMP:  53d0b7c3

IMAGE_VERSION:  6.3.9600.17238

BUCKET_ID_FUNC_OFFSET:  547d3

FAILURE_BUCKET_ID:  0x13C_nt!_??_::NNGAKEGL::_string_

BUCKET_ID:  0x13C_nt!_??_::NNGAKEGL::_string_

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:0x13c_nt!_??_::nngakegl::_string_

FAILURE_ID_HASH:  {81c667ca-13fd-6170-97a0-9063500f733d}

Followup: MachineOwner

---------

3: kd> lmvm nt

start             end                 module name

fffff800`05210000 fffff800`05999000   nt         (pdb symbols)          c:\symbols\ntkrnlmp.pdb\6E60CE642F39465DAF09219706DE11471\ntkrnlmp.pdb

    Loaded symbol image file: ntkrnlmp.exe

    Mapped memory image file: c:\symbols\ntoskrnl.exe\53D0B7C3789000\ntoskrnl.exe

    Image path: ntkrnlmp.exe

    Image name: ntkrnlmp.exe

    Timestamp:        Thu Jul 24 08:37:39 2014 (53D0B7C3)

    CheckSum:         0071597C

    ImageSize:        00789000

    File version:     6.3.9600.17238

    Product version:  6.3.9600.17238

    File flags:       0 (Mask 3F)

    File OS:          40004 NT Win32

    File type:        1.0 App

    File date:        00000000.00000000

    Translations:     0409.04b0

    CompanyName:      Microsoft Corporation

    ProductName:      Microsoft® Windows® Operating System

    InternalName:     ntkrnlmp.exe

    OriginalFilename: ntkrnlmp.exe

    ProductVersion:   6.3.9600.17238

    FileVersion:      6.3.9600.17238 (winblue_gdr.140723-2018)

    FileDescription:  NT Kernel & System

    LegalCopyright:   © Microsoft Corporation. All rights reserved.

It looks like something low level (perhaps a memory leak) so not easy to identify. I'm just digging a bit deeper into the dump to see if anything stands out.

You don't have a full memory.dmp by chance?

Cheers,

Jon

[Alistar]

If you take a look at both crash dumps, the stack just before kernel panic is exactly the same:

nt!KeBugCheckEx

nt! ?? ::NNGAKEGL::`string'+0x547d3

nt!ObpRemoveObjectRoutine+0x64

nt!ObfDereferenceObjectWithTag+0x8f

nt!PspReaper+0xa1

nt!ExpWorkerThread+0x293

nt!PspSystemThreadStartup+0x58

nt!KiStartSystemThread+0x16

EDIT: And loaded modules show:

fffff801`0d118000 fffff801`0d184000   TmXPFlt    (deferred)            

    Image path: \??\C:\Program Files (x86)\Trend Micro\OfficeScan Client\TmXPFlt.sys

    Image name: TmXPFlt.sys

    Timestamp:        Sat Aug 30 15:11:38 2014 (5401CD8A)

    CheckSum:         0005DDB6

    ImageSize:        0006C000

    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

fffff801`0d184000 fffff801`0d193000   TMEBC64    (deferred)            

    Image path: \SystemRoot\system32\DRIVERS\TMEBC64.sys

    Image name: TMEBC64.sys

    Timestamp:        Mon Jul 01 15:02:09 2013 (51D17DD1)

    CheckSum:         00019AD9

    ImageSize:        0000F000

    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Can you try getting a newer version of this "Early Boot Clean Driver" or remove it altogether and see if the server is stable without it? Perhaps some MS patch is interfering, or maybe a component of VMware tools?...

[jrmunday]

Hi Alistar,

I see those in the loaded modules, but I'm interested to know how you picked these out specifically - is this something you have seen before?

: kd> lmv m TmXPFlt

start             end                 module name

fffff801`b2c91000 fffff801`b2cfd000   TmXPFlt    (deferred)            

    Image path: \??\C:\Program Files (x86)\Trend Micro\OfficeScan Client\TmXPFlt.sys

    Image name: TmXPFlt.sys

    Timestamp:        Sat Aug 30 14:11:38 2014 (5401CD8A)

    CheckSum:         0005DDB6

    ImageSize:        0006C000

    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

3: kd> lmv m TMEBC64

start             end                 module name

fffff801`b101b000 fffff801`b102a000   TMEBC64    (deferred)            

    Image path: \SystemRoot\system32\DRIVERS\TMEBC64.sys

    Image name: TMEBC64.sys

    Timestamp:        Mon Jul 01 14:02:09 2013 (51D17DD1)

    CheckSum:         00019AD9

    ImageSize:        0000F000

    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

3: kd> lmv m TmPreFlt

start             end                 module name

fffff801`b299d000 fffff801`b29ad000   TmPreFlt   (deferred)            

    Image path: \??\C:\Program Files (x86)\Trend Micro\OfficeScan Client\TmPreFlt.sys

    Image name: TmPreFlt.sys

    Timestamp:        Sat Aug 30 14:11:25 2014 (5401CD7D)

    CheckSum:         0001438F

    ImageSize:        00010000

    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

3: kd> lmvm TmPreFlt

... all loaded modules;

3: kd> lm

start             end                 module name

fffff800`041b6000 fffff800`041bf000   kd         (deferred)            

fffff800`05210000 fffff800`05999000   nt         (pdb symbols) c:\symbols\ntkrnlmp.pdb\6E60CE642F39465DAF09219706DE11471\ntkrnlmp.pdb

fffff800`05999000 fffff800`05a09000   hal        (deferred)            

fffff801`b0e00000 fffff801`b0e88000   CI         (deferred)            

fffff801`b0e8d000 fffff801`b0ef3000 mcupdate_GenuineIntel (deferred)            

fffff801`b0ef3000 fffff801`b0f01000   werkernel   (deferred)            

fffff801`b0f01000 fffff801`b0f62000   CLFS       (deferred)            

fffff801`b0f62000 fffff801`b0f84000   tm         (deferred)            

fffff801`b0f84000 fffff801`b0f99000   PSHED      (deferred)            

fffff801`b0f99000 fffff801`b0fa3000   BOOTVID    (deferred)            

fffff801`b0fa3000 fffff801`b1000000   msrpc (deferred)            

fffff801`b1000000 fffff801`b101b000   mountmgr   (deferred)            

fffff801`b101b000 fffff801`b102a000   TMEBC64    (deferred)            

fffff801`b102a000 fffff801`b1034000   atapi      (deferred)            

fffff801`b105f000 fffff801`b112e000   Wdf01000   (deferred)            

fffff801`b112e000 fffff801`b113f000   WDFLDR     (deferred)            

fffff801`b113f000 fffff801`b1157000   acpiex     (deferred)            

fffff801`b1157000 fffff801`b1162000   WppRecorder (deferred)            

fffff801`b1162000 fffff801`b11ec000   ACPI       (deferred)            

fffff801`b11ec000 fffff801`b11f6000   WMILIB     (deferred)            

fffff801`b1200000 fffff801`b125f000   volmgrx    (deferred)

fffff801`b125f000 fffff801`b126e000   PCIIDEX    (deferred)            

fffff801`b1274000 fffff801`b1300000   cng        (deferred)            

fffff801`b131b000 fffff801`b1337000   pdc        (deferred)            

fffff801`b1337000 fffff801`b134f000   partmgr (deferred)            

fffff801`b134f000 fffff801`b13b8000   spaceport   (deferred)            

fffff801`b13b8000 fffff801`b13cd000   volmgr     (deferred)            

fffff801`b13cd000 fffff801`b13e6000   vmci       (deferred)            

fffff801`b13e6000 fffff801`b13fc000   vsock      (deferred)            

fffff801`b1400000 fffff801`b1478000   NETIO      (deferred)            

fffff801`b1478000 fffff801`b1482000   msisadrv (deferred)            

fffff801`b1482000 fffff801`b14ca000   pci        (deferred)            

fffff801`b14ca000 fffff801`b14d7000   vdrvroot   (deferred)            

fffff801`b14d7000 fffff801`b14e0000   intelide   (deferred)            

fffff801`b14e5000 fffff801`b15fd000   NDIS       (deferred)            

fffff801`b1600000 fffff801`b164c000   netbt      (deferred)            

fffff801`b167a000 fffff801`b16af000   ataport    (deferred)            

fffff801`b16af000 fffff801`b16cc000   lsi_sas    (deferred)            

fffff801`b16cc000 fffff801`b172b000   storport   (deferred)            

fffff801`b172b000 fffff801`b1787000   fltmgr     (deferred)            

fffff801`b1787000 fffff801`b17b5000   quota      (deferred)            

fffff801`b17b5000 fffff801`b17d7000   datascrn   (deferred)            

fffff801`b17d7000 fffff801`b17e5000   cbafilt    (deferred)            

fffff801`b1800000 fffff801`b1861000   dxgmms1    (deferred)            

fffff801`b1861000 fffff801`b1a57000   Ntfs       (deferred)            

fffff801`b1a57000 fffff801`b1a73000   ksecdd     (deferred)            

fffff801`b1a73000 fffff801`b1a83000   pcw        (deferred)            

fffff801`b1a83000 fffff801`b1a8e000   Fs_Rec     (deferred)            

fffff801`b1a8e000 fffff801`b1abf000   ksecpkg    (deferred)            

fffff801`b1abf000 fffff801`b1b14000   CLASSPNP   (deferred)            

fffff801`b1b14000 fffff801`b1b42000   cdrom      (deferred)            

fffff801`b1b42000 fffff801`b1bb8000   dedup      (deferred)            

fffff801`b1bb8000 fffff801`b1bcf000   ahcache    (deferred)            

fffff801`b1c00000 fffff801`b1c4f000   volsnap    (deferred)            

fffff801`b1c4f000 fffff801`b1c66000   mup        (deferred)            

fffff801`b1c66000 fffff801`b1c72000   ndistapi   (deferred)            

fffff801`b1c72000 fffff801`b1c8e000   disk       (deferred)            

fffff801`b1c8e000 fffff801`b1ca3000   crashdmp   (deferred)            

fffff801`b1ccc000 fffff801`b1cd5000   Null (deferred)            

fffff801`b1cda000 fffff801`b1f4e000   tcpip      (deferred)            

fffff801`b1f4e000 fffff801`b1fba000   fwpkclnt   (deferred)            

fffff801`b1fba000 fffff801`b1fdf000   wfplwfs    (deferred)            

fffff801`b1fdf000 fffff801`b1fed000   BasicRender   (deferred)            

fffff801`b1fed000 fffff801`b1ff8000   ws2ifsl    (deferred)            

fffff801`b2200000 fffff801`b2220000   tdx        (deferred)            

fffff801`b2229000 fffff801`b23aa000   dxgkrnl (deferred)            

fffff801`b23aa000 fffff801`b23bc000   watchdog   (deferred)            

fffff801`b23bc000 fffff801`b23ce000   BasicDisplay   (deferred)            

fffff801`b23ce000 fffff801`b23e2000   Npfs       (deferred)            

fffff801`b23e2000 fffff801`b23ee000   Msfs       (deferred)            

fffff801`b23ee000 fffff801`b23fc000   TDI        (deferred)            

fffff801`b2400000 fffff801`b240c000   mssmbios   (deferred)            

fffff801`b240c000 fffff801`b2432000   dfsc (deferred)            

fffff801`b2439000 fffff801`b24cb000   afd        (deferred)            

fffff801`b24cb000 fffff801`b24f5000   pacer      (deferred)            

fffff801`b24f5000 fffff801`b2506000   netbios    (deferred)            

fffff801`b2506000 fffff801`b2553000   tmcomm     (deferred)            

fffff801`b2553000 fffff801`b25c3000   rdbss      (deferred)            

fffff801`b25c3000 fffff801`b25dc000   wanarp     (deferred)            

fffff801`b25dc000 fffff801`b25ea000   nsiproxy (deferred)            

fffff801`b25ea000 fffff801`b25f6000   npsvctrig   (deferred)            

fffff801`b2600000 fffff801`b2606380   CmBatt     (deferred)            

fffff801`b2607000 fffff801`b2613000   BATTC      (deferred)            

fffff801`b2613000 fffff801`b2631000   intelppm   (deferred)            

fffff801`b2631000 fffff801`b263b000   pnpmem     (deferred)            

fffff801`b263b000 fffff801`b265c000   raspptp    (deferred)            

fffff801`b265c000 fffff801`b2680000   rasl2tp    (deferred)            

fffff801`b2680000 fffff801`b268b000   NdisVirtualBus   (deferred)            

fffff801`b268b000 fffff801`b26a6000   raspppoe   (deferred)            

fffff801`b26a6000 fffff801`b26e3000   ndiswan (deferred)            

fffff801`b26e3000 fffff801`b2700000   rassstp    (deferred)            

fffff801`b2700000 fffff801`b271f000   AgileVpn   (deferred)            

fffff801`b271f000 fffff801`b272e000   CompositeBus   (deferred)            

fffff801`b272e000 fffff801`b2739000   kdnic      (deferred)            

fffff801`b2739000 fffff801`b274a000   umbus      (deferred)            

fffff801`b274a000 fffff801`b2769000   i8042prt   (deferred)            

fffff801`b2769000 fffff801`b2779000   kbdclass (deferred)            

fffff801`b2779000 fffff801`b2781000   vmmouse    (deferred)            

fffff801`b2781000 fffff801`b2791000   mouclass   (deferred)            

fffff801`b2791000 fffff801`b27dc000   vm3dmp     (deferred)            

fffff801`b27dc000 fffff801`b27f3000   vmxnet3n61x64   (deferred)            

fffff801`b27f3000 fffff801`b27fd000   vmgencounter   (deferred)            

fffff801`b27fd000 fffff801`b27fe600   swenum     (deferred)            

fffff801`b2800000 fffff801`b28a9000   peauth (deferred)            

fffff801`b28ca000 fffff801`b2918000   ks         (deferred)            

fffff801`b2918000 fffff801`b2923000   rdpbus     (deferred)            

fffff801`b2923000 fffff801`b293a000   NDProxy    (deferred)            

fffff801`b293a000 fffff801`b2946000   dump_diskdump   (deferred)            

fffff801`b2946000 fffff801`b2963000   dump_LSI_SAS   (deferred)            

fffff801`b2963000 fffff801`b296af00   HIDPARSE   (deferred)            

fffff801`b296b000 fffff801`b2979000   monitor    (deferred)            

fffff801`b2979000 fffff801`b299d000   luafv      (deferred)            

fffff801`b299d000 fffff801`b29ad000   TmPreFlt   (deferred)            

fffff801`b2a00000 fffff801`b2a39000   mrxsmb20   (deferred)            

fffff801`b2a59000 fffff801`b2c91000   VSApiNt    (deferred)            

fffff801`b2c91000 fffff801`b2cfd000   TmXPFlt    (deferred)            

fffff801`b2cfd000 fffff801`b2d11000   lltdio (deferred)            

fffff801`b2d11000 fffff801`b2d29000   rspndr     (deferred)            

fffff801`b2d29000 fffff801`b2d49000   bowser     (deferred)            

fffff801`b2d49000 fffff801`b2d60000   mpsdrv     (deferred)            

fffff801`b2d60000 fffff801`b2dcc000   mrxsmb     (deferred)            

fffff801`b2e00000 fffff801`b2e09000   vmmemctl   (deferred)            

fffff801`b2e09000 fffff801`b2e54000   mrxsmb10   (deferred)            

fffff801`b2e54000 fffff801`b2e64000   condrv     (deferred)            

fffff801`b2e64000 fffff801`b2e6f000   secdrv     (deferred)            

fffff801`b2e6f000 fffff801`b2eb2000   srvnet     (deferred)            

fffff801`b2eb2000 fffff801`b2ec4000   tcpipreg   (deferred)            

fffff801`b2efe000 fffff801`b2ff8000   HTTP       (deferred)            

fffff801`b3038000 fffff801`b30e4000   srv2       (deferred)            

fffff801`b30e4000 fffff801`b3172000   srv        (deferred)            

fffff801`b3172000 fffff801`b319f000   tunnel     (deferred)            

fffff801`b319f000 fffff801`b31aa000   rdpvideominiport   (deferred)            

fffff801`b31aa000 fffff801`b31de000   rdpdr      (deferred)            

fffff801`b31de000 fffff801`b31eb000   terminpt   (deferred)            

fffff960`0014b000 fffff960`0055a000   win32k     (deferred)            

fffff960`006cd000 fffff960`006d6000   TSDDD      (deferred)            

fffff960`00838000 fffff960`00873000   cdd        (deferred)            

Unloaded modules:

fffff801`b301b000 fffff801`b3024000   cpuz136_x64.

fffff801`b3012000 fffff801`b301b000   cpuz136_x64.

fffff801`b3009000 fffff801`b3012000   cpuz136_x64.

fffff801`b3000000 fffff801`b3009000   cpuz136_x64.

fffff801`b31f4000 fffff801`b31fd000   cpuz136_x64.

fffff801`b31eb000 fffff801`b31f4000   cpuz136_x64.

fffff801`b1ca3000 fffff801`b1caf000   dump_storpor

fffff801`b1caf000 fffff801`b1ccc000   dump_LSI_SAS

fffff801`b1c66000 fffff801`b1c72000   hwpolicy.sys

fffff801`b1300000 fffff801`b131b000   sacdrv.sys

3: kd> lm TmXPFlt

Cheers,

Jon

[Roland_Schauer]

@Jon

oh yes, I have full memory dumps... I'll prepare a download for you tomorrow (about 2GB rar file)

@cos

Currently build 11.0.1454 (11 + patch 1) is installed, as far as I know there's nothing new online yet

Same problem with version 10.6 SP3

@alistar

As the newest version of TM OfficeScan is installed, it'll be difficult to get the new driver, because it's part of the suite. But of course I'll check tomorrow.

so thanks again for all your help and good night!

I'll get back as soon as possible.

best regards

Roland

[Alistar]

Hi Jon,

I have checked out devidmiatello's crash dump and these two modules have collided with each other. This was not so clear in Roland's dump, but as I pointed out earlier, the stack trace is exactly the same in both cases, and these modules were also found on both of the guys' dump. So there is definitely something interfering with the security software there.

Just a long shot - do you guys have VMCI or vSHIELD enabled or a driver installed in VMware tools? I'd go with their clean, minimal install with just the most necessary drivers. Also what about storage controllers, are you using paravirtual on your fileservers compared to the rest of your environment? I've noticed these in loaded modules as well.

[Roland_Schauer]

Hi Alistar,

I've tried with and without VMCI driver, didn't change anything. vShield has never been installed in my production environment.

Currently this fileserver is running LSI Logic SAS storage controller, but I've also tried the Paravirtual SCSI Controller. Actually my other fileservers are still on Paravirtual Controller (as almost all virtual servers in our environment).

Trend Micro seemed to have some problems with the Early Boot Clean Driver in 10.6, but it should be possible to disable TMEBC, so I'll test this next time. (I would like to run the server with 1 vCPU at least a week to see if this helps already)

Do you have a mail address via pm for me?

@devidmiatello:

For my interest: Has your faulty server configured multiple vCPUs?

thanks again!

best regards

Roland

[Roland_Schauer]

Hi Jon,

you should already have received mail at jonmunday.net with some information for downloading the dump.

thanks a lot!

best regards

Roland

[commerzbank]

Thanks, downloading it now :smileyhappy:

[Roland_Schauer]

Currently it seems that it has something to do with multiple vCPUs (and Trend Micro OfficeScan or any VMware driver). The server has been running since 2014-12-17 without any crash after reconfiguring to 1 vCPU.

I'll check again next week.

[Roland_Schauer]

Hey guys!

as I'm back in Austria again, I have some new information: unfortunately reducing vCPUs didn't help much, so I decided to uninstall OfficeScan for further testing.

Waiting for the next BSOD...

best regards

Roland

[Dany_Bouchard]

Hello Roland

Did you experience any BSOD since you removed TrendMicro?

I too have the same issue with a new file server with DFS running Windows 2012 R2 with Trendmicro 10.6. (2 BSOD in 24 hours)

thanks

[Roland_Schauer]

Hi Dany!

The server has been running without Officescan about 4 weeks without any BSOD.

5 weeks ago I've installed same version of OfficeScan again to get things clear, but no BSOD too. Maybe any Windows update fixed the issue in the meanwhile.

best regards

Roland