Lsass.exe is stuck waiting in Symantec Drivers

[Rizwan Shaikh]

HI team,

All our environment servers were not accessible post regular updates from Symantec server.

For resolution we only have to restart the server. and after every update issue reoccurs on srver.Post anaylsys from Azure memory dump it shows below error

"

Lsass threads are stuck waiting in Symanted drivers

0: kd> !mex.t ffffe0001ec11080

Process                      Thread           CID       UserTime KernelTime ContextSwitches Wait Reason         Time State

lsass.exe (ffffe00011b2b900) ffffe0001ec11080 2a0.16b0      31ms         0s              69 WrPushLock  1h:26:59.468 Waiting"

As for now we have uninstall Symantec from all the servers as it was causing downtime. Request to please help us with guidance what is the above error and how we can resolve it.

 

Regards,

RIzwan

[ℬrίαη]

Exact version of SEP? I'd suggest getting a case opened up now if this is critical.

[Christopher Clark]

We also have the same issue on Azure VM's using SEP RU1,  we have a case open and are waiting on the full memory DMP from Microsoft so we can upload it to the case.

 

The data indicates we are running in to an issue that we have seen before where a 3^rd party component tried to thread synchronize in a process notify routine. This leads the system in to a dead lock where the only recovery option is to reboot. I recommend engaging Symantec for an updated version of the software.

 

0: kd> !di

Dump Name: memory.dmp

Computer Name: xxxxxxxxxxxxxxx

Windows 8.1 Kernel Version 9600 MP (2 procs) Free x64

Product: WinNt

Built by: 9600.18969.amd64fre.winblue_ltsb.180309-0600

Kernel base = 0xfffff801`aea12000 PsLoadedModuleList = 0xfffff801`aecde570

System Uptime: 2 days 2:47:55.151

SystemManufacturer = Microsoft Corporation

SystemProductName = Virtual Machine

Processor: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz

Keyboard Bugcheck: E2

Kernel Complete Dump File: Full address space is available.

 

Lots of threads stuck waiting on LPC replies.

 

0: kd> !tl -t

PID            Address          Name                    Rn Ry Rc IO Er Lc Mu Ps Th Su Pl Bk Tr In !!

============== ================ ======================= == == == == == == == == == == == == == == ==

0x0    0n0     fffff801aed61300 Idle                     2  .  .  .  .  .  .  .  .  .  .  .  .  .  .

0x4    0n4     ffffe0011c8405c0 System                   .  .  .  .  .  3  .  .  .  .  2  .  .  .  .

0x278  0n632   ffffe0011e146900 lsass.exe                .  .  .  .  .  .  .  .  .  .  2  .  .  .  .

0x37c  0n892   ffffe0011e4b2080 svchost.exe              .  .  .  .  .  2  .  .  .  .  .  .  .  .  .

0x3b4  0n948   ffffe0011e4db080 svchost.exe              .  .  .  .  .  5  .  .  .  .  .  .  .  .  .

0x520  0n1312  ffffe0011e5bd900 avagent.exe              .  .  .  .  .  .  .  .  1  .  .  .  .  .  .

0x558  0n1368  ffffe0011e76e080 svchost.exe              .  .  .  .  .  1  .  .  .  .  .  .  .  .  .

0x56c  0n1388  ffffe0011e7a1580 HealthService.exe        .  .  .  .  .  .  .  .  .  .  5  .  .  .  .

0x600  0n1536  ffffe0011e7df900 WaAppAgent.exe           .  .  .  .  .  1  .  .  .  .  .  .  .  .  .

0x660  0n1632  ffffe0011cac5340 ccSvcHst.exe             .  .  .  .  .  1  2  .  .  .  2  .  .  .  .

0x6d0  0n1744  ffffe0011e7d1900 svchost.exe              .  .  .  .  .  1  .  .  .  .  .  .  .  .  .

0xb6c  0n2924  ffffe0011e673900 WmiPrvSE.exe             .  .  .  .  .  1  .  .  .  .  .  .  .  .  .

0xde0  0n3552  ffffe0011eaf2900 MonitoringHost.exe       .  .  .  .  .  .  .  1  .  .  .  .  .  .  .

0x27fc 0n10236 ffffe00127d36840 cscript.exe              .  .  .  .  .  .  .  .  .  .  .  .  .  1  .

0x1560 0n5472  ffffe00127130080 cmd.exe                  .  .  .  .  .  .  .  .  .  .  .  .  .  1  .

0x2ff8 0n12280 ffffe00126968300 avupdate.exe             .  .  .  .  .  .  .  .  .  .  .  .  .  1  .

0x1e58 0n7768  ffffe00123c7b880 sppsvc.exe               .  .  .  .  .  .  .  .  .  .  .  .  .  1  .

0x54   0n84    ffffe00127a6a900 CollectGuestLogs.exe     .  .  .  .  .  .  .  .  .  .  .  .  .  1  .

0x1d90 0n7568  ffffe00127c5e0c0 taskhostex.exe           .  .  .  .  .  .  .  .  .  .  .  .  .  1  .

0x1d3c 0n7484  ffffe00126546900 Ccm32BitLauncher.exe     .  .  .  .  .  .  .  .  .  .  .  .  .  1  .

0x1cb4 0n7348  ffffe00125fe21c0 cscript.exe              .  .  .  .  .  .  .  .  .  .  .  .  .  1  .

0x21e0 0n8672  ffffe001279a4900 appidcertstorecheck.exe  .  .  .  .  .  .  .  .  .  .  .  .  .  1  .

============== ================ ======================= == == == == == == == == == == == == == == ==

PID            Address          Name                     2  0  0  0  0 15  2  1  1  0 11  0  0  9  0

 

Warning! Zombie process(es) detected (not displayed). Count: 2 [zombie report]

 

The below is not allowed. You cannot thread sync in a process notify routine. See this url for more information on whats allowed inside of these routines : https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/windows-kernel-mode-process-and-thread-manager

 

 

0: kd> !us -k notify

Kernel Summary

============================================================

2 threads [stats]: ffffe0011e89f080 ffffe0011fae6600

    fffff801aeb6a286 nt!KiSwapContext+0x76

    fffff801aea87bae nt!KiSwapThread+0x14e

    fffff801aea87629 nt!KiCommitThreadWait+0x129

    fffff801aea871f5 nt!KeWaitForMultipleObjects+0x9a5

    fffff800c6b29ec9 SRTSP64+0xbdec9

    fffff800c6b277e5 SRTSP64+0xbb7e5

    fffff800c6ad164c SRTSP64+0x6564c

    fffff800c6ac6f11 SRTSP64+0x5af11

    fffff800c6b074b8 SRTSP64+0x9b4b8

    fffff800c6a3ade8 symefasi+0x176de8

    fffff800c6a40e7d symefasi+0x17ce7d

    fffff800c6a402fa symefasi+0x17c2fa

    fffff800c6a3f7a4 symefasi+0x17b7a4

    fffff800c7b07eea IDSvia64+0x34eea

    fffff800c7b078fb IDSvia64+0x348fb

    fffff800c7b076f2 IDSvia64+0x346f2

    fffff800c7b0f326 IDSvia64+0x3c326

    fffff800c7b0f148 IDSvia64+0x3c148

    fffff800c7b15429 IDSvia64+0x42429

    fffff800c7b15b16 IDSvia64+0x42b16

    fffff800c7b1669e IDSvia64+0x4369e

    fffff800c6c0df41 SYMEVENT64x86+0xdf41

    fffff800c6c0e1cc SYMEVENT64x86+0xe1cc

    fffff800c6c0e33e SYMEVENT64x86+0xe33e

    fffff800c6c07d26 SYMEVENT64x86+0x7d26

    fffff800c6c0b041 SYMEVENT64x86+0xb041

    (Inline)         nt!PspCallProcessNotifyRoutines+0x188

    fffff801aeda1cd8 nt!PspInsertThread+0x80c

    fffff801aeddc9d6 nt!NtCreateUserProcess+0x806

    fffff801aeb79513 nt!KiSystemServiceCopyEnd+0x13

 

7 threads [stats]: ffffe0011fbcd080 ffffe00126b9e880 ffffe00126258340 ffffe00128052040 ffffe0012784a080 ffffe001267e5880 ffffe0011fa21880

    fffff801aeb6a286 nt!KiSwapContext+0x76

    fffff801aea87bae nt!KiSwapThread+0x14e

    fffff801aea87629 nt!KiCommitThreadWait+0x129

    fffff801aea871f5 nt!KeWaitForMultipleObjects+0x9a5

    fffff800c6b29ec9 SRTSP64+0xbdec9

    fffff800c6b277e5 SRTSP64+0xbb7e5

    fffff800c6ad164c SRTSP64+0x6564c

    fffff800c6ac6f11 SRTSP64+0x5af11

    fffff800c6b074b8 SRTSP64+0x9b4b8

    fffff800c6a3ade8 symefasi+0x176de8

    fffff800c6a40e7d symefasi+0x17ce7d

    fffff800c6a402fa symefasi+0x17c2fa

    fffff800c6a3f7a4 symefasi+0x17b7a4

    fffff800c6b4fd97 Ironx64+0x13d97

    fffff800c6b5016d Ironx64+0x1416d

    fffff800c6b4fb25 Ironx64+0x13b25

    fffff800c6b4e6e4 Ironx64+0x126e4

    fffff800c7c6e8ec BHDrvx64+0x168ec

    fffff800c7c87ca0 BHDrvx64+0x2fca0

    fffff800c7c701e5 BHDrvx64+0x181e5

    fffff800c7c80c0a BHDrvx64+0x28c0a

    fffff800c7c80a6d BHDrvx64+0x28a6d

    fffff800c7c633fe BHDrvx64+0xb3fe

    fffff800c7c62d7d BHDrvx64+0xad7d

    fffff800c7d118ec BHDrvx64+0xb98ec

    fffff800c7d117c9 BHDrvx64+0xb97c9

    fffff800c7d1203a BHDrvx64+0xba03a

    fffff800c6c0df41 SYMEVENT64x86+0xdf41

    fffff800c6c0e1cc SYMEVENT64x86+0xe1cc

    fffff800c6c0e33e SYMEVENT64x86+0xe33e

    fffff800c6c07d26 SYMEVENT64x86+0x7d26

    fffff800c6c0b041 SYMEVENT64x86+0xb041

    (Inline)         nt!PspCallProcessNotifyRoutines+0x188

    fffff801aeda1cd8 nt!PspInsertThread+0x80c

    fffff801aeddc9d6 nt!NtCreateUserProcess+0x806

    fffff801aeb79513 nt!KiSystemServiceCopyEnd+0x13

 

Threads matching filter: 9 out of 835

[Rizwan Shaikh]

We are using Symantec 14 version .We do have a case with Symantec but yet the issue is not identified.Case # 14440092 (Sanofi EMS AO- Multiple Servers RDP issue : we faced issue in one of our CD servers and all websites stop working. We could only see Symantec updates being downloaded in the server during server is inaccessible.)