NSE not processing

[TeleFragger]

Ok everything was working great friday and came in this morning. Noticed Automation Reports did not go this morning.

checked server and when I go into log viewer I see nothing but the following error... also queue was full so I moved them all out to clear it.. stuff is going back in slowly and not leaving... we rebooted both sql and ns servers...

 

any ideas?

 

 

Log File Name: C:\ProgramData\Symantec\SMP\Logs\a.log

Priority: 2

Date: 3/17/2014 9:40:33 AM

Tick Count: 3685040

Host Name: MyServer

Process: AeXSvc (8132)

Thread ID: 78

Module: AeXSVC.exe

Source: Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatch

Description: Failed to process event, will retry. Attempt: 2

**CEDUrlStart** :http://entced.symantec.com/entt?product=SMP&version=7.1.8400.0&language=en&module=euBqK19WCe54FvBZsBlgjhGXQ8JBeUVjF6DNDyVctJN1pIZUANooXpr85sGhbxcn&error=120685716&build=**CEDUrlEnd**

 

 

( Exception Details: System.IO.FileNotFoundException: Could not find file 'C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue\302d90e0-7be1-4197-82d8-2ef54bf380f0.nse'.

File name: 'C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue\302d90e0-7be1-4197-82d8-2ef54bf380f0.nse'

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)

   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)

   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)

   at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize)

   at System.IO.StreamReader..ctor(String path, Boolean detectEncodingFromByteOrderMarks)

   at Altiris.NS.ClientMessaging.EventQueueDispatcher.ParseMessageHeader(String message, Boolean useFilename, Guid& wrkstaGuid, String& sWrkstaName, String& ScenarioGuid)

   at Altiris.NS.ClientMessaging.EventQueueDispatcher.Dispatch(String message, Boolean useFilename)

   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatch(DispatchSlot slot) )

( Exception logged from:

   at Altiris.Diagnostics.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception)

   at Altiris.NS.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception)

   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatch(DispatchSlot slot)

   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchSingleNSE(DispatchSlot slot)

   at Altiris.NS.ClientMessaging.EventQueueDispatcher.WorkerLoop(Object deliverySlot)

   at System.Threading.ThreadHelper.ThreadStart(Object obj)

 )

( Extra Details:  Type=System.IO.FileNotFoundException Src=mscorlib )

 

[Migration User]

Hi,

Please try the solution described in this KB:

http://www.symantec.com/business/support/index?page=content&id=TECH183959

[ssv1984]

Hi Telefragger, Hope this might help you, NS 7.1 SP2 www.symantec.com/docs/TECH183347 NS 7.5 www.symantec.com/docs/TECH211788 Check with the solution provided in this : http://www.symantec.com/connect/forums/ns-event-queue-evtqfast-full-returning-busy To check with the evnt queue status : http://www.symantec.com/connect/blogs/how-check-ns-various-event-queue-statuses-once How Event Queues in ITMS 7.1 SP2 get processed : www.symantec.com/docs/DOC5480 Regards, Siva

[TeleFragger]

have to take my kid to school - 2hr snow delay.. but will check them out when i get back. Some do look familiar... i always search a lot b4 posting... but appreciate the replies..

 

more to come...

Jeff

[TeleFragger]

I already had it set to 4000000... 

[Migration User]

I faced the similar problem in my environment regarding NSE and for that my NS console became very slow. What I did, I remove all NSE and  I exclude the folder "'C:\ProgramData\Symantec\SMP\EventQueue" from Antivirus autoprotect scan and  then restart the server. After that my problem got resolved.

[RyanSpooner]

I too am having a similar problem today.

Check why active directory imports were failing, and found the queue was full. This is already configured for the 4000000 max size.  So I stopped all the services, moved all files out of EvtQueue into a bad folder on the root of C:.  I then also added an exception to the VirusScan console for the whole c:\programdata\symantec folder and subfolders. I then started the services and all seemed fine.

However then my Log Viewer started getting messages like the below. When I started from blank Queue folders, why is it now complaining about not being able to find files in the queue folder?

Note that my original problem was slow processing of nse files from the queue, now though it's complaining about file snot being found? How can I clear out the queues and make SMP start event processing from scratch? I'm wondering if it's trying to find the files from all the nse's that I moved out?

Log File Name: C:\ProgramData\Symantec\SMP\Logs\a.log

Priority: 1

Help and Support: 

Date: 18/03/2014 14:13:27

Tick Count: 1101559797

Host Name: ROCVSMPNS01

Process: AeXSvc (3380)

Thread ID: 41

Module: AeXSVC.exe

Source: Altiris.NS.ClientMessaging.EventQueueDispatcher.BackupEvent

Description: Unexpected exception while trying to copy processed event to the event copy path. Context: EventFailureBackupFolder

 

 

 

( Exception Details: System.IO.FileNotFoundException: Could not find file 'C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue\5c384b49-7c34-43a2-962d-5ea660ca8d6c.nse'.

File name: 'C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue\5c384b49-7c34-43a2-962d-5ea660ca8d6c.nse'

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

   at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite)

   at Altiris.NS.ClientMessaging.EventQueueDispatcher.BackupEvent(DispatchSlot slot, String coreSettingFolder, String subDirectory, String logMessage) )

( Exception logged from:

   at Altiris.Diagnostics.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception)

   at Altiris.NS.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception)

   at Altiris.NS.ClientMessaging.EventQueueDispatcher.BackupEvent(DispatchSlot slot, String coreSettingFolder, String subDirectory, String logMessage)

   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatch(DispatchSlot slot)

   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchSingleNSE(DispatchSlot slot)

   at Altiris.NS.ClientMessaging.EventQueueDispatcher.WorkerLoop(Object deliverySlot)

   at System.Threading.ThreadHelper.ThreadStart(Object obj)

 )

( Extra Details:  Type=System.IO.FileNotFoundException Src=mscorlib )

 

 

[TeleFragger]

i moved mine out... restarted servers both SQL first.. then Altiris...

once fully up.. moved everything back in..

56k items in 

3k went to bad

2 days later.. all processed and moved bad back in... now all done... and back up and running..

i need to figure out why this happened in the first place. Our full inventories are at the beginning of the month so wasnt that... 

[Igor Perevozchikov]

As far as I remember, I had similar case.

If user will cut all available .nse files from EventQueue folder, while Altiris Client Message Dispatcher, Altiris File Receiver are running and then user will restart these services (Without putting all NSE back prior to services restart), then errors will appear in log as you've mentioned above.

Could not find file 'C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue\302d90e0-7be1-4197-82d8-2ef54bf380f0.nse'.

After restart of these services and putting all nse back, they should be processed.

But why you had fullfilled EventQueue folder, this is another question. Large amount of NSe with big file size? AD Import was done with large amount of imported data from Active Directory? Or Package Server has sent all package statuses to NS?

[TeleFragger]

got a full folder again... ughhhh...

still trying to figure out root cause...

[Igor Perevozchikov]

Check when "AD Import" import rules are scheduled to run

[TeleFragger]

we only do an AD import on a few items...  we do not import in all users or computers...

 

[Igor Perevozchikov]

OK. Then copy one of available .nse file from \EventQueue folder\ and open copied NSE to see from what it relates to.

[TeleFragger]

Same as Ryan.. hmmm.. gonna copy them out.. restart services and drop them back in and see..

I am seeing tons of errors.. and one looks like...

 

Log File Name: C:\ProgramData\Symantec\SMP\Logs\a.log
Priority: 1
Help and Support: 
Date: 3/31/2014 1:09:11 PM
Tick Count: 440068212
Host Name: MyServer
Process: AeXSvc (1360)
Thread ID: 120
Module: AeXSVC.exe
Source: Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatch
Description: Failed to process event.

 

( Exception Details: System.IO.FileNotFoundException: Could not find file 'C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue\01135334-3c5f-4b66-8b54-69b1ca2a0de0.nse'.
File name: 'C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue\01135334-3c5f-4b66-8b54-69b1ca2a0de0.nse'
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize)
   at System.IO.StreamReader..ctor(String path, Boolean detectEncodingFromByteOrderMarks)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.ParseMessageHeader(String message, Boolean useFilename, Guid& wrkstaGuid, String& sWrkstaName, String& ScenarioGuid)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.Dispatch(String message, Boolean useFilename)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatch(DispatchSlot slot) )
( Exception logged from:
   at Altiris.Diagnostics.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception)
   at Altiris.NS.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatch(DispatchSlot slot)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchSingleNSE(DispatchSlot slot)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.WorkerLoop(Object deliverySlot)
   at System.Threading.ThreadHelper.ThreadStart(Object obj)
 )
( Extra Details:  Type=System.IO.FileNotFoundException Src=mscorlib )

 

[Igor Perevozchikov]

Did you copy a single .NSE and open it, to determine where from this is NSE was generated? (Otherwise what solution/plug-in/process generates NSE in such large amount?)

If you could determine who does it, then it will be better to find a way to balance such generation of NSE and avoid further \EventQueue\ folder fulfilling.

[Gavin Sanish]

If you are using 7.1 SP2 MP1.1 you can install v9 which contain the Fixes for the NSE issues.

--Slow full inventory NSE processing affects Basic Inventory processing. 

--Software Inventory is appending "00" to two digit year install dates on 
Windows Patches, thus the NSEs fail to process with error CommitBatch:
Couldn't update table dbo.[Inv_SW_Patch_Windows]..

--Symantec Management Agent should not send all locally queued NSE files 
if the first one is unable to be sent

--Agents outside internal networks not sending inventory NSE's correctly.

These are the issues fixed in previous roll ups 

--FlushAgentEvents registry key stops NSEs from processing

--AD-Import NSE fails to process. The following error message is displayed: 
CommitBatch: Couldn't update table dbo.Evt\_Resource\_Merge.

--Specific NSE files from clients are not processed. These files become stale 
and are never processed.

Please check the article 

http://www.symantec.com/docs/HOWTO81832