"Failed to Process NSE" Errors since upgrade to 7.6

[chrismcevoy72]

I'm a little concerned about the number of "Failed to Process NSE" errors that are appearing in the NS logs since we upgraded to 7.6 HF7 (from 7.5) last week.  The errors all report the same fault.  I've checked the NSE folders on the server and they are all empty (apart from the odd event appearing and then disappearing in the queue).  However, the errors seem to be occuring every minute (with 3 failed attempts each time).  I've not noticed any problems in operation or from the console or client side but the amount of errors and their regularity is worry.  The logs show the following:

[3/3] Failed to process event: # 13942468, 965.00 B, inline
Failed to process NSE.
   [Altiris.NS.Exceptions.AeXException @ Altiris.NS.StandardItems]
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.OnMessageImpl(String message, Boolean useFilename)
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.OnMessageInternal(String message, Boolean useFilename)
   at Altiris.Deployment.Messaging.ILOInventoryCapture.OnMessage(String message)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchWithinContext(String message, Boolean useFilename, IItem item, IItemMessaging iim, MessageHeader header)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchWithinContext(Int32 eventQueueId, String message, Boolean useFilename, Int64 fileSize, Int64 eventId, MessageHeader header)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.TryDispatch(Int32 eventQueueId, String message, Boolean useFilename, Int64 fileSize, Int64 eventId)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatch(DispatchSlot slot, Boolean& fatal)

Exception of type 'Altiris.NS.TaskManagement.TaskStoppedException' was thrown.
   [Altiris.NS.TaskManagement.TaskStoppedException @ Altiris.NS]
   at Altiris.NS.ResourceManagement.MessagingResource.SaveImpl(Boolean allowUpdate)
   at Altiris.Database.DatabaseContext`1.RetryAction(Int32 retries, Boolean transacted, Getter`1 getContext, Action`1 action, Action`1 retry, DeadlockRetryArgs& re, Exception& exception, Boolean inTransaction)
   at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetryHelper(Int32 retries, Boolean inTransaction, Getter`1 getContext, Action`1 action, Action`1 retry, Boolean transacted)
   at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetry(Int32 retries, Boolean startNewTransaction, Nullable`1 isolationLevel, Boolean independentContext, Action`1 action, Action`1 retry)
   at Altiris.NS.ResourceManagement.MessagingResource.Save()
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.SeekResourceImport(String message, Boolean useFilename, String[] conditionBodyInventoryResources, XmlTextReader xmlReader, ArrayList parseStack, MessagingResource& fromResource, ResourceImporter& resourceImporter)
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.OnMessageImpl(String message, Boolean useFilename)

Exception logged from: 
   at Altiris.Diagnostics.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception, String footer)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.HandleException(DispatchSlot slot, Int32 i, Exception ex, String& file, Boolean& fileInline, Int32& retryLimit)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatch(DispatchSlot slot, Boolean& fatal)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.WorkerDispatchStep(DispatchSlot slot, IAutoCallContextSnapshot snap)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchSlotThreadProc(Object state)
   at System.Threading.ThreadHelper.ThreadStart(Object obj)

**CEDUrlStart** :http://entced.symantec.com/entt?product=SMP&version=7.6.1383.0&language=en&module=euBqK19WCe54FvBZsBlgjvQJ267PbV6Sgdiy6+3auzijphNgVSieNz92uK9ygPCSfkXVwdKDinbB/I5D2ba2rQ==&error=-298970188&build=**CEDUrlEnd**

-----------------------------------------------------------------------------------------------------
Date: 11/03/2016 10:17:20, Tick Count: 42037515 (11:40:37.5150000), Host Name: xxxxxx, Size: 3.45 KB
Process: AeXSvc (1268), Thread ID: 43, Module: AeXSVC.exe
Priority: 1, Source: EventQueueDispatcher
File: C:\ProgramData\Symantec\SMP\Logs\a.log

Can anyonce decipher that and point me to the problem?

[Igor Perevozchikov]

1. Do you have a Resource GUID to identify which client PC sent this nse file to NS? Was there any resource merge?

Check Log Viewer on NS, there should be information messages about merges, etc. And ResourceGuid in warning/error message.

2. "Legacy Agent Communication" option is enabled on "Notification Server settings" page?

3. Client agents and Site Servers are already upgraded to latest version of 7.6 HF7?

[chrismcevoy72]

Looks like I have a couple of blacklisted agents.  The lines in the logs preceeding each error point to this:

Policy request failed from: xxx.xxx.xxx.xxx
The specified host resource has been blacklisted
   [Altiris.NS.Exceptions.NSComException @ Altiris.NS]
   at Altiris.NS.AgentManagement.PolicyRequest.GetPoliciesCheckHost(CodeProfiler cp, ParseRequestResult result)
   at Altiris.NS.AgentManagement.PolicyRequest.GetPolicies(String request, Boolean bCompress, String& agentPoliciesXml)
   at Altiris.Web.NS.Agent.GetClientPoliciesHandler.ProcessRequest(HttpContext context)

COM Exception errcode: 0x80041003

Exception logged from: 
   at Altiris.Diagnostics.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception, String footer)
   at Altiris.Web.NS.Agent.GetClientPoliciesHandler.HandleNsComException(HttpContext ctx, NSComException ex)
   at Altiris.Web.NS.Agent.GetClientPoliciesHandler.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

**CEDUrlStart** :http://entced.symantec.com/entt?product=SMP&version=7.6.1383.0&language=en&module=WH8Wg9DRX4Gnw3Ks9xVldjXzeYzGddDrMuxrM5zOLQy8Frf5pIro01ThIi1fSjWR2av1P1cNRqIdVkHO4ghhqQ==&error=1699120817&build=**CEDUrlEnd**

-----------------------------------------------------------------------------------------------------
Date: 11/03/2016 14:52:52, Tick Count: 58569531 (16:16:09.5310000), Host Name: xxxxx, Size: 2.71 KB
Process: w3wp (3520), Thread ID: 15, Module: w3wp.exe
Priority: 1, Source: Altiris.Web.NS.Agent.GetClientPoliciesHandler.HandleNsComException
File: C:\ProgramData\Symantec\SMP\Logs\a21.log

and this

Attempt to save messaging resource 74aaf656-fafe-4411-a790-0d65c8f0fe48 has been blocked by the Host Resource Blacklist.
Exception of type 'Altiris.NS.TaskManagement.TaskStoppedException' was thrown.
   [Altiris.NS.TaskManagement.TaskStoppedException @ ]


Exception logged from: 
   at Altiris.Diagnostics.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception, String footer)
   at Altiris.NS.ResourceManagement.MessagingResource.SaveImpl(Boolean allowUpdate)
   at Altiris.Database.DatabaseContext`1.RetryAction(Int32 retries, Boolean transacted, Getter`1 getContext, Action`1 action, Action`1 retry, DeadlockRetryArgs& re, Exception& exception, Boolean inTransaction)
   at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetryHelper(Int32 retries, Boolean inTransaction, Getter`1 getContext, Action`1 action, Action`1 retry, Boolean transacted)
   at Altiris.Database.DatabaseContext`1.PerformWithDeadlockRetry(Int32 retries, Boolean startNewTransaction, Nullable`1 isolationLevel, Boolean independentContext, Action`1 action, Action`1 retry)
   at Altiris.NS.ResourceManagement.MessagingResource.Save()
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.SeekResourceImport(String message, Boolean useFilename, String[] conditionBodyInventoryResources, XmlTextReader xmlReader, ArrayList parseStack, MessagingResource& fromResource, ResourceImporter& resourceImporter)
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.OnMessageImpl(String message, Boolean useFilename)
   at Altiris.NS.StandardItems.Messaging.InventoryCaptureItem.OnMessageInternal(String message, Boolean useFilename)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchWithinContext(String message, Boolean useFilename, IItem item, IItemMessaging iim, MessageHeader header)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchWithinContext(Int32 eventQueueId, String message, Boolean useFilename, Int64 fileSize, Int64 eventId, MessageHeader header)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.TryDispatch(Int32 eventQueueId, String message, Boolean useFilename, Int64 fileSize, Int64 eventId)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.PerformDispatch(DispatchSlot slot, Boolean& fatal)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.WorkerDispatchStep(DispatchSlot slot, IAutoCallContextSnapshot snap)
   at Altiris.NS.ClientMessaging.EventQueueDispatcher.DispatchSlotThreadProc(Object state)
   at System.Threading.ThreadHelper.ThreadStart(Object obj)


-----------------------------------------------------------------------------------------------------
Date: 11/03/2016 14:52:53, Tick Count: 58570109 (16:16:10.1090000), Host Name: xxxxxx, Size: 2.73 KB
Process: AeXSvc (1268), Thread ID: 37, Module: AeXSVC.exe
Priority: 2, Source: MessagingResource
File: C:\ProgramData\Symantec\SMP\Logs\a21.log

 

I assume this is because of duplicate agent GUIDs?  Is there a way of resolving this via the console?

 

[Igor Perevozchikov]

To reset a GUID on managed client PC, you can manually execute this command line in cmd on client PC.

• "C:\Altiris Agent\AeXAgentUtil.exe" /ResetGuid

Or if you want to do this from SMP Console, then you should create a client task "Run script" task -> put there this cmd line and schedule this task to run on client PC.

[chrismcevoy72]

Thanks Igor.  Can you still run jobs and tasks on a client that has been blacklisted?  I assumed you couldn't...

 

[Igor Perevozchikov]

Maybe you've already checked this KB about Agent blacklisting

• https://support.symantec.com/en_US/article.HOWTO9748.html