Linux - Basic Inventory Not Sending - being held in /opt/altiris/notification/nsagent/var/queue

[Jeff Stanton]

Hello everyone!

We have encountered a strange problem with a significant number of our managed Linux devices.

I started to notice that our Linux devices were becoming "unmanaged".  They are updating the "Agent Communication" date as well as the "Configuration Requests" date but not the "Basic Inventory" date.  After our purging maintenance window closes on the devices (30 days) that hadn't sent a Basic Inventory, these devices became unmanaged/retired.  Upon investigation, we found that the /opt/altiris/notification/nsagent/var/queue directory was full of NSE files (sometimes hundreds, sometimes thousands).  On some devices we were able to mark the system as "Active", clear that directory's contents and successfully get the agent to update its Basic Inventory and thereby the device becomes "managed" again.  On others, this process did not work.  The NSEs continue to pile up once again.  It seems the next step would be to uninstall the agent and reinstall.  Since we currently have around 80 devices in this situation, I am hoping to avoid this if at all possible.  And, what is going to keep this from happening again?  The long-term is a greater concern...I don't want to go through this process again in one month, three months, one year, etc.

It seems there is something else that is keeping the system from sending the Basic Inventory.

Is there another file that should be deleted on the Linux client that might be holding those up?  Once we are able to update the configuration on these, they seem fine...I've fixed a handful using this method.

I currently have a case open, but since there are more added each day, I am trying to exhaust all support options.

Thanks in advance for your help!

[Laurence Shackleton]

Hi Jeff. Just a couple of days back I've noticed the exact same thing happening across a large portion of our Windows fleet too. I haven't raised a case yet as it's a mad time of year but I will ASAP. Keep us posted! Maybe you can message me your case number so I can reference it when I raise one? It looks like ours issues started in mid-late August - attached an example screenshot.

[Igor Perevozchikov]

Hi All!

1. There is a similar problem for Agent Health summary report and is fixed in upcoming ITMS 8.7.1 release https://knowledge.broadcom.com/external/article/272134

2. @Jeff Stanton:/opt/altiris/notification/nsagent/var/queue directory was full of NSE files (sometimes hundreds, sometimes thousands).

Do you have any Agent blockout enabled? 
As example if there is a total blockout, then Agent doesn't send data to Notification Server and keeps all these nse files locally. 

INFO 2023-09-14 13:44:05.381      8389  140303767426624  CTA-comm > Communication to the server (https://IGOR:443/Altiris/ClientTaskServer/GetTasks.aspx?crc=0008000700000835&isInMaintenanceWindow=false&resourceGuid={F2D143D3-3636-473C-BF18-49835F02E352}&sessionId={4BCD7BCF-FA48-4497-8C88-81C4FABFF5ED}) aborted due to an active network blockout.
INFO 2023-09-14 13:44:32.480      8389  140303752214080  INV-Session > Finished instance enumeration: code = 0, msg = Success
INFO 2023-09-14 13:45:00.439      8389  140303766734400  SWUAgent-AgentObject > Housekeeping schedule fired
INFO 2023-09-14 13:45:05.448      8389  140303767426624  CTA-comm > Communication to the server (https://IGOR:443/Altiris/ClientTaskServer/GetTasks.aspx?crc=0008000700000835&isInMaintenanceWindow=false&resourceGuid={F2D143D3-3636-473C-BF18-49835F02E352}&sessionId={4BCD7BCF-FA48-4497-8C88-81C4FABFF5ED}) aborted due to an active network blockout.
INFO 2023-09-14 13:45:08.011      8389  140303752214080  SMF-SoftwareCache > Sending information about known software to NS...
WARN 2023-09-14 13:45:08.029      8389  140303763498560  EventQueue > TrySendEvent(): Unable to send queued event /opt/altiris/notification/nsagent/var/queue/200.000000000169468818500001.nse: 2165768228,  Total blockout is enabled.
INFO 2023-09-14 13:45:08.938      8389  140303752214080  INV-Session > Finished NSE generation
WARN 2023-09-14 13:45:08.939      8389  140303763498560  EventQueue > TrySendEvent(): Unable to send queued event /opt/altiris/notification/nsagent/var/queue/200.000000000169468818500001.nse: 2165768228,  Total blockout is enabled.

WARN 2023-09-14 13:49:53.672      8389  140303752214080  PolicyManager > RefreshPoliciesImpl(): UpdatePolicies() failed with the following error: 2214592525, Failed to update policies from the Notification Server, message:  Total blockout is enabled
INFO 2023-09-14 13:49:53.675      8389  140303752214080  PolicyManager > Set policy refresh task to start at 2023-09-14 13:53:53 with interval of 60 minutes, deviation 0 minutes.
INFO 2023-09-14 13:49:53.676      8389  140303752214080  PolicyManager > Final results of the policy refresh:

[root@Linux IP]# aex-refreshpolicies 
Unable to refresh Policies due to active Notification Server blockout.
[root@Linux IP]# aex-sendbasicinv 
Unable to send Basic Inventory due to active Notification Server blockout.

Did you check Linux Agent logs ( /opt/altiris/notification/nsagent/var/aex-client.log ) to determine what was a reason that it didn't send nse files to NS? 

Is there any information what happened on this affected Linux client computer on NS side from "Altiris Log Viewer"? Is there information about managed Linux Computers merge events?

"Disable resource keys sent by Agent" option is enabled or disabled? by default it is disabled like on picture below

Best regards,
IP.

[Jeff Stanton]

Igor, thanks for the reply.  

1. Do you have any Agent blockout enabled?  Blockouts are not enabled.
2. Did you check Linux Agent logs ( /opt/altiris/notification/nsagent/var/aex-client.log ) to determine what was a reason that it didn't send nse files to NS?   Yes, logs were examined and also sent in via the support ticket
3. Is there any information what happened on this affected Linux client computer on NS side from "Altiris Log Viewer"? Is there information about managed Linux Computers merge events?  I queried the merge tables and did not see any merge events related to these devices.
4. "Disable resource keys sent by Agent" option is enabled or disabled?  "Disable resource keys sent by Agent" checkbox is NOT checked.
5. There is a similar problem for Agent Health summary report and is fixed in upcoming ITMS 8.7.1 release https://knowledge.broadcom.com/external/article/272134 - Will there be a pointfix for those not yet on 8.7.1?  Does this known issue also apply to ULM devices?

I have been able to fix each but still don't know the root cause.

Thanks for any additional insight.

Jeff

[Roman Nedugov]

Hi,

The Linux agent could mark some events as bad and put them to the end of the sending list or postpone sending them if the initial sending was failed by some reason. If NS server answered not in the expected way or sending has failed due to transfer issues, the event could be moved to "back-off" list. 
All this is not persistent and should reset upon agent (re)start.
In order to workaround the issue, just try to restart the agent and see whether event files start to disappear from the <install-path>/var/queue/ folder.

Of course, in order to find the root cause, need to collect the "devnote" level logs which will include agent start and sending failures.

Regards,

Roman.

[Jeff Stanton]

Thanks for the reply Roman.  Can you instruct me how to collect devnote level logs?  I will have to have my Unix team gather these.

Right now, it seems that with the support of the Unix team, we have fixed the ~80 devices with this issue, but I expect it to come up again.

Thanks in advance.

[Roman Nedugov]

>>Can you instruct me how to collect devnote level logs?

Type in the console: "aex-helper debug on". The logs become "devnote" level. Pay attention that the log will grow in size fast and could start "rotating". For more fine tune, seek "debug_file_size" in <install-path>/nsagent/etc/client.conf, where you can edit the limit of the log file size.

[Roman Nedugov]

Just put it here for history reasons.

The issue was related to the flaw in logic when zero-size NSE's are encountered in the <install-path>/var/queue/ folder. The workaround is to remove zero-size files with ".nse" extension from the "queue" folder. The rest of the NSE's will be send automatically when agent (re)starts or wake-up from previous back-off.

https://knowledge.broadcom.com/external/article/274064