Managed software delivery policy timeout - Part 2

[deemacgee]

Hi all,

Revisiting a similar issue from an earlier thread, but this time with a different cause. The fun never ends. This is ITMS 8.5 RU3.

This time, when trying to save a Managed Software Delivery policy, the Console will throw a timeout error if the policy contains targets. Software, scheduling, remediation parameters - all can be added and the policy saved without difficulty, almost instantaneously.

Once a target is added - even an empty target - and policy saved, the Console will eventually throw a timeout error.

Log viewer indicates three errors triggered by saving:

- "Exception occurred while opening SLIC store. Thread was being aborted".
I assume SLIC is a licensing component but I can't see how that would be an issue if we are under our allotment count.
Initially I thought it was this issue, but following the instructions made no difference.
https://knowledge.broadcom.com/external/article/174297/getclientpoliciesaspx-requests-time-out.html

- "Unhandled exception. Request timed out. [System.Web.HttpException] @" from (w3wp) Altiris.Web.NS.Agent.Global.Application_Error
Reading the log entry indicates a HTTP POST was made to https://[server]:4726/altiris/NS/Agent/GetClientPolicies.aspx. We do run CEM, but it seems odd that a request would be going to port 4726.

- "HTTP Request failed: /Altiris/SoftwareManagementSolution/ManagedDelivery/TaskDetails.aspx?ItemGuid=[guid]"
The [guid] value corresponds to the policy in question, and manually navigating to the constructed URL returns the Console as expected.

The result is the same irrespective of the client or browser.

I can clone existing policies without difficulty - target and all - but they all generate the same timeout when edited and saved.

MSD creation worked perfectly well during the last set of software assignments, about a month ago. The only change between now and then has been the addition of our renewed licences.

Any advice would be appreciated.​

------------------------------
Tech Monkey/IT Primate
------------------------------

[deemacgee]

Quick update to the above: I ran a platform repair from Symantec Installation Manager, and this appears to have rectified the issue. Policies can now be created and saved with targets included.

The log displays "Initializing NSConfigurationWebService Url: http://evtpraltns01.ahlnet.local/Altiris/NS/Services/LicensingWebService.asmx" at 30 second intervals, which seems a little high, but I'm willing to live with it for now.

------------------------------
Tech Monkey/IT Primate
------------------------------

[deemacgee]

Second update: the issue has resurfaced. No changes have been made to the server in the six hours since my last post.
Needless to say, this is immensely frustrating.

The log indicates this error was thrown:
Killing task 'Altiris.NS.ItemTaskManagement.ItemTaskManagerServiceHost' because it has exceeded its wait time.

Immediately after this error, a series of license failure messages are written and continue as I write this post.

The same error "Killing task" error is thrown within seconds of restarting Altiris services.

------------------------------
Tech Monkey/IT Primate
------------------------------

[deemacgee]

Hi all,

Revisiting this thread because the issue has become quite frustrating.

After careful monitoring since my last post, there is definitely a correlation between the licence count and the number of connected machines, however, it is not always consistent. We recently added a fairly large number (100) of ITMS licences, and within two days, we were again receiving the "Exception occurred while opening SLIC store. Thread was being aborted." message in the logs, along with the inability to save updated Managed Software Delivery policy (as mentioned above), and the failure of API calls to the engine by our deployment scripting.

I can guarantee we did not add 100 new machines in two days. There is no reason we should be experiencing allocation issues.

Even more confusingly, the Product Licencing panel in SIM shows the Used value for all products is well below the Count value for those products - in some case, more than 200 seats below our allotment - so not only are we running into inexplicable resourcing issues, we have no reliable way to meter and plan our usage because the base figures are incorrect.

I run a pretty aggressive cleanup of our client population every week or two, and the platform returns to normal functioning for a few hours at most, but after that we're back to SLIC errors and timeouts and API misses.

There has to be more to it. Has anyone seen this behaviour before? Are there any diagnostics I can run? Why are the SIM licencing panel figures so wildly detached from each other if we're (allegedly) out of licences again?

------------------------------
Tech Monkey/IT Primate
------------------------------

[JOE.PS1]

I have never seen any of these symptoms before.  The only thing close I can think of is recently I started seeing timeouts while attempting to edit or even view tasks which have a very long history of executions, such as tasks scheduled to run every hour.  Even though the portal groups the task history into bundles of 100, it does not handle it well when it gets to a certain number.  I ended up writing a custom SQL purge command to clean these up individually.  However, as I recall, my errors were the console just became unresponsive, I did not see the types of errors you are getting.  

The only recommendation I have for you is to bump up your logging levels to enable trace and debug logs and then check the additional log entries around the errors you are getting to see if you see something useful.  I seem to recall some of the license maintenance tasks only show up in debug logging.  

Have you opened a case with Broadcom?  If you are dreading this, I can tell you the last case I opened was handled very well.  The case handling was really bad after the Broadcom move  but it appears they have their stuff together now, or at least that was my experience.

------------------------------
Joe
------------------------------

[deemacgee]

Another update for the forums, and if a dev or rep could jump in and acknowledge these issues, I'd be grateful. This situation continues, and every interruption to our usual services creates difficulties in arguing for the continuation of ITMS as our preferred management system.

We added more licences recently, taking our seat count to its highest-ever, and for about a month we had almost no difficulties. Last week, however, we ran headlong into the Exception occurred while opening SLIC store log message and Console timeouts when assigning machine targets to MSD policy.

I have retired machines, purged machines, I've run the NS.Daily scheduled task after clearing a batch of inactive endpoints, I've restarted the Altiris Service, and I've tried re-adding our licences through SIM. Nothing so far has worked. We had a brief period of functionality late last week after removing a small batch of endpoints (<10). Since then, ITMS has refused to assign targets to an MSD or allow a targeted MSD to be activated.

There are no problems that I can see with Patch Management; all SOE and software installations executed with Deployment Solution have been successful. Monitor, where used, is not experiencing issues. Asset is fine. The Console does not display errors and we've detected no inaccuracies in our metering or reporting. Existing MSDs with targets still function and are dynamically updated as filters detect new candidate endpoints added to our environment. Our issues seem directly and solely related to the spontaneous failure of Software Management Solution as a result of this SLIC service, which itself must be tied to some licencing issue.

I have run this query against our database to search for licences in use, and the returned number for Altiris Software Management Solution 8.5 RU3 is 210 below our seat count. This matches precisely the seat count in SIM --> Product Licencing; we are at ~4200 clients, of a maximum ~4400. Expanding the licence panel indicates we have a number of licences with part or no use of the assigned seats - in one case, a 100-seat Software Management Solution licence is showing zero use. (Licence counts for other solutions are also in the same range, roughly 4200 of 4400 seats.) This is an expensive waste and one which needs resolving.

I am at a real loss to explain or rectify any of this. By all measure, we should have leeway of ~200 additional clients beyond our current fleet, which is quite sufficient room for annual expansion, but we have hit an unknown limitation which undermines our ability to make use of this platform. It's not a sustainable circumstance and one which may see the end of the platform in our environment if not resolved soon.

Any advice (beyond "enable debugging mode") - or better still, an actual fix - would be appreciated.

Courtesy TL;DR version:
- Software Management Solution policy creation/update fails with Exception occurred while opening SLIC store message in log.
- MSD policy fails to create/update if targets are assigned.
- SMS licence count in SIM is ~200 seats below maximum,
- No issues with other Solutions.
- No issues with MSD policy created during period immediately after adding new licences when SLIC errors were not being generated.
- No change after following https://knowledge.broadcom.com/external/article/158855/retiring-a-computer-does-not-reclaim-an.html.

Thanks
DMcG

------------------------------
Tech Monkey/IT Primate
------------------------------