Patch Management Solution

 View Only
  • 1.  tangled tasks - this policy is being reconstructed by the patch management distribution task

    Posted Dec 21, 2016 08:51 AM

    I have been trying to get patches downloaded and policies made for said patches. I go into the list of patches the system has found, say there's a security patch, pick anything, it does not matter at all, for Microsoft, Adobe, Google, whatever, pick one, right click, and choose distribute. It used to flag that patch for download, then the next day I could go in and tweak things - there would be a policy that was turned off, with the default settings as far as who and when. However, now it does nothing but say start pending - for minutes, hours, even days, until I kill it.
    I found in the logs that a group of computers I had used once was messed up, so went to that policy that distributed several Microsoft updates and tried to remove that list of targets so I could DELETE it, and use the current better list of targets. I removed the target (list) from the policy, chose a new target and chose to save. It said saved, but sticks on start pending, since yesterday.
    I have found several articles pointing to troubles downloading new patches, tasks getting stuck and never completing, etc. and ran reports and found MANY MANY failed tasks since at least December 8th. Holy cow, no wonder things have stopped working!
    The issue is - after spending 3 hours here yesterday, forums (connect) blogs, KB articles and documents, nothing at all for this version - all dates back to 7.1 and PRIOR. All are from 2014 or earlier. I don't dare run SQL queries or changes against a database that's years advanced of those old articles.

    Help, please! I've seen almost every single document published on task issues, start pending, etc. so please don't send me to the most common. I found a dozen or more messages or threads in Connect here and I followed every link people here suggested, as I said, most were OLD and for OLD versions. We are at 7.6, the VERY latest. I have all fixes and patches applied to the management server, patch management and so on.
    Symantec helped me with a corrupted policy a few weeks ago and said we had the latest of the 7.6 stuff, so it's current (although 8 sounds interesting, not sure if our license permits a move to 8 and I'd never in a million try a full upgrade on a corrupted system!
    I have seen these messages and have followed all the links -
    https://www.symantec.com/connect/forums/patch-management-tasks-stuck-start-pending-and-not-showing-under-task-status

    https://support.symantec.com/en_US/article.HOWTO54534.html

    I ran the report for the above using the currently running known tasks XML included and it showed a ton of failures, etc. A LOT.

    I looked at this one as well but it's 6 bloody years old, for OLD versions!
    https://support.symantec.com/en_US/article.TECH40277.html

    And this one has a current date, but that's the modified date - it was written years ago and applies to 7.1 and 7.5 and we have 7.6 and it's not quite the same, close, but not exactly what I'm seeing - or is it?? and with our newer version, there's no way I'm going to try old scripts or queries for 7.1 against 7.6 database!
    https://support.symantec.com/en_US/article.TECH46342.html

    Would this one apply? To our version? And what does it mean to run those lines shown "truncase table etc."?
    https://support.symantec.com/en_US/article.TECH218556.html

    Any ideas please?

     

    task-instance.jpg

    altiris-start-pending-download-error.jpg

    The start pending NEVER goes away, even a day or more later.

    altiris-start-pending-error.jpg

    And if I stop a process that sits as above (and they ALL do), using the STOP button, then close, I get this -

    altiris-start-pending-download-error-result.jpg

    I cannot distrubute a patch, cannot download a patch, cannot change a patch policy as after saving, it sits with this message
    this policy is being reconstructed by the patch management distribution task

    and it never does.........

     



  • 2.  RE: tangled tasks - this policy is being reconstructed by the patch management distribution task
    Best Answer

    Broadcom Employee
    Posted Dec 22, 2016 03:09 AM

    Hi ShadowsPapa,

    I have some thoughts for your case, just thoughts...

    1. Is there a lot of Server tasks/ClientServer/Jobs with often run schedule and they aren't completed yet and are in time-out queue?

    2. You can check performance counter within Task Server on NS machine to see how much pending tasks are in queue.

    • TS "Server tasks" max threads is '20' by default so probably queue is full of other Server based tasks by unknown for now reason, and therefore your distribution task goes to pending state as well because of full queue.

    PerfMon.jpg

    3. Check what Jobs/Server tasks are currently isn't yet completed to identify a reason of such behavior that they are filling a TS "Server Tasks" queue.

    Here is a information about Tasks Status from "vItemTaskStatusDescription"

    ItemTaskStatus    ItemTaskStatusDescription
    0                         Never Started
    1                         Start Pending
    2                         Running
    3                         Stop Pending
    4                         Stopped
    5                         Killed
    6                         Completed
    7                         Error

    "TaskInstances" SQL Table contains task instances current state/results.

    You can check all current Tasks state from TS web part. Open SMP Console ⇒ "Settings" ⇒ "Console" ⇒ " Web Parts" ⇒ expand "Task Management" folder and click on "Recently Accessed Jobs and Tasks" web part and then click "Show Preview"

     

    Regards,

    IP.

     

     



  • 3.  RE: tangled tasks - this policy is being reconstructed by the patch management distribution task

    Posted Dec 22, 2016 03:22 PM

    Thanks for the detail - I ran a basic SQL query built into the SQL server management studio and I wondered what the 0 and the 2 meant. I chose serverstasihnstancerequests and asked for the first 1,000 rows and got back over 400 results. There seemed to be a lot going on. And they were ALL 2 or 0 with most of them being 0.

    I did stumble around and found that the issue seemed to start on December 8 at a little after 1 pm. So I concentrated on things from that time period and found many many instances of tasks that had failed that day, and sometimes there were tasks that should have run one time, but they failed a dozen times all within a minute. Our SQL server had a little problem one day this month and that may have been when the processor and memory maxed out and it went down hard. So I spend over 8 hours going through every single setting and task I could find, and ran into some I had never seen before, never knew about, some of the automation, and started deleting everything using the interface. I went back and deleted any task that had failed or was successful and when I got up to December 8th, I started finding patterns - and once I deleted all of the pending and queued tasks from that exact hour on that day, suddenly that SQL table shrank fast. It went from over 400 to 20  in the last 4 hours of my cleaning up tasks. I stopped the running tasks, deleted them and left only things set in the future - like tonight or tomorrow.
    There were almost 1,000 tasks related to Wake On Lan - it took almost an hour to open several pages of 100 tasks each, delete old ones, find those running and stopping them and deleting all of the pending or queued WOL tasks. Why they were there and so many, I do not know, but those seemed to be the real problem. Those mystery wake on lan tasks.

    The picture I posted at the top of my first post shows the SQL server - arrow pointing down the number of rows. The remaining 20 are all set at 2, and when I choose a patch to distribute I see 1 or 2 new rows get added with a 0, but soon those go to a 2 and it drops back to 20 again.

    I had wondered WHY 20 - now I know! Thank you for that DETAIL. I love detail and pictures. Now I know I was seeing 20 set at 2 for a reason and anything new came in as a 0 because the max is 20. I did not realize that until now. It must be working normally now, or close to it.

    The server is EXTREMELY busy now, I have never ever seen the logs blow by SO fast - it appears to be download and recreating almost everything, all patches. Weird.  The lines are scrolling by so fast I can't keep up and that is a first. It used to sort of sit there while I read things.
    It's downloading many dozens of files, stating some are already downloaded and so on.
    I am finding just 1 error now in the logs - the red has all but disappeared!
    If I can correct this error, a filter called "All Windows 7 Computers" that seems to have incorrect number of columns, so it says, then I'll be a lot happier.

    Thanks again for the details. You included things I have not been able to find otherwise in 2 days of searching. That list of numbers, the 0 through 7 and what they indicate is very helpful. I'm going to print that for future reference..... although I hope I don't run into this again very soon. Normally this thing simply works, set it and forget it.

    I can actually have it download patches and create patch policies like it used to last month.

    logs-downloading.jpg

     

     



  • 4.  RE: tangled tasks - this policy is being reconstructed by the patch management distribution task

    Broadcom Employee
    Posted Dec 23, 2016 12:24 AM

    Also there is an article for DB health maintenance



  • 5.  RE: tangled tasks - this policy is being reconstructed by the patch management distribution task

    Posted Dec 27, 2016 01:15 PM

    Great - thank you. I was hoping to find something specific to this product as it, as well as the SEP database, are quite active and take a bit more power from our SQL server than even our in-house client management system does. I do not know a lot about SQL, other than knowing it does take some maintenance steps to keep things working smoothly. I've seen poor performance with the management product, even the last tech who remoted in and worked directly on our ITMS/notification server commented that "it should be faster than this" and suspected that SQL was part of the reason.

    Even though I was finally on the right track in getting the tasks working again and getting that table down from over 400 tasks listed, and being unable to do anything with more tasks, downloads and policy reconstruction, your help ensured I was going in the right direction and you provided details I was not otherwise ready to find so as far as I am concerned, you helped me get this solved, so I am marking your prior message as the solution. This way anyone looking for help related to the same issues I was facing can save some time and follow your information and get things working again.

    Thanks.