Stopping Automic db for patching

[monica areni]

Hello everyone,

the database of our Automic's installation will be stopped for three hours due to system's patching activities.

I'm trying to find out which is the fastest method for stopping all the JOBS, JOBP scheduled during the maintenance window, and restarting them at the activities's end.

Thank you in advance!

Monica

[Daryl Brown]

I'm actually interested in how other folks approach these DB outages, too.

In our case, we typically opt to leave the Automic environment running during this DB outage, with the understanding that the application will thrash about being unable to connect to the DB all the while.  But the advantage of this approach is that when the DB connectivity is finally restored, Automic will automatically kick off any scheduled tasks that were missed during that outage.  (We've found that behavior to be preferable to the effort that would be involved to review all active schedules and manually reset any tasks that were missed.)  In practice, we've often found that a lot of our agents will have disconnected during this outage, although they generally all reconnect within a few minutes once the DB connectivity is restored.  Since Automic by that point will be kicking off all the missed workflows, we do frequently wind up with failed tasks because the agent hadn't yet reconnected, but we find the advantage of knowing all scheduled tasks were kicked off to outweigh the hassle of restarting jobs due to the agent not having been present.

[Tony Ferraro]

Very interesting Daryl! Does Broadcom approve that approach? If it works reliably it seems like a decent option.

[Wendy Copeland]

Hi Tony,
This is Wendy not Darryl. I think that they would not have any issues with it since I found some of the Commands on their Website. It works reliably. However, if you are shutting down all of Automic, you would still have to turn off Agents and Service Managers manually. I wrote the Scripts back in September 2019 and we have been using them since then with very little trouble. The only failures have been if an Event or Recursively Running Job or Workflow has been cancelled previous to running the Shut Down Script (as then of course it cannot cancel it).

Have a great day.

Sincerely,
Wendy Copeland
Systems Administrator II
Canon Medical Systems USA, Inc. | Information Technology
2441 Michelle Drive, Tustin, CA 92780
P 714-559-0221
E [e-mail]<mailto:[e-mail]> | W us.medical.canon<http: us.medical.canon/="">
[cid:[e-mail]79D0]<https: [e-mail]79D0]<https:">www.linkedin.com/company/canon-medical-systems-usa-inc="">[cid:[e-mail]79D0]<https: twitter.com/canonmedicalus="">[cid:[e-mail]79D0]<https: www.youtube.com/canonmedicalus="">
[Mail_footer_CMSCgroupmark_en]

[Matthias Schelp]

Hi Monica,

Have you considered stopping all clients in the administrativ view of client 0? That prevents the engine from starting new Jobs. The downside is that running jobs are not effected and may end up "lost" if the operation system process ends while the engine is in maintenance. So we usally wait 5 - 10 min for running processes to end after stopping the client.

Regards, Matthias

[Wendy Copeland]

Hi Monica,
I have written scripts that hold the Queues, then they count the active Jobs every 60 Seconds and when the Last Active Job is Completed, they E-Mail us saying that the Queues are down,  I also have ones that do the same thing but in addition they Cancel the Events, Recursive Jobs and Workflows, and the Capture Job for Oracle Jobs.  I also have Post-Outage Scripts that Release the Queues and Restart the Capture Job, Events and Recursive Jobs and Workflows.

[Tony Ferraro]

"'Im trying to find out which is the fastest method for stopping all the JOBS, JOBP scheduled during the maintenance window, and restarting them at the activities's end."
I've always wondered why there is not an out of the box Broadcom process/GUI button to do this given that 100% of customers will have this issue to some extent numerous times per year.

[Paul Sellan]

My suggestion is you may stop the JSCH if you have added all your JOBS and
JOBP inside JSCH.
So, once the maintenance is done you may GO the JSCH and resume all the
auto-trigger jobs. You may REST TASK any timeout jobs.
This method seems to be easy and fast to control the jobs during the
maintenance period.

[Prakash Subramanian]

Here, one dropback is that the workflow will be skipped if any JOBP start time is crossed. Hence have a separate queue for JSCH and a separate queue for all scheduled workflows in JSCH. Here WFs will wait for the queue to be released until DB patching is done.

[Wendy Copeland]

Wanted to make it clear.  Our Scripts do not stop Agents or Clients.  If Automic is coming down, we do that part manually.  The Script simply Holds the Queues and Queries until all Active Jobs have finished.  Once all Active Jobs have finished it stops Events, Recursively Executing Jobs and Workflows and the Oracle Capture Job.  After the Outage the Script Releases the Queues, and restarts the Oracle Capture Job, all Events and the Recursively Executing Jobs and Workflows.
You are right about the skipped Jobs and Workflows.  After an Outage, we always run missed Jobs and Workflows.

[Frank Muffke]

Hi

For us the most reliable method is stopping the queues (and clients) and wait a bit until currently running jobs finish. Unfortunately this is necessary because Jobs on Java Agents (we have a lot of SQL Agents) will crash if you stop the agent. All OS Agents can remain running, theres no need to stop them.
Usually all objects keep in the state they were before in.

Basically you could keep the clients and queues running when shutting down the AE - the only risc is if at the moment of shutdown any script is being calculated.....

Last year we had a DB Outage on our Sandbox env for half a day, there were several issues with CPs and JCPs (CP3 was a real CP and after Db was back a JCP, agents tried to reconnect recurringly and dumped a trace) we had to restart the whole system...
Out of my experience, a DB outage of <10min is no big deal for the AE, for longer ones I would shut down the AE.

cheers, Wolfgang

------------------------------
Support Info:
if you are using one of the latest version of UC4 / AWA / One Automation please get in contact with Support to open a ticket.
Otherwise update/upgrade your system and check if the problem still exists.
------------------------------