Maintenance mode for AWA agents

[Stephan Schiller]

Hello Community,

for our automated patch process for servers i´m trying to create a "maintenance-mode" for the awa-agents.

High-Level Idea:

1. set agent resources to 0
2. check for active tasks
3. wait, till active task are finished
4. run preparing jobs via awa
5. change entry in uc_ex_hostchar to "patch_mode" for involved agents ("workload_max_job = 0" and "workload_max_ft = 0" so that resources of the agent after a restart are still zero)
6. server patching ()
7. run post processing jobs
8. set agent resources to unlimited
9. change entry in uc_ex_hostchar to "default" for involved agents

Details:

The steps 1,2,3,8 and 9 are easy to develop. But with the steps 4 and 7 as well as 5 and 9 i´ve got some problems.


Step 4 and 7

To run jobs during the agent resources are limited isn´t that easy as expected. In the documentation it is written:

"Use the MODIFY_UC_OBJECT function in the scripts of both Workflows in order to set them to Ignore Agent Resource. This step is required because no other tasks must run at this time. Ignore Agent Resource then applies to all objects that are started by these two workflows."
(https://docs.automic.com/documentation/webhelp/english/AWA/12.2/DOCU/12.2/AWA%20Guides/help.htm#_Common/BestPractices/AWA_Admin/BestPractices_ControlledComputerRestart.htm%3FTocPath%3DAdministering%2520and%2520Configuring%7CManaging%2520your%2520AWA%2520System%7C_____6)

So I tried to use :SET &MODOBJ# = MODIFY_UC_OBJECT(&$RUNID#, SET_EXPRESS, "ON") in the process tab of the maintenance workflow. But sadly with no effect on the jobs inside the workflow. They were still waiting for agent resources.
Then I developed a script object, which "scans" the activation report of the maintenance workflow for activated Tasks.

SET &HND# = PREP_PROCESS_REPORT(, &RUNNR#, "ACT","*RUNID*", "COL=DELIMITER", "DELIMITER=*'*")
:PROCESS &HND#
: SET &RET_RUNNR# = GET_PROCESS_LINE(&HND#,4)
: SET &MODOBJ# = MODIFY_UC_OBJECT(&RET_RUNNR#, SET_EXPRESS, "ON")
:ENDPROCESS

This works for workflows, which aren´t nested. For complex ones, you would need recursion, which isn´t possible with uc4 script. Or I need to develop some sql-statement to retrieve all needed RUNIDs.


Step 5 and 9

I would like to change the value in uc_ex_hostchar via uc4 script, but it is not possible to run something on client 0000. So the only way I see here is to change the variable via application interface.

I would like to know if someone has got other solutions for such a use case. What are your best practices?

I´m looking forward to your ideas.

Best regards

Stephan

------------------------------
Application Manager/Admin
Debeka
------------------------------

[Frank Muffke]

Hi

whats the purpose of the status "maintenance-mode"?

should the agents be kind of offline during OS patching activities for UC4 jobs or should they perform any patching activities similar to Cental agent upgrade?

To me its not clear what is (for example) done in steps

6. server patching ()
7. run post processing jobs

cheers, Wolfgang

------------------------------
Support Info:
if you are using one of the latest version of UC4 / AWA / One Automation please get in contact with Support to open a ticket.
Otherwise update/upgrade your system and check if the problem still exists.
------------------------------

[Stephan Schiller]

Hi Wolfgang,

the steps 4 and 7 are used to stop/start applications, databases etc. on the server, which is beeing patched.
The os patching for the servers is started via DSM/Puppet (step 6), so there is no AWA involved.

And yes, the agent should be kind of offline during the os patching, so that no "business jobs" for the agent can run. They should start after the os patching is done and the resources of the agent are released.

Best regards

Stephan

------------------------------
Application Manager/Admin
Debeka
------------------------------

[Carsten Schmitz]

​Hi,

I was kinda wondering the same thing Wolfgang might be getting at:

Whats the advantage over simply stopping the agent? I mean, when you set ressources to 0, it won't accept any new jobs anyway and only finish up old jobs. And that will still happen, afaik, when you terminate the agent ahead of the update. The OS jobs will be detached, finish on their own, and eventually call up the "Jobmelder" / "Job Messenger" to report back, at least on most platforms and again, as far as I know.

Stopping and starting the agent via a service manager (scripted, not dialog) seems a lot easier to me than creating some maintainence mode with the ressource limits (but maybe I am overly sceptical, because at least in previous versions I have seen flaws with the ressource limits and don't trust them anymore ...)

Best,
Carsten

[Frank Muffke]

I was kinda wondering the same thing Wolfgang might be getting at: Whats the advantage over simply stopping the agent?
=> Exactly :-)

The OS jobs will be detached, finish on their own, and eventually call up the "Jobmelder" / "Job Messenger" to report back, at least on most platforms
=> Exactly again :-)
If the OS is shut down during job execution (when the OS agent is shut down yet) this situation should be prevented.

Java agents (SQL, RA_FTP, ..) should be always limited prior agent shutdown, because the job will crash if its currently running and the agent is stopped meanwhile. (had a ticket with support open [when support had product knowledge])  theres no way to prevent/recover jobs from JAVA agents.

In my opinion limit resources & stop agent afterwards should be fair enough.

one more thing to think of is agent- autostart as soon as the OS comes up.
In our case most departments use autostart only a small amount use their own scripts or do it manually (there are some tricky applications running on that need some time to get started up).

So in case of OS patching activities we stop the OS agents manually (and AE if necessary) and run our JAVA agents - limit & stop script manually.

cheers, Wolfgang



------------------------------
Support Info:
if you are using one of the latest version of UC4 / AWA / One Automation please get in contact with Support to open a ticket.
Otherwise update/upgrade your system and check if the problem still exists.
------------------------------

[Stephan Schiller]

Hi Carsten and Wolfgang,

stopping/starting the agent is the procedure we use actually. The problem is, that the awa agent is started during the start procedure of the server. And a server restart is needed for os patching. But then all queued jobs will start directly and I´ve got no chance to run the "post patching jobs" first.... which are needed to run the corresponding business jobs. And the only way I see is to limitate the resources of the agent.

The os patching is orchestrated. And the maintenance windows for the servers are defined in an ucmdb. For us as AWA-Admins it is a blackbox when a server is going to be patched. But we need to ensure that all workload automation (on this agent) still works properly, even when an os patching is executed.

------------------------------
Application Manager/Admin
Debeka
------------------------------

[Carsten Schmitz]

​Hi Stephan,

I'd still be advocating stopping the agent. You could launch Service Manager on the patching machine right when it comes up again, have the agent set to not auto-start, and launch the agent at a later time when you are ready to do so. This can be scripted for a bunch of machines at once, using a central service manager, such as: ./ucybsmcl -c START_PROCESS -h $SERVER:8871 -n $ENVNAME -s "$AGENTTYPE". Or one could manipulate the .smc file with sed to switch on and off autostart as needed, the syntax of the smc is quite parser friendly.

Even if you don't use Service Manager at all you could make startup dependent on some flag set at the stage where you intend to do post-patching steps by wrapping the whole thing into an overarching script and have it loop until the flag gets set. The flag could be anything, such as a local file (which, again, could be placed on multiple machines at once in a distributed way).

That's the route I would take, but others mileage may vary.

Hth,

edit: oh, and that all only counts for OS agents, I didn't consider Java agents. For those, please see Wolfgang's remarks, those may well need to be handled differently.

------------------------------
I will not respond to PM asking for help unless there's an actual reason to keep the discussion off of the public forums.
------------------------------

[Stephan Schiller]

I still see no way to resolve this problem by stopping the agent. For example:

- no workflows/jobs for an awa agent are active
- shutdown of the agent
- os patching is executed
- during the os patching, two workflows A & B for this agent are activated
- these two workflows or rather the jobs are waiting for host
- now the os patching is done
- start of the awa agent
- both workflows will immediately start, after the agent has started

But I need to run "post patching"-jobs first, to e.g. start applications on the server. How can I ensure, that these two workflows A & B will still wait?

------------------------------
Application Manager/Admin
Debeka
------------------------------

[Carsten Schmitz]

Maybe I'm entirely miss-understanding what you are saying, but again: Why not start (at least for OS agents, Java agents may be a different story) the OS agent after you have done your post patching stuff, started all the applications etc?

I mean, if you have sufficient information about the reemergence of the server and the conclusion of the patching to be able to determine that you now want to kick-off a script to raise ressource limits, you might as well kick off a script that connects to ServiceManager and tells it to start the agent instead. That way, your workflows, scheduled in the engine and waiting for the "host" (née agent) to show up as active will still wait until you (or the init system) are done with the prep work.

[Christoph Rekers]

You could use :SET_UC_SETTING to change the WORKLOAD_MAX (for JOBS and JOBF) when your maintenance starts. This sets the resources to zero until the next start of the agent(s).
To prevent the resources beeing set back to unlimited use the parameter EXECUTE_ON_START => UC_EXECUTOR_START in the UC_HOSTCHAR_xxx where UC_EXECUTOR_START can be any activatable object.
Create such an object in any client other than C0000 and let it execute the :SET_UC_SETTING WORKLOAD_MAX = 0 . The object, e.g. a SCRI will be started as soon as the agents starts. If all agents are up and running you can start a script that sets the resources back to unlimited - or do it manually via WebUI.

[Stephan Schiller]

Hi Christoph,

I´m trying to solve my problem by using/setting the agent resources to zero. I have the problem, that the resources after a agent restart still remain at zero.

You are right with your idea that the parameter EXECUTE_ON_START would set the resources to zero. Our problem is that this would apply to all restarting agents and not just to those
which are currently included in an OS patching process. So agents which are restarted because of another reason than an OS patching process would be mistakenly assigned with zero
resourced after an restart too.

My idea is to have two UC_HOSTCHAR variables. "UC_HOSTCHAR_DEFAULT" with parameter WORKLOAD_MAX_JOB and WORKLOAD_MAX_FT set to unlimited and "UC_HOSTCHAR_PATCHMODE" with parameter WORKLOAD_MAX_JOB and WORKLOAD_MAX_FT set to zero. Before the os patching begins I would like to change the entry in UC_EX_HOSTCHAR for the agent from "DEFAULT" to "PATCHMODE". So when an agent restart happens, the agent resources would still be zero. When the patching has finished I would change the entry in UC_EX_HOSTCHAR back to "DEFAULT", so that for upcoming agent restarts the resources wouldn´t be limited.

But I think there is no other way to change entries in UC_EX_HOSTCHAR then using the application interface. Or do you know a different way?

------------------------------
Application Manager/Admin
Debeka
------------------------------

[Frank Muffke]

I second Carstens attempt as its simpler and more failsafe in my opinion...

If you use Service Manager what about disabling Autostart Flag in SMC file (or unchecking it in SMGR dialog)?

As soon as the server is rebooted - no matter how often -  SMGR starts but the agent keeps down until you start it either via script or via AWI or via SMGR dialog

cheers, Wolfgang

------------------------------
Support Info:
if you are using one of the latest version of UC4 / AWA / One Automation please get in contact with Support to open a ticket.
Otherwise update/upgrade your system and check if the problem still exists.
------------------------------

[Stephan Schiller]

@Frank Muffke and @Carsten Schmitz

I think you might not have fully understood my problem (my english isn´t the best) so I´ll explain my problem again in german so that after this it is hopefully more clearly.

Das Problem, dass ich bei dem Stoppen und Starten von den betroffenen Agenten bei einem OS Patching sehe ist, dass fachliche Jobs nach dem Start des Agenten zu früh (bzw. direkt) anlaufen.
Die Agenten werden bei uns über ein selbst geschriebenes Skript gestartet, der Autostart ist standardmäßig ausgestellt. Dieses Startskript der Agenten ist in die Start-Prozeduren eines Servers mit eingebunden.
Wenn nun solch ein OS-Patching-Prozess startet, der über ein Orchestrierungstool gesteuert wird, ist es notwendig direkt vor und nach dem OS Patching noch "Wartungsjobs" für den Server zu starten. Diese werden in AWA getriggert und dienen zum Stoppen/Starten von Anwendungen/DBs/Webserver etc. auf diesem Server.
Ich muss also zunächst sicherstellen, dass keine fachlichen Verarbeitungen mehr laufen. Danach müssen die Wartungsjobs gefahren werden, damit entsprechende Anwendungen etc. auf dem Server beendet werden. Dann erfolgt das OS Patching via DSM/Puppet. In diesem Zeitraum ist es aber möglich, dass fachliche Verarbeitungen für den Server, der gerade gepatched wird, in AWA anlaufen. Wenn ich nun den Agent vor Beginn des Patchings beenden würde, würden diese gequeueten Verarbeitungen (waiting for host) direkt starten, sobald durch bzw. nach Patching ein Server-Restart erfolgt und somit auch der AWA Agent wieder gestartet wird. Ich habe dann keine Möglichkeit vorher noch meine Wartungsjobs (starten von Anwendungen etc.) via AWA zu starten. Und die fachlichen Verarbeitungen würden vermutlich abortieren, da die entsprechenden Anwendungen, DBs, ... noch gar nicht zur Verfügung stehen.

Ich hoffe damit habe ich etwas besser ausgedrückt, wo mein Problem bei einem Agenten Stop/Start in diesem Zusammenhang liegt. Oder vllt. habe ich auch nicht richtig verstanden, wie ihr es gemeint habt ;-) Dann klärt mich bitte, am besten dann auch in deutsch, nochmal auf.

Best regards

Stephan
​​​

------------------------------
Application Manager/Admin
Debeka
------------------------------

[Carsten Schmitz]

​Hallo Stephan,

Ah, okay. Die Info dass Du diese Wartungsjobs über Automic ausführen musst war mir nicht klar, sorry auch von meiner Seite dafür. Du willst quasi erstmal "exklusiven Zugriff" auf den laufenden Agenten, bevor andere Jobs drauf los gelassen werden.

Dann geht das natürlich nicht, den Agenten einfach später zu starten, dann wirst Du tatsächlich irgendwas mit Ressourcen oder so machen müssen. An der Stelle bin ich leider raus weil ich dazu nichts sinnvolles beitragen kann, wir nutzen die Ressourcen-Limits oder ähnliches nicht und haben diesen Anwendungsfall so auch nicht.

Sorry nochmal für eventuell entstandene Konfusion, und hoffentlich kommst Du mit einem der anderen Vorschläge oder was anderem, was noch kommen mag weiter :)

Vg,
Carsten

[Christoph Rekers]

Hi Stephan, dann schau Dir nochmal meinen Vorschlag an. Das was Du vor hast, wird nur mit Resourcen funktionieren. Wir können gerne darüber telefonieren und das durchdiskutieren. VG Christoph

[Christian Boeck]

Hi Stephan,
der Vorschlag von Christoph mit der Resourcen-Steuerung und der Skriptausführung bei EXECUTE_ON_START sollte auf alle Fälle funktionieren, mein Vorschlag mit export/import sollte denke ich auch laufen. Solltest Du es z. B. nicht über die neue API machen sondern noch über Load/Unload Utility könntest du das File auch in einem StorageObjekt vorhalten und zur Laufzeit auschecken, evtl. sogar zur Laufzeit beim Export noch modifizieren je nachdem wie du es im StorageObjekt ablegst usw.

Es gibt auf alle Fälle eine Lösung dafür :-)
​

------------------------------
Thx & rgds
Christian
------------------------------

[Christian Boeck]

Hi Stephan,
I don't know which version you are using, but the import / export function may solve your problem.
https://docs.automic.com/documentation/webhelp/english/AA/12.3/DOCU/12.3/REST%20API/Automation.Engine/index.html?overrideUrls=../Automation.Engine/swagger.json,../Continuous.Delivery.Automation/swagger.json,../Analytics/swagger.json,../Infrastructure.Manager/swagger.json#/
you can export the original vara before you start the patchmode, then change your values and start patchmode and afterwards and before you come back again import the vara again.

------------------------------
Thx & rgds
Christian
------------------------------

[Stephan Schiller]

Great idea :-) .... haven´t thought about it yet. Thank you!

Best regards

Stephan

------------------------------
Application Manager/Admin
Debeka
------------------------------

[Christoph Rekers]

When I use

:SET_UC_SETTING WORKLOAD_MAX, AGENT01, 0 

- or manually with the UI - the resource is always set to "unlimited" after the agent restarts. That's the default behavior.

With what I previously explained the resource will be set to zero when an agent restarts. If there are Jobs waiting for a resource they will be executed with
FIFO as soon as a resource is available. Now comes the trick:

Assign AE Priority = 1 (Attributes tab) to the Jobs that need to be run as Post maintenance jobs. No other job than this type of jobs must have this Prio.
In consequence all Jobs with Prio. higher than 1 must wait until there's a free resource.

To prevent that - after a agent start - the UC_EXECUTOR_START object sets the res. of all agents, you create a VARA that contains all agents in question.
So when the maintenance is to be started you start the SCRI - that loops through the VARA - and sets the res. to zero. You wait until all jobs are finished and - if need be - restart the machine. When the agent returns, the UC_EXECUTOR_START object loops through the VARA to determine all agents to be set to res. 1 or more, depending on the number of jobs running in parallel. Your Post-Maintenance jobs will start, but not the ordinary jobs as their Prio is less than 1 (2-255). The last object of your Post maintenance procedure sets the res. back to unlimited and deletes all agent entries in you VARA to prevent that the res. of any agent will be set in case the agent was restarted for an other reason than maintenance. So the next time an agent (re-)starts the UC_EXECUTOR_START-object will be started but will not set any res. as the VARA doesn't contain any agent entry. Thus the agent res. is "unlimited".

If there's a certain bunch of agents that are in question of the maintenance procedure you can still assign them to a customised UC_HOSTCHAR_xxx

Give it a try ;-)

[Stephan Schiller]

Thank you for this advice. I´m not sure if I can adapt this mechanism completely, but I´ll give it a try. :-)

Best regards

Stephan

------------------------------
Application Manager/Admin
Debeka
------------------------------

[Frank Muffke]

Buenos Morgos,

Noch eine Idee wär mir eingefallen, was ist, wenn Du die Post- Wartungsjobs in einer eigenen Queue ausführst und während der Wartung alle Anderen Queues auf STOP schaltest? Dann laufen die Wartungsjobs los, sobald der Agent wieder da ist und Du kannst dann die Queue auf GO schalten, wenn Du möchtest.


EN:
what about using a seperate queue for the maintenance jobs?
during maintenance you could set all other queues to STOP and after they are finished you can set all queues to GO

cheers, Wolfgang

------------------------------
Support Info:
if you are using one of the latest version of UC4 / AWA / One Automation please get in contact with Support to open a ticket.
Otherwise update/upgrade your system and check if the problem still exists.
------------------------------

[Carsten Schmitz]

Hatte ich auch dran ​gedacht. Scheitert aber m.M. nach dran, dass er immer nur einen Agenten oder ein subset (jedenfalls nicht alle) von "User Jobs" ausnehmen will. Wüsste nicht wie das mit Queues abzubilden wäre ... wenn er alle anderen Queues stoppt, läuft nichts mehr auf seinem zu patchenden Agenten, aber auch nichts mehr auf anderen Agenten?

[Frank Muffke]

Verdammt, richtig...

Dann würd _Ich_ die SMGR Variante bevorzugen :-)

LG & schönes WE!

------------------------------
Support Info:
if you are using one of the latest version of UC4 / AWA / One Automation please get in contact with Support to open a ticket.
Otherwise update/upgrade your system and check if the problem still exists.
------------------------------

[Stephan Schiller]

Exakt! Beim Anhalten der Queues würden gar keine Verarbeitungen mehr laufen, kommt daher nicht in Frage.

Ich werde mich jetzt daran begeben und die Szenarien mal testen.

Szenario1 (Idee von Christoph):
Die Ressourcen des Agenten werden von mir - ich sag jetzt einfach mal auf 100 - begrenzt. Wartungsjobs bekommen dann einen Wert < 100 zugewiesen, damit diese laufen können. Fachliche Verarbeitungen einen Wert > 100, damit diese im Fall der Fälle warten und nicht verfrüht loslaufen.
Vorteil den ich bei dieser Lösung sehe ist, wenn einmal ordentlich implementiert, ist es sehr stressfrei und man hat bei der Entwicklung weniger Scriptaufwand. Nachteil: Ich muss erstmal alle fachlichen Jobs mit Ressourcen belegen.... denke das sollte über eine Massenänderung aber auch recht einfach realisierbar sein.

Szenario2 (Ansatz mit UC_EX_HOSTCHAR):
Die Ressourcen des Agenten werden komplett auf 0 gesetzt. Vor Restart des Agenten Modifizierung der UC_EX_HOSTCHAR mittels Ex-/Import, damit beim Restart eine UC_HOSTCHAR_XXX agezogen wird, die standardmäßig WORKLOAD_MAX_XXX auf 0 gesetzt hat. Damit bleiben die Ressourcen des Agenten auch nach dem Neustart auf 0 limitiert.
Vorteil hierbei wäre, dass alle fachlichen Verarbeitungen unberührt bleiben und unsere internen Prozesse (zur Erstellung von Jobs) nicht angepasst werden müssen. Nachteil: Eindeutlich höherer Scriptaufwand zum antriggern der Wartungsjobs. Hier muss ich mit einem Script auslesen, welche Jobs laufen müssen und dann mit MODIFY_UC_OBJECT ein SET_EXPRESS für diese Jobs durchführen.

Ich danke euch jedenfalls vielmals für den hilfreichen Input!

Ich gebe nochmal bescheid, für welchen Weg ich mich letztendlich entschieden habe.

Gruß

Stephan

------------------------------
Application Manager/Admin
Debeka
------------------------------

[Carsten Schmitz]

​Hi,

Ich will ja nicht nochmal in die Suppe spucken, aber was immer es wird: Teste es gut :)

Ich erinnere mich dass wir vor ein paar Jahren mal mit verschiedenen Mechanismen experiementiert haben, Last auf Agenten zu verteilen. Damals haben wir es mehrfach geschafft, Jobs auf Agenten laufen zu lassen, obwohl die nach dem Ressourcenwert logisch nicht hätten laufen dürfen. Konnte sich damals jedenfalls bei uns niemand erklären. Habe ich ja vorher schon angedeutet, dass ich dem nicht mehr traue, finde aber leider die alten Unterlagen nicht mehr. Das war allerdings mit V10 iirc, vielleicht ist ja seitdem auch irgendwas gefixt.

Vg,
Carsten