Resource allocation across agents

[Legacy User]

I'm struggling how to interpret the Automic documentation concerning resource allocation across agents.  I have a situation where I need to run a job one at a time on any 1 of 5 servers in my host group.  When I start the process, I will create 20 jobs and I'd like the first 5 (or any 5) to start running, one on each of the Linux machines in my host group.  When one finishes, I want another job in the queue to start on the machine where the job just completed, such that only 1 job will run at a time on each machine.  I've tried to do this through a special host group using a "Load Dependent" method, but its not working as several jobs are running on the same machine at one time.  I'd like to explore using resources for the job and agents.  However I am unable to identify how to assign a resource limit to an agent.  The Automic documentation has an example for resources that states "An agent has 100 available resources for job executions".  Yet the documentation on the Agent object does not seem to indicate where this resource limit should be noted.  Can someone help me identify how to make this work or refer me to the appropriate documentation?  I should point out that we will be running many other jobs on the same machines at the same time that will not be constrained by these resource limits.  The resource limits are only needed for this one job across the servers in the host group.

[Anon Anon]

Jeff:

I am relatively new to V11 (V8 is still running our production) so can't really respond specifically to your question and the use of the latest incarnation of Agent Groups.

Perhaps this discussion regarding Getting Started/Best Practices might be applicable and something similar could be adapted to your needs.  While we execute a number of jobs concurrently on each Agent it could probably be made to meet your requirements by perhaps adding Groups, Syncs or maybe Variables to control for single threading.

[Legacy User]

Mark,
Thanks for your input.  Your method is pretty flexible, which I like.  I suppose it could be adapted.  But it seems like what I'm asking for is available out of the box from Automic.  I'm really just asking for load balancing across the 5 servers I have.  Their documentation is leading me to believe I can do it.  And I thought I had it working with a set of simple "sleep" jobs that slept a random number of seconds.  However when I applied my method to real production jobs, it did not maintain the load balancing as I expected.  I'm really hoping I can do this without custom variables and scripts.
Thanks again,
Jeff

[Pete Wirfs]

This is not a very robust idea;

Define 5 unique jobs, one for each target agent.  Each of these 5 jobs would have their "Tasks running parallel" set to 001.  Then you simply submit all 20 jobs, each one being submitted 4 times.

If one server completes its 4 jobs early, it sits idle while the other servers continue to chug along.  So this is not true workload balancing.  If you really need an intelligent workload process, you may have to write it yourself with a script object that loops and submits jobs dynamically until all of the work is done(?)

Actually rather than a looping script, it would be better if the job executed a script from the post-process phase.  Then the script would only fire when you knew there was an available agent.

[Jessica Isaacson]

Sorry this comment doesn't come with a good suggestion; I just wanted to mention that you are not alone in your experience. We also need to load balance multiple jobs across multiple servers in an agent group. Both Next and Resource based models are completely unreliable if two jobs start within a few seconds of each other.

If you're scaling a lot of jobs horizontally in a large cloud of ephemeral VMs, you really can't guarantee separate start times or assign specific hosts. The agent group functionality seems so simple and clear, and yet doesn't work as advertised. You are not imagining it, and it's infuriating!

[Jessica Isaacson]

Actually I think I can speak to how to assign a resource limit. I haven't done this myself, so someone may want to fact-check.
First you create a variable in client 0, UC_HOSTCHAR_YOURAGENTNAME (named like the agent).

Then you create a line for WORKLOAD_MAX_JOB in that particular variable. You can set any host characteristics there, to override what you have in UC_HOSTCHAR_DEFAULT. Copy the syntax from UC_HOSTCHAR_DEFAULT.
https://docs.automic.com/documentation/WEBHELP/English/all/components/AE/9.13/All%20Guides/help.htm#ucaabu.htm%3FTocPath%3DAdministration%2520Guide%7CConfiguration%7CSettings%2520in%2520UC4%2520Variables%7C_____9

Also one more nasty surprise for resource-based host assignment: if a task goes into “waiting for resource,” status because the hosts are all too busy, then the task already has an agent chosen. If another agent becomes free, too bad. The task will wait anyway.