AutoSys Workload Automation

 View Only
  • 1.  Virtual machine - forcing priority for submission to a specific real machine

    Posted Jul 25, 2017 01:52 PM

    Hello,

     

    Situation:

    Got a virtual machine (Virtual1), with two real machines(Machine1 and Machine2) attached to it. Machine1 is considered primary, Machine2 is considered fail-over.

     

    Question:

    For jobs that submit themselves to Virtual1, is it possible to force them to try Machine1 first and then hit Machine2, only if Machine1 is down?

     

    Thanks a lot in advance!

     

    Have a great day,

    Tom



  • 2.  Re: Virtual machine - forcing priority for submission to a specific real machine
    Best Answer

    Posted Jul 26, 2017 03:36 AM

    Hello Tom

     

    I see a few options:

     

    - The first one is to use the max_load attribute and put a max load value on Machine1 and low value on Machine2

       When Machine1 goes down, it will be automatically set offline and all your jobs will be running on Machine2

     

    - The second option is to use a machine chooser script to get rid of the virtual machine definition

    machine_chooser

    Defines the name of a program or user-written batch file for the scheduler to execute at run time to determine which machine to use. The scheduler runs this program, writes the name of the machine to its log file, and substitutes this output as the machine name.

    Details HERE

     

    - The third option is to put some logic in your fail-over environment or in the scripts used to start the second node in a cluster environment  to ensure that the first node is offline when you start the fail-over node (Machine2).

     

    This can be done, by running "sendevent -E MACH_OFFLINE -N <node>" 

    Or "sendevent -E MACH_ONLINE -N <node>" to put it back online

     

     

    Regards

    Jean Paul



  • 3.  Re: Virtual machine - forcing priority for submission to a specific real machine

    Posted Jul 26, 2017 08:30 AM

    Hello Jean Paul,

     

    Thanks a lot for all this info!

    Few questions/notes:

    "- The first one is to use the max_load attribute and put a max load value on Machine1 and low value on Machine2

       When Machine1 goes down, it will be automatically set offline and all your jobs will be running on Machine2"

    Does setting a high max load value on Machine1 and a low one on Machine2 guarantee priority job submission on all executions to Machine1, unless that machine is down, or are there still chances that jobs could submit to Machine2?

     

    "- The second option is to use a machine chooser script to get rid of the virtual machine definition

    machine_chooser"

    VERY interesting, was not aware of that functionality. But, I'm unsure how do we pass the machine name to the scheduler log. Is there a variable that the script should populate once it completes it selection? Tried to find example of such a script, not really fruitful.

     

    "- The third option is to put some logic in your fail-over environment or in the scripts used to start the second node in a cluster environment  to ensure that the first node is offline when you start the fail-over node (Machine2)."

    Not really an option in my situation.

     

    Thank you very much!

     

    Have a great day,

    Tom



  • 4.  Re: Virtual machine - forcing priority for submission to a specific real machine

    Posted Jul 26, 2017 09:58 AM

    Hi Tom

     

    max_load works with job_load in the job definition.

    For the first option, if max_load is 100 for example on machine1 and max_load is 1 on machine2, jobs with job_load attribute  more than 1 will be all running on machine1. But if any job is defined with job_load 1, it can run on machine2

     

    About the machine chooser script, the script can pass the machine to the scheduler with the echo command

     

    For example:

    .../...

    ping -c2 server2 >> /dev/null
    if [ $? != 0 ]
    then
    #unable to ping secondary internal machine
    exit 254
    fi
    HOST=server1
    echo "${HOST}"
    exit 0

     

    Regards

    Jean Paul



  • 5.  Re: Virtual machine - forcing priority for submission to a specific real machine

    Posted Jul 26, 2017 12:15 PM

    Jean Paul,

     

    Perfect, got it now.

    For the script's location, I assume I need to place it on the AE machine (localhost) and not on either of the two application servers?

     

    Regards,

    Tom



  • 6.  Re: Virtual machine - forcing priority for submission to a specific real machine

    Posted Jul 26, 2017 12:26 PM

    Hi Tom

     

    You can place the script at any location on the scheduler ( running the event_demon process )

    Don't forget that in HA mode, it has to be placed at same location on your both schedulers

     

    For example on Unix/Linux

     

    machine: `/usr/local/bin/my-machine-chooser`

     

    More details HERE

     

    Have a great day

     

    Jean Paul



  • 7.  Re: Virtual machine - forcing priority for submission to a specific real machine

    Posted Jul 26, 2017 06:32 PM

    Jean Paul,

     

    Went through the CA doc, was just making sure.
    Thanks a lot for all the this. Very useful and appreciated!

     

    Have a great day.

    Tom



  • 8.  Re: Virtual machine - forcing priority for submission to a specific real machine

    Posted Jul 28, 2017 05:16 AM

    To the original question, you could perhaps ditch the VM and put 'machine: host1,host2' in the job definitions.  If host1 disappears, host2 should be used.

    Another option could be using AutoSys virtual resources.

     

    Many sites/people consider the machine chooser mechanism to present a big security hole and ban its usage, because you're running a script on the AE server.  Any scripts need to be present and correct on the Shadow Scheduler too if you're using one.  'ping' is never a good enough test on the availability of a machine.  Personally I'd never use a machine chooser, but I know some places that use this.

     

    Embedding 'sendevent' commands, or any client commands, in scripts or code, is another practice that many places do not allow.