AutoSys Workload Automation

 View Only
Expand all | Collapse all

Bulk ON_HOLD action

  • 1.  Bulk ON_HOLD action

    Posted Oct 11, 2017 02:15 AM

    Hi Autosys Experts,

    What is the best way to put all the jobs of an instance ON_HOLD? Put all the jobs without a box_name ON_HOLD?

     

    eventor -G may not help as it will put only the jobs that were supposed to be run during the instance downtime will be ON_HOLD.

     

    Its a 4.5 instance.

     

    Regards

    Guru



  • 2.  Re: Bulk ON_HOLD action
    Best Answer

    Posted Oct 11, 2017 04:17 AM

    Hi,

     

    U can try to make use of GLOBAL VARIABLE as a condition in the job definitions.

    eg:

    insert_job: JOB A

    conditions: v(GV=1) and s (JOB 123)

    days_of_week: daily

    start_times: "7:00"

    date_condition: 1

     

     

    set the value of GV by using set_global_variable command to 0.

     

     

    so in this case, since the job A has 4 starting conditions to satisfy, if any of the condition is not fulfilled this job will not run.

    1. value of GV shd be 1

    2. Success of JOB 123

    3. time is 7AM

    4. run day is daily

     

    As using the send event command we have set the value of GV to 0, so this job will not run until the value of GV is set to 1, irrespective of other 3 conditions being satisfied. Hence the job will not be running and is kind of ON_HOLD situation.

     

    This is the best possible option that I can think of w.r.t version 4.5, as GV is the attribute which is in picture in autosys from all versions starting from 4.0 till now.

     

    Above is just an example, you can use GV without combining any other conditions as well.

     

    However, a point of caution is:

    if a job is defined to run on the specific value of GV, then that job definition wont be an adhoc job and runs immediately as soon as the value of GV met the job condition.

    **** Job status will be ACTIVATING or INACTIVE and future actions (adhoc batch requests) will be taken according to job status.

     

     

    Also,

    if you would have been using latest version then we cud have used RESOURCE attribute which also works in same way as GV but handling of  adhoc request / forcestarts varies as per resource type used.

     

    **** I am not sure or confused with the job states, I think in above example, job will be in ACTIVATING STATE and be in AC state until the value of GV is set to 1.



  • 3.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 05:02 AM

    Hi Sunish,

     

    Thank you. 

     

    This would require me to update all the job definitions in the instance. Isn't it?



  • 4.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 05:18 AM

    yes, but it wont be a tough task. You can jil the jobs in txt with update_job: and required parameters. and whole jil will be processed in fractions of mins.

    else you can do that in parts and test it and move further.

     

    that is the only option.

     

    or you can issue a sendevent command directly to place the jobs on hold.



  • 5.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 06:56 AM

    Check the auto_hold option as well. It'd involve creating a top level box with start_times and making all your jobs part of the top-box. Once the box goes to running, all the jobs within will be placed on hold.

     

    Thanks & Regards,

    Chandru

    CA Technologies



  • 6.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 07:02 AM

    The question was all jobs. stop and start AutoSys in globalhold mode.

    you want to put all the toip boxes on hold or top jobs? I need to ask why? 

    This sounds like a very ODD requirement.

     

     

     

     

     

    Steve C.



  • 7.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 07:25 AM

    Chandru

    auto_hold only works on child processes.

    I still would like to understand this requirement.. sounds to me contrived. 



  • 8.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 07:44 AM

    Thank you Sunish/Chandru/Steve!

     

    Thank you for the comments.

     

    Here is my requirement:

    I want to put all the jobs of the instance ON_HOLD.

    Then let the application owners force start only the ones they need.

     

    From reading the documentation for globalhold, i was under the impression that only the jobs that have missed their run due to instance being down would be placed ON_HOLD (and the jobs with later timestamp would start as usual). However, it looks like globalhold can be used in this case.

     

     

    This requirement is to stop all the non-required jobs from running on our QA instance, and (hopefully) solve the performance issues.

     

     

    Regards

    Guru



  • 9.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 07:47 AM

    Autosys 4.5? how many Eps did you set up?

    My guess is you have one. If you bump up the number of Eps (check $AUTOUSER/config.$AUTOSERV

    You will get a performance boost.

     

    Also migrate to latest version..

    11.3.6 SP6 CUM1

     

    Good Luck

     

     

    Steve C.

     

     

    Nothing in this message is intended to constitute an electronic signature unless a specific statement to the contrary is included in this message.

     

    Confidentiality Note: This message is intended only for the person or entity to which it is addressed. It may contain confidential and/or privileged material. Any review, transmission, dissemination or other use, or taking of any action in reliance upon this message by persons or entities other than the intended recipient is prohibited and may be unlawful. If you received this message in error, please contact the sender and delete it from your computer.



  • 10.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 07:52 AM

    Hi Steve,

     

    Thank you.

     

    EP count is set to 16.

     

    Whats the maximum that can be used? 

     

    Regards

    Guru



  • 11.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 07:57 AM

    You do know that too many Eps impacts performance as well.

    Also the number of jobs launching at the same time.

     

    I would work on reducing it and see if that speeds things up.

    There is a wall on how efficient EP count may be. 16 seems very HIGH , I have been in some really big shops and never needed to go that high.

     

    If this is a QA instance perhaps people aren’t clearing out old jobs no longer necessary as well.

     

     

     

    Steve C.

     

     

    Nothing in this message is intended to constitute an electronic signature unless a specific statement to the contrary is included in this message.

     

    Confidentiality Note: This message is intended only for the person or entity to which it is addressed. It may contain confidential and/or privileged material. Any review, transmission, dissemination or other use, or taking of any action in reliance upon this message by persons or entities other than the intended recipient is prohibited and may be unlawful. If you received this message in error, please contact the sender and delete it from your computer.



  • 12.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 08:01 AM

    Also 4.5 has been EOL/EOS for some time. why aren't you on R11?



  • 13.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 08:35 AM

    You may need to work with application teams to identify the non running or not required jobs.

     

    This is not the right way to identify the required jobs.

     

    However the option globalhold will hold all the jobs in the instance and you need to manually forcestart the job even the dependencies as well.Having said this.

     

    for globalhold option to work, you first need to stop the event demaon, then modify the config.INSTANCE file, then start the event demaon.

     

    out of curiosity - how come this option will help you identify  the non running or not required jobs.



  • 14.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 09:20 PM

    Sunish_CA

    for globalhold option to work, you first need to stop the event demaon, then modify the config.INSTANCE file, then start the event demaon.

    Back in (good old) AutoSys 4.x days, eventor -G command was used to start the Scheduler (quondam Event Processor) in Global AutoHold mode.

     

    Cheers,

    Chandru



  • 15.  Re: Bulk ON_HOLD action

    Posted Oct 11, 2017 09:24 PM

    Guruprasad_Kulkarni

    Please remember to mark response that answers your query correct. So far, Steve's GlobalHold suggestion sounds like what you are looking at.

     

    Thanks,

    Chandru



  • 16.  Re: Bulk ON_HOLD action

    Posted Oct 12, 2017 05:24 AM

    Thank you Steve/Sunish/Chandru!

     

    Steve,

    I would reduce the number of EPs and try soon.

     

    This being QA instance, application teams usually would test only the jobs that would be relevant. Over time, these jobs, once the testing is completed, have been let to run, resulting in huge load.

     

    My plan is to ask the users to force start only the ones that would be relevant for now.  That would be much easier than reaching out to every job owner and check its relevance.

     

    However, i would consider putting only the top-most level jobs ON_HOLD instead of Globalhold, as GlobalHold would require application owners to force start _every_ job in a flow/box. Also, GlobalHold would place all the new jobs that get created ON_HOLD.

     

    We are at 4.5, with effort to get upgraded to the latest support version 11.3.6.

     

    Chandru,

     

    I have marked the answers as Helpful.

     

     

    Regards

    Guru



  • 17.  Re: Bulk ON_HOLD action

    Posted Oct 12, 2017 05:38 AM

    Steve/Chandru/Sunish,

     

    What are your thoughts on this approach.



  • 18.  Re: Bulk ON_HOLD action

    Posted Oct 12, 2017 06:55 AM

    The point of QA is for them to test their batch flow. They cannot do that if they have to forcestart.

    I do not agree with the approach

     

     

    Steve C.

     

     

    Nothing in this message is intended to constitute an electronic signature unless a specific statement to the contrary is included in this message.

     

    Confidentiality Note: This message is intended only for the person or entity to which it is addressed. It may contain confidential and/or privileged material. Any review, transmission, dissemination or other use, or taking of any action in reliance upon this message by persons or entities other than the intended recipient is prohibited and may be unlawful. If you received this message in error, please contact the sender and delete it from your computer.



  • 19.  Re: Bulk ON_HOLD action

    Posted Oct 12, 2017 07:35 AM

    Hi Steve,

     

    I agree. This would be a one-time activity, their next schedules would be run automatically.

     

    This is to weed out currently non-required jobs.

     

    Regards

    Guru



  • 20.  Re: Bulk ON_HOLD action

    Posted Oct 12, 2017 09:27 PM

    I second Steve. Users are likely to feel crippled if they have to force start every time. I am still not sure what you mean by non-required jobs? Do you mean jobs that are no longer needed, but are still active (scheduled and run) and hence weighing down the Event Processor? If you go down the global auto-hold route, how long will you leave the EP in that mode? How do you exactly identify non-required jobs then? Look at jobs that didn't run during the time-period EP was in global auto hold?

     

    Cheers,

    Chandru



  • 21.  Re: Bulk ON_HOLD action

    Posted Oct 16, 2017 05:48 AM

    Hi Chandru,

     

    Yes, by non-required jobs, i mean the jobs that can be parked (ON_HOLD). This is bringing down performance.

    I would not start the instance in global auto-hold mode.

    I will get a list of all the jobs that are "independent jobs" (there is no box name for these jobs). Put them ON_HOLD. Ideally this should stop all the jobs in the instance, except the ones that depend on external jobs/etc.

    As a one-time activity, instruct the users to start force_start the jobs that they require. From the next schedule, they will be started automatically.

     

    I reduced the EPCount from 16 to 14, to no avail. The performance issue persists.

     

    If I force start a job, it takes around 4 mins to get the STARTJOB event created. Then, 4 mins to execute the command. 

    The STARTJOB event gets processed immediately (I assume this is because the remote agent is directly updating the databases).

    RUNNING and SUCCESS/FAILURE events are delayed.

     

     

    Example:

     

    Job Name Last Start Last End ST Run Pri/Xit
    ____________________________ ____________________ ____________________ __ _______ ___

    GTEST1 10/16/17 04:14:38 10/16/17 04:16:19 SU 406962996/3

    Status/[Event] Time Ntry ES ProcessTime Machine
    -------------- --------------------- -- -- --------------------- -------
    [FORCE_STARTJOB] 10/16/17 02:06:43 0 PD 10/16/17 02:10:06
    <user@machine name>
    STARTING 10/16/17 02:10:06 2 PD 10/16/17 02:10:11 <machine name>
    RUNNING 10/16/17 02:10:09 2 PD 10/16/17 02:14:40 <<machine name>>
    SUCCESS 10/16/17 02:11:49 2 PD 10/16/17 02:16:06



  • 22.  Re: Bulk ON_HOLD action

    Posted Oct 24, 2017 07:35 AM

    Hi Chandru/Steve,

     

    Any thoughts..



  • 23.  Re: Bulk ON_HOLD action

    Posted Oct 24, 2017 07:29 PM

    Hi Guruprasad,

    You need to test various EPCount settings, starting with 6, to arrive at a sweet-soft.

    It appears though the clog is at the database side. Have you re-indexed the database?

    What is the row count on event and proc_event tables? What is the data retention period. Have you been archiving data regularly?

     

    Thanks,

    Chandru



  • 24.  Re: Bulk ON_HOLD action

    Posted Oct 27, 2017 02:48 AM

    Thank you Chandru!

    Here are the counts:

    xql>>[DBServer][DBName> select count(*) from event
    xql>>[DBServer][DBName] 2> ;

    -----------
    21422

     

    xql>>[DBServer][DBName>select count(*) from proc_event
    xql>>[DBServer][DBName] 2> ;

    -----------
    1990571

     

     

    We store data for just 2 days.

     

     

    I shall try the EP count settings and update.

     

    Regards

    Guru



  • 25.  Re: Bulk ON_HOLD action

    Posted Oct 27, 2017 08:40 AM

    Hi Chandru,

     

    I tested setting EPCount value from 6 to 15. The latency is consistent and around 5 mins or more.

     

    I have been collecting latency data, its the difference of event_time_gmt and que_status_stamp in proc_event table.

    For every 3 mins, the records are:

     

    10 27 2017 08:10:17 No of  EVENTS 1012 max LATENCY 458 Average latency 303 seconds
    10 27 2017 08:12:23 EVENTS 927 LATENCY 512 Average 321
    10 27 2017 08:14:30 EVENTS 1176 LATENCY 554 Average 388
    10 27 2017 08:16:37 EVENTS 1127 LATENCY 576 Average 427
    10 27 2017 08:18:42 EVENTS 1077 LATENCY 556 Average 381
    10 27 2017 08:20:48 EVENTS 1067 LATENCY 519 Average 393
    10 27 2017 08:22:54 EVENTS 1149 LATENCY 496 Average 358

     

     

    So there are around 1000 events in 3 mins and latency is around 500 seconds.

     

     

    Compared to other normal loaded instances:

     

    10 27 2017 08:15:59 No of EVENTS: 265 max LATENCY: 13 seconds Average latency: 3 seconds
    10 27 2017 08:18:01 EVENTS 177 LATENCY 9 Average 3
    10 27 2017 08:20:03 EVENTS 547 LATENCY 19 Average 3
    10 27 2017 08:22:04 EVENTS 212 LATENCY 10 Average 3
    10 27 2017 08:24:06 EVENTS 363 LATENCY 45 Average 14
    10 27 2017 08:26:08 EVENTS 563 LATENCY 108 Average 13

     

     

    Load is around 5 times more.



  • 26.  Re: Bulk ON_HOLD action

    Posted Nov 16, 2017 09:45 AM

    Hi Gents,

     

    There are some  (interesting) updates:

     

    We re organized  the event_PUC index on event table on both the databases.

     

    When the instance runs in single server mode, the performance is flawless.No latency.

     

    Once the instance starts running in dual server mode the latency starts again. 

    Not sure if the issue is with the other DB or with the syncing issue between the DBs.

     

    I will try to run the instance from the other DB only. 

     

    As per my understanding, both the DBs are seperately updated by the Event Processor. Is this not the case?

     

    Should I try t reorg indexes on proc_event table?

     

    For now, the instance runs on single server mode, with no performance issues.



  • 27.  Re: Bulk ON_HOLD action

    Posted Nov 16, 2017 09:47 AM

    reorg rebuild event event_PUC