ESP dSeries Workload Automation

 View Only
  • 1.  force job to complete after x minutes and continue on with successors

    Posted Apr 22, 2020 06:11 PM
    We have a requirement that if a Job doesn't finish running after X amount of time, then we should force complete the Job to allow for it's successors to begin running.

    I have tried to setup a test for this like the workflow presented below.

    1. Create e new application with 2 jobs. The first job is a Unix job that sleeps for 2 minutes. The second job is a Unix job that runs an echo command.

    2. In the Unix sleep job, I set a Time Dependency to be marked as Overdue if execution time exceeds 1 minute.

    3. In the Job Notifications, I setup an Alert to monitor OVERDUE states

    4. The OVERDUE Alert condition will trigger a Javascript to run that I thought would force complete the Job.
    // Force complete the current job
    // Tests WOB._qualifier to see if the current job has a qualifier.
    if (WOB._qualifier == null)
    {
       execCommand('%WOB._name', '%(APPL._name).%APPL._gen', 'ACTION COMPLETE');
    }
    else
    {
       execCommand('%(WOB._name).%WOB._qualifier', '%(APPL._name).%APPL._gen', 'ACTION COMPLETE');
    }


    When I trigger the Application to run, at 1 minute I see the Unix Sleep job is marked as Overdue, but the Sleep Job continues to run in EXEC state until it finishes sleeping at 2 minutes.

    Any ideas on what I could be missing here to force this job to stop executing and complete after 1 minute?


  • 2.  RE: force job to complete after x minutes and continue on with successors

    Broadcom Employee
    Posted Apr 23, 2020 05:31 AM
    Hi,

    Looking the command , it looks good. However , if it is not able to force complete the job , the reason will be get only from the server tracelog.txt file for any ERROR  message which can indicate why it failed.

    Ravi Kiran


  • 3.  RE: force job to complete after x minutes and continue on with successors

    Broadcom Employee
    Posted Apr 23, 2020 10:31 AM
    I was curious. I implemented the code.
    Below is the log. The string /MAIN/DPTESTLATEALERT.3/UNIX_1 looks like it has a "."(period) after the job name when there is no qualifier  I added a qualifier and it worked successfully.  I would guess that it is going through the wrong side of the "IF" condition. 

    NOTE:
    This command will COMPLETE the job in dSeries but I believe the job will continue to run on the server.  It may be better to also issue a "kill -9" to the process. 

    ***********************************
    LOG
    ***********************************

    20200423 09:53:34.436 [incoming:messages] [INFO] DM.Appl.DPTESTLATEALERT.3: [2020-04-23_09:53:34.436] Processing AFM: 20200423 09533442+0400 DSERIES_DON DSERIES_DON /MAIN/DPTESTLATEALERT.3/UNIX_1. ACTION COMPLETE Programmatic Userid(user.POWDO03)

    20200423 09:53:34.441 [essential] [INFO] DM.Appl.DPTESTLATEALERT.3: [2020-04-23_09:53:34.441] ERROR: Job DPTESTLATEALERT.3.UNIX_1. is not an active job managed by the Manager.

    20200423 09:53:34.441 [incoming:messages] [INFO] DM.Appl.DPTESTLATEALERT.3: [2020-04-23_09:53:34.441] Finishing AFM: 20200423 09533442+0400 DSERIES_DON DSERIES_DON /MAIN/DPTESTLATEALERT.3/UNIX_1. ACTION COMPLETE Programmatic Userid(user.POWDO03)


    Don


  • 4.  RE: force job to complete after x minutes and continue on with successors
    Best Answer

    Broadcom Employee
    Posted Apr 23, 2020 10:49 AM
    Hi, 
    I think this is the solution.  Change the "IF" statement to the one below. 
    if (WOB._qualifier == "")


    Don


  • 5.  RE: force job to complete after x minutes and continue on with successors

    Posted Apr 23, 2020 11:19 AM
    Thanks for your time Don! This did the trick for me, and it's working with and without a qualifier now.

    I also added in a KILL action to kill the process on the server. Not sure if the end-user will want this or not, but either way this should address their needs.

    Thanks!
    Cory


  • 6.  RE: force job to complete after x minutes and continue on with successors

    Broadcom Employee
    Posted Apr 23, 2020 11:59 AM
    Hello,
    There are couple of ways you can force a job to complete if it is overdue.
    The best way is to use an ALERT (call it WAIT_OVERDUE).  You define a alert with a JavaScript like this:

    execCommand('YOUR_JOB_NAME','%(APPL._name).%APPL._gen','ACTION COMPLETE');​

    Just replace 'YOUR_JOB_NAME'  with actual job name in the above command, leave everything as is.

    Then in the Job, go to Time Dependencies and setup the Overdue to when you want the job to be completed.
    Set overdue in Time Dependencies

    Next, go to Notifications and set your Alert for Overdue


    Set the Alert for Overdue

    When your job runs for, say 1 minute, and has not finished, the job will go in Overdue.  The Notification will the alert for Overdue.  The alert will run the JavaScript which will force complete your job.

    HTH,

    Nitin Pande

    ------------------------------
    Support
    Broadcom
    Toronto
    ------------------------------