Limiting "waiting for end of parallel task(s)" runs

[Michael_Coxson_5769]

We have a number of Schedules for Workflows that, if they become Blocked, will pile up "waiting for end of parallel task(s)" runs.  Is there a way to limit or cancel these parallel runs after a given period in the Queue?  We are coming from Dollar Universe where each execution had a launch window that would stop the execution attempt after a default period, but I haven't found there for this tool yet.

[Pete Wirfs]

So on the Attributes tab you must have set a value in the "Max Number" for "Tasks running parallel", correct?

It defaults to "Wait" when a collision occurs.  You could change this setting to "Abort".  (I wish there was an option to skip(?))

[Michael_Coxson_5769]

Yeah that's the issue, our monitoring tool will collect failures and report them, so we want to avoid that. If there's no built-in tool I can whip something up to clear them once every period but it just seems silly. 

[Pete Wirfs]

What if you set "Max Number" back to zero, and add a new step to the top of the workflow that runs a script (or a precondition rule) to check to see if another instance is active?  If it detects that another instance is active, then it sets a variable that tells all of the remaining steps in the workflow to SKIP.

I don't know if this might be a catch-22 though... it might always detect that itself is running.  So it might require an SQLI database query to return a count of how many instances are running?

[Michael_Coxson_5769]

Yeah another issue is that teams have an expectation coming from Dollar Universe that they will receive a notice after a period when an execution fails due to conflict (provided it's a critical session). If they don't receive that notice they might miss critical windows. I'm considering having a process that checks every minute or two for any "waiting" process, checks their activation time against the current time, and if it exceeds a set period it cancels the process and calls a notification for the team. It sounds goofy but it might be necessary to get an equivalent service. 

[Anon Anon]

Ha? that sounds a bit weird indeed...

If thats a critical session and stops/gets blocked - why don´t you handle this issue but want to handle the next step - the waiting for end of  parallel tasks... ? In my opinion thats to late to react on an error.

The simplest way would be alerting (email, message, ...) as soon as the (one and only) workflow stops running. Then your OPS staff can handle this issue and solve it.

The max parallel 1 clause would be the next step on another level of error handling.

[Michael_Coxson_5769]

We already do alert on aborts for critical processes and for most instances that should be enough. However, what if get application team takes two and a half hours to evaluate the issue on their end and the Job/Workflow is scheduled for once every hour?  Two executions would be queued for processing and could potentially harm data depending on their exact purpose. A similar example would be a Job that normally takes 20 minutes to process data but either due to a large amount of data or a hung process takes an hour and a half. Yes, we can alarm on MRT and notify the team, but are they expected to halt their Schedule immediately upon receipt of the notice?

If this were a matter myself and team were directly administrating it wouldn't be an issue, but we are designing schedules for a rather large enterprise and need automated solutions for potential common problems. 

[Anon Anon]

Michael Coxson said:

 Two executions would be queued for processing and could potentially harm data depending on their exact purpose.

In this case I think the max parallel =1 with else action= ABORT would be the best solution. So you can be sure that no Data damage can happen with an "unwanted" workflow running. This is basically the same as your request - recurrently checking if there are tasks in "waiting for end of  parallel tasks" state.

Otherwiese you could periodically start a script checking if there are Workflows of that kind in "waiting for end of  parallel tasks" state and cancel them via script - but I personally would not recommend that ina PROD env ...

[Michael_Coxson_5769]

This is definitely a solution, though it would alarm on abort. I suppose this is the equivalent of the "Time Overrun" alarm within Dollar Universe, so that may be okay. It would be nice if it had the Skip option as Pete mentioned. I'll discuss with the team and probably make the maintenance process anyway, if only for a good practice exercise. 

[Anon Anon]

Hmmm you could do a trick - starting the Workflow via Script (from the Schedule) and do a check if any Workflow instances ar running e.g.

:SET &OBJECT# = "JOBP.CHECKPARALLEL_WITH_SCRIPT"

:SET &RunID# = GET_UC_OBJECT_NR(&OBJECT# )

:IF &RunID# <> "" OR " "
:  SET &STATUS# = GET_UC_OBJECT_STATUS (JOBP,&RunID#)
:    P "Parallel task &RunID# for OBJ &OBJECT# with state &STATUS# detected."
!    Maybe you can use a CANCEL_UC_OBJECT here...
:  else
:    P "NO Parallel task detected. Starting Workflow...." 
:    S &RET# = ACTIVATE_UC_OBJECT(&OBJECT#)
:    P "&RET# &OBJECT# Started."
:ENDIF

[Michael_Coxson_5769]

Here's a check against objects in "waiting" state which should cancel them if they exceed the threshold (in this instance 10 minutes).  If this is scheduled to run consistently in the background it should allow for a grace period in execution and then cancel them afterward.  I haven't written in the notification to the team, but that would come next.

This uses a VSQLI call for the following:

select EH_NAME,EH_AH_IDNR,EH_ACTIVATIONTIME from EH where EH_CLIENT=&$CLIENT# and EH_STATUS=1709 order by EH_NAME

:SET &PARALLELS# = PREP_PROCESS_VAR(TOOLS.VSQLI.PARALLELRUNS) :SET &CURRENTDATE# = SYS_DATE("YYYY-MM-DD") :SET &CURRENTTIME# = SYS_TIME("HH:MM:SS") !Read through all parallel runs and get their activation times :PROCESS &PARALLELS# :  SET &OBJNAME# = GET_PROCESS_LINE(&PARALLELS#,2) :  SET &OBJRUNID# = GET_PROCESS_LINE(&PARALLELS#,3) :  SET &OBJTIME# = GET_PROCESS_LINE(&PARALLELS#,4) !Compare activation date against current date :  DEFINE &ACTSPLIT#,STRING,2 :  FILL &ACTSPLIT#[] = STR_SPLIT(&OBJTIME#," ") :  SET &DATEDIFF# = DIFF_DATE("YYYY-MM-DD:&ACTSPLIT#[1]","YYYY-MM-DD:&CURRENTDATE#") :  SET &DATEDIFF# = FORMAT(&DATEDIFF#,"000") !Compare activation time against current time :  DEFINE &TIMESPLIT#,STRING,2 :  FILL &TIMESPLIT#[] = STR_SPLIT(&ACTSPLIT#[2],".") :  SET &TIMEDIFF# = DIFF_TIME("HH:MM:SS;&TIMESPLIT#[1]","HH:MM:SS;&CURRENTTIME#") !:  PRINT "ACTIVATION TIME: &ACTSPLIT#[1] &ACTSPLIT#[2] ||| CURRENT TIME: &CURRENTDATE# &CURRENTTIME# ||| DIFFERENCE: &DATEDIFF# &TIMEDIFF#" !If the wait time exceeds 10 minutes, cancel the waiting Object execution :  IF &TIMEDIFF# > "001000" :    SET &DAYHH# = MULT(&DATEDIFF#,24) :    SET &HH# = STR_CUT(&TIMEDIFF#,1,2) :    SET &HH# = ADD(&HH#,&DAYHH#) :    SET &HH# = FORMAT(&HH#,"00") :    SET &MM# = STR_CUT(&TIMEDIFF#,3,2) :    SET &SS# = STR_CUT(&TIMEDIFF#,5,2) :    PRINT "&OBJNAME# (RUNID &OBJRUNID#) HAS EXCEEDED ITS RUN GRACE PERIOD BY &HH#:&MM#:&SS#" :    SET &CANCEL# = CANCEL_UC_OBJECT(&OBJRUNID#,"ENDED_TIMEOUT") :  ENDIF :ENDPROCESS :CLOSE_PROCESS &PARALLELS#

This method has excess information and is rather verbose, but at this point that would be to allow differing grace period options per team if we wanted to sort by team rather than a one-size-fits-all approach.

Thoughts?