Automic Workload Automation

 View Only
Expand all | Collapse all

Dependencies between UC4 systems

  • 1.  Dependencies between UC4 systems

    Posted 24 days ago
    We are in the progress of migrating from one cloud platform to another. During the transition phase, we will have two UC4 systems running side-by-side, and workflows will be gradually migrated from the old system to the new one. It would greatly ease the migration if we could set up dependencies between tasks in one UC4 system and tasks the other.

    Is there a way to do this?


  • 2.  RE: Dependencies between UC4 systems

    Posted 24 days ago

    Hi,

    I do not think that this can be done out of the box. Do you talk about external dependencies set in a workflow?

    Regards,
    Juergen



    ------------------------------
    Juergen Lechner
    Managing Consultant
    setis GmbH
    Germany
    ------------------------------



  • 3.  RE: Dependencies between UC4 systems

    Posted 24 days ago
    Hi,

    I also don´t know a solution for this out of the box.

    But a workaround could be triggering jobs via REST-API, which are set as external dependencies.
    For example:
    Workflow A in system A triggers at some point job A via REST-API in system B
    The workflow in system B has an external dependency for job A.

    Not an elegant way, but it should work.

    Regards,

    Stephan Schiller


  • 4.  RE: Dependencies between UC4 systems

    Posted 24 days ago
    Good morning,

    since this is a short term workaround, it will be the only way of doing this. It is a direct approach that does not need third party tools or data / status storage. I was also thinking about this too. But before that I wanted to know that I have understood the question. :-)

    Regards,
    Juergen

    ------------------------------
    Juergen Lechner
    Managing Consultant
    setis GmbH
    Germany
    ------------------------------



  • 5.  RE: Dependencies between UC4 systems

    Broadcom Employee
    Posted 23 days ago
    Hi Michael,
     
    I recommend creating dummy SCRI objects in the target system and using them as external dependencies to the dependent job. Using the RESTAPI (or any of our APIs) you can then execute the dummy SCRI object to fulfill the external dependency. Make sure you include somehow the name of the predecessor task name in the name of the dummy SCRI so that you can see which predecessor in the calling Automic instance fulfills the dependency.
     
    In theory you could also use SYNC objects (SET_SYNC), but in my opinion the usage of dummy SCRI objects is much more visible for the user. 
     
    Regards, Markus



  • 6.  RE: Dependencies between UC4 systems

    Posted 23 days ago
    Besides the already mentioned REST-API solution, Broadcom still ships the old "CallAPI" utilities for multiple operating systems (API\CallAPI folder from image file).
    Platforms for CallAPIs (automic.com)
    With this, the step supposed to trigger an object on the target system would be an OS step, that runs the CallAPI executable with connection information for the target system, and the script to trigger the object there using AE script language.

    Similar result using REST-API, also not elegant, but also prevents "too much" custom code.


  • 7.  RE: Dependencies between UC4 systems

    Posted 15 days ago
    Edited by Michael Lowry 15 days ago
    My first attempt to post this reply failed, and the content was lost. Trying again.

    As I see it, there are two main approaches to setting up dependencies between jobs in different UC4 systems.
    1. Polling from successor system
    2. Trigger from predecessor system

    Current set-up

    Approach 1: polling from successor system

    Approach  2: trigger from predecessor system



    Approach

    Pros

    Cons

    1. Polling from successor system

     

    • Simpler for batch developers to implement.
    • No changes required to predecessor workflows.
    • Polling means more network traffic and at least some delay between completion of the predecessor job and start of the successor job.
    • Development of status checker job could be somewhat involved.

    2. Trigger from predecessor system

     

    • No polling, just a single trigger upon completion of the predecessor task.
    • Requires changes to both the predecessor and successor workflows.
    • Requires even more work if more than one successor follows the same predecessor.


    Did I overlook anything? Thanks in advance for additional feedback and suggestions.




  • 8.  RE: Dependencies between UC4 systems

    Posted 15 days ago
    Edited by Michael Lowry 15 days ago
    I can already say that we are leaning toward the first approach: polling from the successor system. The main reason is simplicity for the people developing the workflows. There are hundreds of "batch developers" in the company, each responsible for his or her own batches of workflows and related objects. We don't want to force them to make big changes to their workflows. The more painless we can make the migration for them, the more likely they will be to embrace the change.

    This brings me to the status checker job. It probably won't be a job, but rather a workflow of its own, with several child objects that do the work of resolving the remote dependency. The status checker will have to take several parameters. These are derived from the parameters one can specify on external dependencies:
    • Object name
    • Within parent
    • With alias
    • Task status
    • Same logical date
    Additionally. the status checker will need to read several parameters to tell it in what UC4 system to find the predecessor task, how often to check, and so on:
    • Remote UC4 system name
    • Polling interval
    • Maximum number of polling attempts
    • Maximum time to check
    Once we have developed the status checker, it'll be relatively straightforward for batch developers to drop in this object in place of external dependencies setting the required parameters when they do so.

    The parameters will likely be set via object variables overridden at the workflow task level. (We might even use a prompt set for this, to aid input of valid values. The sticking point is probably finding a way to ensure that all prompt set values are set ahead of time, so that execution is not interrupted when the status checker starts.)



  • 9.  RE: Dependencies between UC4 systems

    Posted 15 days ago
    In your polling approach you could get some trouble if "JOB2" runs more often than one time. It could be possible that the wrong "JOB2" is detected. If this doesn´t occur in your system or the parameters in combination (object name, within parent, with alias) are unique, the polling approach should be fine.

    Regards

    Stephan Schiller


  • 10.  RE: Dependencies between UC4 systems

    Posted 15 days ago
    Edited by Michael Lowry 15 days ago
    Yeah, there are options one can specify to fine-tune the behavior of external dependencies. In our case, recreating this logic is probably overkill, because very few people are actually using these options. The MVP will cover the most common use case. We can add features later if necessary. :)



  • 11.  RE: Dependencies between UC4 systems

    Posted 15 days ago

    There are several solutions to the parallel jobs run.
    1. Modify the Alias and add the RunID to the Alias in the pre processing
    2. When triggering the job, add the RunID as Key and the job name as value to a VARA. The status checker checks the VARA and deletes the entry when the job is finished

    We are using both (regarding to the usage) successfully since years.

    Regards

    Peter



    ------------------------------
    Best regards

    Peter Gross
    Senior System Engineer
    Raiffeisen Schweiz
    St. Gallen Switzerland
    ------------------------------



  • 12.  RE: Dependencies between UC4 systems

    Posted 11 days ago
    Edited by Michael Lowry 11 days ago
    I have developed a proof of concept for a workflow that can be used as a drop-in replacement for external dependencies. The dependency options are specified via a prompt set. A REST job fetches the remote task status, and a second REST job fetches parent task info if necessary. Retries are accomplished via task post-conditions.


    All of the checks up to and including the check for same logical date are working. The remainder are yet to be implemented, and may not be depending on how many people are actually using these options today.

    And speaking of that, here's a little SQL query I wrote that lists all but the most commonly used types of external dependencies in the AE DB.

    SELECT OH_Name, JPP_Object, JPP_ParentObject, JPP_Alias, JPP_ParentAlias, JPP_ExtWhen, JPP_ExtSLTType, JPP_ExtSLTWithin,
    JPP_ExtElse, JPP_ExtElseAlarm, JPP_ExtTimeout, JPP_ExtTimeoutE, JPP_ExtTimeoutA, JPP_ExtExecute, JPP_PreCntExt
    FROM OH LEFT JOIN JPP ON OH_IDnr = JPP_OH_IDNR
    WHERE OH_CLIENT = 100
    AND OH_DeleteFlag = 0
    AND ( JPP_ParentObject IS NOT NULL OR
    JPP_ParentAlias IS NOT NULL OR
    JPP_ExtElse <> 0 OR
    JPP_ExtElseAlarm <> 0 OR
    JPP_TimeoutFlag <> 0 OR
    JPP_ExtExecute IS NOT NULL )
    ORDER BY OH_Name, JPP_ParentObject
    This will help us to get an idea of how prevalent the different kinds of options are, so that we can decide which ones are worth the effort of recreating in the dependency checker tool.


    Here are a couple more SQL queries.

    Populate the Check expected status pick list

    SELECT ZUTYP_STATUS
    FROM UC_ZUTYP
    WHERE ZUTYP_ISALIVE = 0
    ORDER BY ZUTYP_STATUS ASC


    Fetch the list of status codes for a given ZUTYP_STATUS

    SELECT ZUTYP_STATUSRANGE
    FROM UC_ZUTYP
    WHERE ZUTYP_STATUS = ?

    The JOBI that expands this into a full comma-delimited list has already been posted to another thread.

    More to come.




  • 13.  RE: Dependencies between UC4 systems

    Posted 2 days ago
    Edited by Michael Lowry 2 days ago
    The PoC has now evolved into a nearly feature-complete remake of the External Dependency feature, but for cross-system dependencies. I'm just working on a few remaining details, and I have a few questions:

    • When the mismatch option is set to wait, how often is the dependency checked? (In one test, it looked like it was every 20 seconds.)
      • When the option to execute an object on mismatch is also enabled, is this object executed every time the dependency is checked and fails to match?
    • There is only one field for object to execute upon mismatch and/or timeout. Is there in way to tell in the called task, whether the task was called due to a mismatch or timeout?
    • If the mismatch or timeout option is set to skip, the external dependency task ends with the status ENDED_SKIPPED. Successor tasks might be configured to act upon the basis of the predecessor's status. Is there really no way to force a task (e.g., a workflow) into the ENDED_SKIPPED status, to mimic the behavior of external dependency tasks?



  • 14.  RE: Dependencies between UC4 systems

    Posted 14 days ago
    Edited by Michael Lowry 11 days ago
    Duplicate post due to Higher Logic bug.


  • 15.  RE: Dependencies between UC4 systems

    Posted 10 days ago
    Edited by Michael Lowry 10 days ago
    I'm trying to find out how prevalent the same logical date option is in the external dependencies in our UC4 systems.
    I cannot find the field in the JPP table that stores this information. Does anyone know?

    Ping @Joel Wiesmann & @Philipp Elmer.
    ​​


  • 16.  RE: Dependencies between UC4 systems

    Posted 10 days ago
    Edited by Christian Boeck 10 days ago
    Hi Michael,
    IMHO - you can find the logical date only in the tables
    EH -> eh_ldate
    respectively
    AH -> ah_ldate

    Missunderstanding, you are looking for the checkbox column. I keep searching

    ------------------------------
    Thx & rgds
    Christian

    ------------------------------
    Managing Consultant
    Setis GmbH
    ------------------------------
    ------------------------------



  • 17.  RE: Dependencies between UC4 systems

    Posted 10 days ago
    Hi Michael,

    so now, I think you are looking for

    jpp_extslttype = 3

    table jpp

    ------------------------------
    Thx & rgds
    Christian

    ------------------------------
    Managing Consultant
    Setis GmbH
    ------------------------------
    ------------------------------



  • 18.  RE: Dependencies between UC4 systems

    Posted 10 days ago
    Edited by Michael Lowry 6 days ago
    Yes, that's it! This is not documented in the JPP Table schema documentation.




  • 19.  RE: Dependencies between UC4 systems

    Posted 10 days ago
    Edited by Michael Lowry 9 days ago
    To clarify, I'm looking for the field in the JPP table (or other table) that stores the status of the checkbox Check if the external task was activated with the same logical date as the workflow in the external dependency.

    Thanks to @Christian Boeck for answering the question. My current understanding is depicted below.




  • 20.  RE: Dependencies between UC4 systems

    Posted 9 days ago
    Edited by Michael Lowry 6 days ago
    Given how the controls work behind the scenes, I've reworked the PRPT a bit, combining the two timing-related controls into one. I also reworded the options to make them easier to understand.

    I'm not sure why the task timing options are implemented via two separate controls in the AE when a single radio button control is sufficient.

    Broadcom, feel free to use the above idea in a future product update. :)




  • 21.  RE: Dependencies between UC4 systems

    Posted 9 days ago
    Edited by Michael Lowry 9 days ago
    After looking more carefully at the documentation page External Dependency Tab, it became clear that implementing the remote dependency checker in way that is faithful to the current behavior of the external dependency feature will require more effort than originally anticipated.

    These bits in particular stood out in the documentation:

    Important! The system is designed to avoid overlapping situations. This means that an instance (an execution) of a task is used in only one instance (execution) of the Workflow in which it is referred to as external dependency.
    ...
    13:00 - The next execution of the Workflow starts but it must wait. Although the latest execution of the external task lies within the designated two hours, that execution has already been considered by the previous Workflow execution. In this case, the designated hour starts with the end of the latest instance (execution) of the Workflow, namely at 12:10.

    This means that the dependency checker will have to keep track of the run IDs of tasks that have previously matched, to avoid the situation wherein a single predecessor task fulfills the dependency checks for two or more instances of the same successor.

    We could track previously-fulfilled dependencies in a STATIC VARA.



  • 22.  RE: Dependencies between UC4 systems

    Posted 7 days ago
    Edited by Michael Lowry 7 days ago

    I'd like to know how the Automation Engine uniquely identifies external dependencies to ensure that subsequent instances of the same external dependency are not resolved by the same external task.

    I'm guessing it's something like the combination of the OH_Name of the workflow plus the EJPP_Lnr of the external dependency task.

    Can someone confirm this, or if it's not correct, let me know what the unique identifier is?




    • 23.  RE: Dependencies between UC4 systems

      Posted 6 days ago
      Edited by Michael Lowry 6 days ago

      In some quick tests, I found some evidence to support my hunch about the unique ID the AE uses for external dependency matching:

      U00011574 External dependency: 'EBM.DUMMY_TASK.SCRI(EBM.DUMMY_WORKFLOW.JOBP)' / Lnr: '00002' satisfied by Workflow RunID: '0434586903' / task RunID: '0434564643' / Lnr: '00002'.




    • 24.  RE: Dependencies between UC4 systems

      Posted 4 days ago
      Edited by Michael Lowry 4 days ago

      AFAIK, there is no predefined variable for the Lnr of a task, so I use this SQL query to look up the Lnr of a workflow task based on its run ID.

      SELECT EH_EJPP_Lnr FROM EH WHERE EH_AH_IDNR = ?


      I'm using this in an SQLI VARA in the process tab of the workflow.

      :SET &INPUT_RUN_ID# = &$RUNID#
      :PSET &STATUS_CHECKER_LNR# = GET_VAR(EBM.GET_WF_TASK_LNR.VARA_SEC_SQLI)

      (The SQLI VARA uses the script variable &INPUT_RUN_ID# in the bind parameter.)




    • 25.  RE: Dependencies between UC4 systems

      Posted 4 days ago
      Hi Michael

      I’m using this in the PreProcess of every JOBS and JOBP:
      :SET &RUNDID# = SYS_ACT_ME_NR()
      :SET &JOB_TYPE# = GET_STATISTIC_DETAIL(&RUNDID#,DST_HOST_TYPE,)
      :SET &TOP_JOBP# = &TOP_JOBP#
      :IF &TOP_JOBP# = ""
      :PSET &TOP_JOBP# = SYS_ACT_TOP_NAME()
      :ENDIF

      This is storing the absolute top level JOBP (regardless how many levels of sub workflows you have).
      It enables you to use the variable &TOP_JOBP# in every subsequent object (in our case for alarming.

      If you only need the runid of the parent workflow, just use:
      https://docs.automic.com/documentation/webhelp/english/AWA/11.2/AE/11.2/All%20Guides/Content/ucaagp.htm


      Viele Grüsse
      Peter