Automic Workload Automation

 View Only
Expand all | Collapse all

Dependencies between UC4 systems

Michael A. Lowry

Michael A. LowryNov 18, 2022 06:03 PM

  • 1.  Dependencies between UC4 systems

    Posted Nov 08, 2022 08:20 AM
    We are in the progress of migrating from one cloud platform to another. During the transition phase, we will have two UC4 systems running side-by-side, and workflows will be gradually migrated from the old system to the new one. It would greatly ease the migration if we could set up dependencies between tasks in one UC4 system and tasks the other.

    Is there a way to do this?


  • 2.  RE: Dependencies between UC4 systems

    Posted Nov 08, 2022 09:52 AM

    Hi,

    I do not think that this can be done out of the box. Do you talk about external dependencies set in a workflow?

    Regards,
    Juergen



    ------------------------------
    Juergen Lechner
    Managing Consultant
    setis GmbH
    Germany
    ------------------------------



  • 3.  RE: Dependencies between UC4 systems

    Posted Nov 09, 2022 03:41 AM
    Hi,

    I also don´t know a solution for this out of the box.

    But a workaround could be triggering jobs via REST-API, which are set as external dependencies.
    For example:
    Workflow A in system A triggers at some point job A via REST-API in system B
    The workflow in system B has an external dependency for job A.

    Not an elegant way, but it should work.

    Regards,

    Stephan Schiller


  • 4.  RE: Dependencies between UC4 systems

    Posted Nov 09, 2022 03:50 AM
    Good morning,

    since this is a short term workaround, it will be the only way of doing this. It is a direct approach that does not need third party tools or data / status storage. I was also thinking about this too. But before that I wanted to know that I have understood the question. :-)

    Regards,
    Juergen

    ------------------------------
    Juergen Lechner
    Managing Consultant
    setis GmbH
    Germany
    ------------------------------



  • 5.  RE: Dependencies between UC4 systems

    Broadcom Employee
    Posted Nov 09, 2022 06:10 AM
    Hi Michael,
     
    I recommend creating dummy SCRI objects in the target system and using them as external dependencies to the dependent job. Using the RESTAPI (or any of our APIs) you can then execute the dummy SCRI object to fulfill the external dependency. Make sure you include somehow the name of the predecessor task name in the name of the dummy SCRI so that you can see which predecessor in the calling Automic instance fulfills the dependency.
     
    In theory you could also use SYNC objects (SET_SYNC), but in my opinion the usage of dummy SCRI objects is much more visible for the user. 
     
    Regards, Markus



  • 6.  RE: Dependencies between UC4 systems

    Posted Nov 09, 2022 06:32 AM
    Besides the already mentioned REST-API solution, Broadcom still ships the old "CallAPI" utilities for multiple operating systems (API\CallAPI folder from image file).
    Platforms for CallAPIs (automic.com)
    With this, the step supposed to trigger an object on the target system would be an OS step, that runs the CallAPI executable with connection information for the target system, and the script to trigger the object there using AE script language.

    Similar result using REST-API, also not elegant, but also prevents "too much" custom code.


  • 7.  RE: Dependencies between UC4 systems

    Posted Nov 18, 2022 01:59 AM
    Edited by Michael A. Lowry Nov 18, 2022 03:06 AM
    My first attempt to post this reply failed, and the content was lost. Trying again.

    As I see it, there are two main approaches to setting up dependencies between jobs in different UC4 systems.
    1. Polling from successor system
    2. Trigger from predecessor system

    Current set-up

    Approach 1: polling from successor system

    Approach  2: trigger from predecessor system



    Approach

    Pros

    Cons

    1. Polling from successor system

     

    • Simpler for batch developers to implement.
    • No changes required to predecessor workflows.
    • Polling means more network traffic and at least some delay between completion of the predecessor job and start of the successor job.
    • Development of status checker job could be somewhat involved.

    2. Trigger from predecessor system

     

    • No polling, just a single trigger upon completion of the predecessor task.
    • Requires changes to both the predecessor and successor workflows.
    • Requires even more work if more than one successor follows the same predecessor.


    Did I overlook anything? Thanks in advance for additional feedback and suggestions.




  • 8.  RE: Dependencies between UC4 systems

    Posted Nov 18, 2022 02:30 AM
    Edited by Michael A. Lowry Nov 18, 2022 03:05 AM
    I can already say that we are leaning toward the first approach: polling from the successor system. The main reason is simplicity for the people developing the workflows. There are hundreds of "batch developers" in the company, each responsible for his or her own batches of workflows and related objects. We don't want to force them to make big changes to their workflows. The more painless we can make the migration for them, the more likely they will be to embrace the change.

    This brings me to the status checker job. It probably won't be a job, but rather a workflow of its own, with several child objects that do the work of resolving the remote dependency. The status checker will have to take several parameters. These are derived from the parameters one can specify on external dependencies:
    • Object name
    • Within parent
    • With alias
    • Task status
    • Same logical date
    Additionally. the status checker will need to read several parameters to tell it in what UC4 system to find the predecessor task, how often to check, and so on:
    • Remote UC4 system name
    • Polling interval
    • Maximum number of polling attempts
    • Maximum time to check
    Once we have developed the status checker, it'll be relatively straightforward for batch developers to drop in this object in place of external dependencies setting the required parameters when they do so.

    The parameters will likely be set via object variables overridden at the workflow task level. (We might even use a prompt set for this, to aid input of valid values. The sticking point is probably finding a way to ensure that all prompt set values are set ahead of time, so that execution is not interrupted when the status checker starts.)



  • 9.  RE: Dependencies between UC4 systems

    Posted Nov 18, 2022 02:49 AM
    In your polling approach you could get some trouble if "JOB2" runs more often than one time. It could be possible that the wrong "JOB2" is detected. If this doesn´t occur in your system or the parameters in combination (object name, within parent, with alias) are unique, the polling approach should be fine.

    Regards

    Stephan Schiller


  • 10.  RE: Dependencies between UC4 systems

    Posted Nov 18, 2022 02:56 AM
    Edited by Michael A. Lowry Nov 18, 2022 03:04 AM
    Yeah, there are options one can specify to fine-tune the behavior of external dependencies. In our case, recreating this logic is probably overkill, because very few people are actually using these options. The MVP will cover the most common use case. We can add features later if necessary. :)



  • 11.  RE: Dependencies between UC4 systems

    Posted Nov 18, 2022 03:06 AM

    There are several solutions to the parallel jobs run.
    1. Modify the Alias and add the RunID to the Alias in the pre processing 
    2. When triggering the job, add the RunID as Key and the job name as value to a VARA. The status checker checks the VARA and deletes the entry when the job is finished

    We are using both (regarding to the usage) successfully since years.

    Regards

    Peter



    ------------------------------
    Best regards

    Peter Gross
    Senior System Engineer
    Raiffeisen Schweiz
    St. Gallen Switzerland
    ------------------------------



  • 12.  RE: Dependencies between UC4 systems

    Posted Nov 21, 2022 06:17 AM
    Edited by Michael A. Lowry Nov 21, 2022 06:22 AM
    I have developed a proof of concept for a workflow that can be used as a drop-in replacement for external dependencies. The dependency options are specified via a prompt set. A REST job fetches the remote task status, and a second REST job fetches parent task info if necessary. Retries are accomplished via task post-conditions.


    All of the checks up to and including the check for same logical date are working. The remainder are yet to be implemented, and may not be depending on how many people are actually using these options today.

    And speaking of that, here's a little SQL query I wrote that lists all but the most commonly used types of external dependencies in the AE DB.

    SELECT OH_Name, JPP_Object, JPP_ParentObject, JPP_Alias, JPP_ParentAlias, JPP_ExtWhen, JPP_ExtSLTType, JPP_ExtSLTWithin,
    JPP_ExtElse, JPP_ExtElseAlarm, JPP_ExtTimeout, JPP_ExtTimeoutE, JPP_ExtTimeoutA, JPP_ExtExecute, JPP_PreCntExt
    FROM OH LEFT JOIN JPP ON OH_IDnr = JPP_OH_IDNR
    WHERE OH_CLIENT = 100
    AND OH_DeleteFlag = 0
    AND ( JPP_ParentObject IS NOT NULL OR
    JPP_ParentAlias IS NOT NULL OR
    JPP_ExtElse <> 0 OR
    JPP_ExtElseAlarm <> 0 OR
    JPP_TimeoutFlag <> 0 OR
    JPP_ExtExecute IS NOT NULL )
    ORDER BY OH_Name, JPP_ParentObject
    This will help us to get an idea of how prevalent the different kinds of options are, so that we can decide which ones are worth the effort of recreating in the dependency checker tool.


    Here are a couple more SQL queries.

    Populate the Check expected status pick list

    SELECT ZUTYP_STATUS
    FROM UC_ZUTYP
    WHERE ZUTYP_ISALIVE = 0
    ORDER BY ZUTYP_STATUS ASC


    Fetch the list of status codes for a given ZUTYP_STATUS

    SELECT ZUTYP_STATUSRANGE
    FROM UC_ZUTYP
    WHERE ZUTYP_STATUS = ?

    The JOBI that expands this into a full comma-delimited list has already been posted to another thread.

    More to come.




  • 13.  RE: Dependencies between UC4 systems

    Posted Nov 30, 2022 01:01 PM
    Edited by Michael A. Lowry Dec 01, 2022 06:30 AM
    The PoC has now evolved into a nearly feature-complete remake of the External Dependency feature, but for cross-system dependencies. I'm just working on a few remaining details, and I have a few questions:

    • When the mismatch option is set to wait, how often is the dependency checked? (In one test, it looked like it was every 20 seconds.)
      • When the option to execute an object on mismatch is also enabled, is this object executed every time the dependency is checked and fails to match?
    • There is only one field for object to execute upon mismatch and/or timeout. Is there in way to tell in the called task, whether the task was called due to a mismatch or timeout?
    • If the mismatch or timeout option is set to skip, the external dependency task ends with the status ENDED_SKIPPED. Successor tasks might be configured to act upon the basis of the predecessor's status. Is there really no way to force a task (e.g., a workflow) into the ENDED_SKIPPED status, to mimic the behavior of external dependency tasks?



  • 14.  RE: Dependencies between UC4 systems

    Posted Nov 18, 2022 06:03 PM
    Edited by Michael A. Lowry Nov 21, 2022 07:08 AM
    Duplicate post due to Higher Logic bug.


  • 15.  RE: Dependencies between UC4 systems

    Posted Nov 22, 2022 05:59 AM
    Edited by Michael A. Lowry Nov 22, 2022 05:59 AM
    I'm trying to find out how prevalent the same logical date option is in the external dependencies in our UC4 systems.
    I cannot find the field in the JPP table that stores this information. Does anyone know?

    Ping @Joel Wiesmann & @Philipp Elmer.
    ​​


  • 16.  RE: Dependencies between UC4 systems

    Posted Nov 22, 2022 08:07 AM
    Edited by Christian Boeck Nov 22, 2022 08:23 AM