Work flow dependencies and coordination

[Nikhil_Karel_9136]

Hi all,
Let me jump into the question right away. We have two different workflows running in different client numbers and clusters. The data generated as output of WF1, WF2,WF3,WF4.....WF60 and so on is needed as input to run the WF100. For which we cross verifying the WF1,WF1, WF2,WF3,WF4....WF60  are running or done and then move the data from 1 cluster to another for which we have a shell command. If we move the data when the tasks are running then there is 100% chance that the data copied is inaccurate/incomplete and input data to WF100 is not reliable and output of  WF100 is inaccurate. To address this issue we are currently going through each status of  WF1,WF1, WF2,WF3,WF4......WF60 is complete or running and then do a data transfer, this manual effort is taking up a lot of time. So is there a way I can wrap this problem up ?

Is there way for WF100 to know that the WF1,WF1, WF2,WF3,WF4....WF60 are running or done as a precondition check? If done then launch the data move-shell command so that the data is copied from 1 cluster to another, the WF100 has the input data needed to perform the task designated to it.



Thanks,
Nikhil

[Anon Anon]

There are several possibilities to fulfill this request.

Let me suggess some which come to my mind:

Uset WF1...WF60 as external dependency for WF100  (if in the same client)

or you could use triggerfiles (WF25_OK.txt, WF32_NOK.txt) or something else)

Also possible would be a VARA object that stores the Status of all Workflows.

Or a script (with SQL VARA) that checks the END_Status of all predecessor Workflows

Is there a possibility to check Data integrity with OS commands?

[Nikhil_Karel_9136]

Hi Wolfgang,

Thanks for the response, the 60 small work flows are in a Client 1 and WF100 is in client 2. I have not set the WF1,WF2...WF60 as external dependency for WF100. Is it possible to do so ? that would be something I can explore.

Doing the data integrity is next step to check if data is accurate in location 1 and location 2 after transfer.

Can you please elaborate on the below,

"or you could use triggerfiles (WF25_OK.txt, WF32_NOK.txt) or something else)

Also possible would be a VARA object that stores the Status of all Workflows.

Or a script (with SQL VARA) that checks the END_Status of all predecessor Workflows"


can you provide me steps or scripts ? screen shots or documentation link to set this up in my case ?
Using Atomic 9.0



Any info is appreciated. Thanks

[Pete Wirfs]

We have systems that use trigger files.  First you set up a shared file folder on the network.  When WF60 finishes, it saves a trigger file called "WF60_OK.txt" in this folder.  WF100 has a task that watches and waits for the arrival of  WF60_OK.txt before running the rest of the workflow.

This could also be solved with CALLAPI.  CALLAPI is an automic batch utility that lets you issue commands to the client of your choice.  For instance WF60 could run a CALLAPI solution that would cause WF100 to execute in the other client.  (However I've never done it this way.)

[Keld Mollnitz]

Have you considered using a Sync object ?

[Anon Anon]

Nikhil_Karel_9136Hmm Iam not sure if external dependency will work across 2 clients.

I like Petes Idea very much. Its simpler and more performant than mine.

You can (as an option) let the last job in the workflow check all prior jobs via dependency and start your WF100 via CALL API.

https://us.v-cdn.net/5019921/uploads/editor/lc/dtrrn9v3ujl4.jpg" width="1344">

CALL API call could be: ucxbxxxc script=script.txt

and script.txt contains your Automic script e.g.
:set &RET# = ACTIVATE_UC_OBJECT(JOBS.WIN.DIR)

more info - installation:
https://docs.automic.com/documentation/WEBHELP/English/AWA/11.2/AE/11.2/All%20Guides/help.htm#ucacht.htm%3FTocPath%3DAdministration%2520Guide%7CInstallation%7CNew%2520Installation%7CInstallation%2520Procedure%7CInstalling%2520the%2520CallAPIs%7C_____10

usage:
https://docs.automic.com/documentation/WEBHELP/English/AWA/11.2/AE/11.2/All%20Guides/help.htm#ucacif.htm%3FTocPath%3DApplication%2520Integration%2520Guide%7CCallAPI%7C_____2

[Nikhil_Karel_9136]

@Pete Wirfs 

We have the current set up as WF1,WF2,......WF60 in client A, all are different workflows and are not related to each other, we have an another big WF in Client B which has sub work flows in them. These sub workflows depend on 1 or more WF's (1to60) in client A.

Can you help me with the below questions?

•  Set up a shared file folder on the network - can you guide me how to do or share a link that can help.
  
  
• Earlier you have mentioned that when WF60 finishes, it saves a trigger file called "WF60_OK.txt" in this shared folder - But its not just the WF60 but I need to make sure WF1....F60 all are completed or not. So can we use WF100 to have a task that watches for all WF_OK.txt or just one WF60_OK.txt only?
  If its possible to have multiple WF_OK.txt, can you please explain.
  
  
• Thirdly, we also want to optimize the run time of WF100 by checking if we can set WF1 as external dependency to the Sub-workflows (within WF100) as external dependency. In this manner the Sub-workflows (within WF100) can complete when ever the workflows are ready (WF1 to WF60 ) rather than waiting for all 60 workflows to complete and then kick off WF100.

[Pete Wirfs]

By "shared file folder", I simply mean a file folder that all of your UC4 clients can update.  This could be put on your UC4 server, or it could be put on a different server.  The best practices at your shop should be followed.

Yes it is possible to have multiple trigger files.  Note that Wolfgang suggested this in his June 21 post above as well.

Regarding creating trigger files and using them at the right time is totally under your control.  You have to decide when to write the file, and you have to decide what event it should trigger.  The devil is in the details!  LOL!

The creation of a trigger file can be done with a very simple DOS command like this one;

echo Completed >WF99_OK.txt