Sync objects and restarts

[Daryl Brown]

Background info: 
I'm working on a workflow that involves downloading two distinct files from a vendor, importing them in a particular order, and then running an extract once both files have been loaded.

In the interest of being able to restart the process to download-and-import each file, I've buried each process in its own subworkflow.  These two subworkflows will run in parallel when the overall workflow kicks off -- this way I can download the two files simultaneously -- but I use an sync object on the import jobs in each subworkflow to ensure the file from subworkflow #1 loads before the file from subworkflow #2.  

I've set up the sync object with the following states:
READY -- ready to start importing
LOCKED_IMP1 -- locked; importing file #1
DONE_IMP1 -- finished importing file #1; awaiting import of file #2
LOCKED_IMP2 -- locked; importing file #2

The import job for the first file will move the sync state from READY -> LOCKED_IMP1 -> DONE_IMP1.  The sync object will remain in LOCKED_IMP1 in the case of failure, as we don't want the second file to start importing until the first one has successfully completed.
The import job for the second file will move the sync state from DONE_IMP1 -> LOCKED_IMP2 -> READY.

Problem:
Now this design works pretty well....but it gets a little messy whenever I run into a problem with the first import.  Since that first failed run has set the sync object to LOCKED_IMP1, when I go to restart the job, it sits in 'Waiting for SYNC' until I manually update the sync object back to 'READY'.

So I'm seeing a few different ways I could tackle this, but wanted to get the community's opinion on the best approach:
1)  Update the import job to set the status back to READY on abend?
2)  Update the import job to ERR_IMP1 on abend, and create a new action to (re)start that job from that starting state?
3)  redesign the workflow entirely so that it doesn't rely on sync objects?

Any other ideas?  Thoughts?
How elaborate are other folks' Sync objects' states and actions?

[Anon Anon]

I’ve done something similar to what you are describing (I think).  A few Script objects were created that did nothing except have their Sync tabs specified with the appropriate Actions.  The actual tasks that I wanted to control had logic to test if they were being restarted or not using a SYS_ACT_RESTART and would then execute the appropriate Script object with ACTIVATE_UC_OBJECT and the WAIT parameter.  The activated Script object would handle the actual Sync processing to wait or end and then return to their activating task to continue execution. 

[Edited for completeness 10/29/2014 07:15 pdt]

You may also need to add Post Process logic to set the Sync after completion of the actual task.  We also sometimes use the SET_SYNC or GET_SYNC functions for additional actions.  Considering your requirements you may also need to include the Sync on the actual task to set the Abend or End actions or a combination of everything.

Hope I’ve described this in an intelligible manner.  Let me know if you need a better or expanded description.

[Pete Wirfs]

Personally I would choose option (3).  This is really just a predecessor/successor type of problem, isn't it?   Long term maintenance of the solution would be simplified.

[Daryl Brown]

@Mark -- I understand exactly what you're talking about, and I do see merit to that approach.  (I'm not sure I'm ready to adopt that approach just yet, but you've got me thinking about it now...)   You'd lose the ability to see something in a blue 'waiting for SYNC' state while monitoring the workflow, though, and I could envision Ops getting confused as to whether the job was running or waiting.  Has that been a problem in your experience?

@Pete -- It would definitely simplify the design if I did away with syncs, yes.  But I see some drawbacks to doing that as well:
1) Merge the two import workflows into a single workflow so that the two imports happen serially -- The danger here is if the first import fails, and the support team decides that we need to download a new file from the vendor.  I'd now be halfway through my workflow, and they'd be asking me to just restart certain steps.  It's doable...and I suppose I could bury those download parts in their own subworkflow... but the point is that restartability becomes uglier with this design.
2) Set the two import subworkflows to run in sequence rather than in parallel -- This would work (and would definitely be simpler), but I'd be wasting time waiting to download a file that we know is already available.  The overall workflow has to be completed by a certain time, so I was hoping to do as much parallel processing as possible.

[Anon Anon]

Daryl:

For us, we work very hard to not have Ops getting confused so we have attempted to implement nothing that requires routine monitoring on their part.  After all this is an automation tool.  

The intent is to have our Ops staff just monitor and react to exceptions or notifications and the overall health of the AE system itself and not that of individual scheduled tasks.  Many of our processes use various forms of AE's Runtime Supervision and in-house written logic to send notifications via email and paging of processes not starting or ending on time.  We also use BMC's Patrol to detect blocked process flows and request notifications and escalation.

Just as a note, we often add Archive Key and Comments at activation of these types of Sync Script objects to help determine who or why they were executed.  It assists in the rare case that we need to perform some type of diagnosis or recovery. 

So I guess my response is no, this has not been a problem in my experience.

[Legacy User]

Hi Daryl, 

I'd go with option 3 and solve it like that. 

- 2 separate download workflows that run in parallel
- 2 separate wrapper workflows that run in parallel and contain the
- 1 import workflow

The import workflow gets its parameters by object-variables set in its wrapper. By setting the "Max Parallel" Option of the import workflow to 1 and else "wait" you can make sure one import is running at a time. 

Hope I got your use case right .. 

[Daryl Brown]

Let me provide a little more detail on these two workflows; I think I may have confused people by trying to oversimplify them.

There are two files that I need to import here -- a "customer" file, and a "check transaction" file.  The customer file needs to be loaded first, because the check file may reference customers in the customer file; the transactions won't load properly if the corresponding customers don't already exist in the database.

Now the two import workflows as I've designed them so far are basically identical, with the following steps:

• FTP job -- download the file from the client to our local sftp server
• script -- mark the file as received in a variable object.  (I have a late file alarm script that I schedule separately to look at this variable object to verify if this file was marked as received; if not, it'll send an alert email.)
• File Transfer -- move the file from our sftp server to the application server where it will be imported
• Linux job -- import the file  ** This is the only job in this workflow that needs to be run in sequence -- the customer import file needs to be imported before the check import file.  **
• Linux job -- encrypt and archive the import file

@Geunter -- Hopefully this clarifies things.  The "max parallel" setting won't work because it's not just that I only want to run one import at a time -- I actually have to perform the import on a particular file first.

[Anon Anon]

Daryl:

Would the following work now that I think I understand your requirement a bit better?

1 - Mainworkflow – Set Sync Status to “incomplete”

2 - Subworkflow1  - Set Sync to Status “complete” when Import1 completes normally.

3 - Subworkflow2 – Import2 waits on Sync if Status not “complete”.

I don’t think that I see a reason to have the Sync’s Status have anything more than two states.  It appears to me that the only issue is that second load does not execute until the first load has completed. 

As usual, I could be missing some subtlety but I thought I’d offer this approach.

[Pete Wirfs]

If I understand correctly, the Linux job in the check transaction workflow needs to wait for successful completion of the Linux job in the customer workflow.

 This type of rule can be done as an external predecessor in V9. 

[Daryl Brown]

@Mark -- Very clever.  I'm used to locking and unlocking the sync object on only the particular tasks that need it, so I hadn't considered using them the way you're proposing here.  But it definitely looks like it would work.  

@Pete -- Yeah, I guess that would work also.  I haven't used external dependencies a lot because I often couldn't configure the lead time satisfaction the way I wanted, or else ran into problems finding the right dependency status to use.  But in this case I guess that's pretty straightforward...  I'll need to experiment with this one a bit, but this may actually turn out to be the easiest way to go.

[Anon Anon]

Not to denigrate External Dependencies, but we try to avoid them whenever possible though we do have a few.  Under OM V8 they pose too many restrictions and challenges for us.  We often have problems related to various exceptions such as early and late starts, restarts, certain calendar conditions, manual and "extra" executions and lead times as you noted.

Hopefully they have addressed some of these issues in more current versions and we will re-evaluate their usage.  Until then, our normal approach is to often use Syncs in the manner that I last described.

[Daryl Brown]

So out of the three of us, I guess that makes you the most comfortable with external dependencies, @Pete.  (Like Mark, we do use them here in a few places, but have generally found that they don't offer quite the level of control that we could get from a sync object.)

After some testing, I think they'll actually work quite well in this scenario, but let me ask you -- how would you configure the lead time satisfaction here?
I'm leery of the 'since last workflow execution' option, on the off chance that someone had to run the customer import manually at some random time.  That would leave the 'Within <hh:mm:ss> before Workflow start' as the most logical fit, but what time would you use for that?  (I'm thinking of using 10 minutes.)  The two workflows will be activated in parallel, so in theory the start time should be nearly identical, but theory and practice are two different things...

[Pete Wirfs]

In this case we would schedule both workflows to activate at the same time and use the "After Workflow start" option within the external dependency.

But if you want to be able to launch it with an unpredictable time delay, then I would set a "within" of an hour or two to support the biggest use-case time delay you think could happen.