Notify if job didn't started at specific time

[Amey .]

HI team, 

We need help on a  common scenario which many of you already resolved - 
Problem statement - We have JOB A and JOB B in a WORKFLOW, where JOB B runs at everyday 10:00 AM, we want to send a notification if JOB B didn't start executing at 10:00 AM, similar notification we want to  implement at WORKFLOW level as well. 

Solution we have tried - on time and dependency we have added time dependency as 10:00 AM and also at precondition we have added "if time is before 10:00 AM execute Task else notification object and execute task" 

Observations -  we are getting delay notification only when JOB B is getting active, but we want notification to be sent out as soon as time goes past 10:00 AM even though JOB B is in waiting for predecessor state.

Similarly for workflow , we have created a JOB schedule and added daily applications but due to some activity if we hold/Suspend the application then also we want to create a notification that workflow didn't starts on a specific day.

Thanks in advance
Regards,
Amey Deshpande

[Pete Wirfs]

So you have this WORKFLOW;

          START -> JOBA -> JOBB

I would try adding a JOBC that also runs immediately after START, but give it an earliest start time of 10:01, and give it a precondition rule that checks to see if JOBA has finished or not.

......... 

------------------------------
Pete Wirfs
SAIF Corporation
Salem Oregon USA
------------------------------

[Amey .]

Hi Pete,

We have a Daily workflow consisting of 150 jobs and out of that few jobs which are having multiple dependencies along with time checks. 

so having such JOB C will make it more complicated  .

Regards,

Amey Deshpande

 

  

[Matthias Schelp]

Hi Amey,

you could use the new SLO (Service Level Objective) feature if you're on version 12, but it is quite limited in it's functionality. However, it may be enouph in your case. Here is an example configuration from the Automic help link:

My personal prefered solution is to write a sql statement that checks in the database (ah table) if the job was started an pass its result to a monitoring system like Nagios, Icinga or what ever you're using.

Regards, Matthias

[Amey .]

Hi Matthias, 

SLO objects can be useful in Workflow management, but what about the JOBS which are under the workflow and not started but passed the time limit.

Regards,

Amey Deshpande

[Matthias Schelp]

Hi,

SLO works for any kind of Automic object regardless if it's a workflow or a jobs. You can check for the latest start and end time and you can specify a call object or a Job that will e.g. send you an e-mail if the job didn't start in time.

Regards, Matthias

[Amey .]

Hi Matthias, 

I have one more thing to check, can we add Alias name in SLO to monitor JOBS or JOBP.

Regards,

Amey Deshpande

[Matthias Schelp]

Hi,

hm, good question. Haven't tried that. Honestly I'm not using the alias feature in our environment. So best thing would be if you tests that on your own.

Regards, Matthias

[Michele Adamski]

Have you successfully gotten an SLO to work? As time allows, I've been trying to test one but I just can't get it operable.

[Matthias Schelp]

Hi Michele,

Yes, it's not much. I have exactly 1 SLO running. But before we continue let me suggest to open a new thread or send me a pm.

Regards, Matthias

[Jared Kessans]

For this scenario, the Time Checkpoint on the properties/Time and Dependencies tab is what you need to use.  If JOB B doesn't start at 10am, a notification can be sent at 10am if it isn't active.

[Amey .]

Hi Jared,

It worked...!!! Thank you

Is there anyway we can alert on workflow as well if it didn't starts on a specific time.

Regards,

Amey Deshpande

 

[Jared Kessans]

What scenario are you concerned about?  I'm thinking that if the workflow is triggered in by a schedule object it will start at the expected time unless it faults due to some problem with the set up.

For a fault in a schedule, you could look at using the result tab on the properties of the workflow within the schedule object.

Using the ANY_EXCEPT_FAULT for the OK Status else alert should capture faults if you only want to be alerted to faults and not workflow cancellations.

If you are concerned about a schedule being stopped/quit accidentally, I put a process in place to keep track of active schedules and any time a schedule is stopped or quit we get notified about it and can check into it to verify it was proper.  And yes, it happened a couple of times by an operator, hence the reason I implemented the schedule check.

[Mick Solliday]

I too have been unable to get an SLO to cover the missed start time scenario. 

I had to build a workflow that makes an API call to get all the start times for schedules.  Then it puts that info in an array, had to do some format conversion in the post-processing to make the date/time format match usable predefined formats.  Then the second step is a script object that takes a list of objects in a VARA and compares the start time with the previous start time.  If it falls outside a time range it activates our workflow to create a servicenow ticket.
More than happy to share the solution I built if someone would want to look at it and see if it could be adapted to their needs.

You would think for such a common scenario there would be a basic feature that could monitor start times out of the box, but I couldn't make anything work correctly.  If that changes, I'll update here. :)

[Pete Wirfs]

I played with the SLO feature a little bit, but we had already created our own best practice that is still working well for us. If our JOB.A needs to have a 10AM SLA test, we would build a new workflow that we would call JOB.A.10AM.CHECKER that runs at 10AM to check the status of JOB.A.

One thing I prefer about this home-grown practice, is it is more visible/intuitive to everyone that this check is taking place, complete with run history for the check activity.  The SLO feature is stored under a different area of the AWI and staff are less likely to be aware (without training) of what is and isn't getting checked.

------------------------------
Pete Wirfs
SAIF Corporation
Salem Oregon USA
------------------------------

[Amey .]

Hi Pete,

I have few things to check for creating status check workflows - 

1. we have monthly code releases and patching activities and during those we need to suspend/Hold jobs. so if we create status check workflows which means we need to run them on some agent to check the status of the job or application. so in that case if job or workflow are in suspend/hold state how it will provide delay notifications.

2. we are having over 500 workflows which executes on daily, hourly, half hourly basis so adding more applications to check the status can be a tedious task for monitoring team. 

I think SLO may help only if it allows to track using Alias name.

Regards,

Amey Deshpande

[Jared Kessans]

Amey, did you check out the Checkpoint option?  It's used for this exact scenario based on your original post.

[Amey .]

Hi Jared,

I am currently exploring Automic workload automation but never heard of Checkpoint. 

It will be really helpful if you provide an example for the same.

Regards,

Amey Deshpande

[Pete Wirfs]

Very Interesting.  I have never used the checkpoint feature before, but it looks useful for your usecase.  It is a workflow task property setting;



------------------------------
Pete Wirfs
SAIF Corporation
Salem Oregon USA
------------------------------

[Shani Kushwaha]

Hi Amey,

for your requirement, "checkpoint" is the preferred and simplest option as suggested by @Jared Kessans. this can be enabled in all the workflows and it will send the notification even if JobB is in "Waiting for Predecessor" condition.

-
Shani

​

------------------------------
Thanks
ShaniK
Automic - HCL
------------------------------

[Amey .]

Hi Shani,

Yes it worked for the JOBS which are in workflow. 

What about Workflow , suppose if workflow is scheduled to start at 6:00 AM Daily and due to some reason like suspend or hold if it didn't execute then can we get a notification.

Regards,

Amey Deshpande

[Shani Kushwaha]

you would need to test this option under schedule object for the workflow.



-
Shani

------------------------------
Thanks
ShaniK
Automic - HCL
------------------------------

[Matthias Schelp]

Hm,

this would check for the jobs result, but if I understand correctly @Amey . would like to check the starttime.
​
Regards,

Matthias

[Amey .]

Exactly, suppose a workload is scheduled at 6:00 AM and due to some patching activity or maintenance or Agent down situation  workflow didn't trigger at 6:00 AM 

we need an alert to be sent out. 

[Shani Kushwaha]

Hi Amey,

 

A workflow inside another workflow can have the checkpoint option enabled.

A workflow within a JSCH object doesn't have any such option and a JSCH/JOBP Object doesn't affect by agent down but only occur in scenarios like AE down/DB errors which will affect the entire AE system not only the workflow and there are other options to do the AE health checks.

 

The scenario which you are mentioning is very unlikely to happen because a workflow in a JSCH object gets enabled as soon as the start time and other condition matches.

 

You can use SLO or can have a workflow Start check job but again, creating that for all the workflows being used in a JSCH will be a tedious job.

 

**** Following the post for other suggestions ****

------------------------------
Thanks
ShaniK
Automic - HCL
------------------------------

[Amey .]

Hi Shani,

We have observed that during some maintenance activity teams have suspended a few workflows and due to negligence if they didn't release it after activity. 

We observed panic situations among teams, so just to avoid trying to check if we can have such utility which will help to resolve such things.

Regards,

Amey Deshpande

[Pete Wirfs]

You said:

"during some maintenance activity teams have suspended a few workflows and due to negligence if they didn't release it after activity. "

I would be studying HOW are they suspending these workflows such that it allows them to miss desired run schedules.  I would think these processes need correcting or enhanced to automatically re-activate.

Creating a bunch of SLO solutions is still a smart idea.  But they aren't the root of the problem that needs solving either.

Regarding platform failures (Agent is unavailable, etc) We've installed an alert system that traps these errors from the MSG table, so even in the rare event that a resource becomes available, someone gets pinged so it can be resolved quickly.

------------------------------
Pete Wirfs
SAIF Corporation
Salem Oregon USA
------------------------------

[Michele Adamski]

@Shani Kushwaha:  Have you successfully used SLOs? I've been trying to get one working, but haven't yet succeeded. ​

[Shani Kushwaha]

Classification: Confidential

Hi Michele,

 

I have had some use cases but never explored it completely and we are not using it in our environment.

 

Thanks,

Shani Kushwaha