CAWA DE 12.4 - Notification options

[Kenneth Chrapczynski]

Hello brilliant people. We had a situation where a Windows server appeared to be hung. From a CAWA standpoint we didn't see any job failures, none of the jobs being delayed had an overdue on them so workload just backed up. I do not recall what the state of the jobs were, I would guess they were in a ready state. When I looked in the waiting view I saw 900+ jobs. Eventually the server just cleared up and things started flowing again. Our Win Admin could not find anything in any log that explains this unexpected delay.

Now the powers to be would like us to send a notification when something like this happens.

I looked into "not started by" on jobs that run against this server with "realnow plus 90 minutes", but that doesn't seem practical. We do have a job flow of 6 jobs that runs hourly on this server but adding the "not started by" on one of those jobs isn't practical either because it is know to fail daily for known reason.

Is there a way to configure an overdue notification based on server not flowing?

Any thoughts/insights on how to monitor when work is queuing up and not running would be appreciated.

Ken Ski

------------------------------
DARDEN Corporation
Orlando, FL
------------------------------

[Ravi Kiran Kunduri]

Hi,

Can you please look at the Dueout Times Propagation

Broadcom		remove preview
Dueout Times Propagation The server can propagate dueout times, up-stream, to all predecessors of a job that has a dueout time. The server sets the dueout times of predecessors based on historical average elapsed times, a job's execution time averaged over previous runs. By using dueout propagation, you can avoid specifying dueout times for all jobs in an Application. View this on Broadcom >

 

Hope it helps!

Ravi Kiran

[Kenneth Chrapczynski]

Thanks Ravi! I see Dueout Times Propagation can be applied individually to an Application or on all Applications. This looks promising.

What happens if a predecessor job in the flow fails and remains in that state well past the calculated dueout time?

The reason I ask is the Application I identified as a good candidate has six jobs that run against the suspected Windows server.

This Application is setup with 11 Event file monitors; within the application we use the "runtime name" and javascript to make each application and jobs unique based on the name of the file. They trigger into the active queue prefixed with one of those 11 names. Once a day at 1 am over 2000+ applications are triggered into the active queue, then from 4 am through 11 pm a smaller increment of the same are triggered in hourly. On a daily basis a few jobs fail for known reasons and sit in the queue until the application is manually forced complete.  

------------------------------
DARDEN Corporation
Orlando, FL
------------------------------

[Ravi Kiran Kunduri]

Thank you for sharing the details.If the job names are changing on the daily basis , in that case may be the propagate due out time may not work as it depends on the history of job run to anticipate the end time of a job.

[Kenneth Chrapczynski]

Hi Ravi,

Sorry I have been Out of the Office the past 10 days.

With this Application the job names remain the same within the 11 unique Event triggers. What happens if a predecessor job in the flow fails and remains in that state well past the calculated dueout time?

Of the 2000+ possible Applications that could be triggered in, we have a few daily that fail for known reasons. What happens to the over-due logic with those will they send out the over-due notification? On those occasions we would not want to. We’re really just trying to detect the anomaly where no jobs failed, they were just queuing up and not running.

Ken

[SHARON SHIMANEK]

Hi,

This will not help from an alerting standpoint but a visual monitoring it helps me.  I have 2 custom views I keep open 

1. jobs in READY, RESWAIT or SUBDELAY sorted to keep READY at the top.  Truly a job should not be in ready state more than seconds so if I see something I check what time it triggered.
2. I have my WAITING jobs custom view sorted by state so APPLWAITs are at the top.  when I see orange I know some application backed up.

This only helps if you use desktop client not web.  The custom view capabilities in the desktop client are why we keep using that vs web version. :)

Would be a great enhancement it be able to alert on jobs stuck in ready status more than x minutes.

Sharon

[Kenneth Chrapczynski]

Hi Sharon,

Thanks for responding! I agree it would be a nice enhancement. The due-out time would be helpful if we could only send the notification when a real issue arose, and not because a job is delayed for a known issue, like a failure, or a timing delay like patching, scheduled outages etc.

We are also leveraging the custom views (waiting, held, ready, failed, etc.), that is what led me to see I had an erroneous number of jobs queued/waiting, then we sprang into action trying to determine the cause.

We identified the situation, now we’re being asked to find a way to notify someone sooner.

Ken Ski

[SHARON SHIMANEK]

another option:  What about scheduling a SQL job to run every couple hours, whatever fits your needs with Email notification. If the report returns results you have to look into.  Hard to tell if this will work until you run it with a job stuck in READY status it could be enhanced if needed. 

select WA.esp_RTWOB.name,WA.esp_RTWOB.READYTIME,WA.esp_RTWOB.STARTTIME,WA.esp_RTWOB.thestate,WA.esp_RTWOB.status
from wa.esp_RTWOB
WHERE wa.esp_RTWOB.thestate = 'READY' and WA.esp_RTWOB.STARTTIME is null;

[Kenneth Chrapczynski]

Thanks Sharon. That was where I was heading next. 😊

I had really hoped our Windows Admin had a solution with SCOM or some other tool in their toolbox. I guess the only thing they know is to reboot, LOL!

Ken