Lag time Between Job Ending on Server and End time on Statistics

[Anon Anon]

Hi.  We are in the process of migrating to a new data center.  The UC4 application servers (AE's) and db are still located at the "old" data center.  There is about 40 miles between the 2 data centers.  However, we have many agents installed at the "new" data center as the migration occurs.

I've had a user inquire about differences they are seeing between the information in the job trailer & the statistics end time.  For example here's what I see on the details for the job:

Activation Time = 10:33:40pm

Start Time = 10:33:41pm

Post Processing Time = 10:44:40pm

End Time = 10:44:41pm

However, if I look at the job report, the trailer step shows that it ended at 10:44:23.

Can anyone explain and/or have any ideas on what the delay is between 10:44:23 and 10:44:40?  I realize that this is only 17 seconds.   :)  But the users are concerned and would like an explanation of the delay.

My initial thought was travel time between the 2 data centers, but I thought that there were other customers out there with much further distances between agents & AE's, etc.

I did check to see how many other processes were running on this server and it doesn't seem like a huge number, but I guess it's all relative - about 95 jobs in a 13 minute window.  However, wouldn't that delay/slow down the entire job, not just from when it ended to the start of the Post Process?

Thoughts?

Thanks!

Laura Albrecht

[Alain Moisy]

Hi Laura,

Did yu check that all machines are effectively on the same central clock ? May be simply a time difference between agent time and server time ... Sometimes the obvious ....

Anyway there is some delay, around 3 to 5 seconds minimum, to activate the Post-Processing if there is any one defined in a job. Try to run a job with and without a Post-Processing to check this. Add to it the time for transfering the report (dynamic file transfer in the background) between end of the job (trailer) and the beginning of the post-processing and you may found your 17 seconds.

The problem here is that you do not have the detail of all durations of each step in a job processing unless you dig into the child process in the statistics for example to find how long it takes to get back the report into the DB before starting the post-process. May be also delayed if you have some firewall in the middle to makes things easier ...

Alain

[Anon Anon]

Yes, the servers here all sync up with central ntp servers, so good idea, but no.

I dug deeper in the child statistics as you said, but there still is an unexplained delay.

Activation Time = 10:33:40pm

Start Time = 10:33:41pm

Trailer Step End Time = 10:44:23pm

Post Processing Start Time = 10:44:40pm

REPORT Start Time = 10:44:40pm

EVENT.UNIXCMD Start Time = 10:44:40pm

REPORT End Time = 10:44:41pm

EVENT.UNIXCMD End Time = 10:44:41pm

PREP_PROCESS Start Time = 10:44:41pm

PREP_PROCESS End Time = 10:44:41pm

End Time = 10:44:41pm

So the way I am seeing it, there is still an unexplained lag time of actually 21 seconds.  All the child statistics start / end times occur within 1-2 seconds.

There are  no firewalls involved in this particular scenario.

Thank you for the excellent ideas.  I did open up a ticket with Support.  So we'll see if they discover anything.  Will keep you posted.

[Louis Barroso]

Were you and Automic support able to figure out what was causing the lag time?

[Anon Anon]

No, not really.  We were chalking it up to the data center migration, which is still in progress.  Your question was a reminder to me to check on this since now the majority of agents and the AE's are in the same data center now.  I just checked and we're still seeing occasional lags.  I'll have to revisit this with the user / Support now.