Slow running BG jobs - disparate datacenters

[David Morton]

Hello - in a bit of strange situation that I don't yet know whether is a problem or not.... ;

 

I have a pretty old (long out of support) v8.1 system, it was sat across 3 physical Win2003 server (2xAPPs on 2 of the servers, BG and Actuate (yes very very old and out of support) on the other) - database is on a Unix box with SAN storage.  Everything worked OK, performance generally fine.

 

All the above kit was located in "datacenter #1"

 

For reasons well out of my control, the 3 win servers have just been virtualised and moved to a VM platform running in "datacenter #2" , currently the Unix machine with the database on is at the moment, still in "datacenter #1". The 2 datacenters are not physically close to each other. The Unix machine/SAN will very shortly be physically moved to "datacentre #2" though.

 

We anticipated problems with the 2-datacentre situation (just due to latency) ; but to be fair, the front-end application is running OK across 2 virtual Win servers (which gained new IP addresses as part of the process). And I don't know anything about the network links / capacity between the datacenters either.

 

However, a couple of key BG jobs are behaving a little bit oddly on the third sever ; specifically POST TIMESHEETS and TIME SLICING.

 

POST TIMESHEETS used to run in a few minutes (say 10) under the old-setup, now it is typically seeming to take >1.5 hours before we see any sign of it starting to do any update to timesheets on the database to posted, it seems to then be able to trawl through the timesheets it needs to 'post' in about the usual time - so its almost as if the "start-up phase" of the POST TIMESHEETS job is where the bottleneck is?

 

TIME SLICING just seems unusually slow, it completes eventually though.

 

So at the moment the backend is dragging a bit; the backlog of stuff to do catches up overnight, but its just a little bit concerning (albeit not unexpected) that something has changed!

 

So I don't yet know whether to be concerned about this long-term, if (once the database has moved to the new datacentre) everything is happy, then I am happy - I am just curious if there is anyone who understands the architecture of these jobs who could say "yeah that would be expected in this situation".

 

So any wisdom out there?

 

ta,

Dave.

[Joshua Leone]

Interesting scenario Dave...

Are the jobs only slow from that third server... if they execute from the other two, they are ok? (I'm not clear on that.)

 

I would actually expect much more to be slow then just these jobs.  It's been a couple (ok, many) years since I've touched v8 so I'm trying to remember the capabilities. 

Can you SQL trace the jobs to find out if the time is being consumed in the fetch to be able to even pin to a latency problem?

 

Another possibility... the network link isn't actually a problem at all and it is the SAN.  Try doing a copy of the prteam or prassignment table (select into... ) and get the speed those tables can write.  I have seen issues in the past with underperforming SAN's having this kind of impact.  The only real way to prove it though is to replicate a table with BLOBS.  A straight file copy or even a table copy without blobs doesn't show it.  You get the size of the table and do the copy to get the time to figure out how fast that SAN is writing.

 

Good luck!

Josh

[David Morton]

Ta Josh, you were certainly one of the wise ones who I'd hoped this might intrigue!

 

I've not tried fiddling with anything in the setup (like starting a BG on one of the other 2 virtualised servers) yet.  (BG is only running on the one sever, APPs on the other 2)

If, once the DB/SAN has moved, I am still getting problems then that is when I will start digging. SQL Tracing the POST TIMESHEETS job would certainly be on a list of things to do, but I'm holding off anything too (my) time-consuming just yet.

 

I also expected some slowness in the APP services, but they do seem fine, possibly even quicker user-experience than the 'old' hardware ; this is actually perhaps why I am slightly nervous ; i.e. if the APP is OK then why isn't the BG.

[Joshua Leone]

The only thought process I would have... UI operations to tend be smaller data payloads.  So, if there is pipe constriction, maybe you only see it on the jobs returning lots of data.

But, I go back to that SAN throughput.  I worry about that being an underlying issue that is getting masked by the noise around the datacenters. I would definitely try out that table copy to get the timings.  If your SAN is slow, it will show it.

[David Morton]

SAN should be OK; its not changed at all (when we put the SAN in, it was doing that which fixed our 'old' performance issues, TIMESLICING went down from 10s of minutes to 1 or 2 - happy days!). I do worry a bit about what happens when they move the SAN next week though!

[David Morton]

Just to close the loop...

 

Database server was moved over the weekend to the new data-center, and all is good, in that my BG jobs are now running through quick enough again. (POST is back to a few minutes, TIME SLICING running thru OK)

 

Happy days!

 

(well "happy" as far as; I'll stop worrying about this now)

[Joshua Leone]

Great news Dave!