All,
I'm using a 9.7 agent with a 10.3 EM.
We're using Oracle RAC. I believe it's 12g with an 11g driver. APM identifies the different RAC nodes individually in the metric tree using their host names, so I might have db01, db02, and db03, which are all part of the same logical DB. However some of these nodes are in different data centers than my app server. The response times are noticeably different. That's no surprise, but what IS a surprise is that the response times are consistent for long periods of time, then suddenly change.
For example, suppose the investigator says I'm connected to db01 and I've been getting response times <10ms for hours or days. Then all of a sudden those same queries start returning response times that are much higher, say 50ms. Coincidentally, the other db nodes are about 40-50ms away from my app server.
I have a strong suspicion that new connections are being established to a node in another data center but are still reported under the previous/wrong node. This is especially obvious when the connection switches from a far away data center to a closer one. Suddenly my response times are coming back faster than the speed of light!
Does anyone know what's going on here? I'm not super familiar with Oracle RAC, but it looks like connections are not always getting routed to the same node but APM reports them as if they were.
thanks