Hello all folks,
a short time ago we moved our application environment from Websphere to Liberty Profile. Unfortunately the CPU usage increase more than expacted. Any idea how to calculate the Java Agent footprint in a JVM (we use IBM Java 8, AIX 7.1, APM 10.5.1, the "Default Agent" package) in a easy and usefull way.
Thanks for any suggestion,
We use the javacore files (taken two times a day) to get a CPU usage overview (see "3XMCPUTIME") and assign all "Agent*" and "*Hub*" threads to the Java Agent footprint.
3XMTHREADINFO "UnknownHub Hub Receive 189" J9VMThread:0x0000000034F4A400, j9thread_t:0x0000010043B443F0, java/lang/Thread:0x00000000F4BD6D20, state:R, prio=53XMJAVALTHREAD (java/lang/Thread getId:0x5DE, isDaemon:true)3XMTHREADINFO1 (native thread ID:0x9504B9, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000020)3XMCPUTIME CPU usage total: 0.191125000 secs, user: 0.076544000 secs, system: 0.114581000 secs, current category="Application"3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)3XMTHREADINFO3 Java callstack:3XMTHREADINFO "Agent Heartbeat" J9VMThread:0x000000003064E100, j9thread_t:0x00000100225AE350, java/lang/Thread:0x00000000E01AF0B8, state:R, prio=53XMJAVALTHREAD (java/lang/Thread getId:0x17, isDaemon:true)3XMTHREADINFO1 (native thread ID:0x1A80457, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x80000001)3XMCPUTIME CPU usage total: 14.973503000 secs, user: 14.859944000 secs, system: 0.113559000 secs, current category="Application"3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)3XMTHREADINFO3 Java callstack:
Maybe "footprint" is not the best term.
Footprint usually means "space on disk" or "space in memory". The agent is of little consequence, in either case.
Anything related to CPU is going to be "overhead".
For the agent, overhead is completely configuration and application dependent. First, check that the agent configuration, between the two environments, has not changed. The quickest way to do this is via the APM console, navigating to the agent and see how many metrics is is generating. More than 20% difference probably means the application changed and the agent configuration should be reassessed.
If the agent configuration is the same, then we are on a hunt to find "what changed". Since this is a new platform, new JVM, new OS settings, etc., the compatibility guide is something to check first. Then to understand the differences between the old and new platform.
My favorite is to confirm that the "debug mode" for the app is NOT enabled. This was often overlooked on the WAS platform. No reason to assume "it can't happen here'!
If the agent configuration has "gone bad" for whatever reason, this is usually reflected in a very high invocation count for a particular metric or SQLcall. This can happen when there is a significant feature upgrade in the application.
You have the evidence, before and after, in the Smartstor. You just need to look at the data and you should find "what changed'.
yes, "footprint" is not the best term, "overhead" fits better. But we have a look to the Memory, CPU, etc. and the only open question is the "used CPU".
The number of collected metrics is not the problem, I tkink. In the past a (Websphere) JVM collected 2000 to 3000 metrics, the new (Liberty Profile) JVM collect 250 to 800 metrics only (in general). Some JVMs with DB connections collect up to 5000 metrics, like in the past. And the "problem" JVMs collect 250 to 800 metrics.
Unfortunately I can't compare the old and the new environment because we split "fat" JVMs into "small" JVMs, the configuration was changed significant therefore.
When you say "Default Agent" package does that mean with WLP you are not using the WebSphere agent package and also not using AgentNoRedefNoRetrans.jar and IntroscopeAgent.NoRedef.profile?
Also referring to your previous thread: Comment to the IBM Java support, see TEC1176763
sorry, we use the AgentNoRedefNoRetrans.jar and IntroscopeAgent.NoRedef.profile.
But the used metrics are based on the default-typical.pbl and some/several definitions are take from the IntroscopeAgent.default.profile.
Unfortunately we reduce the collected metric data in general, but the number of JVMs increase.
The expected memory fits, but the used CPU does not, that is the reason I'm interresting in getting some more information about the used CPU inside a JVM.
In some JVMs (Liberty Profile environment) the "agent" thread consume 30% to 40% of the CPU and collect 250 to 800 metrics only.
In the past (Websphere environment) the JVMs use 5% of the CPU and collected 2000 to 3000 (5000) metrics.
The problem seems to me, we split "fat" JVMs (60) into "small" JVMs (160) and nobody calculate/estimate the basic load of the JVMs. And the Agent is a part of the basic load and consume CPU independent of the JVM usage/workload.