DX Application Performance Management

  • Private Community

  • 1.  How to estimate the APM Agent footprint

    Posted Jun 27, 2018 01:34 PM

    Hello all folks,

    a short time ago we moved our application environment from Websphere to Liberty Profile. Unfortunately the CPU usage increase more than expacted. Any idea how to calculate the Java Agent footprint in a JVM (we use IBM Java 8, AIX 7.1, APM 10.5.1, the "Default Agent" package) in a easy and usefull way.

    Thanks for any suggestion,

    Lutz

     

    p.s.

    We use the javacore files (taken two times a day) to get a CPU usage overview (see "3XMCPUTIME") and assign all "Agent*" and "*Hub*" threads to the Java Agent footprint.

     

    For example,

    3XMTHREADINFO      "UnknownHub Hub Receive 189" J9VMThread:0x0000000034F4A400, j9thread_t:0x0000010043B443F0, java/lang/Thread:0x00000000F4BD6D20, state:R, prio=5
    3XMJAVALTHREAD            (java/lang/Thread getId:0x5DE, isDaemon:true)
    3XMTHREADINFO1            (native thread ID:0x9504B9, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000020)
    3XMCPUTIME               CPU usage total: 0.191125000 secs, user: 0.076544000 secs, system: 0.114581000 secs, current category="Application"
    3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
    3XMTHREADINFO3           Java callstack:

    3XMTHREADINFO      "Agent Heartbeat" J9VMThread:0x000000003064E100, j9thread_t:0x00000100225AE350, java/lang/Thread:0x00000000E01AF0B8, state:R, prio=5
    3XMJAVALTHREAD            (java/lang/Thread getId:0x17, isDaemon:true)
    3XMTHREADINFO1            (native thread ID:0x1A80457, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x80000001)
    3XMCPUTIME               CPU usage total: 14.973503000 secs, user: 14.859944000 secs, system: 0.113559000 secs, current category="Application"
    3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
    3XMTHREADINFO3           Java callstack:


  • 2.  Re: How to estimate the APM Agent footprint
    Best Answer

    Posted Jun 27, 2018 01:51 PM

    Lutz:

    Maybe "footprint" is not the best term.

    Footprint usually means "space on disk" or "space in memory".  The agent is of little consequence, in either case.

    Anything related to CPU is going to be "overhead".

    For the agent, overhead is completely configuration and application dependent.  First, check that the agent configuration, between the two environments, has not changed.  The quickest way to do this is via the APM console, navigating to the agent and see how many metrics is is generating.  More than 20% difference probably means the application changed and the agent configuration should be reassessed.

    If the agent configuration is the same, then we are on a hunt to find "what changed".  Since this is a new platform, new JVM, new OS settings, etc., the compatibility guide is something to check first.  Then to understand the differences between the old and new platform.

    My favorite is to confirm that the "debug mode" for the app is NOT enabled.  This was often overlooked on the WAS platform.  No reason to assume "it can't happen here'!

    If the agent configuration has "gone bad" for whatever reason, this is usually reflected in a very high invocation count for a particular metric or SQLcall.  This can happen when there is a significant feature upgrade in the application.

    You have the evidence, before and after, in the Smartstor.  You just need to look at the data and you should find "what changed'.

    Cheers,

    Mike.



  • 3.  Re: How to estimate the APM Agent footprint

    Posted Jun 28, 2018 08:27 AM

    Hello Michael,

    yes, "footprint" is not the best term, "overhead" fits better. But we have a look to the Memory, CPU, etc. and the only open question is the "used CPU".

    The number of collected metrics is not the problem, I tkink. In the past a (Websphere) JVM collected 2000 to 3000 metrics, the new (Liberty Profile) JVM collect 250 to 800 metrics only (in general). Some JVMs with DB connections collect up to 5000 metrics, like in the past. And the "problem" JVMs collect 250 to 800 metrics.

    Unfortunately I can't compare the old and the new environment because we split "fat" JVMs into "small" JVMs, the configuration was changed significant therefore.

     

    With regards,

    Lutz



  • 4.  Re: How to estimate the APM Agent footprint

    Broadcom Employee
    Posted Jun 27, 2018 06:19 PM

    Hi Lutz,

    When you say "Default Agent" package does that mean with WLP you are not using the WebSphere agent package and also not using AgentNoRedefNoRetrans.jar and IntroscopeAgent.NoRedef.profile? 

    Also referring to your previous thread: Comment to the IBM Java support, see TEC1176763 

     

    Thanks

     

    Lynn



  • 5.  Re: How to estimate the APM Agent footprint

    Posted Jun 28, 2018 07:06 AM

    Hello Lynn,

    sorry, we use the AgentNoRedefNoRetrans.jar and IntroscopeAgent.NoRedef.profile.

    But the used metrics are based on the default-typical.pbl and some/several definitions are take from the IntroscopeAgent.default.profile.

    Unfortunately we reduce the collected metric data in general, but the number of JVMs increase.

    The expected memory fits, but the used CPU does not, that is the reason I'm interresting in getting some more information about the used CPU inside a JVM.

     

    In some JVMs (Liberty Profile environment) the "agent" thread consume 30% to 40% of the CPU and collect 250 to 800 metrics only.

    In the past (Websphere environment) the JVMs use 5% of the CPU and collected 2000 to 3000 (5000) metrics.

    The problem seems to me, we split "fat" JVMs (60) into "small" JVMs (160) and nobody calculate/estimate the basic load of the JVMs. And the Agent is a part of the basic load and consume CPU independent of the JVM usage/workload.

     

    With regards,

    Lutz