DX Application Performance Management

  • 1.  Clarity - Track out of memory error with Wily

    Posted Feb 16, 2010 07:13 AM
    Hi,  we are using Clarity 7.5.3 FP05 and monitoring the application with Wily.  We have often out of memory erros in some of our application services (we have 8 application services). We set up alerts in Wily on each JVM of each app. When we reach 95%, we restart the service or empty the cache.  Is there a way with Wily to understand what is causing the out of memory errors? Does anyone implement that?  Best regardsItaï


  • 2.  Re: Clarity - Track out of memory error with Wily

    Broadcom Employee
    Posted Feb 16, 2010 03:06 PM
    Hi Itai,  This is probably related to  high number of metrics reported or being monitored by Introscope: please run the same test using only the base agent without any agent extension as explained below. If the problem does not occur with the base agent, you must then introduce back each component one by one until you reproduces the problem:- Disable any customer pbd.
    - using InstanceCounts on too many classes.
    - Disable any additional agent extension such as leakhunter, ChangeDetector and any PowerPack, by removing the appropriate pbds and agent jar extensions. LeakHunter will certainly increase the memory requirements
    - Disable JMX metric collection in the IntroscopeAgent.profile:
    introscope.agent.jmx.enable=false
    - Disable PMI metric collection in the IntroscopeAgent.profile:
    introscope.agent.pmi.enable=false
    - Disable platform monitor   - see Agentguide.pbd, Configuring platform monitor chapter.
    - Disable SQLAgent by removing the AGENT_HOME/ext/SQLAgent.jar out of the AGENT directory.
    - Set clamps for metrics and sql metrics in the IntroscopeAgent.profile:
    introscope.agent.metricClamp=15000If you are not using  Introscope 8.2.2, I would advice you to upgrade to this latest release and make sure the new the "introscope.agent.reduceAgentMemoryOverhead" agent property is enable and set to true.  If after applied all the above changes the problem persist, please open a support incident.I hope this helps,  Regards,  Sergio


  • 3.  Re: Clarity - Track out of memory error with Wily

    Posted Feb 17, 2010 04:08 AM
    Where is the documentation on what exactly this setting changes?   "introscope.agent.reduceAgentMemoryOverhead"


  • 4.  Re: Clarity - Track out of memory error with Wily

    Broadcom Employee
    Posted Feb 17, 2010 08:09 AM
    At the present time, this  8.2.x Agent property is only documented in  8.2.x README files - see below.  This new property attempt to reduce the agent memory overhead introduced by architectural improvements to the 8.x Agent.Until 7.2, a WeakElementArryHashMap was used to map metric names to metric instances, but this was replaced by a Threadlocal pool and a Master pool in 8.0. This was done for performance reasons, especially in the case of multiprocessor machines where multiple threads would be locked on the hashmap when reads or writes were done. This was slowing it down quite a bit, and hence the new design in 8.0.So, we started caching lookup maps in 8.0, during when we make call to getMetricFromCache() method to see if it already exists in the cache or not. First we check the Threadlocal cache, if it's not there, we check the master pool, and if it's still not there, we create a new metric key and put it into the pool. If one exists in master, but not threadlocal cache, we reference that key into the threadlocal cache for future hits.The lookup for the metric in the cache is made using the key, which is a AgentMetricKey, and this instance is created using the metric name. In the case of SQL metrics, the names could be very long queries, and using this long string in the threadlocal cache is causing huge memory footprint problems, especially when there are many threads in the threadpool holding references to threadlocal variables.  Please also refer to: https://support.wilytech.com/cgi-bin/wilytech.cfg/php/enduser/std_adp.php?p_faqid=2394  I hope this answer your question,  Regards,  Sergio        From Introscope8.2.2.0_README.pdf, page 3:--------------------   The Introscope 8.2 agent has a new property that allows you to balance memory use against response time. Using the property could potentially lower memory consumption at the expense of a possible increase in response time. Add the following to your agent’s IntrscopeAgent.profile file, set the property to true, and uncomment it. You must restart the managed application before changes to this property take effect. ################################Agent Memory Overhead Setting## =======================# Set to true if you want to attempt to reduce the agent memory overhead introduced by architectural improvements to the 8.x Agent.# Increased Agent memory overhead only occurs in certain extreme cases.# The trade-off for the possible lower memory consumption is a possible increase in response time.# Each application is unique and will experience different variations in Memory vs. Response Time trade-offs.# This property is set to false by default and out of the box is commented out. This is not a hot property# and the managed application needs to be restarted for this flag to take effect.#introscope.agent.reduceAgentMemoryOverhead=true