DX Application Performance Management

Expand all | Collapse all

Thread Dumps in waiting state...Causing increase in memory%

Jump to Best Answer
  • 1.  Thread Dumps in waiting state...Causing increase in memory%

    Posted 09-03-2018 11:37 AM

    Hi All,

     

    We have APM 10.7 in our environment hosted on redhat linux 7.4. 

    We are getting high memory % due to increase in threaddumps on the on of the collector.

    Found maximum of the thread dumps are in waiting state or in aged waiting state.

    Kindly suggest me to minimize the memory % issue

     

    Thanks. .

    Regards,

    Anuranjan Kumar



  • 2.  Re: Thread Dumps in waiting state...Causing increase in memory%

    Posted 09-03-2018 09:02 PM

    Hi Anuranjan,

    You mean an increase in the actual number of Collector threads rather than thread dumps - correct?

    Is it definitely the Collector Java process that is showing the increased memory usage? 

    (ps -ef | grep java | grep Introscope_Enterprise_Manager.lax)

     

    The thread and memory usage increase may be expected if more agents are connecting to the Collector and thus it is doing more work.

    The maximum heap you have assigned to the Collector Java process in file Introscope_Enterprise_Manager.lax will not get exceeded.

    You can view Collector Supportability metrics for memory usage on the WebView/Workstation UI under the Health and GC Heap nodes of *SuperDomain*|Custom Metric Host (Virtual)|Custom Metric Process (Virtual)|Custom Metric Agent (Virtual) (collector_name)|Enterprise Manager

    Cluster Supportability Metrics - CA Application Performance Management - 10.7 - CA Technologies Documentation 

     

    Hope that helps.

     

    Thanks,

     

    Lynn



  • 3.  Re: Thread Dumps in waiting state...Causing increase in memory%

    Posted 09-04-2018 09:18 AM

    Hi Lynn,

     

    It is Collector Java process that is showing the increase in memory usage.

    (ps -ef | grep java | grep Introscope_Enterprise_Manager.lax)

     

     As of now, we are in implementation phase and currently no agents are reporting to it.

     

    Although we have increased the heap size to 4 GB. And we have  observed  that memory is gradually increasing day by day. 

     

    Might be Thread dump issue as logs are pointing

     

    Thanks for quick response

     

    Regards,

    Anuranjan Kumar



  • 4.  Re: Thread Dumps in waiting state...Causing increase in memory%

    Posted 09-04-2018 12:47 PM

    Before you finish your tests, I would recommend you run your numbers through the sizing utility to ensure you have enough capacity to handle the workload you're anticipating.

     

    https://sizer.ca.com/APMSizingTool/index.html 



  • 5.  Re: Thread Dumps in waiting state...Causing increase in memory%

    Posted 09-06-2018 02:22 PM

    Hi Haruhiko, Lynn,

     

    Thanks for the quick response & guidance!!!

     

    I have increased no. of days to 90 days (default 30 days) and found the memory utilization % to be static now.

     

    introscope.enterprisemanager.threaddump.storage.clean.disk.olderthan.days=90 days

     

    Thanks for your support..

     

    Regards,

    Anuranjan Kumar



  • 6.  Re: Thread Dumps in waiting state...Causing increase in memory%

    Posted 09-04-2018 08:18 AM

    Hi Anuranjan:
    Was Lynn's response helpful? Please let us know when you cam

    Thanks

    Hal German



  • 7.  Re: Thread Dumps in waiting state...Causing increase in memory%

    Posted 09-04-2018 05:34 PM

    Dear Anuranjan:

       This is something that may be less impacted in a later release. There may be only such much that can be done on an issue like this using screenshots . Perhaps Lynn has additional insights

     

    Thanks

    Hal German



  • 8.  Re: Thread Dumps in waiting state...Causing increase in memory%
    Best Answer

    Posted 09-05-2018 12:50 AM

    Hi AnuRanjan,

    By default the thread dump purge check for older files (default 30 days old per property introscope.enterprisemanager.threaddump.storage.clean.disk.olderthan.days) will occur once per day per property:

    introscope.enterprisemanager.threaddump.storage.clean.disk.freq.days=1

    So if the logs are only showing the same frequency as that property value then that is expected.

     

    So is Xmx currently set to 4GB in Introscope_Enterprise_Manager.lax?

    The top RES value for the process shows higher but that could be explained by other non heap memory usage. 

    How is the heap usage looking from the Supportability metrics?

     

    If there are no agents connected to the Collector have you tried another Collector restart to see if the same symptoms occur?

    If the problem still persists it is probably best to open a Support Case so we can review log and config files in more detail.

    Are you using 10.7 GA or a later HF or SP?

     

    Thanks

     

    Lynn