DX Application Performance Management

Expand all | Collapse all

How to Monitor Active Hung Threads using Introscope Agent?

Jump to Best Answer
  • 1.  How to Monitor Active Hung Threads using Introscope Agent?

    Posted 12-12-2011 12:45 PM
    Hello,

    Is there any possibility to capture Active hung threads using Introscope Agent? If so, how can we achieve this?

    Thanks,


  • 2.  RE: How to Monitor Active Hung Threads using Introscope Agent?

    Posted 12-13-2011 03:36 PM
    What application server/service did you want to monitor hung threads?

    I know in WebSphere you can tap into the WebSphere PMI and there is a metric for "ConcurrentHungThreadCount". In Tomcat you can tap into JMX, but I don't know if it has a hung thread metric by default.


  • 3.  RE: How to Monitor Active Hung Threads using Introscope Agent?

    Broadcom Employee
    Posted 12-19-2011 05:32 AM
    OPTION 1: Use EPAgent LogReader scripts.

    a) If you are using Weblogic and Webpshere, you can use WebsphereLogReader or Weblogic LogReader. For example, for WebsphereLogReader, update the websphereLogReader.cfg as below:

    Files to be modified:
    <EM_HOME>/config/IntroscopeEPAgent.properties

    In "Stateful Plugins" section:
    introscope.epagent.plugins.stateful.names=WEBSPHERE
    introscope.epagent.stateful.WEBSPHERE.command=perl 'C:/Program Files/CA Wily/Introscope8.1.0.0/epaplugins/websphere/websphereLogReader.pl'

    <EM_HOME>/epaplugins/websphere/websphereLogReader.cfg

    LOGFILEDIR = 'C:/IBM/WebSphere61/AppServer/profiles/AppSrv02/logs/server1'
    LOGFILE = 'SystemOut.log'
    SLEEPTIME =300

    {
    match => 'WSVR0605W',
    metricType => 'IntAverage',
    metricName => 'Thread Errors',
    metricValue => '$CURRENTCOUNT'
    },

    The paths above are for my environment on windows. Your's may be different.

    b) If you are not using weblogc or websphere and you need a generic log reader you might want to use logreader_package.zip that you can customize it as you need, however code modification might be required. I have attached to the documentation library.

    NOTE: Unless you know the generated code error/message in the appserver log, you cannot use EPAgent is thisfashion, otherwise you might want to use stall counts, which detects stalls of a given method.

    OPTION 2: Use Stall Metrics:

    The StalledMethodTracer is used to identify any "hung threads", which are threads that get stuck in a method call
    StalledMethodTracer traces the number of method invocations that have exceeded a user-defined execution threshold value and have not finished. This number serves as an approximation of the number of hung threads. The result is reported in the Metrics tree using the fully qualified Metric name specified in the directives file. If desired, users can set up Alerts on this Metric.

    -Stall metrics are turned on by default. If you navigate through the Investigator tree of a particular agent you will see a stall count. You might want to create a metric group based on the stall count metrics. If a stall is discovered send an alert. Stall are generally associated with a hung thread
    -Stall events, on the other hand, are enabled with ErrorDetector. You will have to install and configure error detector of you are using v8.x
    -Stalls are stored as ordinary errors, and will be visible in the Errors TypeView, or in the historical query viewer by querying for "type:errorsnapshot"

    For more information on the StalledMethodTracer refer to the JavaAgent.pdf

    OPTION 3: Use PMI data - This will give you the number of hung threads, I suggest you to discuss this with your appserver admin person.

    OPTION 4: Turn on ThreadTracing - Uncomment "ThreadTracing" property in your toggles-full and toggles-typical pbd files. This will monitor each thread.

    Regards,

    Sergio


  • 4.  Thread Splitted

    Broadcom Employee
    Posted 02-15-2013 12:17 PM
    This thread's discussion was not a part of the original.

    The new thread can be found at 100261087.


  • 5.  RE: How to Monitor Active Hung Threads using Introscope Agent?
    Best Answer

    Posted 02-15-2013 10:06 AM
    A simpler option might be to use the Field Agent extension here:

    ExtendedMemInfo.zip

    Unzip in your Agent root folder, eventually app the .profile.part to your Agent.profile, restart, you should see the info you need, and you can use traditional Introscope goodies (Metric groupins, dahsboards, alerts, etc...) on the metrics. Take a look at the image file for the sample metrics.

    Regs,

    Florian.


  • 6.  Re: How to Monitor Active Hung Threads using Introscope Agent?

    Posted 03-20-2017 05:28 AM

    Hi Florian,

     

    Can you please share ExtendedMemInfo.zip.I am not able to download the ExtendedMemInfo.zip.We want to achive Hogging/Stuck threads in CA APM .

     

    Regards

    Vinay Nayak



  • 7.  Re: How to Monitor Active Hung Threads using Introscope Agent?

    Broadcom Employee
    Posted 03-20-2017 09:01 AM

    Please open a support ticket to get a copy.



  • 8.  RE: How to Monitor Active Hung Threads using Introscope Agent?

    Posted 02-15-2013 01:09 PM
    If the thread is not exactly hung but is instead a "runaway" thread and is eating up lots of CPU, I have a custom extension that can help with that. It may not help with what you're trying to achieve, but I figured I'd mention it.

    By deafult, it outputs three metrics:

    [font=Courier New]Threads:Max CPU %
    Threads:Max CPU Thread
    Threads:Max CPU Thread Delta Flag[font]

    The first metric is the largest percentage of system CPU being used by a thread. The second is a string containing the ID and name of the thread with the max cpu %. The third metric is a flag indicating whether the top thread has changed. You can use these to create two alerts: 1) Max CPU % when over 90% (or whatever) for three out of three periods and 2) Max CPU Thread Delta Flag when 0 (not changed) for three out of three periods. Then create a summary alert for when both of the other alerts are red, which results in having an alert for when the same thread is using >= 90% CPU consistently for 45 seconds. You can then use the Thread Dump ability of Introscope to track down the thread and find its stack trace, etc.

    The extension is also capable of outputting CPU Utilization metrics by Thread Name, Thread ID, or the Thread Class Name, but those modes can cause metric explosions if used incorrectly or on very dynamically threaded JVMs.

    If this sounds useful to your scenario, let me know and I'll upload it.


  • 9.  RE: How to Monitor Active Hung Threads using Introscope Agent?

    Posted 02-15-2013 01:16 PM
    Hi Jak,
    Can you please upload your extension? Appreciate it...
    Zhang


  • 10.  RE: How to Monitor Active Hung Threads using Introscope Agent?

    Posted 02-15-2013 01:17 PM
    Thanks Jak,Florian for your input.

    Could you please upload the extension? it will be help to get the metrics you mentioned.

    Thanks,
    Madhu


  • 11.  RE: How to Monitor Active Hung Threads using Introscope Agent?

    Broadcom Employee
    Posted 02-15-2013 01:24 PM
    Love it! I'm with everyone else...share the love and upload it!


  • 12.  RE: How to Monitor Active Hung Threads using Introscope Agent?

    Posted 02-15-2013 04:11 PM
    Ok, I have uploaded the agent extention, called JTop, to the Community Documents; it is available here. It includes three files:

    [font=Courier New]JTop.jar
    readme.txt
    threads-cpu-tv.xml[font]
    The first is the agent extention itself, the second contains information on how to install and use, and the third is an optional type view if you use one of the agent's more verbose modes. I would greatly appreciate any feedback you're able to provide.


  • 13.  RE: How to Monitor Active Hung Threads using Introscope Agent?

    Posted 02-19-2013 09:15 AM
    Thanks Jakbutler for the extension.


  • 14.  RE: How to Monitor Active Hung Threads using Introscope Agent?

    Posted 02-19-2013 10:55 AM
      |   view attached
    Folks,

    I found this metric in WebSphere PMI that goes by the name "waitingThreadCount". We had some issues the other day and was looking at some of the metrics that could make sense for the Hung Threads. Surprisingly, the number of threads hung from the WebSphere Logs matched the value of the waitingThreadCount.

    This metric can be found under WebSphere PMI node (See Screenshot)

    Thanks,

    KIRAN T


  • 15.  RE: How to Monitor Active Hung Threads using Introscope Agent?

    Posted 02-19-2013 04:34 PM
    Hi Jak,
    Thanks for your share... A quick question:
    Does this extenstion work on EPAgent?
    Thx..

    Zhang


  • 16.  RE: How to Monitor Active Hung Threads using Introscope Agent?

    Posted 02-19-2013 04:52 PM
    Zhang,

    I don't see any reason why it wouldn't; however, the threads it would monitor would be of the EPAgent itself, not a process being monitored by the EPAgent, because it would become part of the EPAgent rather than act as a plugin.

    - Jack