How to Monitor Active Hung Threads using Introscope Agent?

[madhu1512]

Hello,

Is there any possibility to capture Active hung threads using Introscope Agent? If so, how can we achieve this?

Thanks,

[Anon Anon]

What application server/service did you want to monitor hung threads?

I know in WebSphere you can tap into the WebSphere PMI and there is a metric for "ConcurrentHungThreadCount". In Tomcat you can tap into JMX, but I don't know if it has a hung thread metric by default.

[Sergio Morales Correa]

OPTION 1: Use EPAgent LogReader scripts.

a) If you are using Weblogic and Webpshere, you can use WebsphereLogReader or Weblogic LogReader. For example, for WebsphereLogReader, update the websphereLogReader.cfg as below:

Files to be modified:
<EM_HOME>/config/IntroscopeEPAgent.properties

In "Stateful Plugins" section:
introscope.epagent.plugins.stateful.names=WEBSPHERE
introscope.epagent.stateful.WEBSPHERE.command=perl 'C:/Program Files/CA Wily/Introscope8.1.0.0/epaplugins/websphere/websphereLogReader.pl'

<EM_HOME>/epaplugins/websphere/websphereLogReader.cfg

LOGFILEDIR = 'C:/IBM/WebSphere61/AppServer/profiles/AppSrv02/logs/server1'
LOGFILE = 'SystemOut.log'
SLEEPTIME =300

{
match => 'WSVR0605W',
metricType => 'IntAverage',
metricName => 'Thread Errors',
metricValue => '$CURRENTCOUNT'
},

The paths above are for my environment on windows. Your's may be different.

b) If you are not using weblogc or websphere and you need a generic log reader you might want to use logreader_package.zip that you can customize it as you need, however code modification might be required. I have attached to the documentation library.

NOTE: Unless you know the generated code error/message in the appserver log, you cannot use EPAgent is thisfashion, otherwise you might want to use stall counts, which detects stalls of a given method.

OPTION 2: Use Stall Metrics:

The StalledMethodTracer is used to identify any "hung threads", which are threads that get stuck in a method call
StalledMethodTracer traces the number of method invocations that have exceeded a user-defined execution threshold value and have not finished. This number serves as an approximation of the number of hung threads. The result is reported in the Metrics tree using the fully qualified Metric name specified in the directives file. If desired, users can set up Alerts on this Metric.

-Stall metrics are turned on by default. If you navigate through the Investigator tree of a particular agent you will see a stall count. You might want to create a metric group based on the stall count metrics. If a stall is discovered send an alert. Stall are generally associated with a hung thread
-Stall events, on the other hand, are enabled with ErrorDetector. You will have to install and configure error detector of you are using v8.x
-Stalls are stored as ordinary errors, and will be visible in the Errors TypeView, or in the historical query viewer by querying for "type:errorsnapshot"

For more information on the StalledMethodTracer refer to the JavaAgent.pdf

OPTION 3: Use PMI data - This will give you the number of hung threads, I suggest you to discuss this with your appserver admin person.

OPTION 4: Turn on ThreadTracing - Uncomment "ThreadTracing" property in your toggles-full and toggles-typical pbd files. This will monitor each thread.

Regards,

Sergio

[Haruhiko Davis]

This thread's discussion was not a part of the original.

The new thread can be found at 100261087.

[Anon Anon]

A simpler option might be to use the Field Agent extension here:

ExtendedMemInfo.zip

Unzip in your Agent root folder, eventually app the .profile.part to your Agent.profile, restart, you should see the info you need, and you can use traditional Introscope goodies (Metric groupins, dahsboards, alerts, etc...) on the metrics. Take a look at the image file for the sample metrics.

Regs,

Florian.

[nayakvinay]

Hi Florian,

 

Can you please share ExtendedMemInfo.zip.I am not able to download the ExtendedMemInfo.zip.We want to achive Hogging/Stuck threads in CA APM .

 

Regards

Vinay Nayak

[Haruhiko Davis]

Please open a support ticket to get a copy.

[Legacy User]

If the thread is not exactly hung but is instead a "runaway" thread and is eating up lots of CPU, I have a custom extension that can help with that. It may not help with what you're trying to achieve, but I figured I'd mention it.

By deafult, it outputs three metrics:

[font=Courier New]Threads:Max CPU %
Threads:Max CPU Thread
Threads:Max CPU Thread Delta Flag[font]

The first metric is the largest percentage of system CPU being used by a thread. The second is a string containing the ID and name of the thread with the max cpu %. The third metric is a flag indicating whether the top thread has changed. You can use these to create two alerts: 1) Max CPU % when over 90% (or whatever) for three out of three periods and 2) Max CPU Thread Delta Flag when 0 (not changed) for three out of three periods. Then create a summary alert for when both of the other alerts are red, which results in having an alert for when the same thread is using >= 90% CPU consistently for 45 seconds. You can then use the Thread Dump ability of Introscope to track down the thread and find its stack trace, etc.

The extension is also capable of outputting CPU Utilization metrics by Thread Name, Thread ID, or the Thread Class Name, but those modes can cause metric explosions if used incorrectly or on very dynamically threaded JVMs.

If this sounds useful to your scenario, let me know and I'll upload it.

[Anon Anon]

Hi Jak,
Can you please upload your extension? Appreciate it...
Zhang

[madhu1512]

Thanks Jak,Florian for your input.

Could you please upload the extension? it will be help to get the metrics you mentioned.

Thanks,
Madhu

[Haruhiko Davis]

Love it! I'm with everyone else...share the love and upload it!

[Legacy User]

Ok, I have uploaded the agent extention, called JTop, to the Community Documents; it is available here. It includes three files:

[font=Courier New]JTop.jar
readme.txt
threads-cpu-tv.xml[font]
The first is the agent extention itself, the second contains information on how to install and use, and the third is an optional type view if you use one of the agent's more verbose modes. I would greatly appreciate any feedback you're able to provide.

[madhu1512]

Thanks Jakbutler for the extension.

[Anon Anon]

Folks,

I found this metric in WebSphere PMI that goes by the name "waitingThreadCount". We had some issues the other day and was looking at some of the metrics that could make sense for the Hung Threads. Surprisingly, the number of threads hung from the WebSphere Logs matched the value of the waitingThreadCount.

This metric can be found under WebSphere PMI node (See Screenshot)

Thanks,

KIRAN T

[Anon Anon]

Hi Jak,
Thanks for your share... A quick question:
Does this extenstion work on EPAgent?
Thx..

Zhang

[Legacy User]

Zhang,

I don't see any reason why it wouldn't; however, the threads it would monitor would be of the EPAgent itself, not a process being monitored by the EPAgent, because it would become part of the EPAgent rather than act as a plugin.

- Jack