I could find difference between the actual thread count and the thread count in thread dump. How to interpret both of them. Thread count shows value around 7K and in thread dump, we couldn't find all the references of the 7000 threads. Please find the screenshots of both below. Please assist.
Actual Thread Count
Thread dump from Introscope
I can't see the full metrics name for "Active Threads". Where is that metrics coming from. It may be the case that it is incremented when a new thread is created but never decremented. Can you show it over a longer time period (from server start)?
Kindly open a support issue and provide thread dumps, agent logs and other necessary details.
There have been bugs around Thread monitoring that were addressed in later versions making count reported by Agent more accurate vs. thread dumps.
Which is the accurate one, actual thread count reported or the thread dump. We are using version 9.5.
Of course count of threads reported in the thread dump is accurate and based on timing there could be little difference in count between agent and actual dump, but not as much as you are seeing which is definitely a bug. A bug was addressed in 9.5.3 and higher versions for active threads count so I would recommend upgrading to at least 9.5.3 Agent (compatible with 9.5.x EM) and see if that addresses the issue. If not kindly open a support issue so we can track it further.
I am marking this issue as solved now.
I did research on the thread count and thread dump. We are using 220.127.116.11 version of wily. I could see a differencial behavior of Performance and Production environment. In Performance, there is only nominal difference between the thread count and thread dump but in Production we can see a huge difference.
For eg. Performance shows activemq thread count as only 40 and Production shows the count as 15000 for the same thread though the code in performance and production is same. Also same version of wily exists in both the environments.
Please help in analysis.
For a detailed analysis, do as Kulbir suggests above and open a case providing what he recommended.
Sharing the screenshot for 7 days period. The count shows it has increased over successive period. But when we tried to analyse as to why its increasing we couldn't find the exact no. of threads in the thread dump. The version we are using is 9.5. Is it a bug and if it is, any other way we can analyse the threads. Please assist.