We have a Tomcat agent within a Tomcat instance that hosts an application. We have seen when the application starts to have a large number of open file handles, the application operational performance begins to suffer.
The Tomcat process owned by the Tomcat administrator account (non-root) is the process that owns the file handles.
Is there a way to build a custom extension to the Tomcat agent that will run a system command (lsof - p <tomcat pid>|wc -l) and publish the number of file handles?
We do have an epagent, but the agent admin account (non-root) can not access the tomcat process open file count. The tomcat agent, since it is within the tomcat JVM, the extension would run within the tomcat process and hopefully should be able to run the lsof command.
If it is possible, is there any documentation, examples or guidance on how to build the custom extension?
This is a job for the Infrastructure Agent (IA) or EPAgent. It would be trivial to build an IA/EPAgent extension to run lsof and report the metrics back to APM, where you could alert on them or include them in dashboards, etc. The beauty of using an IA/EPAgent extension is that you can use a bash/perl/python/whatever script and simply output the metric you want to report to standard out.
You can find more information on creating an EPAgent plugin at EPAgent Plug-ins - CA Application Performance Management - 10.7 - CA Technologies Documentation .
You also might want to check out
Convert a Standalone EPAgent Script to Run On the Infrastructure Agent - CA Application Performance Management - 10.7 - …
Hope this helps,
Converted to discussion for community input on any experiences of similar custom requirements.
Cross-posted in APM Development to get answers
I checked internally with Surendhar Thallapelly from Sustaining Engineering and he confirmed that although we don't have any agent extension for this at the moment, he can help you with this requirement, you would need to open a support case.
Thank you Barry!
Yes, I looked into attempting to run the command line under the same user that our epagent is running under, and since the tomcat process is not accessible to the epagent user, the epagent can not query on the number of file handles that are opened by the tomcat instance.
With the tomcat agent (java agent), it is within the JVM which is within the same user context as the tomcat process and hopefully should have the ability to execute the command and publish the results.
Is it feasible to modify the Tomcat startup script to startup a separate epagent instance to do this task? Then the epagent process ownership should match the tomcat JVM process ownership and have access to query the number of file handles. Similarly the Tomcat shutdown script can stop the separate epagent instance.
I didn't think of running a second epagent in the same context as the tomcat process.
Adding an extension to the Java Agent would be an easier sell than the second epagent, but I'll ask about the options and opposition to a second epagent to see what and how much push back I get.