Instrumentation
Here are some instrumentation best practices:
- Use APM Studio!
- Reference your <application>.pbl in the introscope.autoprobe.directivesFile property in your IntroscopeAgent.profile. Relieves you of the need to update the profile.
- Reference your custom pbds in your <application>.pbl. You can add any number of pbd files independently of the profile. See example below.
- Separate the TurnOn directives into a <application>-toggles.pbd
- Automate deployment of your custom pbds and <application>.pbl to your test server (e.g. using ant, maven, jenkins, chef, )
- Add your APM config files to version control (cvs, subversion, …)
- Automate your (UAT/Prod) agent deployments using existing processes/tools (e.g. application deployment, release automation, software delivery)
- 3 step pbd creation:
- Optional (usually done by CA Services if needed): Implement and test custom nameformatters and tracer, define tracer mappings
- Create (SetFlag) and enable (TurnOn) your tracer group
- Add classes to tracer groups (Identify…)
- From classes in Eclipse (drag & drop, right mouse click – context menu “Introscope”)
- From AutoProbe.log file
- Manually
- Add methods, tracers and metric names to tracer (Trace…)
- Use TraceComplexMethods... if you don't know what to instrument. Don't ever use TraceAllMethods!
- Optional: add skips (Skip…) to reduce number of instrumented methods/classes
- Evaluate and reduce your instrumentation:
- Monitor only methods that add significant time to a transaction: use search tab on a high level node, search for "Average Response" and enable "Show min.max, count". You get all relevant info in one sortable table!
- Remove (or skip) methods with high invocation count and low (near zero) response time (e.g. getters/setters, utility methods). Focus on methods like execute, processXXX, ...
Related Links
Custom pbl/pbd example
IntroscopeAgent.profile:
[...]
introscope.autoprobe.directivesFile=tomcat-typical.pbl,myapplication.pbl
[...]
myapplication.pbl:
myapplication-errors.pbd
myapplication-pojos.pbd
myapplication-backends.pbd
myapplication-toggles.pbd
myapplication-errors.pbd:
# report exception thrown from MyController.handle(), Error message = e.getMessage()
TraceOneMethodWithParametersIfFlagged: MyAppErrorTracing handle ExceptionErrorReporter “My|Controller|Had|An|Error:Errors Per Interval”
# report creation of com.ca.MyCustomException, Error message = e.toString()
TraceOneMethodWithParametersIfFlagged
: MyAppExceptionTracing
<init> ThisErrorReporter "My|Exception|Was|Invoked:Errors Per Interval"
# report invocation of MyController.onError(), Error message =~ "MyController.onError()"
TraceOneMethodWithParametersIfFlagged
: MyAppErrorTracing
onError MethodCalledReporter “My|onError|Was|Invoked:Errors Per Interval"
myapplication-toggles.pbd:
TurnOn: MyAppPojoTracing
TurnOn: MyAppEJBTracing
#TurnOn: MyAppEJBDetailTracing
TurnOn: MyAppBackendTracing
TurnOn: MyAppErrorTracing
TurnOn: MyAppExceptionTracing
Making sense of your AutoProbe.log
This script will extract a few helpful files from your Autoprobe.log. This will work with any agent version and configuration. The only requirement is that the code that you want instrument needs to be executed - so especially on test systems make sure that your test case is executed at least once. Only then will the classes be loaded by the JVM and written to the Autoprobe.log.
#!/bin/sh
if [ -z "$1" ]
then
echo "Usage: $0 <AutoProbe.log>"
exit 1
fi
#Get all classes in AutoProbe.log:
cat $1 | grep 'Processing class' | sed 's/^Processing class //' | sort > classes.$1
#Get all packages in AutoProbe.log:
cat classes.$1 | sed 's#/[a-zA-Z0-9\$_]*$##g' | uniq > packages.$1
#Get all frameworks in AutoProbe.log:
cat classes.$1 | sed -n 's/\([a-z]*\/[a-z]*\/[a-z0-9]*\)\/.*/\1/p' | uniq > frameworks.$1
#Find all classes ending in Manager, Service, Servlet (and not containing impl or Impl in class or package name):
cat classes.$1 | grep -E '(Manager|Service|Servlet)$' | grep -iv impl | uniq > services.$1
#Find all classes ending in Exception or Error
cat classes.$1 | grep -E '(Exception|Error)$' | grep -iv impl | uniq > exception.$1
#Find all instrumented classes:
cat $1 | grep -B 1 "inserted method tracer" | grep 'Processing class' | sed 's/^Processing class //' | sort | uniq > instrumented.$1
Is there a volunteer for a similar Windows script?
Generating a pbd from your AutoProbe.log
I wrote the attached generate_pbd.pl perl script that creates a pbd from the services.<agent>.log created with the scripts above. Do not use a plain Autoprobe.log because it will generate too many classes many of which are already monitored by the default APM configuration (e.g. j2ee.pbd). The same restriction applies as stated above.
usage: generate_pbd.pl <inputfile> <outputfile> [<applicationname> <author>]
E.g. perl generate_pbd.pl services.AutoProbe.TomcatAgent.log myApp.pbd MyTestApp guenter.grossberger@ca.com
Have fun!