DX Application Performance Management

Instrumentation

Here are some instrumentation best practices:

1. Use APM Studio!
2. Reference your <application>.pbl in the introscope.autoprobe.directivesFile property in your IntroscopeAgent.profile. Relieves you of the need to update the profile.
3. Reference your custom pbds in your <application>.pbl. You can add any number of pbd files independently of the profile. See example below.
4. Separate the TurnOn directives into a <application>-toggles.pbd
5. Automate deployment of your custom pbds and <application>.pbl to your test server (e.g. using ant, maven, jenkins, chef, CA Release Automation)
6. Add your APM config files to version control (cvs, subversion, …)
7. Automate your (UAT/Prod) agent deployments using existing processes/tools (e.g. application deployment, release automation, software delivery)
8. 3 step pbd creation:
   1. Optional (usually done by CA Services if needed): Implement and test custom nameformatters and tracer, define tracer mappings
   2. Create (SetFlag) and enable (TurnOn) your tracer group
   3. Add classes to tracer groups (Identify…)
      1. From classes in Eclipse (drag & drop, right mouse click – context menu “Introscope”)
      2. From AutoProbe.log file
      3. Manually
   4. Add methods, tracers and metric names to tracer (Trace…)
      1. Use TraceComplexMethods... if you don't know what to instrument. Don't ever use TraceAllMethods!
   5. Optional: add skips (Skip…) to reduce number of instrumented methods/classes
9. Evaluate and reduce your instrumentation:
   1. Monitor only methods that add significant time to a transaction: use search tab on a high level node, search for "Average Response" and enable "Show min.max, count". You get all relevant info in one sortable table!
   2. Remove (or skip) methods with high invocation count and low (near zero) response time (e.g. getters/setters, utility methods). Focus on methods like execute, processXXX, ...

Related Links

• Full ProbeBuilder Directives List in the APM documentation
• APM Studio: http://apmstudio.org
• Custom Instrumentation with CA APM

Custom pbl/pbd example

IntroscopeAgent.profile:

[...]
introscope.autoprobe.directivesFile=tomcat-typical.pbl,myapplication.pbl

[...]

myapplication.pbl:

myapplication-errors.pbd
myapplication-pojos.pbd
myapplication-backends.pbd
myapplication-toggles.pbd

myapplication-errors.pbd:

# report exception thrown from MyController.handle(), Error message = e.getMessage()

TraceOneMethodWithParametersIfFlagged: MyAppErrorTracing handle ExceptionErrorReporter “My|Controller|Had|An|Error:Errors Per Interval”

 

# report creation of com.ca.MyCustomException, Error message = e.toString()

TraceOneMethodWithParametersIfFlagged
: MyAppExceptionTracing
 <init> ThisErrorReporter "My|Exception|Was|Invoked:Errors Per Interval"

 

# report invocation of MyController.onError(), Error message =~ "MyController.onError()"

TraceOneMethodWithParametersIfFlagged
: MyAppErrorTracing
 onError MethodCalledReporter “My|onError|Was|Invoked:Errors Per Interval"

myapplication-toggles.pbd:

TurnOn: MyAppPojoTracing
TurnOn: MyAppEJBTracing
#TurnOn: MyAppEJBDetailTracing

TurnOn: MyAppBackendTracing
TurnOn: MyAppErrorTracing
TurnOn: MyAppExceptionTracing

Making sense of your AutoProbe.log

This script will extract a few helpful files from your Autoprobe.log. This will work with any agent version and configuration. The only requirement is that the code that you want instrument needs to be executed - so especially on test systems make sure that your test case is executed at least once. Only then will the classes be loaded by the JVM and written to the Autoprobe.log.

#!/bin/sh

if [ -z "$1"  ]
then
    echo "Usage: $0 <AutoProbe.log>"
  exit 1
fi

#Get all classes in AutoProbe.log:
cat $1 | grep 'Processing class' | sed 's/^Processing class //' | sort > classes.$1

#Get all packages in AutoProbe.log:
cat classes.$1 | sed 's#/[a-zA-Z0-9\$_]*$##g' | uniq > packages.$1

#Get all frameworks in AutoProbe.log:
cat classes.$1 | sed -n 's/\([a-z]*\/[a-z]*\/[a-z0-9]*\)\/.*/\1/p' | uniq > frameworks.$1

#Find all classes ending in Manager, Service, Servlet (and not containing impl or Impl in class or package name):
cat classes.$1 | grep -E '(Manager|Service|Servlet)$' | grep -iv impl | uniq > services.$1

#Find all classes ending in Exception or Error
cat classes.$1 | grep -E '(Exception|Error)$' | grep -iv impl | uniq > exception.$1

#Find all instrumented classes:
cat $1 | grep -B 1 "inserted method tracer" | grep 'Processing class' | sed 's/^Processing class //' | sort | uniq > instrumented.$1

Is there a volunteer for a similar Windows script?

Generating a pbd from your AutoProbe.log

I wrote the attached generate_pbd.pl perl script that creates a pbd from the services.<agent>.log created with the scripts above. Do not use a plain Autoprobe.log because it will generate too many classes many of which are already monitored by the default APM configuration (e.g. j2ee.pbd). The same restriction applies as stated above.

usage: generate_pbd.pl <inputfile> <outputfile> [<applicationname> <author>]

E.g. perl generate_pbd.pl services.AutoProbe.TomcatAgent.log myApp.pbd MyTestApp guenter.grossberger@ca.com

Have fun!