DX Application Performance Management

Instrumentation Best Practices 

03-17-2015 07:48 PM

Instrumentation

Here are some instrumentation best practices:

  1. Use APM Studio!
  2. Reference your <application>.pbl in the introscope.autoprobe.directivesFile property in your IntroscopeAgent.profile. Relieves you of the need to update the profile.
  3. Reference your custom pbds in your <application>.pbl. You can add any number of pbd files independently of the profile. See example below.
  4. Separate the TurnOn directives into a <application>-toggles.pbd
  5. Automate deployment of your custom pbds and <application>.pbl to your test server (e.g. using ant, maven, jenkins, chef, CA Release Automation)
  6. Add your APM config files to version control (cvs, subversion, …)
  7. Automate your (UAT/Prod) agent deployments using existing processes/tools (e.g. application deployment, release automation, software delivery)
  8. 3 step pbd creation:
    1. Optional (usually done by CA Services if needed): Implement and test custom nameformatters and tracer, define tracer mappings
    2. Create (SetFlag) and enable (TurnOn) your tracer group
    3. Add classes to tracer groups (Identify…)
      1. From classes in Eclipse (drag & drop, right mouse click – context menu “Introscope”)
      2. From AutoProbe.log file
      3. Manually
    4. Add methods, tracers and metric names to tracer (Trace…)
      1. Use TraceComplexMethods... if you don't know what to instrument. Don't ever use TraceAllMethods!
    5. Optional: add skips (Skip…) to reduce number of instrumented methods/classes
  9. Evaluate and reduce your instrumentation:
    1. Monitor only methods that add significant time to a transaction: use search tab on a high level node, search for "Average Response" and enable "Show min.max, count". You get all relevant info in one sortable table!
    2. Remove (or skip) methods with high invocation count and low (near zero) response time (e.g. getters/setters, utility methods). Focus on methods like execute, processXXX, ...

Related Links

Custom pbl/pbd example

 

IntroscopeAgent.profile:

[...]
introscope.autoprobe.directivesFile=tomcat-typical.pbl,myapplication.pbl

[...]

 

myapplication.pbl:

myapplication-errors.pbd
myapplication-pojos.pbd
myapplication-backends.pbd
myapplication-toggles.pbd

 

myapplication-errors.pbd:

# report exception thrown from MyController.handle(), Error message = e.getMessage()

TraceOneMethodWithParametersIfFlagged: MyAppErrorTracing handle ExceptionErrorReporter “My|Controller|Had|An|Error:Errors Per Interval”



# report creation of com.ca.MyCustomException, Error message = e.toString()

TraceOneMethodWithParametersIfFlagged
: MyAppExceptionTracing
<init> ThisErrorReporter "My|Exception|Was|Invoked:Errors Per Interval"



# report invocation of MyController.onError(), Error message =~ "MyController.onError()"

TraceOneMethodWithParametersIfFlagged
: MyAppErrorTracing
onError MethodCalledReporter “My|onError|Was|Invoked:Errors Per Interval"

 

myapplication-toggles.pbd:

TurnOn: MyAppPojoTracing
TurnOn: MyAppEJBTracing
#TurnOn: MyAppEJBDetailTracing

TurnOn: MyAppBackendTracing
TurnOn: MyAppErrorTracing
TurnOn: MyAppExceptionTracing

 

Making sense of your AutoProbe.log

This script will extract a few helpful files from your Autoprobe.log. This will work with any agent version and configuration. The only requirement is that the code that you want instrument needs to be executed - so especially on test systems make sure that your test case is executed at least once. Only then will the classes be loaded by the JVM and written to the Autoprobe.log.

 

#!/bin/sh

if [ -z "$1"  ]
then
    echo "Usage: $0 <AutoProbe.log>"
  exit 1
fi

#Get all classes in AutoProbe.log:
cat $1 | grep 'Processing class' | sed 's/^Processing class //' | sort > classes.$1

#Get all packages in AutoProbe.log:
cat classes.$1 | sed 's#/[a-zA-Z0-9\$_]*$##g' | uniq > packages.$1

#Get all frameworks in AutoProbe.log:
cat classes.$1 | sed -n 's/\([a-z]*\/[a-z]*\/[a-z0-9]*\)\/.*/\1/p' | uniq > frameworks.$1

#Find all classes ending in Manager, Service, Servlet (and not containing impl or Impl in class or package name):
cat classes.$1 | grep -E '(Manager|Service|Servlet)$' | grep -iv impl | uniq > services.$1

#Find all classes ending in Exception or Error
cat classes.$1 | grep -E '(Exception|Error)$' | grep -iv impl | uniq > exception.$1

#Find all instrumented classes:
cat $1 | grep -B 1 "inserted method tracer" | grep 'Processing class' | sed 's/^Processing class //' | sort | uniq > instrumented.$1

 

 

Is there a volunteer for a similar Windows script?

 

Generating a pbd from your AutoProbe.log

I wrote the attached generate_pbd.pl perl script that creates a pbd from the services.<agent>.log created with the scripts above. Do not use a plain Autoprobe.log because it will generate too many classes many of which are already monitored by the default APM configuration (e.g. j2ee.pbd). The same restriction applies as stated above.

 

usage: generate_pbd.pl <inputfile> <outputfile> [<applicationname> <author>]

 

E.g. perl generate_pbd.pl services.AutoProbe.TomcatAgent.log myApp.pbd MyTestApp guenter.grossberger@ca.com

 

Have fun!

Statistics
0 Favorited
5 Views
1 Files
0 Shares
0 Downloads
Attachment(s)
zip file
generate_pbd.pl.zip   1K   1 version
Uploaded - 05-29-2019

Tags and Keywords

Comments

11-18-2016 08:33 AM

APM Studio is now version at 3.8.0.3 if you go to the Eclipse - Help - Add New Software - Work with: http://apmstudio.org

11-17-2016 11:44 AM

Hi everybody,

it seems to me the APM Studio is still working with CA APM 10.3.

 

The following files are used in the Bundle:

com.ca.apm.crypto_10.3.0.jar
com.wily.core_10.3.0.jar
com.wily.introscope.bindings_10.3.0.jar
com.wily.introscope.common_10.3.0.jar
com.wily.introscope.em.client.clw_10.3.0.jar
com.wily.introscope.em.client14_10.3.0.jar
com.wily.introscope.em.client_10.3.0.jar
com.wily.isengard.client_10.3.0.jar
com.wily.localizationservice_10.3.0.jar
com.wily.log4j_1.2.8.jar
com.wily.osgi.logservice_10.3.0.jar
com.wily.utilities_10.3.0.jar
org.apache.commons_logging_1.0.4.jar
org.jasypt_1.5.jar
org.owasp.esapi_2.0.0.jar

 

Bye, Lutz

10-13-2015 08:43 AM

Hi Guenter,

the apmstudio.org homepage is gone again.

I got a blue page from a QNap Turbo NAS box only, any idea.

Remove Andreas the APMStudio support/homepage, the page is not available since last week.

Bye, Lutz

05-26-2015 06:14 PM

Every thing works well now.

I can install APMStudio on my Eclipse Kepler with Java 6, but Eclipse Luna with Java 8 doesn't work (hangs all the time I try to open the APM Studio Workspace) on my OS X Yosemite.

 

Nice to see,

the Site is back again,

bye, Lutz

 

p.s.

For backup purpose, I create a APMStudio installer package myself.

05-25-2015 12:37 PM

Hi all folks,

apmstudio.org is back again now, but my eclipse installation hangs at 49% and doesn't continue.

Yes, we know. It is Andreas' "private" server. Only he can fix it/have it fixed.

Nice to know this is a private project from Andreas Reiss, I was unaware of this.

Bye, Lutz

05-21-2015 09:30 PM

I can confirm that is the case. I've used the 9.1 bundle with all versions of 9.1, 9.5, and 9.6. I've not tested with 9.7, but there's no reason why it wouldn't.

Better yet, follow Andrea's directions on the site and create your own bundle.

05-21-2015 01:56 PM

As far as I know the 9.1 bundles should work for all versions >= 9.1.

05-21-2015 01:54 PM

Thanks, Guenter.

I also noticed that there are APM bundles available for only APM v9.0 and v9.1. Will APM Studio work with APM v9.7 also or do we need new bundles created for APM v9.5+.

Thanks,

Bryan

05-21-2015 01:43 PM

Hi,

 

Yes, we know. It is Andreas' "private" server. Only he can fix it/have it fixed.

 

Ciao,

Guenter

05-21-2015 01:31 PM

Just a FYI. The "http://apmstudio.org/" site is not responding and times out. Is there another location where we can download it?

04-09-2015 04:29 PM

Hello Haruhiko,

 

Thanks a lot for your reply, Can you tell me something about the

documentation, where is the new documentation?

 

Thanks,

04-09-2015 04:25 PM

All of the installation instructions are available on the site. Setting up a new installation site in your Eclipse setup will ensure you're getting the latest stable version.

04-09-2015 03:48 PM

Hello guys,

 

This comment was created by Guenter Grossberger the 17/03/2015 04:48 PM,

and I don´t understand what it mean "Use APM Studio: there's a new

update available on the Update Site, Andreas wrote a new documentation

that is awesome!".

 

I read the News session in "http://apmstudio.org/" and the last comment was done

in "Wednesday , November, 2012".

 

The last versión of APM Studio is 2.0, if there is a new version, where

I can get it?, if there is a new documentation, where I can get it?.

 

I believe is so important to share at least one video to install this

tool in the Educate session in youtube.

 

I am sorry, but I am new one in this amazing world of application

performance management and I will appreciate every help that you can

provide for every new person that is starting.

 

Thanks,

03-20-2015 05:37 PM

Good observation! Best to remove Servlets from this command or generate another file without the servlets.

03-20-2015 05:20 PM

You can go to APM Studio website and it shows you the files you need that you can get from your EM installation.

 

You don't really need to get it from CA directly.

03-20-2015 03:31 PM

In regards to the generate_pbl, for educational purpose, can you explain the choice of using the services file.

e.g. aren't out-of-the-box all Servlets already instrumented by the blame tracer ?

03-20-2015 03:18 PM

Hallo Hr Grossberger,

ja, das habe ich gesehen, die Beschreibung zur Installation von APMStudio

setzt nur voraus, das man dies auch kann/darf.

Dies ist bei uns nicht gegeben. Man kann zwar grundsaetzlich Elipse um

einige Tools erweitern, nur nicht ueber die Update/Install Funktion vom

Eclipse selber.

 

Nun ja, ist halt so.

 

You also need the "CA Technologies" bundle which are basically some of

the plugins from the EM.

PM me your email and I'll send it to you.

 

Nehme ich, wenn Sie den uebrigen Teil der Eclipse Installation ebenfals

anhaengen koennen, an die Mail.

 

Danke fuer die Antwort/Muehe,

Lutz Mader

03-20-2015 09:26 AM

Hallo LutzM

Go to APMStudio and click on installation:

Configure a new update sie

    Name: APM Studio Update Site
    Location: http://apmstudio.org

You also need the "CA Technologies" bundle which are basically some of the plugins from the EM. PM me your email and I'll send it to you.

 

Beste Gruesse,

Guenter

03-20-2015 04:43 AM

Hi,

nice idea to use the autoprobe data to create a new pbd.

Easy way to get a good base for a more detail application/prolem based pbd.

Lutz

 

p.s.

I can't find a download lik for the apmstudio and can't download via Eclipse "install new software".

03-20-2015 12:14 AM

Wow! Another great update!
I can't wait to try this out!

03-18-2015 02:14 PM

Thanks then here we go; appending the count number to each instrumented class:

cat $1 | grep -B 1 "inserted method tracer" | grep 'Processing class' | sed 's/^Processing class //' | awk '{++S[$1]} END { for (a in S) print a, S[a]}' | sort > instrumented.$1

03-18-2015 01:39 PM

HI Fred.K,

 

run cat $1 | grep -B 1 "inserted method tracer" | grep 'Processing class' | sed 's/^Processing class //' | sort

 

this will give you all processed classes. Without "uniq" duplicate entries won't be removed. I have no simple solution to count the numbers of identical rows in the output, though

 

Ciao,

Guenter

03-18-2015 08:53 AM

I like the idea of making sense of the autoprobe log.

I am curious, how do you detect the re-instrumentation of the same ?dynamic? classes in short periods of time which acan lead to poor applicaiton performance?

03-17-2015 08:47 PM

Also for helpful tips on instrumentation, please go to this page on the CA Developer APM wiki.

03-17-2015 08:39 PM

Awesome, Guenter_Grossberger!

Always coming up with the good stuff!

Related Entries and Links

No Related Resource entered.