AS PROMISED - you get me 500 views... I share the little piece of heaven I built to get ABA into a more usable configuration.
Some caveats:
1. When I was a development director (before I found APM), I could get my teams to double their output at the mere 'threat' of a direct contribution to the project source, from yours truly! ;-)
- it is really best when I 'talk' about what the code should do... and leave implementation to the professionals!
2. This is from the 'front-lines' of a rogue project. The ship is floundering, the lifeboats have all launched, the crew and officers have all departed... and SWAT is trying to restart the reactor.
- this is not 'GA quality', it is not accepted to be built into the product or supported in any way. There is no roadmap. There is no documentation (other than blog posts - pretty thin!). This is a science project where all the regular suspects have already moved to a "safe-distance".
3. This is completely unsupported, in any way shape or form. Sure, it's "my baby" and I'll probably fix something, if I have the time. I'm giving you all the code so that you can do it yourself! Make it better - please! ;-) But don't invest a lot of time because...
4. This is a "throw-away" project. All of this functionality and process that is today manual - SHOULD be built into the standard product and completely transparent to the user. Provided that it actually works, of course. Specifically, that it makes ABA work more effectively. Once it gets into the product, NONE of this ridiculous parsing will be necessary and all of the parsing defects will go away.
5. I haven't actually proven that it will help ABA work better, worse or no change. By using this code, you agree to become an APM lab rat, and must send me the results of your crater (failure) - or triage gem (success).
6. I'm still finding cases where the generated regex is unacceptable to the MOM. When this happens, comment out the offending entry and restart. Be sure to send me your Analytics.properties file when it starts clean, so that I can continue to resolve the expression failures.
7. This is python code. This is command-line. If you don't know what those mean, hit the wikepedia, hit the google and RTFM.
Using the process for identifying KPIs, discussed here, you will have one or more test files of candidate KPIs. You can generate either a "broad" or "narrow" configuration. It's really quick, so generate both of them and try the broad one first. I think that the narrow will be the best but I really don't have any evidence (yet) either way. So here is the command line:
$ ./build_regex.py -list sap_list -output sap_broad -exceptions sap_broad_exceptions
...writing to sap_broad
...writing exceptions to sap_broad_exceptions
...processing ['./CA-GIS_beta/SAP_average.txt', ''] Index:: 0 Lines Written:: 0
...processing ['./CA-GIS_beta/SAP_errors.txt', ''] Index:: 436 Lines Written:: 2616
...processing ['./CA-GIS_beta/SAP_responses.txt', ''] Index:: 907 Lines Written:: 5442
...processing ['./CA-GIS_beta/SAP_specific.txt', ''] Index:: 1319 Lines Written:: 7914
...processing ['./CA-GIS_beta/SAP_stall_count.txt', ''] Index:: 1323 Lines Written:: 7938
...processing ['./CA-GIS_beta/SAP_stalled.txt', ''] Index:: 1816 Lines Written:: 10896
>>> Evaluated 2505 expressions and got 2316 valid with 189 invalid.
>>> Next index is 2316 and 13896 total lines were written to output file.
With the new configs, in this case sap_broad, copy paste the goodies into your Analytics.properties file and save it with a unique name. This will allow you to move quickly between original, broad and narrow configurations.
For the invalid configurations, you can try and resolve these manually, using the Search tab and checking the "use regex" box. I just ran out of time to figure out why the otherwise valid regex is not palatable to the MOM and ABA.
You might find that you already have a pretty good ABA configuration and want to preserve that. So find out the last index that you used, add one and then you can use that to set the starting point for all the ABA config that you are about to generate. Here is that command line variant, this time working with a single file:
$ ./build_regex.py -fileSAP_average.txt -index 23 -output sap_broad -exceptions sap_broad_exceptions
...writing to sap_broad
...writing exceptions to sap_broad_exceptions
...processing ['SAP_average.txt'] Index:: 23 Lines Written:: 0
>>> Evaluated 501 expressions and got 436 valid with 65 invalid.
>>> Next index is 459 and 2616 total lines were written to output file.
You can also find all of the current capabilities by using the -help or -h parameter:
$ ./build_regex.py -help
usage: build_regex.py [-h] [-list LIST] [-output OUTPUT]
[-exceptions EXCEPTIONS] [-depth {broad,narrow}]
[-debug {none,low,high,all}] [-file FILE] [-index INDEX]
build_regex.py: error: argument -h/--help: ignored explicit argument 'elp'
=======================================================================
$ ./build_regex.py-h
usage: build_regex.py [-h] [-list LIST] [-output OUTPUT]
[-exceptions EXCEPTIONS] [-depth {broad,narrow}]
[-debug {none,low,high,all}] [-file FILE] [-index INDEX]
optional arguments:
-h, --help show this help message and exit
-list LIST supplying a list of filenames of candidate metrics to
be processed into regex suitable for ABA
-output OUTPUT file to capture regex suitable for ABA
-exceptions EXCEPTIONS
file to capture any unexpected metric types
-depth {broad,narrow}
controls quantity of regex (more/less metrics)
-debug {none,low,high,all}
controls debug info level into output file
-file FILE supplying a single file of candidate metrics to
process
-index INDEX set the starting index for the regular expressions
Some Friday afternoon projects you 'll want to consider:
1. just generate the Analytics.properties file and avoid the whole mess with intermediate output files and copy/paste.
2. Write the EM extension that overloads the Search TypeView (SearchKPIs) and goes against the internal data structures (avoiding risky test parsing) and generates the Analytics.properties file directly. This also lets us sidestep the 500 metric display limit, provided we allow for a KPI validation method that picks the best KPIs from the suspect pool. I'll be working on the KPI validation prototype next but its just implementing the rules I established in my book, which is how we pick KPIs manually.
Get your code here.