DX Application Performance Management

Starting v9.5 the prune is done by default in an EM nightly process hence we shall not be concerned about and ignore "prune"

Thanks. For the past weeks I read your post (and other smartstor related) over and over.

We are about to embark on a historical metrics cleanup.  Since we have a fully loaded cluster it is going to take time and we need to ensure that this effort will have a (positive) impact.

Hence we are trying to estimate how much data canwe  easily remove?  We have reduced the agent clamp to 25K (from the v9.5 default 120K) and tackled a few agents configurations.

At first glance adding up all agents with >5K of historical metrics will reduce the overall collector's historical metrics by 10% ... not much at all.

My original question was why the metrics line count was so much smaller than its historical count. David explains that many of the metrics are listed once, but common to many or all the agents .... hence in my small example above 58% is common e.g. I see listed only once "CPU:Utilization" which we know is common.

grrr we are missing flexibility with Smartools. Note to the readers: please tell all your friends to vote up A GUI SmartStor administration tool

Fred:

Regards the difference in metrics you are seeing, here is a portion of an older thread that covers that point (Historical Metrics Limit):

"Live" metrics are those that are being generated by the agent and reporting every 7.5 seconds to the Collector.  This is later aggregated to give the 15 sec reporting interval.  Check out the APM Performance and Capacity Management Guide... on the BookShelf.

Historical metrics are ANY metric that has been "aged out" of the current display.  This happens after 60 minutes, when a metric goes from "live" to "historical".  This is simply metadata about the agent, that is preserved, so that IF and WHEN an agent reconnects with that metric, that the Collector 'knows' that metric and reuses the metadata.  This scheme works great... until it doesn't... and then excess METADATA (historical metrics count) accumulates until the working memory of the Collector is compromised... and bad things happen.

The SmartSotr also stores metric data (no strings) for up to one year.  This is, technically, a "historical" data... but this IS NOT the same thing as the historical-metric-meta-data we are talking about.

Cheers,

Mike.

Hi Fred,

The explanation is that what you get directly out of Smartstor are all of the individual metrics, however you don't have them associated to any agents.

The values you would see in Introscope itself take into account the agents as well, so it just tells you that you have some agents that have the same metrics, which is very likely.

Prune is generally removing any metrics referenced in the metrics.metadata that are without data - why would we want to store information about metrics that have no data.

You can check the supportability metrics in Enterprise Manager | Data Store | MetaData for information about any agents/metrics without data.

Thanks,

David

Hi All,

I was wondering about the following. When I count the metrics lines returned by

./SmartStorTools.sh test_regex -metrics ".*" -src /mycol01/smartstor/data | wc -l

55745

(the first 2 lines are not metrics so really 55743)

In the workstation I search for "Historical Metrics" and for that collector at that time: Number of Historical Metrics 129,952  and the Number of Metrics is 12,329

My question is: why and what is the difference between the two numbers "SmartStorTools metrics line count (~55K) vs Historical Metrics (~129K)?

Depending on the explanation, I’ll have another question, what are the metrics  prune removes, any example?

... I ran it and did not really see a difference in the resulting smartstor size the next day (after the nightly process ran).

Thanks - Fred

Hi Gokhan,

The IndexRebuilder script is for the traces database, that is the store for transaction information, error snapshots, not the metrics which go into Smartstor, so no you don't have to run it related to this article.

if you look into the traces/index folder of each of your Enterprise Manager and see that the files are very large (in Gb) or you have files that are outdated (considerably older than 14 days), it is worth running the script as a separate maintenance task at your convenience.

The script should be run with the Enterprise Manager stopped and run on each of your Enterprise Managers.

If you do decide to use it, you should review the script itself to increase the heap that the process will use because it can run out of heap - ideally you allocate the same amount of Java heap that the Enterprise Manager uses.

For example from the Windows script:

"%JAVA_HOME%\bin\java.exe" -Declipse.consoleLog=false -jar "%INSTALLDIR%\launcher.jar" -install "%INSTALLDIR%\product\enterprisemanager" -application com.wily.introscope.eclipseapp.IndexRebuilder %1

you could add a value to use 4GB of heap like this

"%JAVA_HOME%\bin\java.exe" -Xmx4096m -Declipse.consoleLog=false -jar "%INSTALLDIR%\launcher.jar" -install "%INSTALLDIR%\product\enterprisemanager" -application com.wily.introscope.eclipseapp.IndexRebuilder %1

Thanks,

David

Hello Günter,

Is it necessary to run IndexRebuilder utility after cleanup?