finding the historical metrics

[pivst01]

hi All,

 

how can i find out which metrics are the historical ones?

 

we have a continuous historical metric increase so i'd like to find out which metrics cause it (and modify the instrumentation after that to fix it).

 

i cant see any grey metrics in the investigator.

 

 

thank you,

stefan

[Legacy User]

would  KPI Analysis And Collection.v2.docx help?

[Guenter DI Grossberger]

Hi Stefan,

 

go to Agents and search for ":Historical Metric Count". The truth is already there in the table or you can display the graph of the top N. E.g.

 

In this example I would focus on the first two agents. To get rid of the metrics altogether you need to use SmartStor tools or just delete the whole SmartStor and metadata if it's a non-prod system.

[pivst01]

Guneter,

thank you for your tip.

i already got the "mailicous" agents, but i still dont know the metrics.

1. i suppose the backend metrics will be the wanted, but i couldnt find even 1 "grayed(=historical) metric in the investigator". do you know how i can filter the historical metrics please?

2. i cant drop the whole smartstor as it's a prod env and i need to keep all of the other metrics = so i have to remove just these specific (hopefully founf) historical metrics. is it possible please?

 

thank you,

s.

 

[Viruvaibhav_apm]

Hi Stefan,

 

By the time Guenter replies, can you enable or do some SQL Normalization process ?
It is a common metric leak reason which happens quite many a times.

 

(You will find details in agent Installation guide.)

 

Hope it helps.

 

Regards,
Vaibhav

[pivst01]

Hi Vaibhav,

 

yes, i will do the SQL Normalization now that i see this behavior. however i still

1. need to get  the historical metrics.

2. and remove them.

 

any idea for that please?

 

thank you,

stefan

[Guenter DI Grossberger]

Hi Stefan,

 

to find out which metrics use CLW: Extract Persistent Data from SmartStor and diff historic vs current data - there is not much automated help here, maybe you can come up with some scripts and post them here?

 

to remove metrics use SmartStor tools: Using SmartStor tools to tune SmartStor data

 

If the customer does not really need all the data the easiest and fastest approach is to backup the data directory and start from scratch after the SQL normalization. They can always have a VM (with no live agents so this can be a very light setup) to access the backup data - and they almost never will!

 

Ciao,

Guenter

[pivst01]

1. i think we cant filter the historical data as it needs enormous heapsize to be allocated for the CLW if i need to display the historical metrics for the last 3 months (when the metrics started growing). in addition, i tried it just for 1min timeframe for 1 agent (less than 1% of the metrics) and i got hundreds of lines that would be really difficult to filter.

 

nevertheless i decided to remove all the backend metrics for the 20% of the agents that the historical metrics are generating, but i ran into some difficulties here again:

 

2. i can specify either the metrics (that i'd put as kind of "backend" in global, but if i do it like that i will remove the backend metrics for the other 80% of the agents that i dont want as they use the same backend calls) or the agents (but in that case i will remove all the metrics for these agents, not just the backend metrics and i cant do that as i'd loose all the application metrics)

 

maybe i missed something or forgot about something that could solve this puzzle and if you warn me to it, i'd be thankful.

 

cheers,

stefan

[pivst01]

Hi Guenter/All,

 

it's not the exact requested solution, but it was at least enough to solve our issue, so im copying it below, if it helps to somebody else in similar case.

 

our issue was that the backend metrics on s08982.* agents have been increasing dramatically and caused historical metrics.

 

with description provided by Michael Cook i was able to remove (at least) all of the backend metrics (not just the historical ones, but we could accept this limitation) for the affected agents.

 

here is the guide for it:

 

mkdir ../datanew

mkdir ../datanew1

mkdir ../datanew2

mkdir ../datanew3

mkdir ../datanew4

./SmartStorTools.sh keep_agents -agents "SuperDomain\\|s08982.*" -src ../data -dest ../datanew

./SmartStorTools.sh prune -src ../datanew -dest ../datanew1

./SmartStorTools.sh test_regex -metrics "Backends.*" -src ../datanew1 > metrics3.out

./SmartStorTools.sh remove_metrics -metrics "Backends.*" -src ../datanew1 -dest ../datanew4/

./SmartStorTools.sh remove_agents -agents "SuperDomain\\|s08982.*" -src ../data -dest ../datanew2

./SmartStorTools.sh prune -src ../datanew2 -dest ../datanew3

./SmartStorTools.sh merge ../datanew3 ../datanew4

rm -arv ../data

mv ../datanew4/ ../data

 

thank you (Michael and Guenter) for all your help,

stefan