DX Application Performance Management

Expand all | Collapse all

finding the historical metrics

Jump to Best Answer
  • 1.  finding the historical metrics

    Posted 05-21-2015 05:45 AM

    hi All,

     

    how can i find out which metrics are the historical ones?

     

    we have a continuous historical metric increase so i'd like to find out which metrics cause it (and modify the instrumentation after that to fix it).

     

    i cant see any grey metrics in the investigator.

     

    historical_metrics-01.png

     

    thank you,

    stefan



  • 2.  Re: finding the historical metrics

    Posted 05-21-2015 12:54 PM


  • 3.  Re: finding the historical metrics

    Posted 05-21-2015 02:16 PM

    Hi Stefan,

     

    go to Agents and search for ":Historical Metric Count". The truth is already there in the table or you can display the graph of the top N. E.g.

    Historical Metric Count.png

     

    In this example I would focus on the first two agents. To get rid of the metrics altogether you need to use SmartStor tools or just delete the whole SmartStor and metadata if it's a non-prod system.



  • 4.  Re: finding the historical metrics

    Posted 05-27-2015 04:58 AM

    Guneter,

    thank you for your tip.

    i already got the "mailicous" agents, but i still dont know the metrics.

    1. i suppose the backend metrics will be the wanted, but i couldnt find even 1 "grayed(=historical) metric in the investigator". do you know how i can filter the historical metrics please?

    2. i cant drop the whole smartstor as it's a prod env and i need to keep all of the other metrics = so i have to remove just these specific (hopefully founf) historical metrics. is it possible please?

     

    thank you,

    s.

     

    historical_metrics_count.png

    metric_count-agent-pd.png



  • 5.  Re: finding the historical metrics

    Posted 05-27-2015 05:10 AM

    Hi Stefan,

     

    By the time Guenter replies, can you enable or do some SQL Normalization process ?
    It is a common metric leak reason which happens quite many a times.

     

    (You will find details in agent Installation guide.)

     

    Hope it helps.

     

    Regards,
    Vaibhav



  • 6.  Re: finding the historical metrics

    Posted 05-27-2015 07:51 AM

    Hi Vaibhav,

     

    yes, i will do the SQL Normalization now that i see this behavior. however i still

    1. need to get  the historical metrics.

    2. and remove them.

     

    any idea for that please?

     

    thank you,

    stefan



  • 7.  Re: finding the historical metrics

    Posted 05-27-2015 12:22 PM

    Hi Stefan,

     

    to find out which metrics use CLW: Extract Persistent Data from SmartStor and diff historic vs current data - there is not much automated help here, maybe you can come up with some scripts and post them here?

     

    to remove metrics use SmartStor tools: Using SmartStor tools to tune SmartStor data

     

    If the customer does not really need all the data the easiest and fastest approach is to backup the data directory and start from scratch after the SQL normalization. They can always have a VM (with no live agents so this can be a very light setup) to access the backup data - and they almost never will!

     

    Ciao,

    Guenter



  • 8.  Re: finding the historical metrics

    Posted 06-01-2015 10:09 AM

    1. i think we cant filter the historical data as it needs enormous heapsize to be allocated for the CLW if i need to display the historical metrics for the last 3 months (when the metrics started growing). in addition, i tried it just for 1min timeframe for 1 agent (less than 1% of the metrics) and i got hundreds of lines that would be really difficult to filter.

     

    nevertheless i decided to remove all the backend metrics for the 20% of the agents that the historical metrics are generating, but i ran into some difficulties here again:

     

    2. i can specify either the metrics (that i'd put as kind of "backend" in global, but if i do it like that i will remove the backend metrics for the other 80% of the agents that i dont want as they use the same backend calls) or the agents (but in that case i will remove all the metrics for these agents, not just the backend metrics and i cant do that as i'd loose all the application metrics)

     

    maybe i missed something or forgot about something that could solve this puzzle and if you warn me to it, i'd be thankful.

     

    cheers,

    stefan



  • 9.  Re: finding the historical metrics
    Best Answer

    Posted 06-03-2015 07:17 AM

    Hi Guenter/All,

     

    it's not the exact requested solution, but it was at least enough to solve our issue, so im copying it below, if it helps to somebody else in similar case.

     

    our issue was that the backend metrics on s08982.* agents have been increasing dramatically and caused historical metrics.

     

    with description provided by Michael Cook i was able to remove (at least) all of the backend metrics (not just the historical ones, but we could accept this limitation) for the affected agents.

     

    here is the guide for it:

     

    mkdir ../datanew

    mkdir ../datanew1

    mkdir ../datanew2

    mkdir ../datanew3

    mkdir ../datanew4

    ./SmartStorTools.sh keep_agents -agents "SuperDomain\\|s08982.*" -src ../data -dest ../datanew

    ./SmartStorTools.sh prune -src ../datanew -dest ../datanew1

    ./SmartStorTools.sh test_regex -metrics "Backends.*" -src ../datanew1 > metrics3.out

    ./SmartStorTools.sh remove_metrics -metrics "Backends.*" -src ../datanew1 -dest ../datanew4/

    ./SmartStorTools.sh remove_agents -agents "SuperDomain\\|s08982.*" -src ../data -dest ../datanew2

    ./SmartStorTools.sh prune -src ../datanew2 -dest ../datanew3

    ./SmartStorTools.sh merge ../datanew3 ../datanew4

    rm -arv ../data

    mv ../datanew4/ ../data

     

    thank you (Michael and Guenter) for all your help,

    stefan