Historic Metric Growth - Locating

[Billy Cole]

Hi Community,

 

Does anyone have a couple of smartstor tool queries that might help track down the historic metric growth from a cross-enterprise agent?

 

It has been several years since I had to deal with an agent (Cross-Enterprise agent / Sysview) and that agent is being upgraded from 9.6 to 10.0.  During this, we are seeing the historic metric count grow by about 2,000 metrics per day, specifically between 10:30 pm and 12:00 am.  

In the past, I would just start expanding each of the metric groups under the agent looking for grayed out, but I am not seeing any.

 

My guess, is by the time I look, the metric has already reported gray for an hour and now is no longer being reported to the metric browser.

 

I would like to run a smartstor tool query to return the unique historic metric meta data and then another query for live metric meta data, do a compare to see what the differences are so I can attempt to have the cross-enterprise agent to stop generating the additional historic metrics that we are not able to use/alert on/display

 

Thank you,

 

Billy

[Junaid Wahab]

I found following link will help you working on smartstor

 

https://docops.ca.com/ca-apm/10-5/en/administrating/configure-enterprise-manager/configure-and-manage-smartstor-data#ConfigureandManageSmartStorData-UsetheSmartStorCommand-LineTools

 

https://communities.ca.com/docs/DOC-231167841

[Billy Cole]

Thank you Junaidwily,

 

I've attempted to use the smartstor tool by issuing the "smartstortool.sh -help" to then get a list of commands, then try to weed through them to try to find what I hope is accessible.

 

 

./SmartStorTools.sh list_agents -agents "(.*)SERVER(.*)" -src /wilyCollector/data >> agents.txt

 

This did give me the entire agent name so I know that I'm on the right collector.

 

./SmartStorTools.sh test_regex -agents "(.*)SERVER(.*)"  -src /wilyCollector/data >> herametrics.txt

 

This did not do what I expected it to do.  When I used the -metric option, the command returned that I could use either -agent or -metric but not both.

 

The documentation is focused on using the smartstor tool to prune and not so much to try to determine the metric growth of an agent.

 

Does anyone have a way to locate no longer reporting metrics other than find them through the metric browser, as they gray out?

 

Thanks

 

Billy

[Matthew Muskaloon]

Have you looked into utilizing the APM SQL Server?

 

CA APMSQL Cookbook - CA Application Performance Management - 10.5 - CA Technologies Documentation 

[Haruhiko Davis]

The only hurdle is it requires a security token from Team Center.

[Matthew Muskaloon]

True, thanks for clarifying that.

[Billy Cole]

Thanks Musma03.

 

Sadly we do not have the APM SQL server installed since I just requested 25 servers (RHEL) to replace our 25 SLES servers for our upgrade/re-platform for 10 to 10.5.2.

 

Additionally, this is version 10.0 which does not have the APM SQL component.

 

It would take me a few weeks to get all the requests created, approved and actually get a server so I could install and try to configure the APM SQL.

[Junaid Wahab]

bwcole

For APMSQL Server you don't need to dedicated whole server.

I am using my Secondary MOM for APMSQL server. if you don't have Secondary MOM you can use ATC/WebView server. You just need to configure [EM_HOME]/config/APMSqlServer.properties and start APMSQL server from [EM_HOME]/APMSqlServer/bin/apmsql &.

if you want to use any other server just copy entire APMSqlServer folder and start apmsql &. 

[Billy Cole]

Thanks Junaidwily.

 

On investigating the APM SQL server isn't available for APM 10.0.0.12.  We are moving to 10.5.2 but the current issue of trying to find the historic metrics that are no longer live is on our 10.0 environment.

 

The documentation "We recommend that you install the CA APMSQL Server on a separate server to avoid a memory overhead on the Enterprise Manager."

 

Our EMs are "right sized" so they have only the CPU/memory for what it does and nothing extra.

[Billy Cole]

After a bit of research, found no real way to query the meta data from an agent to determine the difference between the live and historic metrics in v10.0.0.12.

 

We are seeing about a 2,000 historic metric growth every day and the agent is restarted every 2 weeks which keeps the total historic metrics below 36,000 metrics.  This is below the 50,000 historic metric clamp.  We are going to try to catch the agent at midnight to see if we can find the grayed out metrics in the metric browser then if we can find them, look into the sysview/cross enterprise agent configuration to see if we can turn those metrics off.

 

So, for versions before the release of the APMSQL server, didn't find an answer and since we are not running the APM SQL server, don't know if that will provide a way to query the differences between the live versus historic metrics.

[Legacy User]

Hello Billy,

 

By deafult CE APM try to recover "all" the metrics from sysview that you know you should recover.
Using the CE APM conficuracion file, Cross-Enterprise_APM_Dynamic.properties, you can enable or disable the collection of a certain category of metrics or define through regular expressions, to filter specific elements of a specific category.

The above will not prevent you from having metrics in gray in APM for metrics generated from Sysview, since this will depend on whether the metric generated in Sysview has persistence or not. Since sysview many alerts are generated that in APM you will see them active only while the condition is maintained, then they will turn to gray immediately, since sysview does not generate the metric when the alerted condition changes.

 

Alerts from each cics regions

 

In general, the gray metric disappears if the EMs involved are restarted.

Remember that Sysview, generate metrics by default and others that is necessary configurate  to that is generate

It is not clear to me which components changed version, so check compatibility between

Sysview - CE APM - CA APM

My customer do upgrade Sysview to 14.1 release and is necessary do upgrade CE APM to the last versión and  do upgrade EM collector, in my case to the 10.5.2 release. 

[Billy Cole]

Thank you Nando,

 

We attempted to expand all of the metric groupings under then Cross Enterprise Agent, and found no grayed out metrics.  

As you can see from the historic metric count, there are steps of about 2,000 metrics each day.  The growth occurs around midnight and we suspect that it is a batch process that generates the non-perm metrics and after a few hours, they are cleared.  When we check on the agent the following morning, we do not see any grayed out metrics.

 

We are hoping that the growth rate over a month stays below 50,000 and when the agent restarts, the historic metrics will clear.

 

But it would be nice to know what is being generated that is causing this to happen.

 

The metric counts tend to stay around the, 8,100 level while the historic metric count continues to grow.

[Legacy User]

Hi Billy,

 

Have you gone through the exercise as mentioned: Using test_regex to find a SmartStor Data Problem?

 

Configure and Manage SmartStor Data - CA Application Performance Management - 10.5 - CA Technologies Documentation 

 

Otherwise, as smartstor tools has a requirement of needing the EM to be down, I would suggest setting up a script to query against the APM API to grab a list of metrics. (Metrics List Web Service WSDL - CA Application Performance Management - 10.5 - CA Technologies Documentation )

 

To automate this process and try to catch it in the act of occurring, I would suggest setting up an alert of Number of Raw Metrics or Number of Historical Metrics for that specific agent.

 

In doing so, you can set up an action to execute a script, in this case you could use it to run a script make the API call to get a dump of the metrics when the metrics start to rise and also set up another alert/action to execute when the # of metrics drops.

 

This should give you a good depiction of whats going on during your 10:30 pm window.

 

Andrew