1) How can I determine if a metric can be removed?
This is entirely subjective to your monitoring need. In general, backends(SQL),Sockets, JMX metrics are the ones that most people would consider removing. Having said that, different customer has different view on the significance of the metric data collected.
2) What is the difference between the SmartStorTools command 'remove_metrics' and 'prune'?
remove_metrics
Removes metrics matching the given regular expression. It remove the metrics data, but does not delete the metadata.
prune
Finds agents that have no historical data in SmartStor, and removes all associated metric metadata. It remove the metrics metadata. It does not actually edit or delete from the current .metadata file, but instead create a new file with the pruned metadata. You can then rename the new one for it to be used by the EM moving forward.
For more details, please see SmartStor Tool Commands
3) How can I determine whether there is a need to use the prune command?
Prune is used to delete metrics metadata (for example, the metric name). It may be beneficial to use on an inactive agent, which does not have any metrics left associated with it, but the previously created metrics meta data were still kept in the .metadata file. It's more like a house keeping task. You don't need to specify specific agent/metrics expression for the prune command, it will scan through the entire SmartStor database and prune wherever necessary.
4) What is the difference between remove, prune and changing the SmartStor tiers?
Please refer to above answers for the difference between remove and prune. As for the data tiers, it determines when data in each tier will be aged out (removed) from the SmartStor database. SmartStor data is set to age out over time automatically according to the data tiers setting, so the data store will not get excessively large. For example, introscope.enterprisemanager.smartstor.tier1.age specifies the number of days after which data in first tier ages out of SmartStor.
For more details, please see Configure SmartStor Tiers
5) If the Smartstor database's number of historical metrics is growing, how can I determine if I should remove metrics, change the data tiers or add a collector?
You can include all these steps as part of your Smartstor maintenance exercise/practice. Removing metrics will eliminate metrics data that are no longer necessary/important to keep. Changing the data tiers will help SmartStor to automatically maintain the size as it runs.
To consider whether additional collector is required, you will need to access the overall performance sizing of the APM cluster, to determine whether the allocated capacity is still sufficient to cater the incoming load. For example, if your existing collectors are running on powerful servers with good specifications, and the resources (heap and CPUs) allocated to the EM are still sufficient to cater more load, then you can consider to increase historical metric limit on the Collector to allow more metrics, instead of adding new Collectors. Vice versa, if your Collector is already showing sign of overloading or exceeded capacity, and no further resource could be allocated on that server, then it may be time to consider adding new collector.
For more information, please see Enterprise Manager and cluster sizing
6) Can I remove a specific metric under the JMX node?
Yes, as long as you specify the regular expression that would return only that specific metric you would like to remove. In other word, it can remove whatever metrics that is returned by the regular expression you specify in the command.
7) What do the Number of Historical Metrics, Number of Metrics, Number of Metrics Handled (under Custom Metric Host (Virtual) | Custom Metric Process (Virtual) | Custom Metric Agent (Virtual) * | Enterprise Manager | Connections) metrics mean and how do they correlate to each other?
Ø Number of Historical Metrics refers to the Number of Historical Metrics metric displays the total number of metrics an Enterprise Manager is tracking across all agents.
Ø Number of Metrics displays the total metric load on the Enterprise Manager
Ø Number of Metrics Handled refers to the number of incoming Metrics processed per 15-second interval. This number varies, but could be around the same value as Number of Metrics. A value less than Number of Metrics indicates the Enterprise Manager might be overloaded and is not processing incoming data fast enough.
For more details, please see Cluster Supportability Metrics
8) What does the individual metric count value returned by test_regex in the SmartStorTools mean?
The result of the SmartStorTools.sh test_regex -metrics command is a categorization of different type of metrics metadata that are there in the SmartStor. The count reflects the number of metrics metadata (metric name) that matches the given regular expression.