I have maxed out on my live metrics and historical metrics. I have a pretty good idea on how to purge the historical metrics. I have no clue on what to do with the live metrics which have hit their set value of 500,000. Secondly in light of this, I am not seeing live traces. I have purge data from my trace directories across my collectors thinking it might help but to no avail. I am scared to touch smartstor especially purging live metrics. What is your suggestion if anyone has any?What I plan doing with the historical is to shorten storage for 30 seconds frequency and 900 second frequency so during reperiodization it will age out. I am not sure about the live metrics. Please help.
Reducing the live metrics would mean either taking agents away from that EM or getting the agents to report fewer metrics.
Moving the agents would be OK if you have another EM they can go to, and you can restart the applications so that you can update the agent profile to point at a different EM.
For lowering the amount of live metrics
You could run a search on 'Metric Count' for agents on your EM and sort by agents with the highest value.
Then you have to go in one-by-one to your top offending agents and check their Metric Count pie chart and see what is the highest metric values.
Obvious ones are SQL, Sockets, JMX and PMI, but sometimes it can be things like JSPs, it will depend on the nature of the application.
Depending on which type of metrics are prevalent determines the solution to the issue, assuming that you will be able to reduce the amount of metrics without affecting dashboards, alerts, reporting etc.
the other point I should have made at the time is that eventually live metrics become historical metrics, so if you get the live metrics under control, it also helps you to avoid the historical metric clamp.
It's just unfortunately easier said than done..
After reading David's note, please let us know how to proceed with this thread
This was very helpful.