After upgrading from 9.5.1 to 9.7.1 I am getting inconsistent data points from investigator to management module. Secondly the dashboard data does not populate. It takes a while to populate but does not take into account when I opened it (1 minute) lagging. Has anyone seen this before?
This can be a few things.
1. Check if you're getting actual data gaps. This could be due to a performance issue.
2. Check if the metric in the metric group isn't getting pushed out. The metric group will show the top N values of the metric regex
3. For the dashboard not populating fast enough, sounds like you have quite a bit of metrics in your smartstor.
Are you seeing this on Workstation (thin or thick client), or Webview or both?
Your best bet is to open an issue and upload the logs directory of the MOM to the issue.
Indeed, the problem seems to related to a performance issue in the cluster.
If you can, have a look at the the EM(s)s logs, search for the following kewords: capacity, too many, clamped, reached, combining
Othewise open a support case and attach the entire logs directory from all EMs (MOM and collectors)
Please note that we have started documenting all common errors as part of our documentation, you can refer to the below links:
And: CA Tuesday Tip: An update on the APM Troubleshooting guide
I hope this helps,