I am using CA APM 10.5.1. Currently we have a problem with team center that showing no data.
It happened intermittently, and in the IntroscopeEnterpriseManager.log I see some exceptions like this:
8/18/17 11:55:50.654 AM ICT [WARN] [Central Fast Correlation incubator queue consumer thread] [Manager.AppMap] Exception when processing Central Fast Correlation incubatorjava.lang.NullPointerException8/18/17 11:55:54.399 AM ICT [WARN] [Central Slow Correlation incubator queue consumer thread] [Manager.AppMap] Exception when processing Central Slow Correlation incubatorjava.lang.NullPointerException
And there are many message saying vertex association not found like this:
8/20/17 02:48:18.978 AM ICT [WARN] [AppMapAsyncExecutor-7] [Manager.AppMap] Vertex association not found: 128588/20/17 02:48:20.293 AM ICT [WARN] [AppMapAsyncExecutor-8] [Manager.AppMap] Vertex association not found: 14066
I found the KB about that message: What does the "Vertex association not found" [WARN] message in the IntroscopeEnterpriseManager.log mean? But we didn't change the data source for team center. I attached the enterprisemanager log file. Any suggestions?
I noticed some other exceptions in the log, "[Manager.AppMap] Error happened during controller invocation!com.wily.introscope.appmap.rest.exceptions.RestApiException: ACCESS_FORBIDDEN", but there is only 1 on August 18 (previous August 1) and that was for the "guest" user who might have no access.
As the problem is intermittent the root cause may be related to the MOM/cluster performance.Have you noticed high heap usage from the MOM java process? (the perflog.txt might also show symptoms of that)Can you enable DEBUG on the MOM to see if the EM log shows more useful information?
The log starts around mid-May but does not have the last restart timestamp so I cannot see exact 10.5.1.x version/Build. Do you know if this is 10.5.1 GA (10.5.1.6) or 1HF1 (10.5.1.8) or later HotFix version?Ultimately this maybe a scenario that is best handled by a support case where more detailed research can be done.
Follow up: Yan has raised a support case (00822834) for these symptoms as it is impacting Production Environment
** RESOLUTION DETAILS:
On Team Center UI also seeing: Error reading data from polling URL: ../apm/appmap/private/permission: Retrying in 60000 ms. Attempt 1
Stopping WebView, clearing its osgi cache (WEBVIEW_HOME\product\webview\configuration EXCEPT config.ini) & restart did not help
Only after the MOM itself was restarted was the problem resolved (last restart was mid May). At this stage root cause not known & will post here if determined **
Let me give the detailed steps to solve this issue.
1 - Stop the webview follow by EM2 - Go to the \product\webview\configuration directory 3 - Go to the \product\enterprisemanager\configuration directory Delete all directories only : ( do NOT delete the config.ini )4 - Go to the \work directory and delete its contents.5 - Clear the appropriate browser cache. 6 - Start EM, follow by webview
And yes, I still need to know what is the root cause of this issue.
Thanks Yan for posting the details
Please respond to this thread what the case resolution is. This will help others that may have the same issue.
Anyone can help how to perform step 1 and step 6 ?
Are you having a similar problem and you want to try the above steps?
Do you mean how to perform just the stop and start steps?
What OS platform are your EM and WebView running under?
Yes i have similar problem. i want to try above step. Please guide how to perform stop and start step?Stop webview follow by em and Start em follow by webview.OS: Window Server 2012
APM Version: 10.5.2.15
The start stop procedure for the EM is documented here
Start or Stop the Enterprise Manager - CA Application Performance Management - 10.7 - CA Technologies Documentation
similarly on Linux there is a WVCtrl.sh script and windows service if that is ho wit was configured during installation.