Workstation 9.5.2 Memory Leak ?

[Viruvaibhav_apm]

Hi All,

 

Thought to share one of the issue\observation I felt after we upgraded Wily from 9.1.6 version to 9.5.2

 

We observed that after the upgrade, the heap\memory required for workstation to open and access has increased drastically.
In the initial phases, even console was not opening. But when we assigned the Heap around 1gigs which opened the console but with very high heap usage.

 

Below is the snapshot of the Task Manager, where you can see the memory utilization reaching 1gigs.

 

Adding to this problem statement, I tried to profile the client\workstation using JProfiler and found that there were mainly two objects which were greater in number as well as utilizing considerable Heap.

1. java.awt.EventQueue
2. com.wily.introscope.soa.dependencymap.ui.DependencyMapViewerPanel

 

Attaching the JProfiler snapshot for the heap analysis. I checked for the thread details but nothing erroneous I could see.

 

Since I am not much aware about these classes and their functionalities, could anyone help me to know if this is the potential cause for high heap utilization ?
Even if I don't navigate much into the Investigator, heap utilization reaches Max very easily making console unresponsive after sometime.

 

Appreciate your help and suggestions.

 

Regards,

Vaibhav

Attachment(s)

Heap_walker_Biggest_objects_2.html.zip   2 KB 1 version

[Anon Anon]

Vaibhav,

This definitely requires more investigation via support issue however increase of memory usage simply by upgrading(assume no agent\EM changes) seems little suspicious.

SOA Dependency Map does get loaded by default and can use quite a bit of memory depending on amount of data contained or if corrupted can even cause OOM on Workstation side or hangs.

 

Symptoms you describe seem similar so for initial triage try this:

-On each EM\Collector go to <EM_Home>\data\dependencymap folder and check size of files in there

-Shutdown EM\Collector, take a backup of these files and remove them from the folder

-Restart environment and check if WS memory usage normalizes

-If yes then open a support issue and provide us those dependencymap folders for further investigation

 

SOA dependency map will be recreated over period of time as agents notice webservice code getting invoked.

Hope it helps.

 

Regards,

Kulbir.

[Viruvaibhav_apm]

Hi Kulbir,

 

thanks for your quick response.

Let me work on this and get back to you.

 

Regards,

Vaibhav

[nkarthik]

Hello Vaibhav,

 

We are using 9.5.2 on our two production environments and we don't see any memory leak issues. Like kulbir said, you can check on SOA metrics. If SOA metrics are not required on your environment, you can even disable it by adding below property in properties file.

 

com.wily.introscope.soa.deviation.enable=false

 

On other hand, You can even enable the workstation logs using below steps to see what's happening when you logged in.

 

Open IntroscopeWorkstation.properties file under C:\Program Files (x86)\CA APM\Introscope9.6.0.0\config - This location might differ on your desktop

Increase the log level to VERBOSE and look for workstation logs under /logs folder

 

Thanks,

Karthik

[Viruvaibhav_apm]

Hi Karthik\All,

 

I enabled the verbose logging and found the below thing been loggedIn very frequently.

I checked with SuperDoamain if any such kind Webservice of my application is available or not, but it is not from my Applications.

 

Not sure why and from where does this is picking it up. Also same has been observed in the JProfiler logs I had attached initially.

I checked in ws-plugin folder at my workstation where I could see following jar file with similar naming - "com.wily.introscope.soa.dependencymetrics.typeviewer_9.5.0".
Not sure if this is the issue since my workstation version is 9.5.2 where as here I could see'9.5.0'

 

 

 

7/14/15 05:26:11 PM SGT [DEBUG] [Workstation.DependencyMapTypeViewer.TypeViewer] acceptVertex: Cannot instantiate AgentMetricPrefix ("WebServices|Client||getOptionRates" is not a legal Metric name.)

7/14/15 05:26:11 PM SGT [DEBUG] [Workstation.DependencyMapTypeViewer.TypeViewer] acceptVertex: Cannot instantiate AgentMetricPrefix ("WebServices|Client||getSpotRate" is not a legal Metric name.)

7/14/15 05:26:11 PM SGT [DEBUG] [Workstation.DependencyMapTypeViewer.TypeViewer] acceptVertex: Cannot instantiate AgentMetricPrefix ("WebServices|Client||getOptionBuyBackRates" is not a legal Metric name.)

7/14/15 05:26:11 PM SGT [DEBUG] [Workstation.DependencyMapTypeViewer.TypeViewer] acceptVertex: Cannot instantiate AgentMetricPrefix ("WebServices|Client||getSBOptionRates" is not a legal Metric name.)

 

Please suggest any directions.

 

Note:SOA extension\metric is required.

 

Regards,

Vaibhav

[Guenter DI Grossberger]

Looks like an agent is generating a metric name that is missing the namespace: WebServices|Client|<namespace>|getSBOptionRates. And that metric is also used for naming the web service vertex on the app map.

 

Try to track down that agent. It should have "illegal metric name: messages in its log.

 

Ciao,

Guenter

[Viruvaibhav_apm]

Thanks Guenter.

 

But could you please suggest how to track down that agent ?
I tried searching "getSBOptionRates" keyword in the SuperDomain but I was not able to find any.

 

Since the number of agents and domains are many, plz suggest an optimal way to track it down. Appreciate your help.

 

Regards,

Vaibhav

[RICHARD CAHILL]

Try searching your EM IntroscopeEnterpriseManager.log logs for an exception on the last startup.  If there is an error in one of the management module jar files you should see an exception and the name of the offending .jar file.   

[nkarthik]

Hello Vaibhav,

 

Please try accessing APM through webstart. We should be able to isolate whether the issue is on workstation or on EM side if webstart workstation is working properly.

 

Also did you check your EM logs for any noticable errors?

 

Thanks,

Karthik

[Hallett German]

Dear Vaibhav:

   Please let us know if you need further assistance on this thread or are planning to open a support case to investigate?

 

Thanks

Hal German

CA APM Support