Out of memory error while running Coordinator Server

[Sanjupy06]

I'm facing an issue where my coordinator server throws an out of memory error after a couple of days of continues operation. 
The server is running on a Windows machine with 16GB of RAM and no other applications other than coordinator and Registry running. I dont have any vmoptions defined.

I'm using DevTest 9.0.0 version.

Logs of coordinator is attached.

Anybody facing similar issue?

Attachment(s)

Logs_CoordinatorJavaHeapError.txt.zip   6 KB 1 version

[Koustubh Warty]

You need to provide more memory specifically to the coordinator process/service. Check out the link below to add memory settings. Make the change and restart the process/service.

https://docops.ca.com/devtest-solutions/9-5/en/administering/general-administration/memory-settings

[Sanjupy06]

How much memory do you recommend to allot to the coordinator service? 
The service stage a single test with around 100 steps including web service calls and JDBC connection every 15mins.

[Koustubh Warty]

Considering the RAM on thebox - you could do with 2048

[Sanjupy06]

Unable to allot 2048mb.  Could not reserve enough space for xxxxxxxxxxkb object heap.
I can see around 50% of 16GB RAM free at the given point.
I was able to allot 1024mb.

 

Also I believe allotting more memory will only postpone the error. Issue happens after continues operation for few days under the same load. Which kind of points to inefficient GC. Probably references to objects of tests that have ended are alive 

[Legacy User]

Your note indicates that you cannot increase Coordinator memory to 2GB because the system cannot reserve heap; yet, only 8GB of 16GB RAM are in use.  This seems strange to me.   

 

A Simulator should be running the actual test case.  The Coordinator does just that, coordinates.  Simulators pass back information to the Coordinator.  I would look to see if the test case is sharing global information that would be retained inside the Coordinator rather than within a simulator.

 

I am inclined to ask more about how your test case is designed.  Is this a single test case that simply runs for days (with self implemented sleep steps) or is the test case scheduled to run every 15 minutes using CVS or a Monitor?

 

And, are you running using the Derby DB or an enterprise DBMS?  I saw one note in your logs where a query took over 9 seconds to respond.  If using Derby, perhaps the file system cannot keep up.

[Rick.Brown]

If the installation is 32-bit DevTest, the maximum heap size will be restricted. I usually see a maximum of 1.3Gb being available for the -Xmx value for DevTest JVMs.

Here's an explanation, on Stack Overflow:

Maximum Java heap size of a 32-bit JVM on a 64-bit OS - Stack Overflow 

[Sanjupy06]

Hmm..  That's quite possible. Don't remember if the installation is 32 or 64 bit.

 

I will check on it. That should explain the reserve heap issue.  

 

Funny part though is that I was happily running the same cvs monitor on an old LISA 7.5 coordinator /simulator setup on a weak 4gb 32bit machine without any such issues. 

[Sanjupy06]

I was indeed using a 32Bit installation. Installed the 64Bit one and running the coordinator with increased allocation. let monitor for some days and see if the heap size issue resurface.

[Sanjupy06]

Test case is actually a cvs monitor which is scheduled to run in regular intervals. Each run will result in around 100 web service calls and 100 jdbc connections to an external DB. 

 

Im running a derby DB. But i doubt there is enough load in the test run that the file system can be overwhelmed. Tests are executed sequentially by looping through with reporting set to bare minimum summary mode. 

[Sanjupy06]

So the Issue has resurfaced albeit after a longer interval after I increased the allocation.

========================================================================================
2016-11-08 02:24:27,325Z (10:24) [Event Sink Thread Pool Thread 4] ERROR com.itko.util.Pulse - Error calling object pulse()
java.lang.OutOfMemoryError: Java heap space
2016-11-08 02:24:31,312Z (10:24) [ActiveMQ Transport Server Thread Handler: ssl://0.0.0.0:2011/Coordinator] ERROR System.err - Exception in thread "ActiveMQ Transport Server Thread Handler: ssl://0.0.0.0:2011/Coordinator" java.lang.OutOfMemoryError: Java heap space

========================================================================================

This is definitely not due to lack of memory allocation. 2048mb for running a couple of cvs monitors is way more generous.

 

As of now, the only workaround I can think of is to restart the devtest components every alternate days or so.

[mcdbr09]

Sanjupy06 See my comment below to diagnose the issue

[Anon Anon]

This is usual problem which is being faced , you can try by increasing memory in coordinator.vmoptions. try to make it to 3gb and allocate similar or more to registry also.

 

restart the service and it will work.

[Sanjupy06]

Thanks Chandrashekhar. 
Increased memory as suggested. lets monitor for a few days.
But the issue looks more due to bad memory management as it fails due to accumulation, not because of increase in load.

[mcdbr09]

When you ran into the original OutOfMemoryError, it should have generated a .hprof file. If you send that to support (I'm not sure you can attach it here, even compressed it will be large), they can verify what memory is being held onto to see if it's expected or not.

[Sanjupy06]

Thanks Brian.

Yes I have the hrpof and its ~850mb in size. Im thinking how I can send that to support

[mcdbr09]

That's 850mb compressed? If you open a new ticket with support, they will give you instructions on how to attach files to the ticket.