Service Virtualization

Expand all | Collapse all

Out of memory error while running Coordinator Server

  • 1.  Out of memory error while running Coordinator Server

    Posted 10-31-2016 10:34 PM
      |   view attached

    I'm facing an issue where my coordinator server throws an out of memory error after a couple of days of continues operation. 
    The server is running on a Windows machine with 16GB of RAM and no other applications other than coordinator and Registry running. I dont have any vmoptions defined.

    I'm using DevTest 9.0.0 version.

    Logs of coordinator is attached.

    Anybody facing similar issue?

    Attachment(s)



  • 2.  Re: Out of memory error while running Coordinator Server

    Posted 10-31-2016 10:59 PM

    You need to provide more memory specifically to the coordinator process/service. Check out the link below to add memory settings. Make the change and restart the process/service.


    https://docops.ca.com/devtest-solutions/9-5/en/administering/general-administration/memory-settings




  • 3.  Re: Out of memory error while running Coordinator Server

    Posted 10-31-2016 11:06 PM

    How much memory do you recommend to allot to the coordinator service? 
    The service stage a single test with around 100 steps including web service calls and JDBC connection every 15mins.



  • 4.  Re: Out of memory error while running Coordinator Server

    Posted 10-31-2016 11:09 PM
    Considering the RAM on thebox - you could do with 2048




  • 5.  Re: Out of memory error while running Coordinator Server

    Posted 10-31-2016 11:24 PM

    Unable to allot 2048mb.  Could not reserve enough space for xxxxxxxxxxkb object heap.
    I can see around 50% of 16GB RAM free at the given point.
    I was able to allot 1024mb.

     

    Also I believe allotting more memory will only postpone the error. Issue happens after continues operation for few days under the same load. Which kind of points to inefficient GC. Probably references to objects of tests that have ended are alive 



  • 6.  Re: Out of memory error while running Coordinator Server

    Posted 11-01-2016 09:11 AM

    Your note indicates that you cannot increase Coordinator memory to 2GB because the system cannot reserve heap; yet, only 8GB of 16GB RAM are in use.  This seems strange to me.   

     

    A Simulator should be running the actual test case.  The Coordinator does just that, coordinates.  Simulators pass back information to the Coordinator.  I would look to see if the test case is sharing global information that would be retained inside the Coordinator rather than within a simulator.

     

    I am inclined to ask more about how your test case is designed.  Is this a single test case that simply runs for days (with self implemented sleep steps) or is the test case scheduled to run every 15 minutes using CVS or a Monitor?

     

    And, are you running using the Derby DB or an enterprise DBMS?  I saw one note in your logs where a query took over 9 seconds to respond.  If using Derby, perhaps the file system cannot keep up.



  • 7.  Re: Out of memory error while running Coordinator Server

    Posted 11-01-2016 09:35 AM

    If the installation is 32-bit DevTest, the maximum heap size will be restricted. I usually see a maximum of 1.3Gb being available for the -Xmx value for DevTest JVMs.

    Here's an explanation, on Stack Overflow:

    Maximum Java heap size of a 32-bit JVM on a 64-bit OS - Stack Overflow 



  • 8.  Re: Out of memory error while running Coordinator Server

    Posted 11-01-2016 10:45 AM

    Hmm..  That's quite possible. Don't remember if the installation is 32 or 64 bit.

     

    I will check on it. That should explain the reserve heap issue.  

     

    Funny part though is that I was happily running the same cvs monitor on an old LISA 7.5 coordinator /simulator setup on a weak 4gb 32bit machine without any such issues. 



  • 9.  Re: Out of memory error while running Coordinator Server

    Posted 11-03-2016 02:10 AM

    I was indeed using a 32Bit installation. Installed the 64Bit one and running the coordinator with increased allocation. let monitor for some days and see if the heap size issue resurface.



  • 10.  Re: Out of memory error while running Coordinator Server

    Posted 11-01-2016 10:35 AM

    Test case is actually a cvs monitor which is scheduled to run in regular intervals. Each run will result in around 100 web service calls and 100 jdbc connections to an external DB. 

     

    Im running a derby DB. But i doubt there is enough load in the test run that the file system can be overwhelmed. Tests are executed sequentially by looping through with reporting set to bare minimum summary mode. 



  • 11.  Re: Out of memory error while running Coordinator Server

    Posted 11-07-2016 10:01 PM

    So the Issue has resurfaced albeit after a longer interval after I increased the allocation.

    ========================================================================================
    2016-11-08 02:24:27,325Z (10:24) [Event Sink Thread Pool Thread 4] ERROR com.itko.util.Pulse - Error calling object pulse()
    java.lang.OutOfMemoryError: Java heap space
    2016-11-08 02:24:31,312Z (10:24) [ActiveMQ Transport Server Thread Handler: ssl://0.0.0.0:2011/Coordinator] ERROR System.err - Exception in thread "ActiveMQ Transport Server Thread Handler: ssl://0.0.0.0:2011/Coordinator" java.lang.OutOfMemoryError: Java heap space

    ========================================================================================


    This is definitely not due to lack of memory allocation. 2048mb for running a couple of cvs monitors is way more generous.

     

    As of now, the only workaround I can think of is to restart the devtest components every alternate days or so.



  • 12.  Re: Out of memory error while running Coordinator Server

    Posted 11-08-2016 01:10 PM

    Sanjupy06 See my comment below to diagnose the issue



  • 13.  Re: Out of memory error while running Coordinator Server

    Posted 10-31-2016 11:05 PM

    This is usual problem which is being faced , you can try by increasing memory in coordinator.vmoptions. try to make it to 3gb and allocate similar or more to registry also.

     

    restart the service and it will work.



  • 14.  Re: Out of memory error while running Coordinator Server

    Posted 10-31-2016 11:33 PM

    Thanks Chandrashekhar. 
    Increased memory as suggested. lets monitor for a few days.
    But the issue looks more due to bad memory management as it fails due to accumulation, not because of increase in load.



  • 15.  Re: Out of memory error while running Coordinator Server

    Posted 11-03-2016 05:11 PM

    When you ran into the original OutOfMemoryError, it should have generated a .hprof file. If you send that to support (I'm not sure you can attach it here, even compressed it will be large), they can verify what memory is being held onto to see if it's expected or not.



  • 16.  Re: Out of memory error while running Coordinator Server

    Posted 11-09-2016 01:50 AM

    Thanks Brian.

    Yes I have the hrpof and its ~850mb in size. Im thinking how I can send that to support



  • 17.  Re: Out of memory error while running Coordinator Server

    Posted 11-09-2016 02:41 AM

    That's 850mb compressed? If you open a new ticket with support, they will give you instructions on how to attach files to the ticket.