Service Virtualization

 View Only
Expand all | Collapse all

Virtual Service built to support load testing, goes offline in mid of test execution ERROR com.itko.lisa.test.TestEventMgr - EventDeliveryException, probably overload

Jump to Best Answer
  • 1.  Virtual Service built to support load testing, goes offline in mid of test execution ERROR com.itko.lisa.test.TestEventMgr - EventDeliveryException, probably overload

    Posted Jan 01, 2019 09:40 PM

    During load testing, the virtual service that is built to support it is going offline by itself in between when the performance tests are running and hitting 

     

    Other Details:

    Capacity : 50

    Think Time: Average of 300ms

    TPS85TPS

    Type of Virtual Service: SOAP

    Note: We are not using CA DevTest NFT Test cases, instead the test cases hitting this Virtual Service are from Load Runner

    Detailed Error in vse.log:

    2018-12-03 11:57:03,679Z (22:57) [***_V2.0 [VS_***_V2.0_Run]/94] ERROR com.itko.lisa.test.TestExec - Irrecoverable event delivery exception
    2018-12-03 11:58:03,680Z (22:58) [***_V2.0 [VS_***_V2.0_Run]/94] ERROR com.itko.lisa.test.TestEventMgr - EventDeliveryException, probably overload
    com.itko.lisa.test.EventDeliveryException: Timeout putting events on internal event handler queue. This usually indicates an overloaded system.
    at com.itko.lisa.simulator.EventHandler.testEvent(EventHandler.java:244)
    at com.itko.lisa.test.TestEventMgr.pushEvent(TestEventMgr.java:321)
    at com.itko.lisa.test.TestExec.raiseEvent(TestExec.java:692)
    at com.itko.lisa.test.TestExec.raiseEvent(TestExec.java:645)
    at com.itko.lisa.test.TestNode.recordResponseTime(TestNode.java:1090)
    at com.itko.lisa.test.TestNode.executeNode(TestNode.java:990)
    at com.itko.lisa.test.TestCase.execute(TestCase.java:1297)
    at com.itko.lisa.test.TestCase.execute(TestCase.java:1198)
    at com.itko.lisa.test.TestCase.executeNextNode(TestCase.java:1183)
    at com.itko.lisa.test.TestCase.executeTest(TestCase.java:1124)
    at com.itko.lisa.coordinator.Instance.run(Instance.java:208)
    2018-12-03 11:58:03,680Z (22:58) [***_V2.0 [VS_***_V2.0_Run]/94] ERROR com.itko.lisa.test.TestExec - Irrecoverable event delivery exception

     

    Sometimes the below error is also logged

    2018-12-03 11:23:42,594Z (22:23) [PortServer:0.0.0.0/0.0.0.0:9443] ERROR com.itko.lisa.vse.sio.PortServer - An error occurred accepting a client connection.
    java.io.IOException: Too many open files
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
    at com.itko.lisa.vse.sio.NIOPortServer.handleAccept(NIOPortServer.java:90)
    at com.itko.lisa.vse.sio.SelectorThread.handleOperations(SelectorThread.java:310)
    at com.itko.lisa.vse.sio.SelectorThread.run(SelectorThread.java:265)
    at java.lang.Thread.run(Thread.java:745)

     

    This blocks the testers from running their testing, if the virtual services goes down in between the test execution.

    Please suggest on how to resolve this



  • 2.  Re: Virtual Service built to support load testing, goes offline in mid of test execution ERROR com.itko.lisa.test.TestEventMgr - EventDeliveryException, probably overload
    Best Answer

    Broadcom Employee
    Posted Jan 02, 2019 12:13 PM

    Is this on a Linux machine?  Seeing error: java.io.IOException: Too many open files

     

    This indicates you are running out of file descriptors.

     

    Please check the ulimit on your machine:

     

    run command:  ulimit -an

     

     

    You will get output like this:

     

    core file size                         (blocks, -c) 0
    data seg size                        (kbytes, -d) unlimited
    scheduling priority                 (-e) 0
    file size                                  (blocks, -f) unlimited
    pending signals                     (-i) 31812
    max locked memory              (kbytes, -l) 64
    max memory size                  (kbytes, -m) unlimited
    open files                               (-n) 1024
    pipe size                               (512 bytes, -p) 8
    POSIX message queues       (bytes, -q) 819200
    real-time priority                      (-r) 0
    stack size                               (kbytes, -s) 8192
    cpu time                                  (seconds, -t) unlimited
    max user processes                (-u) 31812
    virtual memory                         (kbytes, -v) unlimited
    file locks                                  (-x) unlimited

     

    The two entries you need to be concerned about are open files and max user processes . 

     

    If doing load testing you should have both of these entries set at least 30000.  In the above example, the open files is 1024, which is way too low.

     

    Have your Linux admin increase the soft limit (which is at the user level) and hard limit (machine level) and see if this resolves your issue.

     

     

     



  • 3.  Re: Virtual Service built to support load testing, goes offline in mid of test execution ERROR com.itko.lisa.test.TestEventMgr - EventDeliveryException, probably overload

    Posted Jan 02, 2019 11:53 PM

    Thank you very much Marcy. It was really useful and it worked.