Virtual Service built to support load testing, goes offline in mid of test execution ERROR com.itko.lisa.test.TestEventMgr - EventDeliveryException, probably overload

[saranya_7290]

During load testing, the virtual service that is built to support it is going offline by itself in between when the performance tests are running and hitting 

 

Other Details:

Capacity : 50

Think Time: Average of 300ms

TPS: 85TPS

Type of Virtual Service: SOAP

Note: We are not using CA DevTest NFT Test cases, instead the test cases hitting this Virtual Service are from Load Runner

Detailed Error in vse.log:

2018-12-03 11:57:03,679Z (22:57) [***_V2.0 [VS_***_V2.0_Run]/94] ERROR com.itko.lisa.test.TestExec - Irrecoverable event delivery exception
2018-12-03 11:58:03,680Z (22:58) [***_V2.0 [VS_***_V2.0_Run]/94] ERROR com.itko.lisa.test.TestEventMgr - EventDeliveryException, probably overload
com.itko.lisa.test.EventDeliveryException: Timeout putting events on internal event handler queue. This usually indicates an overloaded system.
at com.itko.lisa.simulator.EventHandler.testEvent(EventHandler.java:244)
at com.itko.lisa.test.TestEventMgr.pushEvent(TestEventMgr.java:321)
at com.itko.lisa.test.TestExec.raiseEvent(TestExec.java:692)
at com.itko.lisa.test.TestExec.raiseEvent(TestExec.java:645)
at com.itko.lisa.test.TestNode.recordResponseTime(TestNode.java:1090)
at com.itko.lisa.test.TestNode.executeNode(TestNode.java:990)
at com.itko.lisa.test.TestCase.execute(TestCase.java:1297)
at com.itko.lisa.test.TestCase.execute(TestCase.java:1198)
at com.itko.lisa.test.TestCase.executeNextNode(TestCase.java:1183)
at com.itko.lisa.test.TestCase.executeTest(TestCase.java:1124)
at com.itko.lisa.coordinator.Instance.run(Instance.java:208)
2018-12-03 11:58:03,680Z (22:58) [***_V2.0 [VS_***_V2.0_Run]/94] ERROR com.itko.lisa.test.TestExec - Irrecoverable event delivery exception

 

Sometimes the below error is also logged

2018-12-03 11:23:42,594Z (22:23) [PortServer:0.0.0.0/0.0.0.0:9443] ERROR com.itko.lisa.vse.sio.PortServer - An error occurred accepting a client connection.
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at com.itko.lisa.vse.sio.NIOPortServer.handleAccept(NIOPortServer.java:90)
at com.itko.lisa.vse.sio.SelectorThread.handleOperations(SelectorThread.java:310)
at com.itko.lisa.vse.sio.SelectorThread.run(SelectorThread.java:265)
at java.lang.Thread.run(Thread.java:745)

 

This blocks the testers from running their testing, if the virtual services goes down in between the test execution.

Please suggest on how to resolve this

[Marcy Nunns]

Is this on a Linux machine?  Seeing error: java.io.IOException: Too many open files

 

This indicates you are running out of file descriptors.

 

Please check the ulimit on your machine:

 

run command:  ulimit -an

 

 

You will get output like this:

 

core file size                         (blocks, -c) 0
data seg size                        (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                                  (blocks, -f) unlimited
pending signals                     (-i) 31812
max locked memory              (kbytes, -l) 64
max memory size                  (kbytes, -m) unlimited
open files                               (-n) 1024
pipe size                               (512 bytes, -p) 8
POSIX message queues       (bytes, -q) 819200
real-time priority                      (-r) 0
stack size                               (kbytes, -s) 8192
cpu time                                  (seconds, -t) unlimited
max user processes                (-u) 31812
virtual memory                         (kbytes, -v) unlimited
file locks                                  (-x) unlimited

 

The two entries you need to be concerned about are open files and max user processes . 

 

If doing load testing you should have both of these entries set at least 30000.  In the above example, the open files is 1024, which is way too low.

 

Have your Linux admin increase the soft limit (which is at the user level) and hard limit (machine level) and see if this resolves your issue.

 

 

 

[saranya_7290]

Thank you very much Marcy. It was really useful and it worked.