Devtest prerequisites to support 5,000 concurrent users for load testing

[harika.gonela1]

Hello Team,

There is a requirement to support up to 5,000 concurrent users for load testing. Below are the configuration details of the Devtest server.

Could you please let us know ,
1. what are the prerequisites to be taken care for the load testing and
2. any performance tuning is required to support this load. 

	Server Configuration
Performance VSE	Yes
CPU	4 CPU
Memory	63 GB Ram
Architecture	x86_64
DevTest  version:	10.2.4
Total No. Of services running in VSE	Total: 686, Running : 549, Offline: 137
Components Running	VSE : 1
OS	RedHat Linux version 2.6.32-754.15.3.el6.x86_64

Regards,
Harika Gonela.

[Danny Saro]

Hi,

 

I assume we are talking about Service Virtualization for Performance (and not about Application Test load testing).

 

5000 concurrent users are hitting the front-end of your system under test, all depends on the behavior programmed into the automated tests, if they mimic real-time behavior of a life end-user then maybe there is a UI action every 5 or 10 seconds, so you'll end up with 1000 to 500 tps hitting the front-end. Back-end transactions per second for this front-end 1000 tps always tend to be a fraction of front-end traffic. (this is because only the business transactions make it to the back-end; UI behavior, navigation, etc... are handled by the SUT internally; also due to caching of data in your SUT the back-end tps might be lower, etc...)

 

Anyways, just a long explanation in order to come to the conclusion that support up to 5000 users at the front-end doesn't really say anything about the load on the Performance VSE.

The throughput of the Performance VSE will only be limited by the resources of the host that runs it. Therefor best practice is to run a performance vse on a dedicated server with no other software running (no other DevTest components or anything else).

 

Only deploy and run the virtual services that are used by the testcases in your load-testing suite. You mention that you have 549 virtual services running, are all of them needed for running the load-test?

Only use standard virtual service models, run them in "efficient" mode, so no customized workflows that read responses from databases or that manipulate files, etc... because then you can have latency originating outside of DevTest (and outside its control)

There should be a list somewhere with some further tweaks that one can apply to the VSE in order to sustain higher throughput (tps):

• Disable VSE logging
• ...

(sorry, that's the only one that comes to mind right now, there are some others)

 

Cheers,

Danny

 

::DISCLAIMER::

---

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.

---

[harika.gonela1]

Hi Danny,

Thanks you for the detailed note.

Below are the answers for your queries. 

• Yes, its Service Virtualization for Performance. It is a dedicated Devtest server with no other software's running. 
• Not all 549 of them are required to run the load tests. 
• Services under test are in Efficient mode. 

As suggested we will be disabling the logs to ERROR mode and setting the concurrent capacity.

Any other suggestions on performance tuning which is required from Devtest Server end?

Regards,
Harika Gonela

[Danny Saro]

Hi,

 

I would expect that you already did or would do below:

• only run the virtual services that are needed for the load test.
• Give the JVM the maximum amount of memory, eg. 48GB if host memory is 64GB and nothing else is running
• Set the JVM's minimum amount of memory equal to the maximum amount of memory so the JVM will not spent resources extending it heap size during the load testing

 

My suggestion would be to be somewhat proactive, and benchmark your VSE for Performance environment specifically for your upcoming load test: using Application Test, create some testcases that directly call your virtual services in the VSE for Performance environment. Put those testcases in a Suite and repeatedly run them as a load test against your Performance VSE each time using more concurrent vUsers. It will show you at what point your Performance VSE setup is breaking and you will know how much TPS you can support. You can go back to the team and decide if the environment is ok for the upcoming test, or if you need additional tweaking.

It's a best practice, should you lack the time to do it for this load test, then for any future load test try to put "Benchmark VSE for Performance" in your planning.

 

Cheers,

Danny

 

::DISCLAIMER::

---

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.

---

[harika.gonela1]

Thank you Danny for the detailed note.

[Karl Miller]

Additional Performance VSE property settings can be found at Sample Performance Benchmarks for Basic Service Virtualization