Service Virtualization

Expand all | Collapse all

VSE Response time spikes

Jump to Best Answer
  • 1.  VSE Response time spikes

    Posted 02-28-2019 03:51 PM

    Hi,

     

    I have created virtual service for Performance testing purpose and deployed in PERF VSE. Virtual Service simple one having one request and response pair. Web Application is under performance test which uses Virtual Service in back end. Service will be hit by 60 TPS. Virtual Service is responding properly for all transactions.

    But when i checked "Virtual Service Response Time" Report from portal, I could see Spikes intermittently during the load test. Not able to find any error or info regarding why Spikes occurred. Below is an example.

     

     

    In the above graph, During 8hr run, few spikes occurred. 

    How to identify what caused these spikes in response times and what might be the reasons. 

    We have high configuration server, 24 core processor and 64 GM RAM.

     

    Thank You

    Rajasekhar



  • 2.  Re: VSE Response time spikes

    Posted 03-01-2019 11:45 AM

    Some random thoughts on this:

    • You show the response time graph, What does the number of transactions per second graph show aroound those period? Just your average 60 tps or does it also show spikes?
    • JVM garbage collection is something that could cause JVM to temporarily do much less work/processing and subsequently let the average response time spike. You mention your system has 64GB RAM but how much memory has been configured for the VSE process?
    • If your average transactions per second graph is steady, and the JVM garbage collection is not the cause, then you would have to know what is happening to your system resources at those particular periods of time. Is some other process kciking in and consuming CPU and/or Memory?

     

    Cheers,

    Danny 



  • 3.  Re: VSE Response time spikes

    Posted 03-01-2019 12:51 PM

    I have also experienced similar behaviour during performance tests- Do you have anything special in your virtual service like match scripts or custom coding?



  • 4.  Re: VSE Response time spikes

    Posted 03-01-2019 04:16 PM

    Thank you Jamesdunn and Danny,

     

    Added my comments. Could you please check and advice.

     

    • You show the response time graph, What does the number of transactions per second graph show aroound those period? Just your average 60 tps or does it also show spikes?

           Average transactions per second graph is steady during the spike time. 

    • JVM garbage collection is something that could cause JVM to temporarily do much less work/processing and subsequently let the average response time spike. You mention your system has 64GB RAM but how much memory has been configured for the VSE process?

            8 GB Memory is configured for VSE process. And i am not sure how to check is there any JVM garbage                 collection activity occurred during that time.

    • If your average transactions per second graph is steady, and the JVM garbage collection is not the cause, then you would have to know what is happening to your system resources at those particular periods of time. Is some other process kciking in and consuming CPU and/or Memory?

            I have consulted Admin team and got confirmation that no other process kicked in during that time. 

             I have seen this kind of behaviour very frequently during performance test runs. I could not see any details in logs also.

     

    I have also experienced similar behaviour during performance tests- Do you have anything special in your virtual service like match scripts or custom coding?

    This particular service doesn't contain any thing like match script or Custom Coding. Can you please let me know, if virtual service contains any Match Script or Custom Coding, will it affect the performance.

     

    Than you

    Rajasekhar

     



  • 5.  RE: Re: VSE Response time spikes

    Posted 01-10-2020 11:07 AM
    Hi Danny,

    i am following up on my earlier request. i am still facing the same issue with my performance runs using virtual services.
    Could you please suggest any other configuration settings to avoid spikes in teh responses.

    Thank you
    Rajasekhar


  • 6.  RE: Re: VSE Response time spikes
    Best Answer

    Posted 01-14-2020 02:54 AM
    Hi,

    You mention system has 64GB RAM and VSE process is configured with 8GB RAM. If there is free memory left on the system then I would suggest to increase the memory made available to the VSE process to 16 or 24GB. And see if that affects the occurrence of spikes or not.

    To do so, edit VirtualServiceEnvironmentService.vmoptions
    Add:
    -Xms24g
    -Xmx24g

    And restart the VSE.

    ------------------------------
    Cheers,
    Danny
    ------------------------------
    Architect
    HCL Enterprise Studio
    ------------------------------
    ------------------------------



  • 7.  Re: VSE Response time spikes

    Posted 03-04-2019 11:54 AM

    I am getting similar behaviour with some of our services and it's also causing issues:

    Response times:

     

    Transactions per Second: