Layer7 API Management

 View Only
Expand all | Collapse all

Socket errors when HTTPS Performance Test

  • 1.  Socket errors when HTTPS Performance Test

    Posted Nov 21, 2017 01:21 AM

    Hi,

     

    I've done some performance tests with API Gateway 9.2.

    Some parameters were tuned well I think.

    The result of the tests was good enough. As expected, HTTP was twice as good as HTTPS,  (simply  http 4000 tps vs https 2000 tps )

    But, there is a odd thing during the tests. When we test with HTTPS, we've got some errors all the time in JMeter(actually BlazeMeter). There is no error with HTTP. 

     

    Is there specific tuning parameters for HTTPS processing?

    Or, Any idea what could be the reason for this?

     

    • [JMeter erros]
    Non HTTP response code: java.net.SocketExceptionNon HTTP response message: Connection reset
    Non HTTP response code: java.net.SocketTimeoutExceptionNon HTTP response message: connect timed out
    Non HTTP response code: java.net.SocketTimeoutExceptionNon HTTP response message: Read timed out

    The errors rate is under 0.2%. (We could ignore these errors, but have to explain)

     

     

    • [Config]

    There is no LB between JMeter and API Gateway node.

    JMeter VUser 2000

     

    io.httpCoreConcurrency  2000

    io.httpMaxConcurrency  2500

    io.httpConnectionIdleTimeout  5

    c3p0DataSource.maxPoolSize 2650

    com.l7tech.common.http.prov.apache.CommonsHttpClient.maxConnectionsPerHost=50000

    com.l7tech.common.http.prov.apache.CommonsHttpClient.maxTotalConnections=100000

    com.l7tech.server.policy.assertion.ServerHttpRoutingAssertion.statePool.enable=true

    ... (and irrelevant params)

     

    note : I've got the same errors when we double them(concurrency).

     

     

    Regards

    Becky



  • 2.  Re: Socket errors when HTTPS Performance Test

    Posted Nov 21, 2017 05:45 AM

    Hello Becky,

     

    Did you see any spike in the memory and/or CPU during the testing?

    Did you see any errors in logs?

    SSG and SSPC logs should indicate if either the services are failing or the process controller isn't responding.

     

    Cheers,

    Adarsh



  • 3.  Re: Socket errors when HTTPS Performance Test

    Posted Nov 21, 2017 07:25 PM

    I didn't see any spike in those resources and errors.



  • 4.  Re: Socket errors when HTTPS Performance Test

    Posted Nov 21, 2017 01:16 PM

    Check your JMeter CPU and Memory usage. I bet its hitting its threshold.



  • 5.  Re: Socket errors when HTTPS Performance Test

    Posted Nov 21, 2017 07:29 PM

    Yes, I checked them again. The cpu usage was under 30%. The memory was under 50% and not changed.



  • 6.  Re: Socket errors when HTTPS Performance Test

    Broadcom Employee
    Posted Nov 21, 2017 04:50 PM

    "JMeter VUser 2000"

    consider TIME_WAIT/CLOSE_WAIT, your "io.httpMaxConcurrency  2500" may not be enough.

    Try Jmeter vuser 1000, you may get  similar throughput, but no error from jmeter.



  • 7.  Re: Socket errors when HTTPS Performance Test

    Posted Nov 21, 2017 08:45 PM

    I tried double http concurrency (4000) and got the same errors. 

     

    My customer wants vuser 2000.

    It was comparison test with AWS api gateway.

    JMeters (4-node) have almost 1900 connections with all the scenarios except this case. It was only 700 connections with CA api g/w https.  And there is sokcet errors.

     

     

    Any ideas?



  • 8.  Re: Socket errors when HTTPS Performance Test

    Broadcom Employee
    Posted Nov 22, 2017 06:37 PM

    Some questions to help point you in the right direction:

    1) Form factor of the gateway: Virtual, Hardware, AWS, Azure, Software, etc

    2) How much RAM and CPU is assigned to the node? If software how much RAM assigned in the node.properties?

    3) What is the output of netstat -tanp during the load test to see what is being left?

    4) What TLS and cipher suite is being used by Blazemeter to connect to the Gateway?

    5) Have you tried just an echo test with either copy request to response assertion or template? This will eliminate the backend latency if threads are tied up.

     

    Other suggestion would be to grab a few thread dumps during the test using the instructions here (Note to increment the file name to represent each thread dump otherwise it will overwrite it): Troubleshooting Gateway application failures, performance concerns, and service outages 

    Thread dump

    A thread dump will provide the viewer with information on what a particular Java application is doing within a particular Java Virtual Machine (JVM). Please perform the following commands from the privileged shell of the API Gateway appliance:

    1. sudo su gateway
    2. ps awwx | grep Gateway.jar | grep -v grep | awk '{print $1}' | xargs -I{} /opt/SecureSpan/JDK/bin/jstack {} > /tmp/thread.tdump

     

    Sincerely,

     

    Stephen Hughes

    Director, CA Support



  • 9.  Re: Socket errors when HTTPS Performance Test

    Posted Nov 23, 2017 03:38 AM

    Hi,

     

    Unfortunately, the AWS environments provided from my customer was closed after performance tests. So we can not grab any status except BlazeMeter reports and logs.

    At first, we ignored the errors because the rate is under 0.2%. So we couldn't think to gather any information.

     

    Anyway, 

    1. AWS AMI image 9.2

    2. system : 8 core , 16G MEM
       gateway 8G memory used.

    3. no info.
        I remember a lot of TIME_WAIT. But, http scenario was the same.

    4. default used : TLS    (=TLSv1 maybe)
    5. Yes, It was just loopback test with return template assertion only. 30 bytes msg.

     

    Gateway node : cpu 50%, memory (8G) stable

    Jmeter 4 node : cpu 30%, memory stable, connection 700

    avg hits : 1980.15

    avg latency : 919.55 ms

    I think the whole resource usage was not bad. the latency was good.

     

    I just want to hear that it's normal result of gateway  or any hints of config. 

    So, you mean it is not normal result, right?

     

    I'd like to check thread dump later if I could build the similar environment.

     

    Thank you.



  • 10.  Re: Socket errors when HTTPS Performance Test
    Best Answer

    Broadcom Employee
    Posted Nov 27, 2017 10:23 PM

    Dear BeckyJang ,

    It a connection timeout on jmeter, it should be 20 sec by default. You may try specify higher connection timeout on your HTTP Request of jmeter test plan. For example, 60000 (milliseconds)

     

    On gateway side, is there any rate limit/throughput quota assertion?

     

    Regards,

    Mark



  • 11.  Re: Socket errors when HTTPS Performance Test

    Posted Nov 27, 2017 11:54 PM

    There was only one assertion - Return template( 30 bytes msg).

    No other things on the gateway. Only tuned parameters applied.

     

    This is my conclusion.

    There is no other tuning options for https.

    To avoid these errors,

    the BlazeMeter should have long enough timeout secs for HTTPS request. Because, the 1node gateway could have only less than 700 HTTPS connections.

     

    Thank you.



  • 12.  Re: Socket errors when HTTPS Performance Test

    Broadcom Employee
    Posted Nov 29, 2017 01:03 AM

    Dear BeckyJang ,

    I done some tests, and use following command to monitor  the connections,

     

    while true; do netstat -atn | grep 8443|grep <jmeter ip>|awk '{print $6}' | sort | uniq -c ; echo "----------" ; date ; echo "--------" ;  sleep 1; done;

     

    I got some thing like this,

    Wed Nov 29 16:12:30 AEDT 2017
    --------
        144 CLOSE_WAIT
        367 ESTABLISHED
        127 SYN_RECV
    ----------

     

    the timestamp is exactly the same as I got error on jmeter.

     

    This is more like a problem on jmeter, SYN_RECV means jmeter send SYN to gateway, gateway reply SYN+ACK, but jmeter never answer ACK. This behavior is similar as SYN-Food attack. I believe the gateway will terminate those kind of connections. So the jmeter get connection reset error.

     

    maybe jmeter needs a more powerful machine?



  • 13.  Re: Socket errors when HTTPS Performance Test

    Posted Nov 29, 2017 08:52 PM

    The low cpu usage of BlazeMeter still makes me confuse.

    But, I think I should check the TCP things as you did. 

     

    Thanks for your kind supports. 



  • 14.  Re: Socket errors when HTTPS Performance Test

    Posted Nov 21, 2017 08:40 PM

    Becky, can you try this How (& Why) to Get More Info on Your Load Test Errors | BlazeMeter  just in case we can do more debugging in your Jmeter



  • 15.  Re: Socket errors when HTTPS Performance Test

    Posted Nov 23, 2017 03:41 AM

    I can get whole information and jtl files in the BlazeMeter report pages. 

    Thank you.