Gateway 11 - Connection Reset Errors during load test

[David Vinell]

Hi,

We are load testing Gateway 11 CR01 and the results are generally good: slightly better response times than we were getting with Gateway 10.

We have two "external" and two "internal" Gateways with the internal one configured with JDBC connections etc.  OAuth Toolkit components are on all Gateways.

Each external Gateway communicates only to it's partner internal Gateway using a hard coded DNS entry - there is no load balancer in place.

But, what we are seeing is a small number (3-5 per minute) of connection reset errors like this:

2023-11-20T10:45:18.159+0000 WARNING 1317 com.l7tech.server.policy.assertion.ServerHttpRoutingAssertion: 4042: Problem routing to https://xxxintgateway.internal:8443/oauth/tokenstore/store. Error msg: Unable to obtain HTTP response from https://xxxintgateway.internal:8443/oauth/tokenstore/store: Connection reset

The corresponding traffic log entry shows that the request has timed out at 60 seconds.

The requests that are reset are random.

We did not see this behaviour in API Gateway 10.

Corresponding logs on the internal Gateways show all request response times are around the 500ms mark.

Keepalives are enabled for most HTTP routing assertions.

Is there a port setting that might help reduce the number of connection resets?

Queued Network Requests on the Internal Gateway are a small number, maybe 10.

tcpdump shows a large number of packets like this, although it might not be relevant.

35970    2023-11-10 09:52:45.068860  <<external gateway IP>> <<internal gateway IP>> TCP    85    3240 → 8443 [PSH, ACK] Seq=1 Ack=1 Win=135 Len=31
35971    2023-11-10 09:52:45.068873    <<external gateway IP>> <<internal gateway IP>>  TCP    54    3240 → 8443 [FIN, ACK] Seq=32 Ack=1 Win=135 Len=0

(repeat several hundred times for different ports)

followed by a very large number of resets issued by the external Gateway

36637    2023-11-10 09:52:45.075450    <<external gateway IP>> <<internal gateway IP>>  TCP    54    7004 → 8443 [RST] Seq=33 Win=0 Len=0
36638    2023-11-10 09:52:45.075455   <<external gateway IP>> <<internal gateway IP>>   TCP    54    9962 → 8443 [RST] Seq=33 Win=0 Len=0
36639    2023-11-10 09:52:45.075456    <<external gateway IP>> <<internal gateway IP>>  TCP    54    3706 → 8443 [RST] Seq=33 Win=0 Len=0

Best Regards,

Dave V.

[Ben Urbanski]

Hello, David. I recommend you open a support case for the connection resets. A connection reset problem was introduced with a Tomcat update in 10.1 CR1. A configurable fix will be included with the upcoming 10.1 CR4 and 11.0 CR2 releases, but we can provide a hotfix if you can't wait. Assuming, of course, your problem is the same problem.

------------------------------
Ben Urbanski
Product Manager, API Gateway
Layer7 API Management
------------------------------

[David Vinell]

Hi Ben,

Thanks for the response - we currently plan to go live with Gateway 11 on November 28th, so I guess the hotfix will be the way to go.

I'll open a support ticket asap.

Thanks,

Dave V.