We are load testing Gateway 11 CR01 and the results are generally good: slightly better response times than we were getting with Gateway 10.
We have two "external" and two "internal" Gateways with the internal one configured with JDBC connections etc. OAuth Toolkit components are on all Gateways.
Each external Gateway communicates only to it's partner internal Gateway using a hard coded DNS entry - there is no load balancer in place.
But, what we are seeing is a small number (3-5 per minute) of connection reset errors like this:
2023-11-20T10:45:18.159+0000 WARNING 1317 com.l7tech.server.policy.assertion.ServerHttpRoutingAssertion: 4042: Problem routing to https://xxxintgateway.internal:8443/oauth/tokenstore/store. Error msg: Unable to obtain HTTP response from https://xxxintgateway.internal:8443/oauth/tokenstore/store: Connection reset
The corresponding traffic log entry shows that the request has timed out at 60 seconds.
The requests that are reset are random.
We did not see this behaviour in API Gateway 10.
Corresponding logs on the internal Gateways show all request response times are around the 500ms mark.
Keepalives are enabled for most HTTP routing assertions.
Is there a port setting that might help reduce the number of connection resets?
Queued Network Requests on the Internal Gateway are a small number, maybe 10.
tcpdump shows a large number of packets like this, although it might not be relevant.
35970 2023-11-10 09:52:45.068860 <<external gateway IP>> <<internal gateway IP>> TCP 85 3240 → 8443 [PSH, ACK] Seq=1 Ack=1 Win=135 Len=3135971 2023-11-10 09:52:45.068873 <<external gateway IP>> <<internal gateway IP>> TCP 54 3240 → 8443 [FIN, ACK] Seq=32 Ack=1 Win=135 Len=0
(repeat several hundred times for different ports)
followed by a very large number of resets issued by the external Gateway
36637 2023-11-10 09:52:45.075450 <<external gateway IP>> <<internal gateway IP>> TCP 54 7004 → 8443 [RST] Seq=33 Win=0 Len=036638 2023-11-10 09:52:45.075455 <<external gateway IP>> <<internal gateway IP>> TCP 54 9962 → 8443 [RST] Seq=33 Win=0 Len=036639 2023-11-10 09:52:45.075456 <<external gateway IP>> <<internal gateway IP>> TCP 54 3706 → 8443 [RST] Seq=33 Win=0 Len=0
Hello, David. I recommend you open a support case for the connection resets. A connection reset problem was introduced with a Tomcat update in 10.1 CR1. A configurable fix will be included with the upcoming 10.1 CR4 and 11.0 CR2 releases, but we can provide a hotfix if you can't wait. Assuming, of course, your problem is the same problem.
Thanks for the response - we currently plan to go live with Gateway 11 on November 28th, so I guess the hotfix will be the way to go.
I'll open a support ticket asap.