Note: Clearly all this is going somewhere, I have another article in work where I am using jmeter to provide load onto API Gateway, and this tool to add delays to the transactions to cause stress on API Gateway - and what happens when limits and timeouts are reached - and how the system recovers.
I should finish the writeup of that article soon & it will complement this one - Cheers - Mark
1. Exceeding Max Capacity and testing under bad conditions
The bottom line is every system has a maximum capacity - and if you exceeded that maximum capacity then something is going to break.
Since API Gateway, as well as being it's own system, is also at heart a proxy to a backend server the API Gateway is very dependent on the performance of that backend server. If that backend server gets overloaded then that quickly cascades to the API Gateway also being overloaded.
The capacity of a system is combination of time-for-a-transaction, and the number of transactions it can run in parallel.
Capacity (TPS) = num-concurrent-transactions / time-for-a-transaction.
But what is often missed, in high load setups is the time-for-a-transaction is very dependent on how well the backend server is performing and the backend server quite often can suffers from delays, such as: from being overloaded; or undergoes a temporary delay such as a restart or database backup; or it cab fails completely.
Then since the backend is in trouble, the problems quickly cascades to the API Gateway, the API Gateway tries to respond to the slower backend server, it will often open more connections, hold more transacitons open for longer, but eventually it will trigger some failure conditions in the API Gateway itself,, perhaps hit some limit (socket or connection count), and eventually request timeouts are reached and transactions start failing.
Often these conditions occur in production, and we want to reproduce them in a non-production environment. This java-testproxy tool allows us to add delays to request, simulating slow, or non responding backend servers, and enabling us to reproduce failure conditions in a controlled manner where we can observe what happens.
Sometimes the administrative response to these issues is to increase connection pool sizes and increase (or decrease) request timeout values. Sometimes that can help give enough resources to get over a hump, but if maximum capacity is exceeded, then often that just delay (for a short time) the time until the system fails, and occasionally the increase in resource can open up a new condition, say a OutOfMemoryException now occurring,
2. Overview of how the delay tool works.
Most transactions run across sockets, such as when the user makes a request, passes it to the API Gateway and the Gateway passes it onto the backend server.
If we can direct the request via our java-TestProxy tool then we can add delays to each packets send over the socket before forwarding the request onto the backend.
That way we can simulate a backend server that is performing badly and has delays, or intermittently drops requests, or never responds.
The java-TestProxy is a java program, it can be run on the API Gateway itself, or on the backend server, or on some other intermediate server.
The content of the packets is not checked, so the tool can be used, with some tricks for hostname, to slow SSL traffic as well.
3. Basic setup & working example
As per the diagram above there are three parts to the setup: the front end User, the API Gateway, and the Backend server.
3.1 User:
My user is going to be a jmeter server, because I want to run this under load, but for a simple test I can use the browser
The URL for my Gateway is : http://odoma04-gw.example.com:8080/testTraceSimple/x1
And you can see the API Gateway forwards it onto an IIS server doing a simple dump.aspx call.
3.2 API Gateway
The API Gateway setup, I have some simple policy that forwards the request onto backend.example.com:8081
3.3 Backend
On the backend server, I have my java-TestProxy tool running in a window, it is listening on port :8081 and forwarding the requests to localhost:80.
And I have an IIS webserver is listening on :80.
3.4 Adding Delays
In the java-TestProxy we can add a 400ms delay to any request that comes in on the :8081 port
And the "status" command then shows that we have the 400ms delay.
We can change that delay at any time during the test, and run a number of other commands to alter the flow of traffic.
There are other settings to influence the traffic flow, but adding and removing the delay is the main one.
4. java-TestProxy Tool
This is the opensource java test tool that I used to add delays to the socket connections in other CA Products (most notably Single-Sign-On) we've used it a few times to simulate issues such as failover or delays in backend processes.
There are other tools that can do similar tricks, this was the one that suited my needs, and I have used on and off for 15+ years or so.
But Other tools exist as well - Linux traffic control (tc) is the main one I hear about :
Purefinity's Blog: Simulating Network Delay Using Linux
Here is a bunch, and I am sure google seach will find more :-)
testing - How do I simulate a low bandwidth, high latency environment? - Stack Overflow
4.1 Install
The java-TestProxy.zip file is attached to this article. Download and unzip it.
In the directory are :
compile.bat
run.bat
proxyinit.txt
You need to edit the .bat files to ensure they point to java sdk location, then run the compile.
Note: There are .sh versions of the files for unix systems as well
compile.bat - will compile the src code and build the TestProxy.jar file
run.bat - will run the tool, and give you a cmd prompt.
When starting up initial commands are read from proxyinit.txt, so you can set some standard setup there.
4.2 Basic Setup
From the current proxyinit.txt
# add simple proxy
createProxy 8081:localhost:80
createProxy 8082:localhost:80
The above create two new "proxy" listens, on :8081 and :8082 both of which forward to localhost:80
(I use this setup often to add delay to one backed, while leaving the other backend unchanged - can be used to test failover or load balancing )
startProxy 8081
startProxy 8082
This starts the proxy servers listening, there is also a stopProxy <port> which will stop them listening.
Then to add a 400 ms delay to any traffic via port 8081 we issue the following command, either in the proxyinit.txt file or at the cmd line prompt.
addSendDelay 8081 400
4.3 Full list of Commands
Typing "help" at the prompt gives a full list of commands.
# Commands to setup proxy ports :
addProxy <localport>:<remotehost>:<remoteport> - Create new proxy service, and start it
removeProxy <localport> - remove proxy service, and close all client connections
startProxy <localport> - Start listenin on localport for new connections (need previously setup service)
stopProxy <localport> - Stop listening for new connection, but leave existing ones open
# Commands to add delays to proxy connections :
setSendDelay <localport> <ms> - Add a delay in client requests before send to backend
setRecvDelay <localport> <ms>- Add a delay in client requests before pass back to client
# Commands to close specific client connections:
close <localport> <num> - Close a specific client connection from 'status' report
closeAll <localport> - Close all client connections for localport (listener remains active for new clients however)
# Commands to list connections :
status <localport> -print status of one port
status -print status of all ports
# Misc commands for batch scripts and init file
. <filename> - take commands from file
source <filename> -takes commands from file
sleep <time-sec> -sleep for time (for use in script file)
# comment - comment line, can also be
<command> # comment - comments can occur on cmd lines.
max <max-connections>
quit
4.4 Examples of using this tool in CA Single-Sign-On
Here are the main articles showing how used this tool in CA SSO
TechTip: Testing Solution to Agent Gateway/SPS with one bad back-end
TechTip - Configure Agent Gateway/SPS to avoid one bad back-end taking down all AG/SPS traffic.
Cheers - Mark