Apply Rate Limit Assertion

[Rekha Kantharaju]

We are using 'Apply Rate limit' assertion on our policies. 
the values provided are as per below 
Max Requests per sec - 100 ( Cluster wide and Spread limit over unchecked)
Limit each - User Or client IP ( selected)
When limit exceeded : Throttle and selected **** for 5 sec
Max Concurrent requests are set to 100.
We are experiencing black out ( service unavailable 503 messages returned)  even when we have not hit 100 requests/sec mark.
we see in ssg logs

• 2024-10-15T07:31:10.388+0100 INFO 1053 com.l7tech.external.assertions.ratelimit.server.ServerRateLimitAssertion: 6950: Rate limit exceeded on rate limiter PRESET(58b4b2cf77e8404e)ClientIp:10.251.69.61
• 2024-10-15T07:31:10.446+0100 INFO 385 com.l7tech.external.assertions.ratelimit.server.ServerRateLimitAssertion: 6955: Failing request because counter is blacked out due to previous failure

would anyone be able to advice the reason? am I missing anything?

[Joseph Fry]

Please review the documentation for "Spread Limit over X seconds":

Clear the check box to disallow bursts. In this scenario, the Gateway only accepts requests arriving no sooner than 1/limit of a second. For example, if the Max requests per second is 100, at least 1/100 second must have elapsed between requests. Requests that arrive sooner are either throttled or shaped (based on the "When limit exceeded" setting).

Disallowing burst traffic is recommended only for advanced users.  It is not recommended to disable burst traffic on a counter that will be servicing multiple concurrent requests, particularly at high rates. Doing so can lead to unintended throttling or delaying of multiple requests that arrive at exactly the same time.

Unless your certain that your requests are sequential and at least 1/100 of a second apart, then you will want to enable bursts or you can get odd behavior.  It is probably failing the the requests because within a few seconds of a hundred requests were closer than 1/100s apart and were throttled for 5 seconds each filling the max concurrent request queue.