In an effort to increase the health of large batch jobs hitting our adhoc endpoint, we have made changes to queue large numbers of requests using that endpoint. This change will reduce the number of active adhoc requests in our system by a single user to 16. If more requests are are part of a larger batch, they will be placed in a queue.
Initial tests of the approach have proven successful. Of 137,896 initial jobs, 10,188 were queued. P95 measurements against an initial baseline suggested only a minimal loss of performance of 5 seconds. 100% these jobs completed successfully. Ultimately users making large adhoc requests to our systems should see less jobs failing and less jobs needing to restart for more data faster.