Issue/Problem/Symptoms:
Secure Proxy Server intermittently reports Noodle_Interupted IOException from SPS logs.
Other times it is seen as Noodle_GenericException in Tomcat server.log.
User may see this from web browser as result of HTTP error 500.
Environment:
Policy server OS: ALL
Secure Proxy server version: R12.5, R12.51, R12.52
Cause:
There can be a few different kinds of Noodle exception message in SPS, basic cause might be different but troubleshooting methods are still the same.
Secure Proxy Server will rely on backend application server availability to serve proxy request.
When backend application server is not available, or down. By default, SPS will attempt 3 times to reconnect, if the connections fail, no "new" connection to the backend server is attempted. Then request will result to an error. SPS HttpClient log could help to narrow down the specific backend application host which was down or slow response at the time.
Resolution/Workaround:
SPS Configuration change to mitigate:
Set http_connection_stalecheck="true" from server.conf, and restart service, by default this value is false. The stalecheck will ensure all the connections in the backend pool are still active, thus when request tries a connection, it will avoid rendering a connection which is already idled out.
How to avoid unfriendly SPS exception page:
In server.conf, you can enable custom error pages in the <customerrorpages> block. Set enable to "yes" (the default is no).
Custom error pages can be defined in SPSErrorMessages.properties and WebServerErrorMessages.properties (located in $NETE_SPS_ROOT/Tomcat/properties/)
SPS Admin Guide explains the details under section “Custom Error Pages Parameters”.
#Configurations related to custom error pages
<customerrorpages>
#possible values are: "yes", "no"
#default value is "no"
enable="yes"
...
locale_country="US"
</customerrorpages>
#Custom error pages configuration end
Additional SPS Tuning:
If for some reason enabling stalecheck introduces performance delay, then additional SPS tuning may be required. In fact, any new SPS setup before going live, should go through a tuning exercise first.
Before doing that, please check proxyrules.xml file, ensure SPS does not forward request back to SPS host itself under default rule. This is common mis-configuration, that does have dire consequences for performance. This can create a looping situation which quickly chews up all the available connection/threads.
Once that is checked, move on the other tuning options:
- a)max_threads=100 (should be slightly > httpd worker threads). It is expected (in normal proxy environment) that the majority of the time, SPS will be waiting for a response from the backend server, so for best throughput it is important that the pipe "width" through each component of SPS has a size that does not force most a wait for a free "thread" or "connection".
For best throughput max_threads needs to be enough worker threads to service ALL apache threads. So this value should be slightly greater than apache worker threads. Apache worker threads are set in httpd-mpm.conf as :
MaxRequestWorkers 400 (for worker mode/unix) ThreadsPerChild 150 (for windows).
Suggest default of 410 greater than worker mode max.
ajp13.max_threads=410
- b) http_connection_pool_max_size="20" (should be > max-threads) As described previously, For best throughput there needs to be a spare backend socket for each worker thread. So connection max pool size should be slightly greater than max_thread setting. Assuming max_thread setting of 410 from prior example, then suggest set :
http_connection_pool_max_size="420"
Your back end server should be configured to handle the load of such thread pool size as well.
- c) http_connection_timeout="0" (should not be zero). Suggest make it some moderately large number rather than wait forever.
http_connection_timeout="3 minutes"
http_connection_pool_connection_timeout="1 minute"
Additional Information:
TEC1871424
http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec1871424.aspx?intcmp=searchresultclick&resultnum=1
TEC607550
http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec607550.aspx?intcmp=searchresultclick&resultnum=5
TEC1929227 Commonly Tuned Parameters in SPS
http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec1929227.aspx?intcmp=searchresultclick&resultnum=1
DISCLAIMER: All performance tuning should be bench tested and verified by client themselves in QA environment before rolling out to production, and due to system setup variations, client may experience different test outcome. If additional help is required, CA service can be contacted on a contract basis. In no event shall CA be liable for any loss or damage of any kind, including but not limited to, special, indirect, punitive incidental or consequential damage, arising out of the delivery, use, evaluation, support or operations provided even if CA has advised client of the possibility of such damage.