Tomcat Connector port 9580 is specified in CSA and it works fine for some days. But after few days(3 to 7 days) the web server starts throwing errors LIKE
[Wed May 20 15:42:18 2015][:] [error] ajp_service::jk_ajp_common.c (2693): (app) connecting to tomcat failed.
[Wed May 20 15:42:18 2015][:] [info] service::jk_lb_worker.c (1519): service failed, worker app is in error state
[Wed May 20 15:42:18 2015][:] [info] service::jk_lb_worker.c (1599): All tomcat instances are busy or in error state
[Wed May 20 15:42:18 2015][:] [error] service::jk_lb_worker.c (1604): All tomcat instances failed, no more workers left
[Wed May 20 15:42:18 2015][:] [info] jk_handler::mod_jk.c (2806): Service error=0 for worker=loadbalancer
[Wed May 20 15:42:18 2015]loadbalancer WEB_SERVER_HOST 42.366119
[apache@WEB_SERVER_HOST conf]$ cat mod-jk.conf
# Load mod_jk module
# Specify the filename of the mod_jk lib
LoadModule jk_module modules/mod_jk.so
# Where to find workers.properties
# Where to put jk logs
# Set the jk log level [debug/error/info]
# Select the log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"
# JkOptions indicates to send SSK KEY SIZE
# 1) Changed from +ForwardURICompat.
# 2) For mod_rewrite compatibility, use +ForwardURIProxy (default since 1.2.24)
# See http://tomcat.apache.org/security-jk.html
JkOptions +ForwardKeySize +ForwardURICompatUnparsed -ForwardDirectories
JkRequestLogFormat "%w %V %T"
# Mount your applications
JkMount /* loadbalancer
# Let Apache serve the images
#JkUnMount /blahblah* loadbalancer
# You can use external file for mount points.
# It will be checked for updates each 60 seconds.
# The format of the file is: /url=worker
# Add shared memory.
# This directive is present with 1.2.10 and
# later versions of mod_jk, and is needed for
# for load balancing to work properly
# Note: Replaced JkShmFile logs/jk.shm due to SELinux issues. Refer to
# Add jkstatus for managing runtime data
Deny from all
Allow from 127.0.0.1
[apache@WEB_SERVER_HOST conf]$ cat workers.properties
# Load-balancing behaviour
# Status worker for managing load balancer
[apache@WEB_SERVER_HOST conf]$ cat uriworkermap.properties
# Mount the Servlet context to the ajp13 worker
1. When I checked netstat -na|grep 9580 it showed 200+ established connections which is weird so I added worker.app.connection_pool_timeout=600 in workers.properties on web server.
2. Same needs to be done on tomcat but I have a question here
Current tomcat settings are
<Connector port="9580" protocol="AJP/1.3" redirectPort="8443" address="WEB_SERVER_HOST_IP" />
which get copied from CSA everytime I restart,
If I have to setup the timeout and stuff like below, where should I set that up so that it does not get overwritten on APP restart?
<Connector port="9580" address="WEB_SERVER_HOST_IP" protocol="AJP/1.3" redirectPort="8443" emptySessionPath="true" enableLookups="false" maxThreads="1200" connectionTimeout="600000" />
Strictly speaking, the file I'm about to mention isn't editable in a way that is supported, so any changes would be an unsupported customization.
However, I think I understand your needs and so you will be looking to make changes to $NIKU_HOME/.setup/scripts/j2ee/tomcat/deploy.xml
This is the file that writes out the tomcat server.xml file with the (connector and other) configuration information inside it. Specifically around this part of it (in order to get the final XML elements/attributes structure in the server.xml as needed):
Hope that helps.
Be careful, the first time I played around with adding new configuration elements, I broke the app server.
Make sure you have a copy of the original and you might want to start the app via a console window with serviceappcmd-debug.exe or was it serviceappcmd-diag.exe.
This will push a lot of logging out for you to review if everything is copacetic.
Thanks for the hint.
Having a web server between Hardware Load Balancer and app server is org standard here and there are firewall as well as zone restrictions that make us have a redundant web server in there.
I see that you mentioned use of reverse proxy instead of mod_jk
Can you tell me more about that solution?
So I suspect you have a load balancer for the URL that then farms out requests to multiple web servers, and each of those web servers is then configured as a reverse proxy (currently using the ajp connector port) to each tomcat service.
In your apache web server configurations, you would use/setup mod_proxy instead of mod_jk and just have it redirect incoming traffic over HTTP. I'm not familiar with mod_jk but it's probable that it already requires and uses mod_proxy too, and just uses the mod_jk part for the conversion of the HTTP text to the binary AJP and sets some additional properties.
There isn't much else I can add/say about it though (lack of implementation exposure) and if other implementations out there are fronting tomcat with apache httpd, they may be able to chime in with their experience and configuration.
I switched to mod_proxy, monitoring the response now.