Change timeout settings on Tomcat. App not responding back to load balancer (mod_jk)

[Suhas Kamble]

Tomcat Connector port 9580 is specified in CSA and it works fine for some days. But after few days(3 to 7 days) the web server starts throwing errors LIKE

 

 

[Wed May 20 15:42:18 2015][:] [error] ajp_service::jk_ajp_common.c (2693): (app) connecting to tomcat failed.

[Wed May 20 15:42:18 2015][:] [info] service::jk_lb_worker.c (1519): service failed, worker app is in error state

[Wed May 20 15:42:18 2015][:] [info] service::jk_lb_worker.c (1599): All tomcat instances are busy or in error state

[Wed May 20 15:42:18 2015][:] [error] service::jk_lb_worker.c (1604): All tomcat instances failed, no more workers left

[Wed May 20 15:42:18 2015][:] [info] jk_handler::mod_jk.c (2806): Service error=0 for worker=loadbalancer

[Wed May 20 15:42:18 2015]loadbalancer WEB_SERVER_HOST 42.366119

 

 

 

 

 

 

[apache@WEB_SERVER_HOST conf]$ cat mod-jk.conf

# Load mod_jk module

# Specify the filename of the mod_jk lib

LoadModule jk_module modules/mod_jk.so

 

 

# Where to find workers.properties

JkWorkersFile conf/workers.properties

 

 

# Where to put jk logs

JkLogFile logs/mod_jk.log

 

 

# Set the jk log level [debug/error/info]

JkLogLevel info

 

 

# Select the log format

JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

 

 

# JkOptions indicates to send SSK KEY SIZE

# Notes:

# 1) Changed from +ForwardURICompat.

# 2) For mod_rewrite compatibility, use +ForwardURIProxy (default since 1.2.24)

# See http://tomcat.apache.org/security-jk.html

JkOptions +ForwardKeySize +ForwardURICompatUnparsed -ForwardDirectories

 

 

# JkRequestLogFormat

JkRequestLogFormat "%w %V %T"

 

 

# Mount your applications

JkMount /* loadbalancer

# Let Apache serve the images

#JkUnMount /blahblah* loadbalancer

 

 

# You can use external file for mount points.

# It will be checked for updates each 60 seconds.

# The format of the file is: /url=worker

# /examples/*=loadbalancer

JkMountFile conf/uriworkermap.properties

 

 

# Add shared memory.

# This directive is present with 1.2.10 and

# later versions of mod_jk, and is needed for

# for load balancing to work properly

# Note: Replaced JkShmFile logs/jk.shm due to SELinux issues. Refer to

# https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=225452

JkShmFile run/jk.shm

 

 

# Add jkstatus for managing runtime data

 

 

<Location /jkstatus>

    JkMount status

    Order deny,allow

    Deny from all

    Allow from 127.0.0.1

</Location>

 

[apache@WEB_SERVER_HOST conf]$ cat workers.properties

worker.list=loadbalancer,status

 

 

worker.app.port=9580

worker.app.host= APP_SRV_HOST

worker.app.type=ajp13

worker.app.ping_mode=A

worker.app.socket_connect_timeout=10000

worker.app.connection_pool_timeout=600

worker.loadbalancer.sticky_session=True

 

 

# Load-balancing behaviour

worker.loadbalancer.type=lb

worker.loadbalancer.balance_workers=app

 

 

# Status worker for managing load balancer

worker.status.type=status

 

[apache@WEB_SERVER_HOST conf]$ cat uriworkermap.properties

 

 

# Mount the Servlet context to the ajp13 worker

/*=loadbalancer

[apache@WEB_SERVER_HOST conf]$

[Suhas Kamble]

1. When I checked netstat -na|grep 9580 it showed 200+ established connections which is weird so I added worker.app.connection_pool_timeout=600 in workers.properties on web server.

2. Same needs to be done on tomcat but I have a question here

Current tomcat settings are

  <Connector port="9580" protocol="AJP/1.3" redirectPort="8443" address="WEB_SERVER_HOST_IP" />

    which get copied from CSA everytime I restart,

If I have to setup the timeout and stuff like below, where should I set that up so that it does not get overwritten on APP restart?

 

  <Connector port="9580" address="WEB_SERVER_HOST_IP" protocol="AJP/1.3"  redirectPort="8443" emptySessionPath="true" enableLookups="false" maxThreads="1200" connectionTimeout="600000" />

[Nick Darlington]

Strictly speaking, the file I'm about to mention isn't editable in a way that is supported, so any changes would be an unsupported customization.

 

However, I think I understand your needs and so you will be looking to make changes to $NIKU_HOME/.setup/scripts/j2ee/tomcat/deploy.xml

 

This is the file that writes out the tomcat server.xml file with the (connector and other) configuration information inside it.  Specifically around this part of it (in order to get the final XML elements/attributes structure in the server.xml as needed):

 

 

Hope that helps.

[Eugene Greiff]

Be careful, the first time I played around with adding new configuration elements, I broke the app server.

Make sure you have a copy of the original and you might want to start the app via a console window with serviceappcmd-debug.exe or was it serviceappcmd-diag.exe.

This will push a lot of logging out for you to review if everything is copacetic.

V/r,

Gene

[Suhas Kamble]

Hi Nick,

 

Thanks for the hint.

 

Having a web server between Hardware Load Balancer and app server is org standard here and there are firewall as well as zone restrictions that make us have a redundant web server in there.

 

I see that you mentioned use of reverse proxy instead of mod_jk

 

Can you tell me more about that solution?

[Nick Darlington]

So I suspect you have a load balancer for the URL that then farms out requests to multiple web servers, and each of those web servers is then configured as a reverse proxy (currently using the ajp connector port) to each tomcat service.

 

In your apache web server configurations, you would use/setup mod_proxy instead of mod_jk and just have it redirect incoming traffic over HTTP.  I'm not familiar with mod_jk but it's probable that it already requires and uses mod_proxy too, and just uses the mod_jk part for the conversion of the HTTP text to the binary AJP and sets some additional properties.

 

There isn't much else I can add/say about it though (lack of implementation exposure) and if other implementations out there are fronting tomcat with apache httpd, they may be able to chime in with their experience and configuration.

[Suhas Kamble]

I switched to mod_proxy, monitoring the response now.