Clarity

Expand all | Collapse all

Change timeout settings on Tomcat. App not responding back to load balancer (mod_jk)

Jump to Best Answer
  • 1.  Change timeout settings on Tomcat. App not responding back to load balancer (mod_jk)

    Posted 05-21-2015 11:14 AM

    Tomcat Connector port 9580 is specified in CSA and it works fine for some days. But after few days(3 to 7 days) the web server starts throwing errors LIKE

     

     

    [Wed May 20 15:42:18 2015][:] [error] ajp_service::jk_ajp_common.c (2693): (app) connecting to tomcat failed.

    [Wed May 20 15:42:18 2015][:] [info] service::jk_lb_worker.c (1519): service failed, worker app is in error state

    [Wed May 20 15:42:18 2015][:] [info] service::jk_lb_worker.c (1599): All tomcat instances are busy or in error state

    [Wed May 20 15:42:18 2015][:] [error] service::jk_lb_worker.c (1604): All tomcat instances failed, no more workers left

    [Wed May 20 15:42:18 2015][:] [info] jk_handler::mod_jk.c (2806): Service error=0 for worker=loadbalancer

    [Wed May 20 15:42:18 2015]loadbalancer WEB_SERVER_HOST 42.366119

     

     

     

     

     

     

    [apache@WEB_SERVER_HOST conf]$ cat mod-jk.conf

    # Load mod_jk module

    # Specify the filename of the mod_jk lib

    LoadModule jk_module modules/mod_jk.so

     

     

    # Where to find workers.properties

    JkWorkersFile conf/workers.properties

     

     

    # Where to put jk logs

    JkLogFile logs/mod_jk.log

     

     

    # Set the jk log level [debug/error/info]

    JkLogLevel info

     

     

    # Select the log format

    JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

     

     

    # JkOptions indicates to send SSK KEY SIZE

    # Notes:

    # 1) Changed from +ForwardURICompat.

    # 2) For mod_rewrite compatibility, use +ForwardURIProxy (default since 1.2.24)

    # See http://tomcat.apache.org/security-jk.html

    JkOptions +ForwardKeySize +ForwardURICompatUnparsed -ForwardDirectories

     

     

    # JkRequestLogFormat

    JkRequestLogFormat "%w %V %T"

     

     

    # Mount your applications

    JkMount /* loadbalancer

    # Let Apache serve the images

    #JkUnMount /blahblah* loadbalancer

     

     

    # You can use external file for mount points.

    # It will be checked for updates each 60 seconds.

    # The format of the file is: /url=worker

    # /examples/*=loadbalancer

    JkMountFile conf/uriworkermap.properties

     

     

    # Add shared memory.

    # This directive is present with 1.2.10 and

    # later versions of mod_jk, and is needed for

    # for load balancing to work properly

    # Note: Replaced JkShmFile logs/jk.shm due to SELinux issues. Refer to

    # https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=225452

    JkShmFile run/jk.shm

     

     

    # Add jkstatus for managing runtime data

     

     

    <Location /jkstatus>

        JkMount status

        Order deny,allow

        Deny from all

        Allow from 127.0.0.1

    </Location>

     

    [apache@WEB_SERVER_HOST conf]$ cat workers.properties

    worker.list=loadbalancer,status

     

     

    worker.app.port=9580

    worker.app.host= APP_SRV_HOST

    worker.app.type=ajp13

    worker.app.ping_mode=A

    worker.app.socket_connect_timeout=10000

    worker.app.connection_pool_timeout=600

    worker.loadbalancer.sticky_session=True

     

     

    # Load-balancing behaviour

    worker.loadbalancer.type=lb

    worker.loadbalancer.balance_workers=app

     

     

    # Status worker for managing load balancer

    worker.status.type=status

     

    [apache@WEB_SERVER_HOST conf]$ cat uriworkermap.properties

     

     

    # Mount the Servlet context to the ajp13 worker

    /*=loadbalancer

    [apache@WEB_SERVER_HOST conf]$



  • 2.  Re: App not responding back to load balancer (mod_jk)

    Posted 05-21-2015 11:19 AM

    1. When I checked netstat -na|grep 9580 it showed 200+ established connections which is weird so I added worker.app.connection_pool_timeout=600 in workers.properties on web server.

    2. Same needs to be done on tomcat but I have a question here

    Current tomcat settings are

      <Connector port="9580" protocol="AJP/1.3" redirectPort="8443" address="WEB_SERVER_HOST_IP" />

        which get copied from CSA everytime I restart,

    If I have to setup the timeout and stuff like below, where should I set that up so that it does not get overwritten on APP restart?

     

      <Connector port="9580" address="WEB_SERVER_HOST_IP" protocol="AJP/1.3"  redirectPort="8443" emptySessionPath="true" enableLookups="false" maxThreads="1200" connectionTimeout="600000" />



  • 3.  Re: Change timeout settings on Tomcat. App not responding back to load balancer (mod_jk)

    Posted 05-21-2015 01:46 PM

    Strictly speaking, the file I'm about to mention isn't editable in a way that is supported, so any changes would be an unsupported customization.

     

    However, I think I understand your needs and so you will be looking to make changes to $NIKU_HOME/.setup/scripts/j2ee/tomcat/deploy.xml

     

    This is the file that writes out the tomcat server.xml file with the (connector and other) configuration information inside it.  Specifically around this part of it (in order to get the final XML elements/attributes structure in the server.xml as needed):

     

     

    Hope that helps.



  • 4.  Re: Change timeout settings on Tomcat. App not responding back to load balancer (mod_jk)

    Posted 05-21-2015 02:35 PM

    Be careful, the first time I played around with adding new configuration elements, I broke the app server.

    Make sure you have a copy of the original and you might want to start the app via a console window with serviceappcmd-debug.exe or was it serviceappcmd-diag.exe.

    This will push a lot of logging out for you to review if everything is copacetic.


    V/r,

    Gene



  • 5.  Re: Change timeout settings on Tomcat. App not responding back to load balancer (mod_jk)

    Posted 05-28-2015 11:36 AM

    Hi Nick,

     

    Thanks for the hint.

     

    Having a web server between Hardware Load Balancer and app server is org standard here and there are firewall as well as zone restrictions that make us have a redundant web server in there.

     

    I see that you mentioned use of reverse proxy instead of mod_jk

     

    Can you tell me more about that solution?



  • 6.  Re: Change timeout settings on Tomcat. App not responding back to load balancer (mod_jk)
    Best Answer

    Posted 05-28-2015 11:56 AM

    So I suspect you have a load balancer for the URL that then farms out requests to multiple web servers, and each of those web servers is then configured as a reverse proxy (currently using the ajp connector port) to each tomcat service.

     

    In your apache web server configurations, you would use/setup mod_proxy instead of mod_jk and just have it redirect incoming traffic over HTTP.  I'm not familiar with mod_jk but it's probable that it already requires and uses mod_proxy too, and just uses the mod_jk part for the conversion of the HTTP text to the binary AJP and sets some additional properties.

     

    There isn't much else I can add/say about it though (lack of implementation exposure) and if other implementations out there are fronting tomcat with apache httpd, they may be able to chime in with their experience and configuration.



  • 7.  Re: Change timeout settings on Tomcat. App not responding back to load balancer (mod_jk)

    Posted 06-05-2015 11:26 AM

    I switched to mod_proxy, monitoring the response now.