What is an optimized mod_jk configuration for use in Apache with JBoss?

Issue
What is an optimized mod_jk configuration for use in Apache with JBoss?

Environment
Red Hat Application Stack 1 Red Hat Application Stack 2 JBoss Enterprise Web Server 1.0.0

Resolution
Introduction There are many potential problems associated with the default configuration of mod_jk. While the default configuration may be adequate for a very low traffic website, when pushing any moderate to high load through mod_jk, there will be connection issues. This is not due to any bug in mod_jk, it is because the default configuration makes no assumption about your existing hardware or potential load, so, therefore, it is not tuned accordingly.

The community webapp http://lbconfig.appspot.com helps to create an optimized mod_jk configuration. It will generate an optimized configuration based on environment parameters along with the math and explanation behind the configuration.

Note that the configuration recommendations here are to be used as an optimized base reference to avoid many of the common problems users experience with mod_jk. The optimizations here don't cover every single optimization but provide a solid base. Refer to http://tomcat.apache.org/connectors-doc/reference/workers.html for details on all available mod_jk properties.

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 1

What is an optimized mod_jk configuration for use in Apache with JBoss?

Let's take a look at a typical default configuration for Apache/Tomcat/mod_jk:

workers.properties typically located at APACHE_HOME/conf/workers.properties)

worker.list=loadbalancer,status worker.node1.port=8009 worker.node1.host=node1.mydomain.com worker.node1.type=ajp13 worker.node1.lbfactor=1 worker.node2.port=8009 worker.node2.host= node2.mydomain.com worker.node2.type=ajp13 worker.node2.lbfactor=1 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1,node2 worker.status.type=status

JBoss Web's (Tomcat) server.xml AJP snippet

<Connector port="8009" address="${jboss.bind.address}" protocol="AJP/1.3" emptySessionPath="true" enableLookups="false" redirectPort="8443" />

Apache's httpd.conf (on RHEL /etc/httpd/conf/httpd.conf, otherwise APACHE_HOME/conf/httpd.conf):

<IfModule prefork.c> StartServers 8 MinSpareServers 5

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 2

What is an optimized mod_jk configuration for use in Apache with JBoss?

MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 </IfModule>

The above configuration, under load, may cause mod_jk to be very slow and unresponsive, cause http errors, and cause half-closed connections. These problems can arise because there is no connection timeout specified to take care of "orphaned" connections, no error handling properties defined in workers.properties, and no connection limits set in Apache and Tomcat.

Configuring server.xml (JBOSS_HOME/server/CONFIGURATION/deploy/JBOSS_WEB/ server.xml):

The main concern with server.xml is setting the connectionTimeout which sets the SO_TIMEOUT of the underlying socket. So when a connection in Tomcat hasn't had a request in the amount of time specified by connectionTimeout, then the connection dies off. This is necessary because if the connection hasn't been used for a certain period of time, there is the chance that it is half-close on the mod_jk end. If the connection isn't closed there will be an inflation of threads which can over time hit the maxThreads count in Tomcat. If this occurs, Tomcat will not be able to accept any new connections. A connectionTimeout of 600000 (10 minutes) is a good number to start out with. There may be a situation where the connections are not being recycled fast enough; in this instance the connectionTimeout could be lowered to 60000 (1 minute).

The recommended value of maxThreads is 200 per CPU, so here we assume the server is a single core machine. If it had been quad core, we could push that value to 800 or more depending on RAM and other machine specs. The total threads is an aggregate value. If Apache and JBoss are on the same server, and that server has four cores, then you would halve the maxThreads and MaxClients to 400 each. Always ensure the total threads possible doesn't exceed 200 times the number of CPU cores.

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 3

What is an optimized mod_jk configuration for use in Apache with JBoss?

<Connector port="8009" address="${jboss.bind.address}" protocol="AJP/1.3" emptySessionPath="true" enableLookups="false" redirectPort="8443" maxThreads="200" connectionTimeout="600000"/>

Configuring workers.properties:

See comments inline:

worker.list=loadbalancer,status

worker.template.port=8009 worker.template.type=ajp13 worker.template.lbfactor=1 #ping_mode was introduced in 1.2.27, if not using 1.2.27 please specify connect_timeout=10000 and prepost_timeout=10000 as an worker.template.ping_mode=A worker.template.socket_timeout=10 #It is not necessary to specify connection_pool_timeout if you are running the worker mpm worker.template.connection_pool_timeout=600 #Referencing the template worker properties makes the workers.properties shorter and more concise worker.node1.reference=worker.template worker.node1.host=192.168.1.2 worker.node2.reference=worker.template worker.node2.host=192.168.1.3 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1,node2 worker.loadbalancer.sticky_session=True worker.status.type=status

The key point in the above workers.properties is that we've added limits for the connections mod_jk makes. With the base configuration, socket timeouts default to infinite. The other important properties are ping_mode and connection_pool_timeout which must be set to equal server.xml's connectionTimeout when using the prefork mpm. When connectionTimeout and connection_pool_timeout are set, after a connection has been inactive for x amount of time, the connection in mod_jk and Tomcat will be closed at the same time, preventing a half-closed connection. Regardless, even if a connection issue happens, with the correct properties like ping_mode=A set, then the bad connection will be detected and handled transparently.

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 4

What is an optimized mod_jk configuration for use in Apache with JBoss?

Handling long running requests

When there are long running requests, for example, greater than 10 seconds, special properties must be set. Ten seconds is used as this is the default recommended value for the socket_timeout. If a request takes longer than 10 seconds the request will be terminated due to the socket_timeout time. The best approach to take in these cases is to forego the socket_timeout and use a combination of reply_timeout and socket_connect_timeout. If there are requests known to take 30 seconds, the configuration would look like so:

worker.template.reply_timeout=30000 worker.template.socket_connect_timeout=10000 #Commented out now as the global socket_timeout is too short #worker.template.socket_timeout=10

The socket_connect_timeout is kept at 10 seconds because it defines the time for a socket to wait to connect, which has nothing to do with the length of the request, just the connect time of the socket. If socket_timeout was blindly set to 30 seconds, then that would increase the socket_connect_timeout to 30 seconds as a consequence, unless it was defined manually, which is not optimal.

Configuring Apache Note that maxThreads for the AJP connection should coincide with the MaxClients set in Apache's httpd.conf. MaxClients needs to be set in the correct module in Apache. This can be determined by running httpd -V (capital V):

# httpd -V Server version: Apache/2.2.3 Server built: Sep 11 2006 09:43:05 Server's Module Magic Number: 20051115:3 Server loaded: APR 1.2.7, APR-Util 1.2.8 Compiled using: APR 1.2.7, APR-Util 1.2.7 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count)

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 5

What is an optimized mod_jk configuration for use in Apache with JBoss?

Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/etc/httpd" -D SUEXEC_BIN="/usr/sbin/suexec" -D DEFAULT_PIDLOG="logs/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="logs/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf"

This shows that the Server MPM is prefork. This is not always 100% accurate so you should also view the output of /etc/sysconfig/httpd and check for the line: HTTPD=/usr/sbin/ httpd.worker. If it is commented out you are running prefork, otherwise, if uncommented, worker.

From httpd.conf:

<IfModule prefork.c> StartServers 8 MinSpareServers 5 MaxSpareServers 20 MaxClients 200 MaxRequestsPerChild 0 </IfModule>

Or if Apache is using worker, it is:

<IfModule worker.c> StartServers 2

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 6

What is an optimized mod_jk configuration for use in Apache with JBoss?

MaxClients 200 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 </IfModule>

MaxRequestsPerChild is 0, this is the recommended value when using mod_jk as mod_jk keeps open persistent connections. The key values in the above configuration are MaxClients and MaxRequestsPerChild, the rest of the values are left as default. Note that MaxRequestsPerChild is recommended to be 0, however the value may need to be greater than 0 depending on whether Apache is used for other modules also, especially in the case of resource leakage. But for the vast majority of use cases, MaxRequestsPerChild should always be 0.

Advanced worker-mpm Configuration

To get the most out of your mod_jk setup you should be using Apache's worker MPM which provides a definite performance improvement over the Prefork MPM. The following section will detail how to configure Apache/mod_jk/Tomcat with the worker MPM and the math behind the configuration.

Let's start out with the worker MPM configuration:

<IfModule mpm_worker_module> ThreadLimit 100 StartServers 5 MaxClients 1000 MinSpareThreads 100 MaxSpareThreads 1000 ThreadsPerChild 100 MaxRequestsPerChild 0 </IfModule>

The optimal configuration completely depends on the hardware being used and the load requirements. As a general rule of thumb, keep processes low and thread count high.

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 7

What is an optimized mod_jk configuration for use in Apache with JBoss?

To determine the number of processes Apache will use simply divide MaxClients by ThreadPerChild. So in this case MaxClients (1000) / ThreadsPerChild (100) = Processes (10), so Apache will allocate a maximum of 100 threads per each 10 child processes resulting in a total of 1000 possible clients.

Now to translate this to mod_jk, mod_jk maintains a connection pool for each worker defined in workers.properties. By default with Apache mod_jk sets connection_pool_size to ThreadsPerChild, so in the above case that would translate to 100, giving 1000 possible connections to JBoss. This may or may not be desired.

Let's take a common example, there will be three JBoss servers that combined needed to be able to handle 900 concurrent connections:

worker.list=loadbalancer,status worker.template.type=ajp13 worker.template.port=8009 worker.template.ping_mode=A worker.template.connection_pool_size=30 worker.template.socket_timeout=10 worker.node1.reference=worker.template worker.node1.host=192.168.0.101 worker.node2.reference=worker.template worker.node2.host=192.168.0.102 worker.node3.reference=worker.template worker.node3.host=192.168.0.103 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1,node2, node3 worker.loadbalancer.sticky_session=True worker.status.type=status

The above configuration tells mod_jk to multiplex 30 connections to the available Apache processes, which is 10 processes. This means that 30 connections multiplexed over 10 processes gives 300 possible connections to each backend worker. Furthermore, the total

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 8

What is an optimized mod_jk configuration for use in Apache with JBoss?

connections able to be used in this configuration from Apache is 900, which means 100 connections will be left over for static content and whatnot.

Next, configure maxThreads in each AJP connector to match the above.

Nodes 1,2 and 3 AJP connectors:

<Connector port="8009" address="${jboss.bind.address}" protocol="AJP/1.3" emptySessionPath="true" enableLookups="false" redirectPort="8443" maxThreads="300" connectionTimeout="600000" />

The last recommended setting is if the worker MPM is being used, then the following should be added to the httpd.conf file:

# Use a value of 600 with mod_jk 1.2.27 and a value of 60 with 1.2.28 and higher, there was a bug in 1.2.27 that calculated the valu JkWatchdogInterval 60

Enabling the JkWatchdogInterval allows workers to be maintained periodically by a background thread. The maintenance includes checking for idle connections, correcting load status, and detecting back end health status. Note that JkWatchdogInterval was introduced and mod_jk 1.2.27 and will have no effect if used in prior versions. Please note that the JkWatchdogInterval is not to be used with the Prefork MPM.

Important Notes

As of mod_jk 1.2.27 with ping_mode, if set, all of the timeouts are set to 10000 by default, including the reply_timeout. This can have an adverse affect for long running servlets that may not have been encountered pre 1.2.27 as the reply_timeout is typically not set and

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 9

What is an optimized mod_jk configuration for use in Apache with JBoss?

therefore defaults to 0/infinite. Remember though, reply_timeout deals with per packet response, not per request, but in a long running servlet, it may take longer than 10000 (10 seconds) to initially respond, and that is the problem.

Copyright (c) 2011 by Red Hat, Inc. All rights reserved. 10