Java is a very popular development language for web-based applications, and many software vendors and authors of custom web-based applications use the combination of Apache Software Foundation’s Tomcat application server and Apache’s HTTP Serverto support Java servlet and JSP code. In order to run such applications in a typical 3-Tier environment (Presentation Layer, Application Layer, and Data Layer), an Apache server running mod_proxy in reverse proxy mode is placed in the Presentation Layer. The diagram below illustrates how the Tomcat application server in such an environment serves an HTTP request.

One of Moser’s largest clients uses a web-based Learning Management System (LMS) to support the training needs of its staff. The LMS is a Current Off The Shelf (COTS) product, which includes a pre-configured version of Apache Tomcat at the Application Layer, and requires an Oracle database at the Data Layer. In order to comply with the customer’s standard Web Application architecture, Moser added to the Presentation Layer an Apache web server running mod_proxy and configured as a reverse proxy. As part of the integration of the LMS, our customer requires that we performance test the application to ensure that it can handle several thousand concurrent connections.

Background: Configuring the Apache Reverse Proxy

Let’s begin with a little background information on how the Apache reverse proxy is configured. The LMS application uses the HTTP protocol between the client browser and the Tomcat Application server. In order to properly secure the application, SSL is used to encrypt the connection traffic between the client browser and the Apache Reverse Proxy in the Presentation layer. The Apache Reverse Proxy then passes those requests to the Tomcat Application Server in unencrypted form and using port

8080, which is the TCP port that the application vendor has configured Tomcat to listen on for incoming connections.

In addition, we have compiled the Apache Reverse Proxy to use the Worker MPM (Apache Multi-Processing Modules). We have configured Apache to handle 12,000 concurrent connections by using the following in the httpd.conf file:

<IfModule worker.c>
ServerLimit               150
ThreadLimit               120
StartServers              100
MaxClients              12000
MinSpareThreads             5
MaxSpareThreads         12000
ThreadsPerChild           120
MaxRequestsPerChild         0
</IfModule>

This configuration tells Apache to start 100 child processes, each of which is capable of handling 120 connection (threads). Each child process will leave a minimum of 5 threads free to handle additional incoming connections. Apache will handle 12,000 concurrent connections (MaxClients) and will start up to 150 child processes (ServerLimit) in order to handle that load. This configuration handles incoming connections only. Outgoing connections (connections between Apache’s mod_proxy module and the Tomcat Application server) are handled differently, as we will see shortly.

Now let’s take a look at how the Apache reverse proxy module is configured:

ProxyPass /learning http://tomcatapp:8080/learning disablereuse=on retry=0
ProxyPass /help http://tomcatapp:8080/help disablereuse=on retry=0
ProxyPass /pqe http://tomcatapp:8080/pqe disablereuse=on retry=0

This configuration maps requests for /learning, /help, and /pqe made to the Apache Proxy server to be sent to the Tomcat Application server.

Finally, let’s take a look at how the Tomcat Application Server’s HTTP connector is configured. This connector is responsible for handling incoming HTTP requests and passing them to the JVM for processing.

<Connector port="8080" protocol="HTTP/1.1" scheme="http"
connectionTimeout="20000"
redirectPort="8443"
maxThreads="48"
URIEncoding="UTF-8"
allowTrace="false"
xpoweredBy="false"
maxHttpHeaderSize="8192"
server="undisclosed"
maxParameterCount="30000" />

The important setting here is the maxThreads setting, which determines the number of concurrent connections that may be processed by the Tomcat HTTP connector.

Initial Performance Testing Results

We set up and ran the performance test against the LMS application. The test simulated typical usage of the application by 1,000 concurrent users performing a variety of normal tasks, such as logging in, viewing training, taking exams, etc. While the test was executing, we monitored the Apache web server logs, the Tomcat application server logs, and key statistics of the operating system on both servers.

The results of the first performance test were not good. Application response times were high, and the automated testing software logged many errors where application functions did not complete properly. In addition, we noticed the following:

  • CPU and memory on both servers were slightly elevated, but well within the capabilities of both servers
  • The Apache Reverse Proxy error log contained a lot of errors, such as:
    • (99)Cannot assign requested address: proxy: HTTP: attempt to connecto to 10.x.x.x:8080 (tomcatapp) failed

    • ap_proxy_connect_backend disabling worker for (tomcatapp)

    • (104)Connection reset by peer: proxy: error reading status line from remote server tomcatapp:8080, referrer: https://www.example.net/learning/user/catalogsearch/getresults.do

  • The netstat utility showed thousands of connections between the Apache reverse proxy and the Tomcat application server in TIME_WAIT state

Based on these results, we concluded that there is a performance bottleneck between the Apache Reverse Proxy server and the Tomcat Application Server. Our analysis of the log entries and netstat data led to the following conclusions:

  • Every HTTP request by a browser means several connections between the reverse proxy and the application server. There is a limit to the number of connections that the reverse proxy can open to the application server, and once this limit is surpassed, the “Cannot assign requested address” message is the result.
  • The Tomcat server is limited by its current maxThreads setting to being able to process just 48 concurrent connections. Once that limit is reached, many of the connections will be dropped, which leads to the “disabling worker” error message.
  • Every connection from the reverse proxy, whether or not it was serviced by Tomcat eventually ends up in a TIME_WAIT state. This is normal, but the connection stays in this state for 4 minutes. This is an eternity and while the connection is in that state, its slot in the reverse proxy connection table cannot be used for a new connection. Once the table is out of slots, no more connections may be opened, which leads to 500 and 503 return codes on the client browser.

Workaround to Handle Thousands of Concurrent Connections

Here is what we did to tune both the Tomcat Server and the Apache Reverse proxy to handle the large number of requests between the two:

  • The maxThreads setting on the Tomcat Server is much too low. We started by bumping this up to 500 and eventually bumped it up again to 1,000.
  • Implemented the minSpareThreads=”100” setting on the Tomcat server. This forces Tomcat to always keep 100 threads free for incoming requests. Tomcat will continue to add threads up to the maxThreads setting in order to achieve this.
  • Implemented acceptorThreadCount=”4” on the Tomcat server. This setting allows the Tomcat HTTP connector to take full advantage of the multiple CPU cores in the hardware upon which it runs.
  • Implemented a connection queue on the Tomcat server by setting acceptCount to 2000. This creates a queue of 2000 connections that will be accepted by the HTTP connector, but not processed until a thread becomes available.
  • Set maxConnections to 5,000 on the Tomcat server. This will allow 5,000 connections to the Tomcat Server from the Apache Reverse Proxy.
  • Set keepAlivetimeout to 60,000 (60 seconds) which gives the Tomcat Connector the ability to implement connection pools from the Apache Reverse Proxy. The Tomcat Connector will wait 60 seconds for another HTTP request on any given connection before closing that connection.
  • Implemented a connection pool on the Apache Reverse Proxy by eliminating the “disablereuse=on” parameter. With “disablereuse=on”, Apache will use a new TCP connection for each HTTP request. This means that it must open a lot of connections to Tomcat and wait 4 minutes after the transaction is complete for each connection to close. This led to the large number of connections in TIME_WAIT state and the eventual failure of the proxy to be able to open additional connections.
  • Added the “ttl=50” parameter to the ProxyPass statement. This means that each connection in the pool will be used for 50 seconds before it is closed. Because this timeout is 10 seconds less than the keepAlivetimeout setting on the Tomcat server, the Apache reverse proxy will close the connection first. This allows the Apache reverse proxy to have some control over the number of connections that it must open to send its incoming requests to the Tomcat application server.

Final Results

The intention of these settings is to ensure a more efficient use of the network connections between the Apache reverse proxy and the Tomcat application server. Rather than opening a new connection for every request, Apache will reuse connections for 50 seconds before opening new ones. Opening a new TCP connection takes time, memory, and processing power, so this configuration will make better use of network resources and will perform much better under a heavy load of incoming requests. In addition, the Tomcat server will increase its ability to service large numbers of concurrent incoming connections and have a queue in place in the event that the number of incoming connections is so large that there are not enough threads available to process them.

After implementing these changes, we saw a very dramatic difference in the performance test results. The netstat utility showed only a few hundred TCP connections in TIME_WAIT state rather than thousands. The Apache error logs were completely empty, and the access logs showed no abnormal HTTP return codes.

For More Information

Click the links below to get more insightful articles related to this subject: