mod_mpmstats provides three capabilities:
Often this type of monitoring has been performed with mod_status, which requires that a request be sent to the web server from a browser or script, and then be viewed or parsed to record the web server activity. mod_mpmstats can eliminate the need to constantly monitor the mod_status report to determine the current thread usage.
The ability to report the plug-in module where requests are stalled allows the administrator to quickly determine what components to check when the web server is unresponsive.
mod_mpmstats is provided for AIX, Linux, Solaris, HP-UX, Windows, and z/OS.
There are two levels of support, based on the capabilities of your level of IHS.
With this level of support, the reports of busy conditions and the
number of threads in each state can be generated. The additional
module tracking capability enabled by the TrackModules
directive is not available.
The additional information available is the name of the module of request processing, in addition to the information provided with the base level of support.
The Recommended Updates for IBM HTTP Server page has links to download the latest maintenance.
Enable this module if you need to track web server thread usage over time but you don't need extensive details such as what requests are being processed.
If your level of IHS includes module-tracking support, the module tracking feature can result in quicker diagnosis of web server hangs.
Customers with third-party modules and prior problems with unresponsive behavior need to use the module tracking feature.
Note that mod_status provides more details, including actual requests being processed, so we recommend that mod_status be enabled as well.
IBM HTTP Server 6.1 and earlier: Copy mod_mpmstats.so for your platform to the modules directory in the web server installation location (e.g., to /opt/IBMIHS/modules).
IBM HTTP Server 7.0 and later: On supported platforms, this module is provided with
the product in the modules/debug/ subdirectory.
loadmodule mpmstats_module modules/mod_mpmstats.so
IBM HTTP Server 7.0 and later: See conf/httpd.conf.default for the LoadModule and
NetTrace directives referencing this module from the modules/debug/ subdirectory.
LoadModule,
ReportInterval, and TrackModules
directives added as part of the activation step.Whenever enabled, mod_mpmstats will report to the error log when most or all of the web server threads are busy handling clients. Here is an example from Unix:
[Thu Aug 19 14:14:00 2004] [notice] mpmstats: approaching MaxClients (48/50) [Thu Aug 19 14:14:05 2004] [error] server reached MaxClients setting, consider raising the MaxClients setting
(The second message would appear even without mod_mpmstats.)
Here is an example from Windows:
[Wed Aug 17 08:47:29 2005] [notice] mpmstats: approaching ThreadsPerChild (23/25) [Wed Aug 17 08:47:42 2005] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
(The second message would appear even without mod_mpmstats.)
Without this module, the web server will only report the busy condition when it actually reaches the maximum clients setting, and it will only report it once per restart of the server. mod_mpmstats will check every second, and will report the busy condition as often as once every 90 seconds.
The ReportInterval directive specifies the interval in seconds between reports of the number of threads in different states. If the ReportInterval directive is not specified, reports of thread states will not be written.
Example:
# mod_mpmstats will write a report of thread usage every 30 seconds ReportInterval 30
Here is an example section of the error log when
mod_mpmstats has been configured to report the thread usage every two
seconds with the ReportInterval 2 directive:
[Thu Aug 19 14:01:52 2004] [notice] mpmstats: rdy 712 bsy 312 rd 121 wr 173 ka 0 log 0 dns 0 cls 18 [Thu Aug 19 14:01:54 2004] [notice] mpmstats: rdy 718 bsy 306 rd 120 wr 169 ka 0 log 0 dns 0 cls 17 [Thu Aug 19 14:01:56 2004] [notice] mpmstats: rdy 729 bsy 295 rd 123 wr 135 ka 0 log 0 dns 0 cls 37 [Thu Aug 19 14:01:58 2004] [notice] mpmstats: rdy 759 bsy 265 rd 130 wr 67 ka 0 log 0 dns 0 cls 68 [Thu Aug 19 14:02:00 2004] [notice] mpmstats: rdy 809 bsy 215 rd 131 wr 44 ka 0 log 0 dns 0 cls 40 [Thu Aug 19 14:02:02 2004] [notice] mpmstats: rdy 781 bsy 243 rd 117 wr 106 ka 0 log 0 dns 0 cls 20 [Thu Aug 19 14:02:04 2004] [notice] mpmstats: rdy 813 bsy 211 rd 118 wr 72 ka 0 log 0 dns 0 cls 21 [Thu Aug 19 14:02:07 2004] [notice] mpmstats: rdy 685 bsy 339 rd 177 wr 122 ka 0 log 0 dns 0 cls 40 [Thu Aug 19 14:02:09 2004] [notice] mpmstats: rdy 707 bsy 317 rd 193 wr 97 ka 0 log 0 dns 0 cls 27 [Thu Aug 19 14:02:11 2004] [notice] mpmstats: rdy 731 bsy 293 rd 196 wr 39 ka 0 log 0 dns 0 cls 58 [Thu Aug 19 14:02:11 2004] [error] [client 9.27.164.15] File does not exist: /home/trawick/testihsbuild/install/htdocs/en_US/jakdsljadslkfjalkfdsasf [Thu Aug 19 14:02:13 2004] [notice] mpmstats: rdy 747 bsy 277 rd 186 wr 71 ka 0 log 0 dns 0 cls 20 [Thu Aug 19 14:02:15 2004] [notice] mpmstats: rdy 749 bsy 275 rd 162 wr 89 ka 0 log 0 dns 0 cls 24 [Thu Aug 19 14:02:17 2004] [notice] mpmstats: rdy 764 bsy 260 rd 156 wr 83 ka 0 log 0 dns 0 cls 21 [Thu Aug 19 14:02:19 2004] [notice] mpmstats: rdy 784 bsy 240 rd 158 wr 33 ka 0 log 0 dns 0 cls 49 [Thu Aug 19 14:02:21 2004] [notice] mpmstats: rdy 788 bsy 236 rd 159 wr 59 ka 0 log 0 dns 0 cls 18 [Thu Aug 19 14:02:23 2004] [notice] mpmstats: rdy 735 bsy 289 rd 161 wr 117 ka 0 log 0 dns 0 cls 11
Note: The report will not be written when the web server is not actively processing any requests. This information does not require ExtendedStatus to be enabled.
The fields logged are described in the table below:
| field | description |
| rdy (ready) | the number of web server threads started and ready to process new client connections |
| bsy (busy) | the number of web server threads already processing a client connection |
| rd (reading) | the number of busy web server threads currently reading the request from the client |
| wr (writing) | the number of busy web server threads that have read the request from the client but are either processing the request (e.g., waiting on a response from WebSphere Application Server) or are writing the response back to the client |
| ka (keepalive) | the number of busy web server threads that are not processing a request but instead are waiting to see if the client will send another request on the same connection; refer to the KeepAliveTimeout directive to decrease the amount of time that a web server thread remains in this state |
| log (logging) | the number of busy web server threads that are writing to the access log |
| dns (dns lookup) | the number of busy web server threads that are performing a dns lookup |
| cls (closing) | the number of busy web server threads that are waiting for the client to acknowledge that the entire response has been received so that the connection can be closed |
Here's a comparison of displays by mod_mpmstats vs. mod_status for the same server state:
mod_mpmstats would simply write
[Thu Aug 19 15:22:43 2004] [notice] mpmstats: rdy 5 bsy 45 rd 10 wr 3 ka 0 log 0 dns 0 cls 31
The mod_status report would look like this:
CCCCCCCCRRCCRR___RWCCCCCCCCCCW_CCRCWRC_CCCCCRCCRRC.............. ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................
Scoreboard Key:
"_" Waiting for Connection,
"S" Starting up,
"R" Reading Request,
"W" Sending Reply,
"K" Keepalive (read),
"D" DNS Lookup,
"C" Closing connection,
"L" Logging,
"G" Gracefully finishing,
"I" Idle cleanup of worker,
"." Open slot with no current process
If ExtendedStatus On were specified in httpd.conf,
the actual requests being processed would be listed in the mod_status
report.
The mod_status report is easier to read but it is not in a form that is easy to save at intervals and easily see the thread use.
The TrackModules directive is used to enable the reporting of how many requests are currently in each plug-in module. This directive has the following requirements:
Example:
# mod_mpmstats will write a report of thread usage and counts by # module every 15 seconds ReportInterval 15 TrackModules On
Here is an example section of the error log when mod_mpmstats has been configured to report the thread and module usage every 15 seconds with the prior example:
[Sat Mar 19 07:47:12 2005] [notice] mpmstats: rdy 250 bsy 50 rd 0 wr 50 ka 0 log 0 dns 0 cls 0 [Sat Mar 19 07:47:12 2005] [notice] mpmstats: bsy: 50 in mod_cgid.c [Sat Mar 19 07:47:27 2005] [notice] mpmstats: rdy 250 bsy 50 rd 0 wr 45 ka 5 log 0 dns 0 cls 0 [Sat Mar 19 07:47:27 2005] [notice] mpmstats: bsy: 45 in mod_cgid.c [Sat Mar 19 07:47:42 2005] [notice] mpmstats: rdy 250 bsy 50 rd 0 wr 44 ka 6 log 0 dns 0 cls 0 [Sat Mar 19 07:47:42 2005] [notice] mpmstats: bsy: 44 in mod_cgid.c [Sat Mar 19 07:47:57 2005] [notice] mpmstats: rdy 250 bsy 50 rd 0 wr 44 ka 6 log 0 dns 0 cls 0 [Sat Mar 19 07:47:57 2005] [notice] mpmstats: bsy: 44 in mod_cgid.c
In this example, the predominant type of request is a CGI request, which is handled by mod_cgid. Usually, only a subset of active threads will be attributed to a particular plug-in module. Threads not attributed to a plug-in module are busy performing I/O with the client or are handling basic request processing.
When web server requests are handled with reasonable response time, the module counts are not so important, and will reflect the types of processing which is being performed. If there is a meaningful number of CGI requests, then some number of IHS threads will typically be busy in mod_cgid.c. If there is a meaningful number of WebSphere requests, then some number of IHS threads will typically be busy in mod_was_ap20_http.c. If SiteMinder is handling authentication, then some number of IHS threads will typically be busy in mod_sm.c.
When response time starts to deteriorate or the web server stops responding altogether, the module counts are more important. Check the reports of module counts for the time leading up to the deteriorated or unresponsive behavior. If a particular module becomes the predominant module where IHS threads are busy, that is the likely cause of the response problem. For example, if a problem occurs with a WebSphere application which prevents it from responding as expected, the number of IHS threads busy in mod_was_ap20_http.c will begin increasing. If there are enough users making requests to that application, all IHS threads could eventually become busy in mod_was_ap20_http.c. Another example is with authentication requests. If SiteMinder is used and a problem occurs with a policy or LDAP server it uses, the number of IHS threads busy in mod_sm.c will begin increasing. If enough requests arrive before the server problem is resolved, all IHS threads could eventually become busy in mod_sm.c.