mod_whatkilledus

mod_whatkilledus for IHS 2.0 and above writes information about the current connection and request to the error log after an IHS child process crash. This provides important information for interpreting a core dump, and even without a core dump it can be used to determine what types of requests lead to crashes. (Is the connection leading to a crash always to an SSL port? Is the request leading to a crash always for an object handled by a certain plug-in module? Is the connection leading to a crash always from the same client?)

Here is an example report:

[Fri Mar 11 14:25:18 2005] pid 16934 mod_whatkilledus sig 11 crash
[Fri Mar 11 14:25:18 2005] pid 16934 mod_whatkilledus active connection: 127.0.0.1:43923->127.0.0.1:10101 (conn_rec 30683a38)
[Fri Mar 11 14:25:18 2005] pid 16934 mod_whatkilledus active request (request_rec 30689978):
GET /silly/?fn=sigsegv HTTP/1.0|Connection:close
[Fri Mar 11 14:25:18 2005] pid 16934 mod_whatkilledus Active module: mod_silly2.c
[Fri Mar 11 14:25:18 2005] pid 16934 mod_whatkilledus end of report

The detection of the active module, as shown in the highlighted text above, is only possible with cumulative e-fix PK01070 or later for IHS 2.0.42.x and IHS 2.0.47.x, or 6.0.2 or later. If the level of IHS in use does not have the capability, or the active module cannot be determined automatically, the report will not contain that information.

Customers with third-party modules in the IHS configuration should check the active module in the report before calling IBM support. If the active module at the time of the crash is from another vendor, report the problem to the vendor and provide them with any supporting materials they require to diagnose the problem.

Important note for IBM HTTP Server 7.0 and later users

On any platform where mod_whatkilledus is supported, it is now shipped as part of the product starting in IBM HTTP Server 7.0 and later. On these releases, diagnostic modules should not be downloaded or installed from this mustgather.

Supported server versions:

mod_whatkilledus is provided for AIX, Linux, Solaris, HP-UX, and z/OS.

Minimal Level of Support: IHS 2.0.42.2 and 2.0.47.0

With this level of support, the active connection (if any) and the active request (if any) will be reported.

Recommended Level of Support:

With this level of support, the active module at the time of the crash can sometimes be determined automatically.

Release Fix level
2.0.42.2 PK01070 or later
2.0.47.1 PK01070 or later
6.0 6.0.2 or later

IBM recommendation

Any installation of IHS 2.0.42.2 or above on AIX, Linux, Solaris, HP-UX, or z/OS should have this module installed and activated.

(Note: The version of this module for IHS 1.3 is described here.)

Installation

IBM HTTP Server 6.1 and earlier users: Copy mod_whatkilledus.so for your platform to the modules directory in the web server installation location (e.g., to /opt/IBMIHS/modules).

IBM HTTP Server 7.0 and later: On supported platforms, this module is provided with the product in the modules/debug/ subdirectory.

Activation

IBM HTTP Server 6.1 and earlier: Add the following directive after existing LoadModule directives:

LoadModule whatkilledus_module modules/mod_whatkilledus.so

IBM HTTP Server 7.0 and later: See conf/httpd.conf.default for the LoadModule and NetTrace directives referencing this module from the modules/debug/ subdirectory.

The determination of the module active at the time of the crash requires that mod_status is loaded and the ExtendedStatus directive is set to On.

Restart the server so that the updated configuration takes effect.

After requests are received, but before most plug-in modules run, the module saves information about the request in case the child process crashes while processing that request. This introduces a small amount of overhead into request processing, but the overhead is hard to measure and certainly is not noticeable.

Deactivation

Comment out the LoadModule directive added as part of the activation step.

Restart the server so that the updated configuration takes effect.