Crash mustgather (Windows)

System preparation

If you have not previously setup dump tools to save information about program crashes then refer to 'MustGather setup of error reporting dump tools for Microsoft Windows' for information on performing this setup. This setup step must be performed prior to recreating the crash. Make sure you are aware of the directory into which the tools will save log and dump data.

Remove any existing log and crash dump files from the output directory (after backing up as appropriate).

mod_whatkilledus

Note: PI97314 added a diagnostic module named mod_backtrace to Windows installations in 8.5.5.14 and 9.0.0.9. Append the following stanza to httpd.conf to capture a text description of crashes that may speed up analysis by support:

LoadModule backtrace_module modules/debug/mod_backtrace.so
LoadModule whatkilledus_module modules/debug/mod_whatkilledus.so
WKUDisableWindowsErrorBox ON

In the event of a child process crash, logs/whatkilledus.log will contain a description of the crash.

NOTE: You must review the 'Known Windows crashes' section at the bottom of this document before proceeding to make sure your problem can't be solved with one of the solutions documented there.

Obtaining information after the crash occurs

These steps must be performed after the crash occurs.

  1. Gather general system and web server information.

    This information is gathered by running the ihsdiag ServerDoc DescribeConfig tool as described by the instructions in the System and web server information tool documentation.
    This will result in a directory of information named 'ServerConfig.timestamp'.
    That directory should be zipped and sent to IBM using the provided instructions after completing the following steps for obtaining additional information.

  2. Gather crash dump information

    Copy the data (logs and/or crash dump files) saved by the Windows error reporting tool (ex: Dr.Watson), if it exists, into the 'ServerConfig.timestamp' directory.
    These files should be located in the crash dump output directory(s) that were configured in the System preparation section above.
    For earlier Microsoft Windows versions that use Dr.Watson, the data files will usually be named drwtsn32.log and user.dmp.
    For newer Microsoft Windows versions that use the LocalDumps setup, the data files will usually be named liked 'http.exe.4359.dmp'.

  3. Gather any additional error or access log files that might be available.

    The access.log and error.log files will be automatically gathered by the ServerDoc DescribeConfig tool used above, but if the configuration file has been changed to specify different names for these log files then you should copy those log files from the IHS_install_root\logs\ directory to the 'ServerConfig.timestamp\files\logs' directory.

  4. Gather the WebSphere plug-in trace file

    The actual location is specified in plugin-cfg.xml and is generated by configuring LogLevel="Trace".
    Example:
    c:\WebSphere\AppServer\logs\http_plugin.log

  5. Gather Windows Event Viewer information

    • Open the 'Event Viewer' (All Programs -> Administrative Tools -> Event Viewer)

    • Save the data for both the 'Application' and 'System' event logs.
      (For newer versions of Windows, these are located under 'Windows Logs' in the left pane of the Event Viewer.):

      • Select the log in the tree

      • Rt-click and select 'Save Log File As...' or 'Save All Events As...'
        (depending on your version of Windows)

      • Name each file the same as the log (i.e. 'Application' or 'System'), and select a type of 'Event Log (*.evt)' or 'Event Files (*.evtx)'
        (depending on your version of Windows)

      • Save into the 'ServerConfig.timestamp' directory
        (the directory created in the first step by running 'ServerDoc DescribeConfig')

Recap of information to send to IBM support:

Create a zip file of the 'ServerConfig.timestamp' directory as described in the System and web server information tool documentation. Send this ServerConfig.timestamp.zip containing the following to IBM support for analysis:

  • IHS information gathered by the 'ServerDoc DescribeConfig',
    plus Windows system information, as exported to file system_info.txt by the 'System Information' program

  • crash logs and dumps

  • all error and access logs

  • WebSphere plug-in trace file

  • Event Viewer export files

Instructions to send diagnostic information to IBM support.

Known Windows crashes:

IHS fails to start when LDAPCodepageDir is configured

If there is a copy of the IBM LDAP Client or Server library already installed on the same machine as IHS it may have configured the TISDIR variable in the Windows environment. This conflicts with the IHS LDAPCodepageDir directive and may cause IHS to crash without any error or Windows event being logged. If if it is an older version which is no longer required, it is recommended that the IBM LDAP libraries be uninstalled in order to delete the TISDIR environment variable. If that is not possible then the LDAPCodePageDir directive must be removed and the default IHS LDAP codepage directory used.

IHS crashes on startup, or on restart when MaxRequestsPerChild is reached

Certain third party software may add a Winsock Service Provider layer which sits between IHS and the TCP/IP network. If this software does not implement some of the Microsoft advanced Winsock functions correctly, such as AcceptEx, it may cause IHS to crash on startup or restart. A possible fix is to add the "Win32DisableAcceptEx" directive to the IHS configuration file. Setting "MaxRequestsPerChild 0" (the recommended default) may also eliminate the crashes. This crash may be accompanied by the following message in the error log:

[warn] (OS 10038)An operation was attempted on something that is not a socket.  
setsockopt(SO_UPDATE_ACCEPT_CONTEXT) failed.

Excessive ThreadsPerChild values may cause IHS child process crashes

On Windows, IBM HTTP Server has a Parent process and a single multi-threaded Child process. The number of threads in the child process can be tuned according to the expected number of simultaneous connections using the ThreadsPerChild directive. However, attempting to set the number of threads too high can cause excessive memory usage resulting in child process crashes. When approaching the limits, the server may actually run for awhile before the memory limit is exceeded and causes the child process to crash. Values even higher than this would probably cause an immediate crash during startup.

Refer to the Handling enough simultaneous connections with IBM HTTP Server on Windows / ThreadsPerChild setting topic in the 'IBM HTTP Server Performance Tuning' document for information on tuning this value.

Child process exited/crashed and error log has message "unable to init Zlib: deflateInit2 returned -4"

The '-4' return code in the message "unable to init Zlib: deflateInit2 returned -4" is Z_MEM_ERROR and means there was insufficient memory to initialize Zlib.
In this case, some tuning to decrease memory usage is probably needed.

For information on tuning to decrease memory usage, you should refer to the prior Excessive ThreadsPerChild values may cause IHS child process crashes entry in this document, and to the memory references in the IBM HTTP Server Performance Tuning guide, and to the Many RewriteCond/RewriteRule directives topic.