Startup mustgather¶

Introduction¶

IBM HTTP Server (IHS) customers occasionally encounter problems starting the web server. An error message may be displayed on the terminal, or the initial httpd process may crash, or for some other reason the web server fails at startup and does not create child processes to handle client connections. When this type of problem occurs, the documentation required to diagnose the cause includes

operating system, IHS, and WebSphere plug-in configuration information
any messages written to the terminal during startup
any messages written to the IHS error log or WebSphere plug-in log during startup
results of trying to start IHS with/without the WebSphere plug-in
results of trying to start IHS with/without SSL

Some of this information can be gathered automatically. Other information is gathered manually.

Ensure that you are using a supported operating system level¶

Missing operating system prerequisites frequently cause startup failures. Consult this information to find the required operating system levels:

Check for operating system prerequisites

Check for recent/common problems¶

Consult the entries below to rule out a known problem.

misc configuration issues¶

libnsl.so.1: cannot open shared object file: No such file or directory

WebSphere documents the "libnsl" package as a prerequisite on recent Linux distributions but it isn't installed. In the past, it was part of the "glibc" package.

The package must be installed. For more info, find the entry for your OS here
"httpd (pid xxxxx) already running"

IHS uses the path specified by PidFile to keep track of the server is running.
- If multiple instances use the same PidFile path, starting the second one will return this error. Specify a unique PidFile in each instance of IHS.
- If IHS is abruptly killed and the process ID inside the PidFile is reused, the PidFile must be removed manually.
"httpd: bad user name "

This error occurs when the name specified by the User directive cannot be looked up on the system. If the ID can be verified with the id unix command, the user may come from a remote database managed by sssd or NSS. Usernames stored in these databases may require additional packages, especially if IHS is 32-bit. For example, users stored in "sssd" require 32-bit package sssd-client to be installed.

Start failure on Windows¶

Antivirus or other endpoint management software may interfere with IHS startup.

In some cases, IHS may start during a reboot but may not be able to manually be restarted.

Review any denials by your third-party security software and add exceptions for apache.exe/httpd.exe.

Startup crash on AIX ¶

On AIX, ulimit -d can limit the max heap size of IHS processes and can cause out-of-memory startup crashes if set to anything less than hundreds of megabytes.

IHS 9.0 raises the soft limit for ulimit -d to unlimited if the hard limit is unlimited, but does not automatically raise any other combination of limits.

The recommended value is unlimited.

Possible locations of a crash can be in SSL initialization.

Failure with spaces in install path ¶

The default configuration installed by IHS 9.0.0.0 will fail at startup if there are spaces in the installation path. Installing directly to any later maintenance level avoids the issue.

To correct the configuration, find the DirectoryMatch configuration section in httpd.conf to wrap the regular expression in quotes:

-<DirectoryMatch ^c:/Program Files/.../htdocs/$>
+<DirectoryMatch "^c:/Program Files/.../htdocs/$">

Startup hangs on z/Linux ¶

PI90141 (8.5.5.13, 9.0.0.6) is needed to resolve some SSL-related startup hangs on Z14-or-later z/Linux systems.

Problems from WAS admin console / wsadmin¶

Inability to start/stop from the WAS admin console (or wsadmin) can happen for a number of reasons.

Check the deployment managers server.xml entry for the webserver and make sure that there is one <stopArgs> and <startArgs> per command-line argument, and not a single space-delimited entry. Prior to IHS 9.0, a bin/apachectl defect meant this kind of bad tokenization would be tolerated (at the cost of having no control over the tokenization of arguments)

If the start or stop command is modified in the console, separate each argument with a newline. If they are modified with wsadmin, separate them with a semicolon.

'undefined symbol: ap_unixd_setup_child'¶

This error can occur if mod_unixd is not loaded in IHS V9. Consult conf/httpd.conf.default .

Since the default configuration for IHS V9 loads this module, this error will generally only happen when a configuration file from a previous version is used with IHS V9. It's not recommended to start with a verbatim copy of a previous major versions configuration file, but if you do, some details are available here: http://publib.boulder.ibm.com/httpserv/manual24/upgrading.html

SSL0263W after graceful reload¶

SSL0263W can be issued if mod_ibm_ssl goes from being not loaded (via LoadModule) to being loaded across an graceful restart. While in general modules may be removed or added during a graceful restart, mod_ibm_ssl cannot support this sequence.

SSL0166E using IHS archive install on Windows ¶

The IHS archive install on windows depends on a Microsoft package named Visual C++ Redistributable Packages for Visual studio. While the IIM installer has always checked for this package, the archive installer has no opportunity to check.

If the dependency is missing, IHS will fail with SSL0166E and tools like bin\gskcapicmd.bat will generate a popup error complaining about msvcr120.dll.

In 9.0.5.6 and later, both the IIM and Archive installs install the dependency automatically.

https://www.ibm.com/docs/en/ibm-http-server/9.0.5?topic=manager-dependencies-required-by-http-server-windows

Hangs on Fujitsu SPARC-X systems¶

On some Solaris/SPARC systems manufactured by Fujitsu, IBM Global Security Kit (GSKit) can hang during startup. To resolve the issue, take either of the following two steps:

Upgrade IBM Global Security Kit (GSKit) to 8.0.50.88 or later (PI94222 in 9.0.0.8, 8.5.5.14).
Or complete these two steps:
1. Upgrade IBM Global Security Kit (GSKit) to 8.0.50.61 or later (8.5.5.10, 8.0.0.13, or older fixpacks with PI60207)
2. Specify SSLAttributeSet 4007 1 alongside each 'SSLEnable' to prevent the FIPS certified ICC 8.4 from being loaded. This is similar to removing the 'C/' directory but preferred.
  
  Note: If using SSLFIPSEnable, you must also opt-in to using a non-certified cryptograhic module with environment variable ICC_IGNORE_FIPS=YES

8.5.5.4/8.5.5.5 startup failures on Solaris/SPARC with 32-bit IHS¶

On some Solaris/SPARC systems, 32-bit IHS systems fail to start after moving to 8.5.5.4/8.5.5.5. This is resolved with the IBM Global Security Kit (GSKit) update provided in PI36417 (8.0.50.41).

startup errors referencing apr_thread_create()¶

See the dedicated topic here: apr_thread_create.html

Slow startup (or gskcapicmd) with SSL enabled on virtual systems¶

This can happen prior to 8.0.50.13/7.0.4.47. To resolve, set export ICC_IGNORE_FIPS=YES in bin/envvars if not using SSLFIPSEnable.

Startup hangs with mod_proxy_balancer¶

mod_proxy_balancer requires 2 bytes of random data during start/restart which will result in a blocking read of /dev/random. On some virtualized systems, or systems where some other process is exhausting /dev/random, startup may hang until enough entropy can be gathered by the system.

A quick measure of /dev/random response time and variance is the following shell command:

while true; do /usr/bin/time dd if=/dev/urandom of=- bs=2 count=1 2>&1|grep -i Real; sleep 2; done

Consult your operating system vendor if /dev/random response time is an issue for IHS startup, or disable mod_proxy_balancer.

./adminctl stop: admin http (no pid file)¶

In V8R5 and earlier, the adminctl script must be updated if the PidFile in conf/admin.conf is changed. The normal apachectl does not maintain a separate definition of the PID file.

64-bit IHS on Solaris fails to start with Siteminder¶

When running 64-bit IHS on Solaris, IHS cannot be launched if the environment variable LD_LIBRARY_PATH_64 has been set prior to running apachectl and it was not setup to include the $IHSROOT/lib and $IHSROOT/gsk8/lib.

Failures with siteminder on AIX.¶

Siteminder customization to IHS startup scripts can cause startup failures.

Siteminder configuration often adds a file named ca_wa_env.sh to $IHSROOT/bin/envvars or $IHSROOT/bin/apachectl.

This script often modifies the LIBPATH environment variable. It must not prepend any items to the LIBPATH, only append. Particularly risky items prepended to the LIBPATH include e.g. /opt/freeware or /lib (the latter was observed due to an unresolved variable such as ${FOO}/lib).
Siteminder can crash during startup, without providing much feedback, without AIX APAR IV08416 on AIX 6.1 and later (IV16240 for AIX 7.1)

If the APAR cannot be applied, this test/workaround confirms it's the same issue:
```
# in bin/envvars
TZ=America/New_York
export TZ 
```

Various SSL errors¶

SSL0114E: Initialization error, Error processing cryptography.

May occur under the following circumstances:
- If the $IHSROOT/gsk8 directory has been unsafely restored from a backup in a way that results in content from two different IHS fix packs being interleaved (rather than replaced as any fixpack upgrade would naturally do). If this is the case $IHSROOT/bin/gskver will report some "ERROR" entries referencing the included ICC libraries.
- Windows only: If GSKit files have been copied into any directory in the system-wide PATH or LIBPATH environment variables.
- If your KeyFile has an empty password. You must set a non-null password for your KeyFile in Ikeyman.
SSL0115E: Initialization error, Error validating ASN fields in certificate

In IHS v8 or later, with IBM Global Security Kit (GSKit) 8.0.14.11 or earlier, this misleading error can be issued when a KDB stash file is corrupted or has been overwritten by $IHSROOT/bin/sslstash. After IBM Global Security Kit (GSKit) 8.0.14.11, this will be reported as "SSL0104E: GSK could not initialize, Invalid password for keyfile.".

If you're using PKCS11 cryptographic offload, re-stash your secondary KDB password with Ikeyman, and choose a new output location for the $IHSROOT/bin/sslstash command and SSLSTashFile directive.

WebSphere Plugin log has grown to 2GB¶

Truncate or delete the WebSphere Plugin log (http_plugin.log) if it has reached 2GB in size. Does not occur on modern maintenance levels

Module ordering problems¶

Error message referring to the text "ap_cache_cacheable_hdrs_out" when starting IHS.
- Make sure the LoadModule directive for mod_cache is un-commented and precedes the LoadModule directive for mod_mem_cache.
Error message referring to the text "ap_proxyerror" or "proxy_hook_scheme_handler".
- Make sure the LoadModule directive for mod_proxy is un-commented and precedes the LoadModule directive for mod_proxy_http, mod_proxy_connect, and mod_proxy_ftp.

(13)Permission denied: apr_proc_detach failed¶

/dev/null must be writable by the userid that runs apachectl. Generally, /dev/null is world-writable by default.

error invoking `httpd ¶

The httpd command cannot be used directly because library paths must be set up beforehand so that support libraries are found in the IHS lib directory. The apachectl command, in conjunction with the IHSROOT/bin/envvars file, sets up the library paths as necessary.

Depending on what other libraries are installed on the system globally, you may see a variety of errors referencing "missing" libraries or missing symbols within loaded libraries.

On Linux, you can get this same symptom while using apachectl if you make $IHS_ROOT/bin/httpd setuid or setgid because LD_LIBRARY_PATH will be ignored by the operating system. Making IHS binaries setuid or setgid is not supported.

Also on Linux, A similiar limitation exists if "setcap" is run against the httpd binary -- LD_LIBRARY_PATH is ignored by the operating system. This is not compatible with the build/packaging design of IHS. This is not a supported configuration

On AIX, if the httpd binary has been granted extended RBAC access with setsecattr, IHS will not start because the operating system ignores the LIBPATH environment variable. This is not a supported configuration.

`(98)Address already in use: make_sock: could not bind to address [::]:80`¶

# apachectl start
(98)Address already in use: make_sock: could not bind to address [::]:80
no listening sockets available, shutting down
Unable to open logs

Check for overlapping Listen directives in httpd.conf.

Ensure that the port specified in the error message isn't already in use. netstat or lsof can be used for this.

    $ netstat -an | grep :80
    tcp        0      0 0.0.0.0:80                  0.0.0.0:*     LISTEN

    # lsof -ni :80
    COMMAND   PID   USER   FD   TYPE   DEVICE SIZE NODE NAME
    httpd   25810   root    3u  IPv4 53644245       TCP *:http (LISTEN)
    httpd   25812 nobody    3u  IPv4 53644245       TCP *:http (LISTEN)
    httpd   25814 nobody    3u  IPv4 53644245       TCP *:http (LISTEN)
    httpd   25815 nobody    3u  IPv4 53644245       TCP *:http (LISTEN)

Possible initialization problems which can leave an httpd process running include:

In IHS configurations where mod_cgid is loaded, some types of crashes during initialization can leave an httpd process running. In these cases, the stranded httpd process will be running as the non-root userid specified by the User directive and will have a parent pid of 1. This process should be removed with kill -TERM.
The WebSphere plugin logfile specified in the plugin-cfg.xml may have grown to 2 gigabytes, leading to a crash during initialization. If the log file is 2GB or greater in size, remove the log file, terminate any stranded httpd processes, and re-start IHS.

[crit] (78)Connection timed out: alloc_listener: failed to set up sockaddr for ::¶

/usr/IBMHttpServer2/bin#./apachectl start            
[Thu Jul 22 23:47:15 2004] [crit] (78)Connection timed out: alloc_listener: failed to set up sockaddr for ::                  
Syntax error on line 130 of /usr/IBMHttpServer2/conf/httpd.conf:  
Listen setup failed

The text of the message can also be "A remote host did not respond within the timeout period." instead of "Connection timed out".

This is caused by a failure of the AIX resolver function getaddrinfo() when passed certain parameters. It can be avoided by disabling IPv6 listening sockets in IHS. That is done by changing a Listen directive such as

    Listen 80

to

    Listen 0.0.0.0:80

AIX APAR IY57293 resolves this problem for AIX 5.2. AIX APAR IY57365 is the equivalent for AIX 5.1.

If startup or shutdown is very slow (IHS on AIX)¶

This may be caused by IPv6 resolver lookups. Try this possible solution, which is to skip IPv6 lookups for IHS.

Add this to the end of ihsroot/bin/envvars:

  NSORDER=local4,bind4
  export NSORDER

Now try to start IHS and see if the problem is resolved.

IHS won't start with SSL and Siteminder ¶

Apply PM11586 in 6.1.0.33 and later if Siteminder won't start with SSL enabled.

IHS won't start from Start menu shortcut icon (IHS 6.1 on 64-bit Windows)¶

After installing IBM HTTP Server 6.1 onto a Windows 64-bit OS (Windows 2003 or Windows 2008), the web server may not start
when using the 'Start HTTP Server' shortcut icon on the 'Start' menu.
You may also see a message displayed that is similar to the following:

"The item 'apache.exe' that this shortcut refers to has been
changed or moved, so this shortcut will no longer work properly."

When IBM HTTP Server 6.1 is installed to the default location under C:\Program Files on a 64-bit Windows OS, the shortcut icons are created incorrectly.

When this occurs, the icons are created with values such as:

      Target:    "C:\Program Files(x86\IBM\HTTPServer,\)\bin\apache.exe" -w -n  "IBM HTTP Server 6.1" -k start
      Start in:  "C:Program Files(x86/IBM/HTTPServer,\)\bin"

There are several problems in these strings:

The right parentheses for (x86 is in the wrong position
There typically should be a space between 'Program Files' and '(x86)'
The ,\ after 'HTTPServer' should not be present
The 'Start in:' value should have a forward slash after the C:

These can be resolved by editing the values as follows:

Select 'Start->All Programs->IBM HTTP Server V6.1' and right-click 'Start HTTP Server'
Select 'Properties'
Select the 'Shortcut' tab page if not there
Edit both the 'Target:' and 'Start in:' values.
Assuming the original values looked identical to the examples above, then you would fix them by editing to look like:
```
Target:    "C:\Program Files (x86)\IBM\HTTPServer\bin\apache.exe" -w -n  "IBM HTTP Server 6.1" -k start
Start in:  "C:/Program Files (x86)/IBM/HTTPServer/bin"
```
If your actual paths are different, then adjust as needed. You simply need to edit them to be valid paths.
(The Start in: path should work with either forward slashes or backslashes.)
Repeat the above for the 'Stop HTTP Server' shortcut.

You should also check the following items. These may have similar path problems that need to be corrected. Edit them to be valid paths as needed.
(The base part of the values should look similar to the good values shown above.)

httpd.conf file entries:
- 'ServerRoot'
- 'DocumentRoot'
- The Plugin entries
Paths within the Plugin config file
Paths within the Admin Console config

Additional information can be found in the 'Restriction' section of the Installing Web server plug-ins InfoCenter page.

Since the IHS problem is in the install of the base 6.1 package, it cannot be resolved via a fixpack.
The solution is to perform the edits described above, or to install to a path that doesn't contain parentheses in the path string.
The problem does not occur with the newer IBM HTTP Server V7.

[error] SSL0157E: Initialization error, The function call, gsk_attrib_set_buffer(env_handle, GSK_STRICTCHECK_CBCPADBYTES, 1, has an invalid ID¶

This can occur at startup for IHS 6.1.0.47 if the PI31516 interim fix is applied without first upgrading IBM Global Security Kit (GSKit) to a minimum required version as described in the Prerequisites section of the PI31516 interim fix document.

Upgrade the global IBM Global Security Kit (GSKit) as specified to resolve the problem.
This can also occur on 6.1 on Windows if an older IBM Global Security Kit (GSKit) is installed and set to be preferred globally via the PATH environment variable.
This can occur on 7.0 or later on Unix if $IHSROOT/bin/envvars is tampered with or LD_LIBRARY_PATH/LIBPATH is manipulated to find a global IBM Global Security Kit (GSKit) installation from another product.

httpd (pid ...) already running¶

This can occur if any of the following statements is true:

IHS is already running.

Solution: No action required.
A stale httpd.pid file exists in the logs directory, and the PID is already being used by another process. This can happen if IHS was not gracefully shutdown.

Solution: Remove the stale httpd.pid file.
(Linux only) A stale httpd.pid file exists in the logs directory, and the PID is already being used by a Light Weight Process (LWP). This can happen if IHS was not gracefully shutdown. LWP PIDs can be viewed using the command: ps -efL

Solution: Remove the stale httpd.pid file.

apachectl reports "Argument list too long" on RHEL 7 after upgrading kernel to 3.10.0-514.21.2.el7¶

Users that upgrade their RHEL kernel to various 3.10.0 releases will see apachectl fail and report an error similar to the following:

/opt/IBM/HTTPServer/bin/apachectl: line 83: /opt/IBM/HTTPServer/bin/httpd: Argument list too long

The failure is because RHEL no longer allows a small (< 4.5M MB) max stack size limit to be set according to Bug 1463241 from Red Hat. IHS sets the maximum stack size in bin/envvars to 512 KB in 9.0 and above to accomodate customers that configure IHS to use a large number of threads.

The workaround for this issue is to either increase the limit of or to comment out "ulimit -s" set in bin/envvars. If the max stack size limit is increased, it should be set to at least 4200 KB.

CreateThread failed. Unable to create all worker threads. Created X of the Y threads requested with the ThreadsPerChild configuration directive. [Windows only]¶

IHS versions 9.0.0.4 through 9.0.0.6 on Windows may be unable to start up if the server has a high number of threads (> 1500) configured. Users affected by this issue will have an error similar to the following in their error log:

[mpm_winnt:crit] [pid PID:tid TID (OS 8)Unrecognized Win32 error code 8: AH00355: Child: CreateThread failed. Unable to create all worker threads. Created X of the Y threads requested with the ThreadsPerChild configuration directive.

An indication that a user is hitting this issue is that their configuration used to worked on versions 9.0.0.0 through 9.0.0.3 and then started to fail to start up with the above error after upgrading to an affected fixpack.

Users of httpd-la.exe on IHS releases 8.0 and 8.5.5 will also likely hit this issue if they are migrating to an affected 9.0 fixpack.

Solution: Upgrade IHS to 9.0.0.7 or later or apply the iFix for PI87046.

Determine a minimal configuration which has the problem¶

There are several features which commonly are related to startup problems. Determination of which feature, if any, triggers the startup problem will result in faster diagnosis of the problem. These features are:

third-party modules
WebSphere plug-in
mod_ibm_ssl
mod_ibm_ldap

Additionally, the use of https transports in the WebSphere plug-in configuration may trigger the problem.

To find the minimal configuration, disable all of these features. If the problem still occurs, it is not triggered by any of these features, but leave them disabled anyway to keep the configuration simpler. If the problem does not occur, then enable one of these features at a time to find the trigger. When enabling the WebSphere plug-in, try it with and without https transports defined, in order to determine if the use of SSL is the trigger.

If the problem does not occur without a third-party module loaded, contact the vendor of that module for further diagnosis.

Turn on detailed logging until the problem is resolved¶

IBM HTTP Server¶

Edit the configuration file (usually httpd.conf) and set the value of the LogLevel directive to debug. If SSL support is enabled, add the SSLTrace directive to the bottom of the configuration file.

WebSphere plug-in¶

Edit the configuration file (usually plugin-cfg.xml) and change LogLevel to Trace. For example:

<Log LogLevel="Trace" Name="/opt/IBM/WebSphere/AppServer/logs/http_plugin.log"/>

Gathering traces, console messages, and coredumps¶

Make sure coredumps are allowed:

# ulimit -c unlimited

If the problem is related to SSL, set the following IBM Global Security Kit (GSKit) trace environment variables via <ihsinst>/bin/envvars or the windows control panel:

GSK_TRACE_FILE_NUMBER=1
export GSK_TRACE_FILE_NUMBER

# This following setting should only be used for suspected crashes
GSKTRACE_NOBUFFERING=YES
export GSKTRACE_NOBUFFERING

# 250MB
GSK_TRACE_FILE_SIZE=262144000
export GSK_TRACE_FILE_SIZE

Start IHS to create the failure:

# /path/to/apachectl start

(If necessary, substitute your normal way of starting IBM HTTP Server. Note that apachectl is the only supported mechanism for starting IHS 2.0 and above on Unix and Linux systems.)

Cut and paste the actual command used to startup IHS, as well as any messages which appeared on the console, into a file.

Check for a coredump. The location can vary by system configuration and platform. Check in the current directory for a file called core* or search the system for a file called core* and with a current timestamp. On Solaris, run the coreadm command to see if an alternate location or filename pattern has been specified for coredumps. On AIX 5.2 and above, run syscorepath -g to see if an alternate location has been specified for coredumps.

If a coredump was generated, follow the instruction here to extract information from it.

Windows specifics¶

When IHS is run as a service on Windows, special considerations needs to be taken to troubleshoot service startup problems.

After failed startup, capture the event viewer application logs: wevtutil.exe qe Application
Try starting IHS (httpd.exe) from a cmd.exe window
- Without the -k flag, but with -e DEBUG added to allow it to run in the foreground
- It may issue startup messages in the terminal that indicate the area of a problem
- It may work when in the foreground which implies a problem with the service definition itself.
- If startup does NOT fail, the foreground process should be interrupted with Control-C and any remaining httpd processes should be killed with task manager or pskill.

Unix: Get syscall trace of startup¶

Last, get a syscall trace of startup to show what processing occurred before the failure. The following table shows the command to use for your platform.

Platform	Command
AIX 5.2 and above	`truss -o /tmp/ihs_startup -d -f startup-command`
Linux	`strace -v -s1024 -o /tmp/ihs_startup -f startup-command`
Solaris	`truss -o /tmp/ihs_startup -f startup-command`
HP-UX	`tusc -o /tmp/ihs_startup -f startup-command`

Before running this, unset the environment variables GSK_TRACE_FILE and GSKTRACE_NOBUFFERING. Replace startup-command with the command used to start IHS. After the startup failure occurs, the syscall trace will be in file /tmp/ihs_startup.

Sending information to IBM¶

The information to send to IBM includes:

All IHS configuration files.
IHS error_log
System Call trace / Windows event viewer logs if collected
GSkit trace, if collected
copy/paste of any messages issued to the console, dmesg, syslog, etc during the failure.