# Startup mustgather ## Introduction IBM HTTP Server (IHS) customers occasionally encounter problems starting the web server. An error message may be displayed on the terminal, or the initial httpd process may crash, or for some other reason the web server fails at startup and does not create child processes to handle client connections. When this type of problem occurs, the documentation required to diagnose the cause includes - operating system, IHS, and WebSphere plug-in configuration information - any messages written to the terminal during startup - any messages written to the IHS error log or WebSphere plug-in log during startup - results of trying to start IHS with/without the WebSphere plug-in - results of trying to start IHS with/without SSL Some of this information can be gathered automatically. Other information is gathered manually. ## Ensure that you are using a supported operating system level Missing operating system prerequisites frequently cause startup failures. Consult this information to find the required operating system levels: [Check for operating system prerequisites](prepare.html#prereq) ## Check for recent/common problems Consult the entries below to rule out a known problem. ### misc configuration issues 1. libnsl.so.1: cannot open shared object file: No such file or directory WebSphere documents the "libnsl" package as a prerequisite on recent Linux distributions but it isn't installed. In the past, it was part of the "glibc" package. The package must be installed. For more info, find the entry for your OS [here](https://www.ibm.com/support/knowledgecenter/SSEQTP_9.0.5/com.ibm.websphere.installation.base.doc/ae/tins_linuxsetup.html) 2. "httpd (pid xxxxx) already running" IHS uses the path specified by `PidFile` to keep track of the server is running. - If multiple instances use the same `PidFile` path, starting the second one will return this error. Specify a unique `PidFile` in each instance of IHS. - If IHS is abruptly killed and the process ID inside the `PidFile` is reused, the `PidFile` must be removed manually. 3. "httpd: bad user name " This error occurs when the name specified by the `User` directive cannot be looked up on the system. If the ID can be verified with the `id` unix command, the user may come from a remote database managed by `sssd` or `NSS`. Usernames stored in these databases may require additional packages, especially if IHS is 32-bit. For example, users stored in "sssd" require 32-bit package sssd-client to be installed. ### Start failure on Windows Antivirus or other endpoint management software may interfere with IHS startup. In some cases, IHS may start during a reboot but may not be able to manually be restarted. Review any denials by your third-party security software and add exceptions for apache.exe/httpd.exe. ### Startup crash on AIX On AIX, `ulimit -d` can limit the max heap size of IHS processes and can cause out-of-memory startup crashes if set to anything less than hundreds of megabytes. IHS 9.0 raises the soft limit for `ulimit -d` to unlimited if the hard limit is unlimited, but does not automatically raise any other combination of limits. The recommended value is unlimited. Possible locations of a crash can be in SSL initialization. ### Failure with spaces in install path The default configuration installed by IHS 9.0.0.0 will fail at startup if there are spaces in the installation path. Installing directly to any later maintenance level avoids the issue. To correct the configuration, find the `DirectoryMatch` configuration section in httpd.conf to wrap the regular expression in quotes: ``` - + ``` ### Startup hangs on z/Linux PI90141 (8.5.5.13, 9.0.0.6) is needed to resolve some SSL-related startup hangs on Z14-or-later z/Linux systems. ### Problems from WAS admin console / wsadmin Inability to start/stop from the WAS admin console (or wsadmin) can happen for a number of reasons. - Check the deployment managers server.xml entry for the webserver and make sure that there is one \ and \ per command-line argument, and not a single space-delimited entry. Prior to IHS 9.0, a bin/apachectl defect meant this kind of bad tokenization would be tolerated (at the cost of having no control over the tokenization of arguments) If the start or stop command is modified in the console, separate each argument with a newline. If they are modified with wsadmin, separate them with a semicolon. ### 'undefined symbol: ap\_unixd\_setup\_child' This error can occur if `mod_unixd` is not loaded in IHS V9. Consult conf/httpd.conf.default . Since the default configuration for IHS V9 loads this module, this error will generally only happen when a configuration file from a previous version is used with IHS V9. It's not recommended to start with a verbatim copy of a previous major versions configuration file, but if you do, some details are available here: ### SSL0263W after graceful reload SSL0263W can be issued if mod\_ibm\_ssl goes from being not loaded (via `LoadModule`) to being loaded across an *graceful* restart. While in general modules may be removed or added during a graceful restart, mod\_ibm\_ssl cannot support this sequence. ### SSL0166E using IHS archive install on Windows The IHS archive install on windows depends on a Microsoft package named Visual C++ Redistributable Packages for Visual studio. While the IIM installer has always checked for this package, the archive installer has no opportunity to check. If the dependency is missing, IHS will fail with SSL0166E and tools like `bin\gskcapicmd.bat` will generate a popup error complaining about msvcr120.dll. In 9.0.5.6 and later, both the IIM and Archive installs install the dependency automatically. [https://www.ibm.com/docs/en/ibm-http-server/9.0.5?topic=manager-dependencies-required-by-http-server-windows](https://www.ibm.com/docs/en/ibm-http-server/9.0.5?topic=manager-dependencies-required-by-http-server-windows) ### Hangs on Fujitsu SPARC-X systems On some Solaris/SPARC systems manufactured by Fujitsu, IBM Global Security Kit (GSKit) can hang during startup. To resolve the issue, take either of the following two steps: - Upgrade IBM Global Security Kit (GSKit) to 8.0.50.88 or later (PI94222 in 9.0.0.8, 8.5.5.14). - Or complete these two steps: 1. Upgrade IBM Global Security Kit (GSKit) to 8.0.50.61 or later (8.5.5.10, 8.0.0.13, or older fixpacks with PI60207) 2. Specify `SSLAttributeSet 4007 1` alongside each 'SSLEnable' to prevent the FIPS certified ICC 8.4 from being loaded. This is similar to removing the 'C/' directory but preferred. Note: If using `SSLFIPSEnable`, you must also opt-in to using a non-certified cryptograhic module with environment variable `ICC_IGNORE_FIPS=YES` ### 8.5.5.4/8.5.5.5 startup failures on Solaris/SPARC with 32-bit IHS On some Solaris/SPARC systems, 32-bit IHS systems fail to start after moving to 8.5.5.4/8.5.5.5. This is resolved with the IBM Global Security Kit (GSKit) update provided in PI36417 (8.0.50.41). ### startup errors referencing apr\_thread\_create() See the dedicated topic here: [apr\_thread\_create.html](apr_thread_create.html) ### Slow startup (or gskcapicmd) with SSL enabled on virtual systems This can happen prior to 8.0.50.13/7.0.4.47. To resolve, set `export ICC_IGNORE_FIPS=YES` in bin/envvars if not using SSLFIPSEnable. ### Startup hangs with mod\_proxy\_balancer `mod_proxy_balancer` requires 2 bytes of random data during start/restart which will result in a blocking read of `/dev/random`. On some virtualized systems, or systems where some other process is exhausting `/dev/random`, startup may hang until enough entropy can be gathered by the system. A quick measure of /dev/random response time and variance is the following shell command: `while true; do /usr/bin/time dd if=/dev/urandom of=- bs=2 count=1 2>&1|grep -i Real; sleep 2; done` Consult your operating system vendor if /dev/random response time is an issue for IHS startup, or disable `mod_proxy_balancer`. ### ./adminctl stop: admin http (no pid file) In V8R5 and earlier, the `adminctl` script must be updated if the `PidFile` in `conf/admin.conf` is changed. The normal `apachectl` does not maintain a separate definition of the PID file. ### 64-bit IHS on Solaris fails to start with Siteminder When running 64-bit IHS on Solaris, IHS cannot be launched if the environment variable `LD_LIBRARY_PATH_64` has been set prior to running apachectl and it was not setup to include the $IHSROOT/lib and $IHSROOT/gsk8/lib. ### Failures with siteminder on AIX. * Siteminder customization to IHS startup scripts can cause startup failures. Siteminder configuration often adds a file named `ca_wa_env.sh` to $IHSROOT/bin/envvars or $IHSROOT/bin/apachectl. This script often modifies the LIBPATH environment variable. It must not **prepend** any items to the LIBPATH, only **append**. Particularly risky items prepended to the LIBPATH include e.g. /opt/freeware or /lib (the latter was observed due to an unresolved variable such as ${FOO}/lib). * Siteminder can crash during startup, without providing much feedback, without AIX APAR IV08416 on AIX 6.1 and later (IV16240 for AIX 7.1) If the APAR cannot be applied, this test/workaround confirms it's the same issue: ``` # in bin/envvars TZ=America/New_York export TZ ``` ### Various SSL errors - *SSL0114E: Initialization error, Error processing cryptography*. In IHS v8 or later, this can occur if your `KeyFile` has an empty password. You must set a non-null password for your KeyFile in Ikeyman. - *SSL0115E: Initialization error, Error validating ASN fields in certificate* In IHS v8 or later, with IBM Global Security Kit (GSKit) 8.0.14.11 or earlier, this misleading error can be issued when a KDB stash file is corrupted or has been overwritten by `$IHSROOT/bin/sslstash`. After IBM Global Security Kit (GSKit) 8.0.14.11, this will be reported as "SSL0104E: GSK could not initialize, Invalid password for keyfile.". If you're using PKCS11 cryptographic offload, re-stash your secondary KDB password with Ikeyman, and choose a new output location for the `$IHSROOT/bin/sslstash` command and `SSLSTashFile` directive. ### WebSphere Plugin log has grown to 2GB Truncate or delete the WebSphere Plugin log (http\_plugin.log) if it has reached 2GB in size. Does not occur on modern maintenance levels ### Module ordering problems - Error message referring to the text "`ap_cache_cacheable_hdrs_out`" when starting IHS. - Make sure the `LoadModule` directive for mod\_cache is un-commented and precedes the `LoadModule` directive for mod\_**mem**\_cache. - Error message referring to the text "`ap_proxyerror`" or "`proxy_hook_scheme_handler`". - Make sure the LoadModule directive for `mod_proxy` is un-commented and precedes the LoadModule directive for mod\_proxy\_http, mod\_proxy\_connect, and mod\_proxy\_ftp. ### (13)Permission denied: apr\_proc\_detach failed /dev/null must be writable by the userid that runs apachectl. Generally, /dev/null is world-writable by default. ### error invoking `httpd The `httpd` command cannot be used directly because library paths must be set up beforehand so that support libraries are found in the IHS `lib` directory. The `apachectl` command, in conjunction with the `IHSROOT/bin/envvars` file, sets up the library paths as necessary. Depending on what other libraries are installed on the system globally, you may see a variety of errors referencing "missing" libraries or missing symbols within loaded libraries. On Linux, you can get this same symptom while using apachectl if you make $IHS\_ROOT/bin/httpd setuid or setgid because LD\_LIBRARY\_PATH will be ignored by the operating system. Making IHS binaries setuid or setgid is not supported. Also on Linux, A similiar limitation exists if "setcap" is run against the httpd binary -- LD\_LIBRARY\_PATH is ignored by the operating system. This is not compatible with the build/packaging design of IHS. This is not a supported configuration On AIX, if the `httpd` binary has been granted extended RBAC access with `setsecattr`, IHS will not start because the operating system ignores the `LIBPATH` environment variable. This is not a supported configuration. ### `(98)Address already in use: make_sock: could not bind to address [::]:80` # apachectl start (98)Address already in use: make_sock: could not bind to address [::]:80 no listening sockets available, shutting down Unable to open logs - Check for overlapping `Listen` directives in httpd.conf. - Ensure that the port specified in the error message isn't already in use. `netstat` or `lsof` can be used for this. ``` $ netstat -an | grep :80 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN # lsof -ni :80 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME httpd 25810 root 3u IPv4 53644245 TCP *:http (LISTEN) httpd 25812 nobody 3u IPv4 53644245 TCP *:http (LISTEN) httpd 25814 nobody 3u IPv4 53644245 TCP *:http (LISTEN) httpd 25815 nobody 3u IPv4 53644245 TCP *:http (LISTEN) ``` Possible initialization problems which can leave an httpd process running include: - In IHS configurations where mod\_cgid is loaded, some types of crashes during initialization can leave an httpd process running. In these cases, the stranded httpd process will be running as the non-root userid specified by the `User` directive and will have a parent pid of 1. This process should be removed with kill -TERM. - The WebSphere plugin logfile specified in the plugin-cfg.xml may have grown to 2 gigabytes, leading to a crash during initialization. If the log file is 2GB or greater in size, remove the log file, terminate any stranded httpd processes, and re-start IHS. ### \[crit\] (78)Connection timed out: alloc\_listener: failed to set up sockaddr for :: ``` /usr/IBMHttpServer2/bin#./apachectl start [Thu Jul 22 23:47:15 2004] [crit] (78)Connection timed out: alloc_listener: failed to set up sockaddr for :: Syntax error on line 130 of /usr/IBMHttpServer2/conf/httpd.conf: Listen setup failed ``` The text of the message can also be "A remote host did not respond within the timeout period." instead of "Connection timed out". This is caused by a failure of the AIX resolver function getaddrinfo() when passed certain parameters. It can be avoided by disabling IPv6 listening sockets in IHS. That is done by changing a Listen directive such as ``` Listen 80 ``` to ``` Listen 0.0.0.0:80 ``` AIX APAR IY57293 resolves this problem for AIX 5.2. AIX APAR IY57365 is the equivalent for AIX 5.1. ### If startup or shutdown is very slow (IHS on AIX) This may be caused by IPv6 resolver lookups. Try this possible solution, which is to skip IPv6 lookups for IHS. - Add this to the end of ihsroot/bin/envvars: ``` NSORDER=local4,bind4 export NSORDER ``` - Now try to start IHS and see if the problem is resolved. ### [IHS won't start with SSL and Siteminder](SSLSITEMINDER) Apply PM11586 in 6.1.0.33 and later if Siteminder won't start with SSL enabled. ### IHS won't start from Start menu shortcut icon (IHS 6.1 on 64-bit Windows) After installing IBM HTTP Server 6.1 onto a Windows 64-bit OS (Windows 2003 or Windows 2008), the web server may not start when using the `'Start HTTP Server'` shortcut icon on the `'Start'` menu. You may also see a message displayed that is similar to the following: "The item 'apache.exe' that this shortcut refers to has been changed or moved, so this shortcut will no longer work properly." When IBM HTTP Server 6.1 is installed to the default location under `C:\Program Files ` on a 64-bit Windows OS, the shortcut icons are created incorrectly. When this occurs, the icons are created with values such as: ``` Target: "C:\Program Files(x86\IBM\HTTPServer,\)\bin\apache.exe" -w -n "IBM HTTP Server 6.1" -k start Start in: "C:Program Files(x86/IBM/HTTPServer,\)\bin" ``` There are several problems in these strings: - The right parentheses for (x86 is in the wrong position - There typically should be a space between 'Program Files' and '(x86)' - The `,\ ` after 'HTTPServer' should not be present - The '`Start in:`' value should have a forward slash after the `C:` These can be resolved by editing the values as follows: - Select '`Start->All Programs->IBM HTTP Server V6.1`' and right-click '`Start HTTP Server`' - Select 'Properties' - Select the 'Shortcut' tab page if not there - Edit both the '`Target:`' and '`Start in:`' values. Assuming the original values looked identical to the examples above, then you would fix them by editing to look like: ``` Target: "C:\Program Files (x86)\IBM\HTTPServer\bin\apache.exe" -w -n "IBM HTTP Server 6.1" -k start Start in: "C:/Program Files (x86)/IBM/HTTPServer/bin" ``` If your actual paths are different, then adjust as needed. You simply need to edit them to be valid paths. (The `Start in:` path should work with either forward slashes or backslashes.) - Repeat the above for the `'Stop HTTP Server'` shortcut. You should also check the following items. These may have similar path problems that need to be corrected. Edit them to be valid paths as needed. (The base part of the values should look similar to the good values shown above.) - `httpd.conf` file entries: - '`ServerRoot`' - '`DocumentRoot`' - The Plugin entries - Paths within the Plugin config file - Paths within the Admin Console config Additional information can be found in the 'Restriction' section of the [Installing Web server plug-ins](http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/tins_webplugins.html) InfoCenter page. Since the IHS problem is in the install of the base 6.1 package, it cannot be resolved via a fixpack. The solution is to perform the edits described above, or to install to a path that doesn't contain parentheses in the path string. The problem does not occur with the newer IBM HTTP Server V7. ### \[error\] SSL0157E: Initialization error, The function call, gsk\_attrib\_set\_buffer(env\_handle, GSK\_STRICTCHECK\_CBCPADBYTES, 1, has an invalid ID - This can occur at startup for IHS 6.1.0.47 if the PI31516 interim fix is applied without first upgrading IBM Global Security Kit (GSKit) to a minimum required version as described in the `Prerequisites` section of the [PI31516 interim fix document](http://www-01.ibm.com/support/docview.wss?uid=swg24039197). Upgrade the global IBM Global Security Kit (GSKit) as specified to resolve the problem. - This can also occur on 6.1 on Windows if an older IBM Global Security Kit (GSKit) is installed and set to be preferred globally via the PATH environment variable. - This can occur on 7.0 or later on Unix if $IHSROOT/bin/envvars is tampered with or LD\_LIBRARY\_PATH/LIBPATH is manipulated to find a global IBM Global Security Kit (GSKit) installation from another product. ### httpd (pid ...) already running This can occur if any of the following statements is true: - IHS is already running. **Solution:** No action required. - A stale httpd.pid file exists in the logs directory, and the PID is already being used by another process. This can happen if IHS was not gracefully shutdown. **Solution:** Remove the stale httpd.pid file. - **(Linux only)** A stale httpd.pid file exists in the logs directory, and the PID is already being used by a Light Weight Process (LWP). This can happen if IHS was not gracefully shutdown. LWP PIDs can be viewed using the command: *ps -efL* **Solution:** Remove the stale httpd.pid file. ### apachectl reports "Argument list too long" on RHEL 7 after upgrading kernel to 3.10.0-514.21.2.el7 Users that upgrade their RHEL kernel to various `3.10.0` releases will see apachectl fail and report an error similar to the following: > /opt/IBM/HTTPServer/bin/apachectl: line 83: /opt/IBM/HTTPServer/bin/httpd: Argument list too long The failure is because RHEL no longer allows a small (\< 4.5M MB) max stack size limit to be set according to Bug [1463241](https://bugzilla.redhat.com/show_bug.cgi?id=1463241) from Red Hat. IHS sets the maximum stack size in `bin/envvars` to 512 KB in 9.0 and above to accomodate customers that configure IHS to use a large number of threads. The workaround for this issue is to either increase the limit of or to comment out "ulimit -s" set in `bin/envvars`. If the max stack size limit is increased, it should be set to at least 4200 KB. ### CreateThread failed. Unable to create all worker threads. Created X of the Y threads requested with the ThreadsPerChild configuration directive. \[Windows only\] IHS versions 9.0.0.4 through 9.0.0.6 on Windows may be unable to start up if the server has a high number of threads (\> 1500) configured. Users affected by this issue will have an error similar to the following in their error log: [mpm_winnt:crit] [pid PID:tid TID (OS 8)Unrecognized Win32 error code 8: AH00355: Child: CreateThread failed. Unable to create all worker threads. Created X of the Y threads requested with the ThreadsPerChild configuration directive. An indication that a user is hitting this issue is that their configuration used to worked on versions 9.0.0.0 through 9.0.0.3 and then started to fail to start up with the above error after upgrading to an affected fixpack. Users of `httpd-la.exe` on IHS releases 8.0 and 8.5.5 will also likely hit this issue if they are migrating to an affected 9.0 fixpack. **Solution:** Upgrade IHS to 9.0.0.7 or later or apply the iFix for PI87046. ## Determine a minimal configuration which has the problem There are several features which commonly are related to startup problems. Determination of which feature, if any, triggers the startup problem will result in faster diagnosis of the problem. These features are: 1. third-party modules 2. WebSphere plug-in 3. mod\_ibm\_ssl 4. mod\_ibm\_ldap Additionally, the use of *https* transports in the WebSphere plug-in configuration may trigger the problem. To find the minimal configuration, disable all of these features. If the problem still occurs, it is not triggered by any of these features, but leave them disabled anyway to keep the configuration simpler. If the problem does not occur, then enable one of these features at a time to find the trigger. When enabling the WebSphere plug-in, try it with and without *https* transports defined, in order to determine if the use of SSL is the trigger. If the problem does not occur without a third-party module loaded, contact the vendor of that module for further diagnosis. ## Turn on detailed logging until the problem is resolved ### IBM HTTP Server Edit the configuration file (usually `httpd.conf`) and set the value of the `LogLevel` directive to `debug`. If SSL support is enabled, add the `SSLTrace` directive to the bottom of the configuration file. ### WebSphere plug-in Edit the configuration file (usually `plugin-cfg.xml`) and change `LogLevel` to `Trace`. For example: ## Gathering traces, console messages, and coredumps Make sure coredumps are allowed: # ulimit -c unlimited If the problem is related to SSL, set the following [IBM Global Security Kit (GSKit) trace](gather_certificate_doc.html#gskit_trace) environment variables via \/bin/envvars or the windows control panel: GSK_TRACE_FILE_NUMBER=1 export GSK_TRACE_FILE_NUMBER # This following setting should only be used for suspected crashes GSKTRACE_NOBUFFERING=YES export GSKTRACE_NOBUFFERING # 250MB GSK_TRACE_FILE_SIZE=262144000 export GSK_TRACE_FILE_SIZE Start IHS to create the failure: # /path/to/apachectl start (If necessary, substitute your normal way of starting IBM HTTP Server. Note that `apachectl` is the only supported mechanism for starting IHS 2.0 and above on Unix and Linux systems.) Cut and paste the actual command used to startup IHS, as well as any messages which appeared on the console, into a file. Check for a coredump. The location can vary by system configuration and platform. Check in the current directory for a file called `core*` or search the system for a file called `core*` and with a current timestamp. On Solaris, run the `coreadm` command to see if an alternate location or filename pattern has been specified for coredumps. On AIX 5.2 and above, run `syscorepath -g` to see if an alternate location has been specified for coredumps. If a coredump was generated, follow the instruction [here](gather_crash_doc.html) to extract information from it. ### Windows specifics When IHS is run as a service on Windows, special considerations needs to be taken to troubleshoot service startup problems. 1. After failed startup, capture the event viewer application logs: `wevtutil.exe qe Application` 2. Try starting IHS (httpd.exe) from a cmd.exe window - Without the -k flag, but with `-e DEBUG` added to allow it to run in the foreground - It may issue startup messages in the terminal that indicate the area of a problem - It may work when in the foreground which implies a problem with the service definition itself. - If startup does NOT fail, the foreground process should be interrupted with Control-C and any remaining httpd processes should be killed with task manager or pskill. ### Unix: Get syscall trace of startup Last, get a syscall trace of startup to show what processing occurred before the failure. The following table shows the command to use for your platform.
Platform Command
AIX 5.2 and above truss -o /tmp/ihs_startup -d -f startup-command
Linux strace -v -s1024 -o /tmp/ihs_startup -f startup-command
Solaris truss -o /tmp/ihs_startup -f startup-command
HP-UX tusc -o /tmp/ihs_startup -f startup-command
Before running this, unset the environment variables `GSK_TRACE_FILE` and `GSKTRACE_NOBUFFERING`. Replace `startup-command` with the command used to start IHS. After the startup failure occurs, the syscall trace will be in file `/tmp/ihs_startup`. ### Sending information to IBM The information to send to IBM includes: * All IHS configuration files. * IHS error_log * System Call trace / Windows event viewer logs if collected * GSkit trace, if collected * copy/paste of any messages issued to the console, dmesg, syslog, etc during the failure.