Reference: Error log messages

Frequent messages requiring attention

unable to init Zlib: ...

This message generally means that a memory allocation has failed. With IHS, this most commonly happens when some memory limit has been exhausted:

In 32-bit IHS, this is often the total addressable process memory. In more obscure cases, memory may be limited per-process by the system configuration.

A variety of messages with similar repercussions/solutions may occur in the http_plugin.log and share the common phrase "... Failed to allocate...".

• ERROR: lib_htresponse: htresponseGetContentBlock: Failed to allocate the content block

• ERROR: lib_htresponse: htresponseGetChunk: Failed to allocate the chunk

• ERROR: lib_htrequest: htrequestWrite: Failed to allocate memory

• ERROR: as_handler: failed to create pool

Solving the problem

Windows

Increasing ThreadsPerChild beyond even 1000 can result in running out of memory with a 32-bit IHS on Windows. Each thread takes up significant virtual adrress space, even moreso on 64-bit Windows.

• IHS 8.5.5

  • Re-installing the IHS Windows Service with httpd-la.exe instead of httpd.exe will opt-in to using a "large address aware" runtime that nearly doubles the available virtual address space size.

• IHS 9.0

  • IHS 9.0 uses the "large address aware" option by default.

  • The non-IBM Installation Manager IHS Archive install allows a 64-bit IHS to be installed.

If the options above are not viable or not successful:

• Avoid features that require higher memory use such as a large MCacheSize, ESI caching, and mod_deflate.

• Reduce ThreadsPerChild if mod_mpmstats output shows it is oversied.

Linux, AIX, Solaris, z/OS, HP-UX

Run apachectl -V|grep Architecture to determine if IHS is 32-bit or 64-bit

• If using a 32-bit webserver, try the following options in order:

  • Move to 64-bit.

  • Reduce ThreadsPerChild and increase ServerLimit to use fewer memory per-process.

  • Avoid features that require higher memory use such as large MCacheSize

• If using a 64-bit server, ensure no system-wide setting limits the address space size of each process.

  • IHS on AIX ships with such an artificial limit, see gather_crash_doc.html#aixmaxdata

[alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread

Also: [alert] (12)Cannot allocate memory: apr_thread_create: unable to create worker thread

To resolve the crash (usually 32-bit) or high "memory" (really VSZ) symptom, see the dedicated topic here: apr_thread_create.html

[error] server reached MaxClients setting, consider raising the MaxClients setting

At the time the message was written, there were not any worker threads, or child processes in V1.3, available to process additional requests. I.e., all child processes/threads are already in use handling existing requests. The message is only written once during the life of the server, but the peak condition may happen any number of times after the message is logged.

mod_mpmstats for IBM HTTP Server V2.0 and above provides better reporting of this condition; in particular, it can report when the MaxClients condition has been reached as often as once every 90 seconds.

The web server does not take any action when the MaxClients condition is reached. In many cases the condition will be corrected when a temporary application slow-down is resolved and threads, or child processes in V1.3, can finish their current requests and become free to handle new connections. If there is no suspected application slowdown, the customer may need to increase MaxClients to increase the peak capacity of the web server. This type of tuning is described in the IBM HTTP Server Performance Tuning document. As described in that information, the customer can get a good idea of how many of the IBM HTTP Server child processes/threads are in use at a given time by activating mod_status and checking the server-status page, or by monitoring with mod_mpmstats.

This is an important message to watch out for when there is a symptom such as a hung server. In some types of hang conditions (e.g., all IBM HTTP Server child process/threads tied up waiting for the application server to respond, and the application server isn't responding because the application processing the request is hung), this "MaxClients" message will be written to the error log.

In configurations where ThreadsPerChild == MaxClients and a non-zero value is used for MaxRequestsPerChild, this message will be reported between the time that the sole process is exiting due to MaxRequestsPerChild and the time a replacement can be created.

Request processing related messages

[info] (9) bad file number: core_output_filter: writing data to the network.

The significance with this message is the text "bad file number." If it says something like "Broken pipe" or "Connection reset", it is a different issue.

To correct some problems which lead to the "bad file number" message, the customer needs PQ85834 or later for 2.0.42.2, or 2.0.47.

[info] [client 1.2.3.4] (32)Broken pipe: client stopped connection before rflush completed

Messages with similar causes:

• [info] (32)Broken pipe: core_output_filter: writing data to the network

• [info] (73)Connection reset by peer: core_output_filter: writing data to the network

This message is logged when the client closes the connection before IBM HTTP Server responds, or while it is responding.

There are a number of reasons this could happen. Roughly in order of likelihood, some of them are:

• It is to be expected that some subset of clients will drop the connection while IBM HTTP Server is writing to it. The user could have pressed the stop button or closed the browser or there could have been a temporary network problem. If the message is logged infrequently, and users aren't reporting problems, that is the most likely cause, and is not anything to worry about.

• If an application running in WebSphere takes a long time to begin responding to a request, a browser might time out waiting for the response and close the connection. This can happen as quickly as 60 seconds. Then when IBM HTTP Server tries to send the response later, the send will fail with one of the above messages.

  An IP trace and plugin trace should show if this is happening.

  This could be corrected by ensuring the application responds quickly. If that's not always possible, the application can trickle updates back, so the browser knows something is still happening. (How to do that is beyond the scope of IHS support.)

• Another normal condition where this may occur is when a .pdf file is requested from a browser with the Adobe plugin installed. The browser first requests the entire file, then when it sees the type of response the plugin takes over, cancels the initial request (though IBM HTTP Server may not have written the entire response yet), then re*issues the request for the .pdf file in the form of a range request.

• If a connection is abruptly closed this way while the WebSphere Plugin is writing a response, a 400 will be logged. "%X" can be added to the LogFormat directive in use to show these "aborted" connections.

• If a connection is abruptly closed this way while IHS is serving a static file, the original status code (that may or may not have made it to the client, but was written to the IHS hosts TCP stack) will be logged. "%X" can be added to the LogFormat directive in use to show these "aborted" connections.

• A rare condition where these messages may occur is if there is some application layer issue (e.g., browser incompatibility with application response) which leads to the browser abruptly dropping the connection to IBM HTTP Server when it receives certain response data. In order to diagnose the problem, network traces or mod_net_trace are normally needed so that IBM HTTP Server support can see what was sent to the browser. For SSL connections, mod_net_trace is preferred since it traces the data in unencrypted form.

• If you also have 400 errors in the access log around the same time, it's probably another symptom of the same problem. Some versions of IE have a bug when retrying a POST after it times out waiting for a response. IE sends the POST headers again but no body, resulting in a 400 response.

  The Siteminder Proxy Server (SPS) can exhibit the same behavior of retrying a POST without sending the body. Setting the IHS KeepAliveTimeout higher than the SPS connection pool timeout may help to avoid the original "broken pipe" error.

[error] File does not exist: /opt/IBMHTTPServer/htdocs/filename

This message indicates the requested filename, or a directory along the way to the requested filename, does not exist. In the latter case, for IHS 2.x and later, the path displayed stops at the directory name that could not be found.

If a request is intended for the WebSphere Plugin, but isn't handled due to an incomplete plugin-cfg.xml, you might see this message ending in one of your defined application context roots.

[error] (24)Too many open files: file permissions deny server access: /opt/IBMHttpServer/htdocs/en_US/index.html

[error] (24)Too many open files: apr_accept: (client socket)

See Resolving problems with too many open files

[error] [client 1.2.3.4] (11)Resource temporarily unavailable: couldn't spawn child process: /path/to/cgi

This message occurs when the operating system cannot spawn (fork()) another child process for a CGI request. The request will fail with a 500 error.

For more information, refer to fork() failures.

[error] client denied by server configuration: /path

This message is issued when IHS has mapped an incoming request to the filesystem, but IHS has been configured to not allow files from the specified path to be served to clients.

By default, IHS ships with a configuration that doesn't allow files outside of the default document root to be served. When new directories are added into the URL-space via DocumentRoot (possibly in a new VirtualHost) or Alias, a corresponding <Directory> container should be added to the configuation to establish the proper access control.

An example <Directory> container can be viewed in the httpd.conf.default prefixed by the following text:

# This should be changed to whatever you set DocumentRoot to.

Below is an example <Directory> container which allows anyone to access the newly added files:

    <Directory /usr/local/app1>
        Options Indexes FollowSymLinks
        # V855 and earlier:
        Order allow,deny
        Allow from all
        # V9 and later will instead say "Require all granted"
    </Directory>

[error] [client 127.0.0.1] request failed: error reading the headers

This message can be triggered by one of the following protocol errors encountered while reading a request from the client:

• the number of request header fields exceeds the server's limit (400 will be logged in access log)

• the length of a request header field exceeds the server's limit (400 will be logged in access log)

• a request header field is missing a colon separator (400 will be logged in access log)

The exact cause is written in the error message sent to the client, but it is not logged anywhere unless the "error-notes" note for the request is logged via mod_log_config. (This would be rather voluminous and is not recommended except for brief testing periods.) The note can be logged by adding %{error-notes}n to your log format string.

Even without logging the error-notes note, you can consult the access log to see the URL of the request which failed and see what error message was returned to the browser. Look for requests which failed with 400 with the same timestamp as the messages in the error log.

piped log program '(null)' failed unexpectedly

This message can appear with IBM HTTP Server 2.0 and above at shutdown or restart time when piped log programs such as rotatelogs are configured. It does not result in an operational problem.

[client nnn.nnn.nnn.nnn] (13)Permission denied: access to some url failed because search permissions are missing on a component of the path

To find the exact permission problem, first determine which file should be served for the specified url. As an example, if the url is /, this would usually result in serving file index.html in the IBM HTTP Server DocumentRoot directory. With the following configuration in httpd.conf

    DocumentRoot /opt/local/HTTPServer/htdocs/en_US
    ...
    User nobody
    Group nobody

the actual file to be served would be /opt/local/HTTPServer/htdocs/en_US/index.html, and the user/group nobody/nobody must be able to read and search each component of the path up through index.html. So run these commands and verify from the permissions of each component of the path that user/group nobody can read and search that component.

    ls -ld /
    ls -ld /opt
    ls -ld /opt/local
    ls -ld /opt/local/HTTPServer
    ls -ld /opt/local/HTTPServer/htdocs
    ls -ld /opt/local/HTTPServer/htdocs/en_US
    ls -l /opt/local/HTTPServer/htdocs/en_US/index.html

If the directory where you choose to install IBM HTTP Server is not within a directory created by the system, you may need to add read and search permissions to that directory so that the web server user/group can access the files.

[error] [client nnn.nnn.nnn.nnn] (13)Permission denied: unable to connect to cgi daemon after multiple tries: /path/to/cgi

This message can be received for CGI requests with IBM HTTP Server 2.0 or above on Unix and Linux systems. mod_cgid implements the CGI feature in these levels of IBM HTTP Server. At IBM HTTP Server initialization, mod_cgid creates a Unix socket in the filesystem which is used when executing CGI scripts. The default name for this Unix socket is logs/cgisock within the ServerRoot directory. (Example name: /usr/IBMIHS/logs/cgisock) The name of this Unix socket in the filesystem can be changed with the ScriptSock directive. Example: ScriptSock /tmp/scriptsock

There are two requirements for the filesystem path of this Unix socket:

1. Prior to IHS 7.0, if there are multiple instances of IBM HTTP Server running on this machine, the different instances must be configured to use different Unix sockets using the ScriptSock directive. In 8.0 and later, the parent PID is automatically appeneded to the socket path to make it unique.

2. The entire path to the socket must be accessible by the web server user id (e.g., nobody, or whatever is specified by the User and Group directives).

   Assume the following configuration:

        User nobody
        Group nobody
        ScriptSock /usr/IBMIHS/logs/scriptsock

If the permission denied error is written to the error log when a
CGI request is made, the expected cause is that there is a
filesystem permissions issue which prevents the web server user id
from accessing the Unix socket. Perform the following checks:

1.  Run \"ls -ld /usr\" and verify that user/group *nobody* has
    permissions to read (*r*) and search (*x*) the directory.
2.  Run \"ls -ld /usr/IBMIHS\" and verify that user/group *nobody*
    has permissions to read (*r*) and search (*x*) the directory.
3.  Run \"ls -ld /usr/IBMIHS/logs\" and verify that user/group
    *nobody* has permissions to read (*r*) and search (*x*) the
    directory.
4.  Run \"ls -l /usr/IBMIHS/logs/scriptsock\" and verify that
    user/group *nobody* has permissions to read (*r*) and write
    (*w*) the Unix socket file.

Here is an example where the logs directory has been made readable
only by root. In this case, the permission denied error would occur
when trying to process a CGI request. The fix would be to use the
ScriptSock directive to specify a Unix socket file that the web
server user id could access, or to make the logs directory readable
by the web server user id.

        [trawick@b80-2 /]$ ls -ld /usr
        drwxr-xr-x  63 bin      bin            1536 Mar  7 16:15 /usr
        [trawick@b80-2 /]$ ls -ld /usr/IBMIHS
        drwxr-xr-x  22 root     system          512 May 19 2004  /usr/IBMIHS
        [trawick@b80-2 /]$ ls -ld /usr/IBMIHS/logs
        drwx------   2 root     system          512 May 19 2004  /usr/IBMIHS/logs

If the problem persists, send in the following information to IBM support:

1. IBM HTTP Server conf file

2. IBM HTTP Server error log covering a time when CGI requests failed

3. output of ls -ld on every directory component of the path to the Unix socket

4. output of ls -l on the Unix socket itself

[error] [client 127.0.0.1] Invalid method in request unprintable-chars

This problem is almost always caused by an SSL request being handled by a virtual host which is not SSL-enabled. It will be accompanied by a message similar to the following in the access log:

    127.0.0.1 - - [15/Apr/2005:11:42:51 -0400] "\x80\x8c\x01\x03\x01" 501 310 - -

Here are some common points of confusion which can lead to this issue:

1. The client may not have specified the expected server IP address. It may have reached the host using an IP address which is not covered by any of the <VirtualHost> directives.

2. If the <VirtualHost> directive used in httpd.conf specifies a host name instead of an IP address, it may not match all IP addresses for that host name (apply PK01070 or later for IBM HTTP Server 2.0.x to fix that) and/or the web server may get different IP addresses from the DNS for that host name than clients would see.

   One cause for this is a defect in IBM HTTP Server 2.0.x, resolved by PK01070 or later, which respected only the first returned IP address for the host name.

   Another cause for this is when the DNS used by IBM HTTP Server returns different information than the DNS used by the client.

To diagnose the configuration problem, make these configuration changes:

1. Specify only IP addresses in the <VirtualHost > directives for SSL-enabled virtual hosts. Do not use host names.

2. Add the target IP address (%A) to the access log format so that the IP address used by the client on the failing request can be seen.

3. Log which vhost was selected.

   1. Add SetEnv vhostname MAIN to the main scope of httpd.conf.

   2. Add SetEnv vhostname UNIQUE-NAME to each VirtualHost container. Make sure UNIQUE-NAME is unique for each virtual host.

   3. Add the vhostname (%{vhostname}e) to the access log format.

Example log format with both of these changes made:

    LogFormat "%h %l %u %t \"%r\" %>s %b %A %{vhostname}e" common

Example setting of vhostname:

    ...
    SetEnv vhostname MAIN
    ...
    <VirtualHost 192.168.1.15:8443>
    ...
    SetEnv vhostname vhost8443
    </VirtualHost>

    <VirtualHost 192.168.1.1:443>
    ...
    SetEnv vhostname vhost443
    </VirtualHost>

Now restart IBM HTTP Server and try the request again. Check the access log for the destination IP address and the vhost name:

    127.0.0.1 - - [15/Apr/2005:11:58:24 -0400] "\x80\x8c\x01\x03\x01" 501 310 127.0.0.1 MAIN

In this example, the IP address in next to last column is the IP address used by the client. So you have to check if there is an SSL-enabled virtual host which specifies that IP address (or *) on the <VirtualHost > directive.

The last column is the virtual host which was actually chosen. In this example, it is the main scope of httpd.conf, because the IP address used by the client did not match the IP address on any <VirtualHost > directives. Whichever virtual host was chosen, verify that there is no SSLEnable directive for it, and decide if the client sent the request to wrong vhost (due to bad link?) or if the SSLEnable directive is supposed to be specified or if another VirtualHost container is appropriate.

[error] [client nnn.nnn.nnn.nnn] Directory index forbidden by rule: /usr/IBMIHS/docroot/path/

A request has been issued from the browser, and that request maps to an actual directory on the web server machine under DocumentRoot, and there is no active DirectoryIndex directive/document to return a document such as index.html to the browser, so IBM HTTP Server tries to send a directory listing back to the client. However, the Indexes option is not active for the directory so an error is returned.

To resolve the problem, you need to determine what response was supposed to be returned on this request.

If the request is not expected, check for links on your site which specify that URL.

If a directory listing was supposed to be returned, enable the Indexes option for that directory. Example:

    <Directory /usr/IBMIHS/docroot/path/>
    Options +Indexes
    </Directory>

If a static file such as index.html was supposed to be returned, enable that as the DirectoryIndex for that directory. Example:

    <Directory /usr/IBMIHS/docroot/path/>
    DirectoryIndex index.html
    </Directory>

If the request is supposed to map to a WebSphere request, check the WebSphere plug-in configuration file and make sure there is a rule to send requests for that URL to WebSphere.

[warn] [client nnn.nnn.nnn.nnn] mod_was_ap20: /some/URL not found

This message is issued by the WebSphere plug-in, and it indicates a plug-in processing error, due either to the plug-in implementation or to the plug-in configuration. A common cause of this error is fixed by APAR PK01215 (WebSphere plug-in). It can affect ErrorDocument processing as well as some other types of IBM HTTP Server processing. A WebSphere plug-in update including that fix needs to be applied before the error can be investigated further.

[crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock

Related: [emerg] (28)No space left on device: Couldn't create accept lock

Check your system sysv semaphore usage and limits

• Linux: Compare ipcs -l with ipcs -u,

• Solaris: Compare sysdef | grep -i shm with ipcs -s

• HPUX: Compare grep -i SHM /etc/conf/master.d/sysv-sem with ipcs -s

• AIX: (Dynamically adjusted; error should not occur on that platform)

script not found or unable to stat: /path/to/cgi.sh]

If all signs point to the CGI script being present in the correct location, make sure the URL is not being matched by the URI and host_alias patterns in plugin-cfg.xml. Once the Plugin determines it is responsible for a resource, the phase of mapping to the filesysytem that is required for CGI scripts is short-circuited.

You must resolve the overlap of ScriptAlias and the context roots and host aliases in the plugin-cfg.xml.

(12)Cannot allocate memory: cache: Cache provider's store_body failed!

mod_mem_cache generates this debug-level message when a streamed (chunked) response exceeds the value of the MCacheMaxStreamingBuffer directive.

[warn] worker http://example.com/somepath already used by another worker

In terms of Apache HTTP Server proxy, a "worker" is a set of configuration and connection pool for backend connections. When ProxyPass directives point to a common backend URL, this message occurs to warn that the directives will share a common worker.

When a worker is shared in this way, the optional arguments to ProxyPass (such as timeout=xxx) cannot be set independently because the configuration and pool of backend connections are shared.

configuration error: couldn't check access. No groups file?

If using authorization from mod_ibm_ldap, make sure mod_auth is also loaded (mod_auth is loaded by default).

Failed to flush CGI output to client

After PI22070, mod_cgid reports this error if an error is returned while writing or flushing output to the client. It originally went unreported. If you are seeing this error, a good first step is to log %D and %X to see if the source is slow-to-respond scripts and to validate that the connection is aborted.

Other runtime messages

(3448): apr_socket_accept: (client socket)

The operative part of this message is the "(3448)" which is the errno value for EUNATTACH. In 9.0 and later, this message also contains the identifier AH02179.

See zos_questions.html#DVIPA for more info.

[emerg] (9)Bad file number: fcntl: F_SETLKW

The "Bad file number" message indicates that the IBM HTTP Server file mutex descriptor has been closed incorrectly. This has been encountered on Solaris with IBM HTTP Server 1.3 and a third-party module.

Make a change to httpd.conf to work around this problem; add AcceptMutex sysvsem to the bottom of httpd.conf. Also, there are likely changes to the Solaris kernel semaphore tuning. Refer to these notes.

[warn] (22)Invalid argument: setsockopt: (TCP_NODELAY)

[error] (22)Invalid argument: getsockname
[warn] (22)Invalid argument: apr_socket_opt_set: (TCP_NODELAY)
[info] (22)Invalid argument: apr_socket_addr_get(APR_LOCAL)

This message is logged when the client drops the connection right after IBM HTTP Server has started processing it but before IBM HTTP Server has started reading the request. It is to be expected that some subset of clients will drop the connection at unexpected times (this could be a real user at a web browsers pressing the stop button), and this is one of the types of messages that can be logged because of that.

Unfortunately, the error returned from the kernel (EINVAL) has a text description that makes the failure look like a programming error, but that is how these calls are rejected when the TCP connection has already been dropped by the client.

Some load balancers test availability of the server on a regular basis in such a way that IBM HTTP Server often encounters a failure on one of these calls. Current maintenance levels of IBM HTTP Server 1.3 have lowered the severity of these messages and included the client IP address in the log message.

[notice] caught SIGTERM, shutting down

This message is a normal shutdown message for IBM HTTP Server. If a user or automated task (e.g., cron job) does "apachectl stop" or sends SIGTERM to the IBM HTTP Server parent process, this message will be written.

Note: The SIGTERM "signal" shouldn't be confused with other signals that indicate problems, such as SIGSEGV, SIGBUS, SIGABRT, and SIGILL. SIGTERM is the normal mechanism to tell a program to terminate on Unix, so apachectl sends SIGTERM to the IBM HTTP Server parent process to tell it to go away (and to clean up other IBM HTTP Server processes).

[warn] child process 26354 still did not exit, sending a SIGTERM (or SIGKILL)

A child process is not exiting as quickly as expected during shutdown or non-graceful restart processing. During shutdown or non-graceful restart, the parent will tell the child processes to exit. If after a short amount of time a child has still not exited, the parent will send the SIGTERM signal to the child process again and write the message to the log. If after another interval the child process still does not exit, then the parent process forcefully brings down the child by sending the SIGKILL signal to the child process.

Stopping or non-gracefully restarting the web server while it is actively processing client requests will result in this type of message.

[error] could not make child process 11540 exit, attempting to continue anyway

A child process hasn't exited even after sending the kill signal to the process. Web server termination has to continue even though the process hasn't exited, because there is no further action that can be taken automatically.

If the process exits within a few more seconds, the expected cause of the message is high system load at the time of termination, and there is no operational problem.

If the process does not exit on its own within one minute, the expected cause is that a thread in that process is blocked in the operating system kernel. Operating system support may need to investigate if this is a recurring problem. IBM HTTP Server support can investigate if hang documentation is provided. When running the hang collector tool, specify "auto" for the parent process id and "-" for the non-SSL port.

[[debug] worker.c(1042): child 10617008 isn't taking over slots very quickly (1996 of 2000)]

This is a debug message. IHS has tried to start ThreadsPerChild threads in a process, but some subset has not been able to fully initialize. Collect a HangDoc to figure out why threads are slow to initialize.

[alert] No active workers found...  Apache is exiting!

Check for error messages earlier in the error log. One or more of those error messages should explain the problem.

[crit] the listener thread didn't exit

This is sometimes written when a child process exits. It means that one of the threads in a child process didn't exit in the cleanest possible manner, but it doesn't indicate any operational problem. This is a debug message in current levels of IBM HTTP Server.

[error] (12)Not enough space: fork: Unable to fork new process

This message occurs when the parent process tries to create a new child process using the fork() system call, but the operating system returns an error. Clients may be delayed until an existing child process finishes handling an existing connection.

For more information, refer to fork() failures.

[Sat Dec 13 11:52:48 2003] [warn] long lost child came home! (pid 11380)

This can occur with piped loggers (e.g., rotatelogs) during a graceful restart. One of these messages may appear for every piped logger.

With levels of IBM HTTP Server 2.0 prior to interim fix PK01070, it can also occur during steady state for httpd processes which begin exiting due to MaxSpareThreads or MaxRequestsPerChild processing, but take a long time to exit.

[info] server seems busy, (you may need to increase...

This message appears when a lot of child processes have to be created at about the same time to handle increased load. If it occurs very rarely, it may be triggered by a an infrequent but drastic increase in load. If it occurs relatively frequently, the process management tuning needs to be changed, as follows:

Increase MaxSpareThreads to be a larger percentage of MaxClients.

[emerg] (69)Network is down: apr_accept: giving up.

This can be reported on some platforms if the network layer or a specific interface is no longer active. IBM HTTP Server will exit if all child processes receive this same error.

This problem has been encountered on AIX when performing a load balancer takeover. An AIX APAR fix is available to disable reporting the action to applications via the ENETDOWN error:

• AIX 5.2: IY85386

• AIX 5.3: IY85206

Refer to the text of the appropriate AIX APAR for a temporary circumvention.

[alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2513

setuid() can return error code 11 when the NPROC rlimit (ulimit -n) for the user would be exceeded by creating a new thread. This limit is an accumulation of all threads across the entire system for the userid in the error message.

On Linux, first boost the soft limits for NPROC for root and the user in the error message in /etc/security/limits.conf. Second, restart any service/process running under the userid in the error message from a new login shell.

On other platforms, use whatever mechanism is provided to raise the soft limit for NPROC / ulimit -n and recycle all processes running under the userid from new login shells.

[error] mod_ibm_ldap: unable to authenticate the web server for realm 'Example LDAP Realm': Encoding error.

If this is Solaris, verify that the SUNWuiu8 package is installed:

# pkginfo SUNWuiu8
system      SUNWuiu8       Iconv modules for UTF-8 Locale

If this is AIX and IHS is launched with an environment containing LC_MESSAGES=c@lft (e.g. via crontab or /etc/inittab) then either unset LC_MESSAGES or check for the AIX APAR listed below:

  APAR      Release
  --------- ---------
  IY59922   5.1
  IY56520   5.2
  IY59458   5.3

To circumvent the problem without applying the AIX fix, add the following to the bottom of IHSROOT/bin/envvars:

# work around AIX setlocale() problem fixed by IY59922/IY56520/IY59458
unset LC_MESSAGES

[crit] Unable to set default LDAPCodePages directory relative to ServerRoot.

If the ServerRoot directive has been changed from the default value, copy (or symlink) the codepages directory from the installation path into the new ServerRoot

[warn] proxy: No protocol handler was valid for the URL /example/url.js. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

The usual cause of this is the lack of a LoadModule directive to load mod_proxy_http.so when supporting HTTP proxy. Uncomment, or add, the following directive:

    LoadModule proxy_http_module modules/mod_proxy_http.so

If the specified URL is handled via FTP, activate mod_proxy_ftp.so instead.

This message can also be seen with some levels of IBM HTTP Server when SSL is used for the proxy connection, and an error occurs setting up the connection. If the necessary proxy modules are already loaded and the proxy connections are https, apply one of the following fixes:

• IBM HTTP Server 2.0.47.1: PK07831 or later

• IBM HTTP Server 6.0: 6.0.2.1 or later

[notice] suEXEC mechanism enabled (wrapper: /usr/IBMIHS/bin/suexec)

This message indicates that suEXEC could be used if configured in httpd.conf. In order to configure it, the SuexecUserGroup directive is coded. See this suexec how-to for more information.

SSL Proxy requested but not enabled [Hint: use SSLProxyEngine on]

This message is issued when IHS is configured using ProxyPass or mod_rewrite to act as a reverse proxy for an SSL origin server but SSLProxyEngine on is not enabled for the virtual host (or base configuration) handling the request.

child pid 25771 exit signal File size limit exceeded (25)

This fatal message is issued when IHS 1.3.28.1 or earlier has been terminated due to a logfile exceeding 2GB.

taking over scoreboard slot from nnnn (quiescing)

This is a debug message issued when a child process (process A) is taking a long time to exit, and a replacement process (process B) is needed before "process A" can fully exit (due to slow-running requests). Minimizing child process destruction (increase MaxSpareThreads, increase or set MaxRequestsPerChild to 0) to prevent this message and the unnecessary churn it implies.

Not owner: processor unbind failed

IHS attempts to call the AIX bindprocessor subroutine each time a new child process is created. In older releases of AIX (5.2 and earlier), this call was necessary to unbind the children from the CPU the parent process was running on.

On AIX 5.3 and later, as well in WPAR environments, the behavior requested by IHS is already the default. After AIX APAR IZ48332, this no-op call to bindprocessor() returns an error which triggers the warning message. There is no functional problem other than the warning message being logged, which can be confirmed by confirming no CPU # is listed in output of the following cmd:
ps -mp <pid-of-child-process> -o THREAD

This message is logged at debug level after IHS APAR PM09819.

Windows-specific messages

Cannot load IHS_PATH/afpaplugin.dll into server: The specified module could not be found.

This error message may occur when the customer is trying to use IBM HTTP Server with the Websphere Application Server high speed external caching feature and has configured the wrong version of afpaplugin.

The afpaplugin.dll file, which is a component of Webshpere Application Server, is only for use with IBM HTTP Server 1.3.x. The afpaplugin_20.dll file, which is installed via the Websphere 6.x Plugins installation, is required when using IBM HTTP Server 2.0.x and above.

When using IBM HTTP Server 2.0.42 or 2.0.47, cumulative e-fix PQ94086 or later must be applied. With IBM HTTP Server 6.x no additional fix pack is required.

The following error messages indicate that some other vendor's software may be installed which does not correctly implement AcceptEx or other Winsock functions:

• (OS 6)The handle is invalid. : winnt_accept: AcceptEx failed.

• (OS 64)The specified network name is no longer available.: winnt_accept: Asynchronous AcceptEx failed

• (OS 121)The semaphore timeout period has expired. : winnt_accept: Asynchronous AcceptEx failed.

• (OS 10106)The requested service provider could not be loaded or initialized.

• (OS 10038)An operation was attempted on something that is not a socket.

Other symptoms that might be experienced are slow IHS response time or the appearance that IHS has hung.

Try adding Win32DisableAcceptEx to the IHS configuration file and restart IHS.

Note: AFPA must be disabled before adding the Win32DisableAcceptEx directive. See technote Disabling AFPA (fast cache accelerator) in IBM HTTP Server for details of how to disable AFPA.

apache.exe: Could not determine the server's fully qualified domain name. using IP_ADDR for server name

If this message appears in Event Viewer, and you're using the IHS 6.1 Admin Server, add the 'ServerName yourhostname' directive to <ihsinst>/conf/admin.conf.

[warn] (OS 10038)An operation was attempted on something that is not a socket.

If this message occurs just after IHS has begun shutting down ("Child process is ending"), it can be safely ignored.

If this message occurs during steady-state, it is likely the result of buggy third-party firewall or anti-virus software. See other entry.

(OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted. : proxy: HTTP: ....

This error occurs when IBM HTTP Server is asked to rapidly create connections to the same backend server via mod_proxy_http, but windows is configured to only allow a lower number of local (ephemeral) ports. This FAQ is only correct when the error message includes the "proxy" text bolded above, not anytime the preceding part of the message appears as a prefix of other messages.

See the MaxUserPort TCP/IP registry settings to increase this limit.

[crit] (OS 5)Access is denied. : Parent: Failed to create the child process

This can occur if startup fails due to firewall or antivirus software.

Startup messages

/opt/IBMHttpServer/bin/httpd: relocation error: /opt/IBMHttpServer/bin/httpd: undefined symbol: apr_table_compress

This is a symptom of IBM HTTP Server finding the wrong version of the libapr-0.so at runtime. libapr-0.so is provided with the IBM HTTP Server installation in the lib/ subdirectory, and is found by a platform-specific environment variable in IHSROOT/bin/envvars.

• Verify that the library path set in IHSROOT/bin/envvars includes the libs/ subdirectory if you have altered your installation.

• If you've modified the apachectl script, or are using some other custom method to start IHS, make sure it's sourcing the proper IHSROOT/bin/envvars file

[crit] (17)File exists: unable to create scoreboard "/opt/IBMIHS/logs/xxxx

The xxxxx portion of the message varies based on the value of the ScoreBoardFile directive.

Either there are multiple web server instances configured with the same ScoreBoardFile directive or there is a single instance configured with ScoreBoardFile and it did not shutdown cleanly.

The recommended solution in either case is to remove the ScoreBoardFile directive and restart.

The only situation where ScoreBoardFile is needed is if a third-party application which reads the web server shared memory is in use and that application specifies that the ScoreBoardFile directive is required. That is very rare.

[crit] (67)Address already in use: make_sock: could not bind to address nnn.nnn.nnn.nnn port 80

This can occur at startup or restart if some other process is listening to the specified port and optional address.

[error] SSL0157E: Initialization error, The function call, gsk_attrib_set_buffer(env_handle, GSK_STRICTCHECK_CBCPADBYTES, 1, has an invalid ID

This can occur at startup for IHS 6.1.0.47 if the PI31516 interim fix is applied without first upgrading IBM Global Security Kit (GSKit) to a minimum required version as described in the Prerequisites section of the PI31516 interim fix document.

Upgrade the global IBM Global Security Kit (GSKit) as specified to resolve the problem.

SSL-specific messages

SSL0200E: SSL Handshake Failed: ...

SSL0200E is issues for SSL handshake errors that do not have a more specific error code associated with them. Often, a numeric code will be listed that contains additional detail.

• SSL0200E: SSL Handshake Failed: (447)GSK_ERROR_CERTIFICATE_INVALIDSIGALG. GSK_ERROR_CERTIFICATE_INVALIDSIGALG is returned by two separate cases. Both require reviewing a binary, unfortmatted packet capture.

  • If the client sent no list of acceptable signature algorithms during the handshake, the server is only permitted to respond with a relatively weak SHA1 certificate. If the configured certificate is not signed with exactly SHA1, this error is issued.

  • If the client sent a list of acceptable signature algorithms, and the configured certificate does not overlap with the algorithms in the handshake, this error is issued. In later releases, this sub-message is replaced by SSL0280E.

• SSL0200E: SSL Handshake Failed: (454)GSK_ERROR_ILLEGAL_PARAMETER

  • This requires IBM Global Security Kit (GSKit) 8.0.55.13 or later in PH21804.

• SSL0200E: SSL Handshake Failed: (14)GSK_ERROR_UNEXPECTED_INT_EXCEPTION.

  • This often means a certificate in the server certificates trust chain has becom expired after server startup. After a restart, the symptom will turn into SSL0208E: SSL Handshake Failed, Certificate validation error.

  • See https://www.ibm.com/support/pages/node/6218244

• Anything else: Must be diagnosed by support with a binary packet capture and IBM Global Security Kit (GSKit) trace.

SSL0280E: SSL Handshake Failed, the configured certificate chain contains a signature that is not compatible with peers TLS Signature Algorithm requirements.

TLS 1.2 and later allows a client to inform the server what certificate signature algorithms it finds acceptable. Among other things, this allows a server to choose from multiple certificates based on client capabilities.

Under normal circumstances, SSL0280E usually means that the server has no certificate that matches the clients preferences. Browser based clients will often retry with a larger and more lenient set of acceptable signature algorithms, for example adding SHA1 back into the acceptable list.

• On AIX, Linux, and Windows only the leaf/personal certificate is checked by the server, not the entire certificate chain.

• On z/OS, the entire certificate chain is validated against the clients list of signature algorithms.

  • Prior to z/OS APAR OA63632, a behavior change in z/OS 2.5 prevents browsers from performing a retry and can result in the browser displaying an ERR_BAD_SSL_CLIENT_AUTH_CERT error.

SSL0232W: SSL Handshake Failed, Unsupported SSL protocol or unsupported certificate type.

See ssl_questions.html#TLS13

SSL0235W: SSL Handshake Failed, Invalid peer.

• Note: This item has been largely n/a since RC4 was removed in 8.5.5.6 and later.

  In IHS 8.0 and later with SSLProxyEngine enabled, SSL0235W can be issued if SSLCipherSpec has not been configured and the backend negotiates TLS1.2 + an RC4 cipher. The defaults are desgigned for server side usage, not client side as in the proxy.

    SSLCipherSpec ALL -SSL_RSA_WITH_RC4_128_SHA -SSL_RSA_WITH_RC4_128_MD5

If your symptom is not related to SSLProxyEngine, this entry is unlikely to apply.

• If upgrading past 8.5.5.12 with SSLProxyEngine, the failure could be related to connecting to a non-TLS1.2 backend that chooses ECDHE_RSA ciphers added to the TLS1.2 defaults in 8.5.5.12. Due to a limitation of the SSL protocol, the server cannot know that IHS does not suppor the ECDHE_RSA ciphers for TLS1.1 or TLS1.2.

  To resolve the issue, configure the backend server to use TLS1.2 or disable ECDHE_RSA ciphers in IHS.

[crit] SSL0193W: Error setting GSK_ALLOW_ABBREVIATED_RENEGOTIATION to 0 (701)

Starting IHS with LD_LIBRARY_PATH that includes /usr/lib forces a global IBM Global Security Kit (GSKit) to be used rather than the one bundled with IHS and configured via SHLIB_PATH. Remove any directories that contain IBM Global Security Kit (GSKit) or Apache files from any LD_LIBRARY_PATH set before invoking apachectl, or unset it entirely in $IHSROOT/bin/envvars.

SSL0166E Failure attempting to load GSK library

See gather_startup_doc.html#SSL0166E

[error] SSL0104S: GSK could not initialize, Invalid password for keyfile.

The InfoCenter lists this message as SSL0104E instead of SSL0104S.

For each KDB file IHS is configured to use, IHS will also try to open a file with the same file extension changed to .sth. If IHS cannot find, open, or read this file the above error will be reported.

• 9.0.0.8/8.5.5.13 and earlier can fail to properly read stash files if they were created with a later level of IHS (via gskcapicmd) or Java (via WAS, Ikeyman, ikeycmd, or gskcmd). This is related to a flaw in "version 2" stash files.

• 8.5.5.10/9.0.0.2 and earlier can fail to properly read stash files if they were created with a later level of IHS (via gskcapicmd) or Java (via WAS, Ikeyman, ikeycmd, or gskcmd). This is due to the use of "version 2" stash files.

• If the stash file does not exist, it can be created in Ikeyman or gskcapicmd.

• Be sure the userid and group that IHS is configured to run under have read and execute access to every intermediate directory and to the .sth file itself.

• IHS can fail to properly read the stash files if a IBM Global Security Kit (GSKit) other than the one bundled with IHS is being loaded. If the IBM Global Security Kit (GSKit) version reported in the error log is different from that reported by bin/gskver, check bin/envvars and bin/apachectl for any non-IHS paths included in LD_LIBRARY_PATH and LD_PRELOAD.

SSL0154E: Initialization error, A PKCS#11 token is not present for the slot

With IBM Global Security Kit (GSKit) v8, if the SSLServerCert looks correct and verifies in Ikeyman, check the following items:

• ikeyman V8 new configuration steps were used

• LoadFile or LD_PRELOAD used with IHS and gskcapicmd on z/Linux

Depending on the level of Linux used, this may manifest as other SSL0xxxx codes.

[error] SSL0208E: SSL Handshake Failed, Certificate validation error.

IBM Global Security Kit (GSKit) 8.0.50.88 or 8.0.55.11 (IHS 9.0.5.2 and 9.0.5.3)

IHS with IBM Global Security Kit (GSKit) 8.0.50.88 through 8.0.55.11 (inclusive) can erroneously report this error when a certificate in the servers chain does not have explicit "signature algorithm parameters".

Apply 9.0.5.4 or later, or contact support for an interim fix containing a cumulative IBM Global Security Kit (GSKit) update (PH24493 or later).

Typical causes

This message may appear if your KeyFile does not contain all of the necessary intermediate and root certificate authorities necessary for validating your personal certificate.

You must acquire the complete cert chain from your Certificate vendor, and "add" a complete cert chain (from the top down) using Ikeyman.

The following steps illustrate how to determine which certificate may be problematic or missing:

1. Determine the path to your KeyFile by reviewing httpd.conf. For example, conf/key.kdb

2. Determine the certificate label you're using by either finding SSLServerCert in your httpd.conf or viewing the "default certificate" via :

   bin/gskcapicmd -cert -getdefault -db conf/key.kdb -stashed

   For example, "my-certlabel"

3. Validate the certificate being used on the command line:

   bin/gskcapicmd -cert -validate -db conf/key.kdb -label my-certlabel -stashed

4. Look for an error message beginning with GSKKM_VALIDATIONFAIL_SUBJECT and observe the Issuer and Subject of the certificate which cannot be validated.

   GSKKM_VALIDATIONFAIL_SUBJECT: [Class=]GSKVALMethod::PKIX[Issuer=]CN=my-signer[#=]549ab2886a97f86f[Subject=]CN=example.com,O=ibm,C=US

   • This output shows details about the certificate within the chain that failed validation. It may be invalid (rare) or its issuer may not exist in the keystore (common). Use a series of -list and -details to double-check that the signer is present.

     bin/gskcapicmd -cert -list -db conf/key.kdb -stashed
     bin/gskcapicmd -cert -details -db conf/key.kdb -label ... -stashed | egrep '^(Subject|Issuer)'

5. If any signer is not present, obtain it from your certificate authority and add it:

   bin/gskcapicmd -cert -add -db conf/key.kdb -label my-certlabel-ca -stashed -file downloaded-cert.cer

There are some obscure problems which can cause components of a seemingly complete certificate chain to be disregarded by IBM Global Security Kit (GSKit). The gskcapicmd and gskcmd utilities bundled with IBM HTTP Server V8R0 and later understand a "-cert -validate" option which allows the chain to be validated outside of the webserver. If this fails despite a seemingly complete certificate chain, review the chain for the issues below:

1. A certificate in the chain is signed with Signature Algorithm RSASSA-PSS (1.2.840.113549.1.1.10) (visible with gskcapicmd -cert -details ...) which is the default algorithm in some Microsoft based certificate tools but is not supported by IHS. It is also not supported by TLS1.2.

2. What appears to be a valid certificate chain by comparing the Distinguished Name of each issuer, but the actual issuer has a different SubjectKeyIdentifier then the issued certificates AuthorityKeyIdentifier

   This means two issuers exist with the same DN, but different Key ID's, and likely different private keys used for signatures. A IBM Global Security Kit (GSKit) trace might report 'cms_setSSLKey bad rc' and

   'Verify failed: : error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed

[error] SSL0209E: SSL Handshake Failed, ERROR processing cryptography.

• If this message appears during IBM HTTP Server restart or shutdown:

  The message can be ignored. The crypto accelerator operation was aborted during shutdown.

• If this message appears continuously during normal operation:

  1. Verify that the crypto accelerator configuration is correct.
     If the crypto accelerator is accessed using a PKCS11 driver, a common configuration error is that the IHS user id has not been added to the pkcs11 group. Check the User directive in httpd.conf for the relevant user id.

  2. If a configuration problem cannot be found, contact the vendor of the cryptographic accelerator for assistance with diagnosing the error returned by the accelerator.

  3. If a high ThreadsPerChild is used, make sure ulimit -n is more than 3-4 times ThreadsPerChild

  4. If it happens for a particular client with 'SSLClientAuth' configured, that client is likely broken or has an invalid key-pair.

  5. On z/Linux with TLS13, see APAR PH39992 (9.0.5.10)

• If this message appears without PKCS11 crypto hardware, see the final section of the diagnostic info for SSL0212E.

[error] SSL0115E: Initialization error, Error validating ASN fields in certificate.

SSL0115E occurs when the server certificate has a problem that's detected at startup and will result in a startup failure.

See Error processing X509 certificate for some possible causes; they are the same problems that could also be reported when importing or receiving a certificate.

For a historical cause of SSL0115E not directly related to the ASN.1 encoding of certificates, see gather_startup_doc.html#SSLERR

[error] SSL0210E: SSL Handshake Failed, ERROR validating ASN fields in certificate.

SSL0210 occurs when the server certificate has a problem. See Error processing X509 certificate for some possible causes; they are the same problems that could also be reported when importing or receiving a certificate.

[error] SSL0221E: SSL Handshake Failed, Either the certificate has expired or the system clock is incorrect

This means a certificate in the servers certificate chain has expired. It could be the personal certificate of the server, an intermediate, or the root.

• Check that the expected KeyFile is in use

• If SSLServerCert is not in use, verify the default certificate in the key store is not expired

• If SSLServerCert is in use, check that the expected label is used and that it's not expired

• Check the system time

[warn] SSL0222W: SSL Handshake Failed, No ciphers specified.

SSL0222W occurs when the client and server don't share any SSL ciphers, which may be a result of overly-restrictive values for SSLProtocolDisable or SSLCipherSpec

• If a protocol was recently disabled with SSLProtocolDisable, it's likely the client was using that protocol. A binary packet capture will reveal the clients requested protocol.

• If you just updated to 8.5.5.4 or later from an earlier fixpack, your clients likely depend on SSLv3 which i disabled by default in this maintenance level. It is recommended to fix the client rather than re-enabling SSLv3.

• If you've configured a custom cipher list, and enabled ONLY ECDHE_ECDSA ciphers, this failure will occur unless you have a relatively uncommon certificate with an ECDSA private key (instead of an RSA private key). It is not currently portable to use ECDHE_ECDSA if your clients are normal web browsers.

After APAR PI27904, this message has an added suffix indicating that the problem could be any missing overlap in ciphers or protocols.

[error] SSL0212E: SSL Handshake Failed, Internal unknown error. Report problem to service

• If this error occurs after IBM Global Security Kit (GSKit) 8.0.55.31 (IFPH52546, 9.0.5.16, 8.5.5.24) and SSLFIPSEnable is specified, remove any added RSA Key Exchange ciphers. RSA Key Exchange ciphers are disabled by default, and after 8.0.55.31 will fail under FIPS.

• If this error is accompanied by long handshake times, it may be a misreported SSLHandhsakeTimeout which defaults to 10 seconds.

• If this error occurs after moving 9.0.5.2 or 9.0.5.3, and no PKCS11 hardware is in use, apply the cumulative IBM Global Security Kit (GSKit) update PH21804. For relief, disable TLSv13 by setting SSLProtocolDisable TLSv13 after each SSLEnable directive.

This error normally means IHS asked a PKCS11 adapter to perform an operation and it unexpectedly failed. The sequence of PKCS11 calls can be analyzed by IBM support, but it will generally only be useful in giving the PKCS11 vendor a basic description of the error.

Over time, IBM support has noted several PKCS11 misconfigurations of various PKCS11 libraries:

• With Luna adapters, ensure Apache = 1; is set in the Misc section of Chrystoki.conf (or similar Luna configuration file)

• On SLES 11 SP1, overlapping copies of the libica library can break PKCS11 offload in IHS. Remove all versions of libica from the system, then install the packages from the latest major revision of the libica packages for 64-bit and 32-bit. Contact Novell for more information about the coexistence of these two packages that have the same filesystem contents. Note: libica is only used for the built-in crypto offload on z/Linux.

• On Linux, using z/Linux crypto or any other PKCS11 adapter that uses "pkcssslotd", this error message or a crash can occur if the user set in the User directive is not a member of the pkcs11 group, or if the pkcsslotd daemon is not started.

• On Solaris, this error can occur when Niagra-based crypto hardware has not been correctly configured. GSkit trace reveals the following error: [C_Login() returned error 0x32       (CKR_DEVICE_REMOVED)]{style="margin-left; 1em"}.
  Solution: Check the ownership of the directory and contents of the SOFTTOKEN_DIR and make sure the configured "User" and "Group" directives match. On Niagra, these values should generally not be "nobody" but some user/group created to use the crypto offload.

• On Luna (Gemalto) HSM's with High Availability (HA) configured, this can happen if the PKCS11 client library loses connectivity to an HA member. If there is only 1 HA member, connectivity will never be re-restablished without a webserver restart. If this is the cause, a IBM Global Security Kit (GSKit) trace will show C_OpenSession returning CKR_TOKEKN_NOT_PRESENT.

• On HSMs with a hard limit on the number of "open sessions", this error could be returned if IHS exceeds that limit. This should be unlikely as IHS does not use one session per connection. IHS uses one session per SSL enabled virtual host using PKCS11 per IHS child process.

• If no specific matching cause is listed, traces have to be analyzed by support. If the PKCS11 adapter fails to perform some operation that has been delegated to it, only the vendor can debug further.

In some rare cases, this message is issued when no PKCS11 adapter is present.

• On Solaris, IBM Global Security Kit (GSKit) prior to 8.0.55.8 may begin issuing this error after an internal crypto library failure. If applying APAR PH13105 does not resolve the issue, IBM support should be engaged to analyze the results/traces resulting from the following:

  1. Test with gsk8/lib64/C removed (full stop/start required).

  2. Touch the file /tmp/GSKIT_CRYPTO.log and recreate the symptom and share the report with IBM (full stop/start required)

[error] SSL0223E: SSL Handshake Failed, No certificate.

Verify that at least one personal certificate exists in the KDB file specified by the KeyFile directive. If no certificate is marked as default (an asterisk appears next to its label), your configuration must include an SSLServerCert directive for each virtual host containing an SSLEnable directive.

If TLSv1.3 is enabled, this error can be reported if the configured certificate chain uses banned signature algorithms such as MD5 and SHA224, even if the client doesn't try to negoiate TLSv1.3. This is a specification requirement for all negotiated protocols.

Additionally with TLSv1.3, clients specify a list of acceptable signature algorithms. Some browsers will omit the legacy SHA1 signature algorithm, resulting in this error.

Note: Some levels of IHS can report this error instead of SSL0222W.

SSL0234W: SSL Handshake Failed, The certificate sent by the peer has expired or is invalid.

• With SSLProxyEngine ON see the advice above for error SSL0266E.

• This message can be written when OCSP or CRL verification encounters a revoked certificate

• This message can also be written when ASN.1 parsing of a client certificate fails. See gather_certificate.doc.html#T.61 for details

[warn] SSL0226I: SSL Handshake Failed, I/O error during handshake.

The communication between the client and the server failed. This is a common error when the client closes the connection before the handshake has completed.

If a load balancer is configured to try to establish a TCP connection to the SSL port at intervals, the SSL0226I message can occur repeatedly, at the configured interval.

SSL0263W: SSL Connection attempted when SSL did not initialize

In very old releases, SSL0263W could repeatedly occur if there was a fatal SSL configuration error, such as a missing KeyFile. Most of these kinds of errors now result instead in a startup failure.

SSL0263W can be issued if mod_ibm_ssl goes from being not loaded (via LoadModule) to being loaded across an graceful restart. While in general modules may be removed or added during a graceful restart, mod_ibm_ssl cannot support this sequence.

[warn] SSL0265W: Client did not supply a certificate.

This message is received when SSLClientAuth required is specified, but IBM HTTP Server did not receive a client certificate during the SSL handshake. Additionally a 403 Forbidden error will appear at the browser. If the browser has a certificate installed, verify that the certificate authority (CA) which created the client certificate has a signer certificate installed in IBM HTTP Server's key database (.kdb) file. If not, use iKeyman to import the signer certificate information from the client certificate.

Under TLSv13, this can be reported when the client sends a "post_handshake_auth" extension during the handshake. The client certificate is not available with the initial handshake and IHS will not recognize it at this time.

[error] SSL0266E: Handshake Failed, Could not establish SSL proxy connection

This error is reported for almost any SSLProxyEngine error on an outgoing connection. If it is not followed by a more specific error message (on the next line), recreate with LogLevel debug and look for a secondary error message.

• If this message appears when moving to 9.0.5.2 or 9.0.5.3 for the first time, it is likely the backend server has the deprecated SHA1 signature algorithm somewhere in its certificate chain. The message will have a suffix including 'IBM Global Security Kit (GSKit) error 414: GSK_ERROR_BAD_CERT'.

  To resolve, add the following directives after SSLProxyEngine ON:

  SSLAttributeSet proxy:483 1` 
  SSLAttributeSet proxy:4027 1
  

  This allows mod_proxy to accept ALL signature algorithms. The first directive applies to TLS1.2 and earlier and the second directive applies to TLS 1.3.

[error] SSL0224E: SSL Handshake Failed, Invalid or improperly formatted certificate

SSL0224E can have multiple causes. The error message has been lengthened post-GA to include more specific information relevant mostly to TLSv1.2.

1. If the message has trailing information about a "bad_certificate" TLS alert coming from the client, the client may simply not trust the server. In later service levels, this may be reported as SSL0282E.

   User Action: Configure the trust store on the client

2. The servers certificate chain may be using weak signature algorithms (MD5, SHA1) or small key sized.

   User Action: Review the strength of the servers certificate chain and replace if necessary.

3. The client requested a TLS1.2 connection and restricted the set of certificate signature algorithms it was willing to accept, but no cert chain with matching signature algorithms was available in IHS.

   User Action:

   • Obtain a certificate for IHS matching the algorithms requested by the strict client -- consult your CA.

   • Configure the client to be more permissive -- consult the vendor of the client.

   • Disable TLSV1.2 with SSLProtocolDisable to force protocols that do not use the signature algorithm extension.

SSL0279E: SSL Handshake Failed due to fatal alert from client.

This message is issued when the client/browser side of an HTTPS connection terminates the handshake by sending a TLS alert. The message will continue to display the particular alert IHS received that signaled the end of the handshake.

Common alerts and their meanings:

Alert: certificate_unknown

Client sent fatal alert [level 2 (fatal), description 46 (certificate_unknown)]

The "certificate_unknown" alert means the client/browser rejected the servers TLS certificate for unspecified reasons. It is likely that the end user saw a HTTPS-related warning about an untrusted server. Some potential reasons are listed below:

If the hostname requested by the client doesn't match any Subject Alternative Name extension in the servers certificate, recent popular browsers may treat the server as untrusted, even if the hostname requested matches the certificates Common Name (CN). To remedy this situation, access the server by a name listed in the SAN extension or request a new certificate with additional names listed in the SAN extension

This alert can also be sent when the certificate has been revoked by the certificate authority, and the browser checks the online revocation source in the certificate. Since not all browser/releases/configurations check revocation lists, the failures may be grouped among certain users / IP addresses.

Notes:

• When this message is issued, no HTTP request was ever made. The browser may however retry on a new SSL connection after displaying a warning (or maybe not at all).

  • If you are debugging any failure in processing an HTTP request, ignore this error.

• After PH19074, SSL0279E errors that are a result of certificate_unknown alerts will provide additional details under message id SSL0284E.

• Beginning in 9.0.5.5 and 8.5.5.18, SSL0279E and SSL0284E are reduced to LogLevel "info" from "error" when the certificate_uknown alert is received.

Alert: unknown_ca

Client sent fatal alert [level 2 (fatal), description 48 (unknown_ca)]

This alert is sent when the client/browser does not trust the certificate authority that signed the servers certificate. To remedy this situation, obtain a certificate from a trusted authority or establish trust of the current issuer on each system that will access the server.

[error] SSL0267E: SSL Handshake Failed, Timeout during handshake operation.

This message is received when a timeout occurs at any stage in the SSL handshake, indicating that the client did not send an expected message in time. This includes the client sending its initial handshake message after opening the connection.

This is controlled by the core Timeout directive, not SSLV2Timeout or SSLV3Timeout .

Enabling log level debug and SSL trace might provide more information about at what stage the problem is occurring.

[error] SSL0404E: I/O failed RC [504]

This can occur when the client sends an invalid TLS record. In future levels of IHS, it is reported as SSL0412E. Some known defects that trigger this message follow.

1. The TLS specification limits the length of each record to 2^14 / 16384 bytes. If a client specified anything between 2^14 + 1 up to 2^16 (65k), this error will be logged. The client needs to be fixed.

   This can be detected in SSLTrace or a IBM Global Security Kit (GSKit) trace. In SSLTrace you will likely see a message like "SSL read begin bytes [16385]" where the number in brackets is greater than 16384. In a IBM Global Security Kit (GSKit) trace, the equivalent message is "S_Read trying to read 16385"

[error] SSL0404E: I/O failed, insufficient storage.

IHS 2.0.47.1 requires cumulative fix PK07831 or later to be compatible with IBM Global Security Kit (GSKit) versions 7.0.3.12 and higher.

[error] [84a0a58] SSL0409I: I/O Error occurred.

This message can be written to the error log when a client disconnects, and can be safely ignored. The message has been seen on Linux with fix packs 6.1.0.15, 6.1.0.17, or 6.0.2.27. The unnecessary message is removed with APAR PK64092.

[crit] ... SSL0227E: SSL Handshake Failed, Specified label could not be found in the key file.

• This error can occur with SSLFIPSEnable and key stores created with IBM Global Security Kit (GSKit) 8.0.55.29 or later without the -pqc false argument. A later release of IBM Global Security Kit (GSKit) will lift this limitation under FIPS.

  To address, create a new KDB with -pqc false and import the contents from the first key store (or export to the new key store). An example using the CLI is available here

  If using Ikeyman to complete the import step, be sure to not select save as. Changes are implicitly saved, and the "save as" button will strip the the -pqc false setting from the new KDB.

• If SSLServerCert is not present, check that the *.kdb specified by the KeyFile directive has a default certificate specified, or specify SSLServerCert to select a certificate explicitly. KDB files imported from other formats will not typically have a default.

• If SSLServerCert does not contain a colon (':') character, check the the *.kdb file specified by the KeyFile directive contains a certificate with a matching label and that the certificate is marked as a 'personal certificate'.

• If SSLServerCert is of the form token:certlabel, make sure configuration was done using the "PKCS11Config" mechanism in Ikeyman V8 (included with the Java bundled with IHS V7 and later. See cryptohw.html#ikeymanv8 for more information.

• If SSLServerCert is of the form token:certlabel, make sure the "certlabel" component is in the proper case (you can view the proper case with gsk7capicmd/gskcapicmd -crypto but not Ikeyman or gsk7cmd/gskcmd).

[crit] SSL0600S: Unable to connect to session ID cache

Also: [crit] (146)Connection refused: SSL0600S: Unable to connect to session ID cache

Also: [crit] (13)Permission denied: SSL0600S: Unable to connect to session ID cache

See sidd connect failures.

SSL0230I: SSL Handshake Failed, An incorrectly formatted SSL message was received.

• This message can occur when TLSv13 is enabled (9.0.5.2 and later) and the client fails to connect with an SSLv2 record. Many years ago, clients could send SSLv2 records with TLSv10 (or later) handshakes for maximum interoperability. This is not supported when TLSv13 is enabled. In 9.0.5.8 SSL0230I was expanded to cover this case if the first byte of the handshake looks like an SSLv2 record. This applies to Java 6 and IE11.

• This message can appear when SNI (Server Name Indication) is used for certificate selection, but the selected label is not present, failed validation, or does not match the clients signature algorithm requirements. The message was expanded with SNI details in 9.0.5.3.

• With SSLProxyEngine ON and ProxyRemote in IHS 8.5.5 and earlier, this message will appear because IHS does not support using ProxyRemote to contact backend HTTPS servers. This is functionality added to Apache HTTP Server 2.2.15 and not present in IBM HTTP Server.

• In 8.0 and later, many SSL0230I are reported as a different message code #SSL0222W.

SSL0240I: SSL Handshake Failed, Socket has been closed.

This message means that the client (browser) closed the underlying connection unexpectedly during a TLS handshake. It is impossible for the server to know why, it must be debugged on the client side.

• One historical case in which this message is issued is when Windows based TLS clients detected a weak (md2, md5) signature in a certificate chain.

SSL0247E: Handshake Failed, LDAP server not available

If SSLCRLHostname is a hostname or IP address, the server is likely down or firewalled.

If SSLCRLHostname is the special case value "URI", then the CRL Distribution Point in a provided client certificate is either unreachable, returned an invalid response, or returned a response over the default HTTP CRL size limit (200KB). See gather_crl_doc.html#limits to increase the limits.

z/OS-specific messages

core

"apr_pollset_poll failed. Attempting to shutdown process gracefully"

This message may be logged when the server is leaking resources due to PI68803.

mod_auth_saf

[error] [client ip-address] (errno)error-string (errno2=errno2): SAF user userid: authentication failure for "request-uri": Password Mismatch

The server logs this message when an invalid password is specified. The message may be more specific after PI85804.

Common error conditions are:

• 111, 0xnnnn0000 , The password is of a valid length but is not valid.

• 121, 0xnnnn02A7 (JRPasswordLenError), SAF reports that the password has an invalid length. (mod_auth_saf did not make this determination.)

[error] [client ip-address] (errno)error-string (errno2=errno2): SAF user userid not found: request-uri

The server logs this message when an invalid user id is specified. The message may be more specific after PI85804.

Common error conditions are:

• 143, 0xnnnn05DD, The user id is of a valid length but is not defined to SAF.

• 121, 0xnnnn02A6 (JRUserNameLenError), SAF reports that the user id has an invalid length. (mod_auth_saf did not make this determination.) |

[debug] (168)user [username]: SAF authentication failure 401-168 for [request-uri]: password has expired

The server logs this message when a user has entered a correct SAF password, but the password has expired. A call to the function __passwd() returned EMVSEXPIRE. Access is denied.

To enable updating of expired SAF passwords using the server, code the directive "AuthSAFExpiration". For example:

    AuthSAFExpiration on

You may also tailor the message to be presented at the client by coding some brief text instead of "on". For example:

    AuthSAFExpiration "EXPIRED PW: oldpw/newpw/newpw"

You can also use the optional AuthSAFReEnter directive to further tailor the next password message issued at the client. For example:

    AuthSAFReEnter "New PW again:"

[crit] [client ip-address] (157)EDC5157I An internal error has occurred. (errno2=0x090C02AF): user userid: SAF authentication failure 500 for request-uri: function is not supported

The server logs this message when a call to the function __passwd() returns EMVSERR.

Check the errnoJr value for the exact cause. The most common cause, as seen in the example above where errno2 is 02AF, is that you have not marked for program control one or more modules that the server loads. You can use the command "extattr +ap [modulename]" to mark a module program controlled.

[info] [client ip-address] (errno-value)errno-description (errno2=errno2-value) user userid: saf_check_pw=401 for request-uri: Other SAF failure; errno2=yyyyzzzz

The server logs this message when a call to the function __passwd() returns an error which the server does handles with specific messages. The combination of errno and errno2 explain the problem.

UNIX System Services Messages and Codes describes all errno and errno2 values. C/C++ Run-Time Library Reference contains documentation for the __passwd() function.

• z/OS V1R6.0 UNIX System Services Messages and Codes

• z/OS V1R6.0 C/C++ Run-Time Library Reference

[error] [client ip-address] access to [request-uri] failed: [requirement]

The server logs this message when it denies access to a resource based on the options in a "Require" directive in httpd.conf.

If access should have been granted, review the options in the "Require" directive.

[info] [client ip-address] access to [request-uri] failed; reason: user [username] is not in the "Require" list

The server logs this message when it denies access to a resource because a "Require" directive in httpd.conf does not specifically include the username or a group containing the username.

If access should have been granted, review the options in the "Require" directive.

[info] (xxx)rc=[return code] attempting to reset UID

The server logs this message when it has run a request under SAFRunAs control and is attempting to delete the SAF security environment and return to the server's default user ID (UID).

See the documentation for errno xxx and the pthread_security_np() function in the z/OS C/C++ Run-Time Library Reference, document number SA22-7821.

mod_ldap and mod_auth_ldap

[warn] [client 9.37.242.143] [16777781] auth_ldap authenticate: user [xxxx] authentication failed; URI /ldapdir/index.html [User not found][No such object]

The server logs this message if auth_ldap cannot find the requested dn or user, but mod_ldap is not authoritative.

The following messages will also be logged if no module is authoritative and the user is not found by any authorization module:

    [crit] [client 9.37.242.143] check_user_id hook called
    [crit] [client 9.37.242.143] configuration error: couldn't check user. No user file?: /ldapdir/index.html

See http://publib.boulder.ibm.com/httpserv/manual60/mod/mod_auth_ldap.html#authldapauthoritative. Each authorization module only authenticates based on its own knowledge. If directive AuthLDAPAuthoritative off is specified, mod_auth_ldap will not reject the request, but will allow it to be passed to other possible authorization modules. Normally, mod_auth_ldap is the only authorization module configured for a request and AuthLDAPAuthoritative is set to on.

[warn] [client 9.76.147.159] [43] auth_ldap authenticate: user [username] authentication failed; URI [path/to/ldap/directory] [LDAP: ssl connections not supported][N/A]

The server logs this message if an LDAP session was requested but failed.

If an SSL connection to the LDAP server is required, the url specified in AuthLDAPURL must begin with ldaps://, non-SSL must begin with ldap://. Additional messages which describe the problem may have appeared earlier in the error log.

[error] (121)EDC5121I Invalid argument. (errno2=0x07200316): LDAP cache: error while creating a shared memory segment: EDC5121I Invalid argument. (errno2=0x07200316)

Also: [error] (153)EDC5153I Catalog obtain error. (errno2=0xC5C7082A): LDAP cache: error while creating a shared memory segment: EDC5153I Catalog obtain error. (errno2=0xC5C7082A)

Ensure that the size specified by LDAPSharedCacheSize is valid.

Ensure that the file specified in LDAPSharedCacheFile is valid.

Delete the file specified in LDAPSharedCacheFile and try starting IBM HTTP Server again.

If a LDAPSharedCacheFile directive is specified remove it; if it is not specified then add it. If the directive is specified, named shared memory will be used; otherwise, anonymous shared memory will be used.

[crit]  [return code] LDAP cache: failed to set mutex permissions

The server may log this message if the User or Group directive in httpd.conf does not have appropriate permissions.

See the documentation for semctl() IPC_SET operation in the z/OS C/C++ Run-Time Library Reference, document number SA22-7821.

[crit] LDAP: Invalid LDAPTrustedCAType directive - KDB_FILE or SAF_KEYRING type required or mismatched

The server logs this message if the KDB file, SAF keyring, or password file are invalid.

• KDB file requirements: The file extension must be .kdb and the associated associated password file must be have the same name as the kdb file, but with a file extension of .sth.

• SAF keyring requirements: The name must be a maximum of 8 characters and meet all RACF naming conventions. There is no password file.

DSA is unwilling to perform

[error] mod_ibm_ldap: failed to search 'LDAP Realm' with filter '(&(userid=xyzzy)(objectclass=*))': (53) DSA is unwilling to perform [warn] mod_ibm_ldap: LDAP server indicates that password is  expired or user id is locked.`

If the LDAP server is on z/OS with the SDBM or RACF back-end, these two messages are a result of the z/OS LDAP server not supporting compound filters. Simplify the filter to a single condition - e.g., "racfid=&v1" - and if possible, test with ldapsearch to verify it works before using in IHS.

[warn] LDAP: SSL initialization failed OR LDAP: SSL support unavailable

The server logs one or both of these messages if SSL was not requested, or fails. If SSL support is required, check for additional messages in the log and directives such as LDAPTrustedCA and LDAPTrustedCAType.

Linux-specific messages

*** glibc detected *** free(): invalid next size (fast): 0x...

Linux has detected memory corruption in IHS, and this particular child process will likely be terminated. If your system is configured to create coredumps for IHS crashes, the core can be analyzed by the GatherCrashDoc collector tool.

Messages from third-party modules

Contact the supplier of any third-party modules in the configuration in order to address these messages.

• SemCounter ERROR:  Cannot remove  semaphore: 655433

• [notice] msgsnd() FAILED  can't send, ipc msg queue is full (EAGAIN=Resource temporarily  unavailable)

• [error] [client 1.2.3.4] (13)Permission denied: Could not open file /apps/opt/IBMHTTPD-SSO/logs/filter.52954
  This message is known to originate from SiteMinder.

• 1172793600:28168: CSmSharedSegment::smalloc - attached to shared memory segment 5111808 using key 0x6910a016
  This message is known to originate from SiteMinder. It appears to be informational in nature (i.e., not indicate an operational problem), but SiteMinder support would have to give a definitive explanation.

fork() failures

Several web server functions depend on the fork() system call to create a new process. Specific requests can fail or web server performance can be degraded if this system call fails.

The specific cause for failure is usually one of the following:

• limits on processes for the user which started the web server may be exceeded

• system-wide limits on processes may be exceeded

• there may be a system resource shortage (internal kernel tables, paging space)

More information may be available in the operating system documentation for the fork() subroutine.

AIX

The AIX documentation for fork() is http://publib16.boulder.ibm.com/doc_link/en_US/a_doc_lib/libs/basetrf1/fork.htm. The applicable limit is maxuproc. The following commands show how to display the current value and set it to a new value:

    # lsattr -El sys0 -a maxuproc
    maxuproc 2048 Maximum number of PROCESSES allowed per user True
    # chdev -l sys0 -a maxuproc=4096
    sys0 changed