apr_thread_create() failures

The symptom

One of the following messages may occur at any time:

  • [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread

  • [alert] (12)Cannot allocate memory: apr_thread_create: unable to create worker thread

This will often happen after one of the following events:

  • MaxClients or ThreadsPerChild has been increased.

  • New instances of IHS are put into service

  • New colocated instances of WebSphere are put into service.

  • Operating system release is upgraded. For example, RHEL6 reduced roots default process (thread) limit to 1024, so this tuning is required more frequently then it was in the past.

The cause

IHS has failed to create a native thread. The reasons for the failure are documented by the operating system, but generally fall into two high level categories:

  • There is a per-user or per-system limit on the number of threads or processes and it has been exceeded.

  • The system ran out of virtual memory, or the process ran out of addressable memory due to stack space requirements.

The solution

  • Linux:

    • Set ulimit -s 512 in $IHSROOT/bin/envvars and do a full stop and start,

      Linux may be using 8MB per thread of memory without it, which causes startup failures in 32-bit IHS and large ThreadsPerChild

    • Confirm sysctl kernel.pid_max and /proc/sys/kernel/threads-max are much larger than the total number of possible threads on the system

    • Validate the maximum number of threads the system will permit for the user who executes apachectl (typically root) and the configured User.

      From an interactive shell, check the output of ulimit -u to confirm the value significantly exceeds MaxClients + threads from all other sources on the system that share a startup userid.

      Choose one remediation:

      • Set ulimit -u in $IHSROOT/bin/envvars (MaxClients + other users threads on the system)

      • Update the "nproc" soft limit for the user who executes apachectl (typically root) for the required number of system wide threads in /etc/security/limits.conf

        Note: By default, most Linux configurations do not enforce /etc/security/limits.conf settings when a process changes its userid at runtime, which is what prevents you from increasing the limit for justthe configured User directive, e.g. "nobody" or "wasadmin". If you elect to make the change in /etc/security/limits.conf and it does not help, try making the same change for the user specified in the IHS User directive.

        Changing the configured User is especially important if it has large number of non-httpd threads on the system, because the httpd threads will count towards that users limit for starting other software.

  • AIX: On 32-bit instances, the value of LDR_CNTRL variable in IHSROOT/bin/envvars file controls how much of the address space can be used for heap storage, and heap storage is the main requirement on AIX for being able to create a new thread.

    The MAXDATA setting within LDR_CNTRL should be at least 0x60000000 in order to allow a large number of threads. Example:

        LDR_CNTRL="MAXDATA=0x80000000" ; export LDR_CNTRL ...

    On 64-bit IHS instances, LDR_CNTRL should be unset, as it will only limit the amount of memory IHS can use. Similarly, ulimit -d should be either unlimited or at the mimimum several gigabytes.

  • HP-UX: Ensure that the max_thread_proc kernel setting is at least five more than the setting of ThreadsPerChild. max_thread_proc is the maximum number of threads in a process. A related kernel setting is nkthread, the maximum number of threads in the system.
    max_thread_proc and nkthread can be displayed or modified by the sam program. With some levels of HP-UX, changing the value requires a kernel rebuild and a reboot of the system.
    Changing the setting to more than the absolute minimum will allow future web server configuration changes without having to modify this kernel setting. Other applications may benefit as well.

  • Other platforms: If ThreadsPerChild has been dramatically increased, decrease it.

    If MaxClients has been dramatically increased, or there are many other IHS, WAS, or any other threads on the system, increase the users' and systems overall thread limits.