# apr\_thread\_create() failures 

## The symptom

One of the following messages may occur at any time:

  - \[alert\] (11)Resource temporarily unavailable: apr\_thread\_create:
    unable to create worker thread
  - \[alert\] (12)Cannot allocate memory: apr\_thread\_create: unable to
    create worker thread

This will often happen after one of the following events:

  - `MaxClients` or `ThreadsPerChild` has been increased.
  - New instances of IHS are put into service
  - New colocated instances of WebSphere are put into service.
  - Operating system release is upgraded.
    For example, RHEL6 reduced roots default process (thread) limit to
    1024, so this tuning is required more frequently then it was in the
    past.

## The cause

IHS has failed to create a native thread. The reasons for the failure
are documented by the operating system (pthread_create), but generally fall 
into two high level categories:

  - There is a per-user or per-system limit on the number of threads or
    processes and it has been exceeded.
  - The system ran out of virtual memory, or the process ran out of
    addressable memory due to stack space requirements.

## The solution

  - Linux:
    
        
      - Confirm /proc/sys/kernel/threads-max and /proc/sys/kernel/pid_max
        are much larger than the total number
        of possible threads on the system
    
      - Validate the maximum number of threads the system will permit
        *for the user who executes apachectl* (typically root) *and* the
        configured User.
        
        From an interactive shell, check the output of `ulimit -u` to
        confirm the value significantly exceeds `MaxClients` + threads
        from all other sources on the system that share a *startup*
        userid.
        
        Choose one remediation:
        
          - Set `ulimit -u` in $IHSROOT/bin/envvars (`MaxClients` +
            other users threads on the system)
        
          - Update the "nproc" soft limit *for the user who executes
            apachectl* (typically root) for the required number of
            system wide threads in /etc/security/limits.conf
            
            Note: By default, most Linux configurations do not enforce
            /etc/security/limits.conf settings when a process changes
            its userid at runtime, which is what prevents you from
            increasing the limit for *just*the configured User
            directive, e.g. "nobody" or "wasadmin". If you elect to make
            the change in /etc/security/limits.conf and it does not
            help, try making the same change for the user specified in
            the IHS User directive.
            
            Changing the configured User is especially important if it
            has large number of non-httpd threads on the system, because
            the httpd threads will count towards that users limit for

       - 32-bit IHS: Set `ulimit -s 512` in $IHSROOT/bin/envvars and do a full stop
        and start.  Linux may be using 8MB per thread of memory without it, which
        causes startup failures in 32-bit IHS and large
        `ThreadsPerChild`
    
     starting other software.

  - AIX: On 32-bit instances, the value of LDR\_CNTRL variable in `IHSROOT/bin/envvars` file
    controls how much of the address space can be used for heap storage,
    and heap storage is the main requirement on AIX for being able to
    create a new thread.
    
    The MAXDATA setting within LDR\_CNTRL should be at least 0x60000000
    in order to allow a large number of threads. Example:
    
    ``` 
        LDR_CNTRL="MAXDATA=0x80000000" ; export LDR_CNTRL ...
    ```
   
    On 64-bit IHS instances, LDR_CNTRL should be unset, as it will only
    limit the amount of memory IHS can use.  Similarly, `ulimit -d` should
    be either unlimited or at the mimimum several gigabytes.

  - HP-UX: Ensure that the `max_thread_proc` kernel setting is at least
    five more than the setting of ThreadsPerChild. `max_thread_proc` is
    the maximum number of threads in a process. A related kernel setting
    is `nkthread`, the maximum number of threads in the system.  
    `max_thread_proc` and `nkthread` can be displayed or modified by the
    `sam` program. With some levels of HP-UX, changing the value
    requires a kernel rebuild and a reboot of the system.  
    Changing the setting to more than the absolute minimum will allow
    future web server configuration changes without having to modify
    this kernel setting. Other applications may benefit as well.

  - Other platforms: If `ThreadsPerChild` has been dramatically increased, decrease it.
    
    If `MaxClients` has been dramatically increased, or there are many
    other IHS, WAS, or any other threads on the system, increase the
    users' and systems overall thread limits.