apr_thread_create() failures¶
The symptom¶
One of the following messages may occur at any time:
[alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[alert] (12)Cannot allocate memory: apr_thread_create: unable to create worker thread
This will often happen after one of the following events:
MaxClients
orThreadsPerChild
has been increased.New instances of IHS are put into service
New colocated instances of WebSphere are put into service.
Operating system release is upgraded. For example, RHEL6 reduced roots default process (thread) limit to 1024, so this tuning is required more frequently then it was in the past.
The cause¶
IHS has failed to create a native thread. The reasons for the failure are documented by the operating system (pthread_create), but generally fall into two high level categories:
There is a per-user or per-system limit on the number of threads or processes and it has been exceeded.
The system ran out of virtual memory, or the process ran out of addressable memory due to stack space requirements.
The solution¶
Linux:
Confirm /proc/sys/kernel/threads-max and /proc/sys/kernel/pid_max are much larger than the total number of possible threads on the system
Validate the maximum number of threads the system will permit for the user who executes apachectl (typically root) and the configured User.
From an interactive shell, check the output of
ulimit -u
to confirm the value significantly exceedsMaxClients
+ threads from all other sources on the system that share a startup userid.Choose one remediation:
Set
ulimit -u
in $IHSROOT/bin/envvars (MaxClients
+ other users threads on the system)Update the "nproc" soft limit for the user who executes apachectl (typically root) for the required number of system wide threads in /etc/security/limits.conf
Note: By default, most Linux configurations do not enforce /etc/security/limits.conf settings when a process changes its userid at runtime, which is what prevents you from increasing the limit for justthe configured User directive, e.g. "nobody" or "wasadmin". If you elect to make the change in /etc/security/limits.conf and it does not help, try making the same change for the user specified in the IHS User directive.
Changing the configured User is especially important if it has large number of non-httpd threads on the system, because the httpd threads will count towards that users limit for
32-bit IHS: Set
ulimit -s 512
in $IHSROOT/bin/envvars and do a full stop and start. Linux may be using 8MB per thread of memory without it, which causes startup failures in 32-bit IHS and largeThreadsPerChild
starting other software.
AIX: On 32-bit instances, the value of LDR_CNTRL variable in
IHSROOT/bin/envvars
file controls how much of the address space can be used for heap storage, and heap storage is the main requirement on AIX for being able to create a new thread.The MAXDATA setting within LDR_CNTRL should be at least 0x60000000 in order to allow a large number of threads. Example:
LDR_CNTRL="MAXDATA=0x80000000" ; export LDR_CNTRL ...
On 64-bit IHS instances, LDR_CNTRL should be unset, as it will only limit the amount of memory IHS can use. Similarly,
ulimit -d
should be either unlimited or at the mimimum several gigabytes.HP-UX: Ensure that the
max_thread_proc
kernel setting is at least five more than the setting of ThreadsPerChild.max_thread_proc
is the maximum number of threads in a process. A related kernel setting isnkthread
, the maximum number of threads in the system.max_thread_proc
andnkthread
can be displayed or modified by thesam
program. With some levels of HP-UX, changing the value requires a kernel rebuild and a reboot of the system.
Changing the setting to more than the absolute minimum will allow future web server configuration changes without having to modify this kernel setting. Other applications may benefit as well.Other platforms: If
ThreadsPerChild
has been dramatically increased, decrease it.If
MaxClients
has been dramatically increased, or there are many other IHS, WAS, or any other threads on the system, increase the users' and systems overall thread limits.