Solaris SysV semaphore configuration

The default semaphore configuration on Solaris is not adequate for using Apache/IHS with SysV semaphores for the accept mutex for more than a handful of child processes. The problem is that the web server uses a kernel "undo structure" for each child process waiting on the accept mutex, and the Solaris kernel allocates a relatively small number of these structures by default. By design, the web server does not use SysV semaphores by default on Solaris, but there are some situations where the administrator may be advised to use SysV semaphores.

A complicating issue with the semaphore tuning is that there may be other applications running on the system that use semaphore resources. The description here merely suggests what is necessary for the web server's accept mutex. You may need additional kernel resources for other applications. Thus, it is suggested that you allocate plenty of extra kernel semaphore resources because they can't be changed dynamically but instead require editing /etc/system and rebooting the system.

With Apache/IHS 1.3, one of the symptoms of an inadequate kernel SysV semaphore configuration is a message like this in the error log:

accept_mutex_on: No space left on device

(This is the ENOSPC error.) The Apache parent process will exit shortly thereafter.

The active kernel semaphore resources are displayed by the sysdef command. Here is the portion of example output related to semaphores:

    *
    * IPC Semaphores
    *
       150  semaphore identifiers (SEMMNI)
       500  semaphores in system (SEMMNS)
       380  undo structures in system (SEMMNU)
       125  max semaphores per id (SEMMSL)
        50  max operations per semop call (SEMOPM)
        50  max undo entries per process (SEMUME)
     32767  semaphore maximum value (SEMVMX)
     16384  adjust on exit max value (SEMAEM)

(These are just examples, not recommendations.)

The kernel semaphore resources are controlled in the file /etc/system with statements like this, which override the default values.

    set semsys:seminfo_semmns=3000
    set semsys:seminfo_semmni=100
    set semsys:seminfo_semmsl=1200
    set semsys:seminfo_semvmx=32767
    set semsys:seminfo_semopm=100
    set semsys:seminfo_semmnu=500

(These are just examples, not recommendations.)

When the settings are modified in /etc/system, the system must be rebooted for the change to take effect.

Specifc parameters

The parameters that are of interest when running Apache/IHS are listed below, along with requirements for AcceptMutex sysvsem. This does not take into account anything other than the accept mutex. Note that 3rd party modules used with the web server can use semaphores on their own. "Apache/IHS" refers only to the server and standard modules, when the directive "AcceptMutex sysvsem" has been chosen.

semsys:seminfo_semmni - SEMMNI - semaphore identifiers

The web server will only use one per web server instance for the accept mutex. If there are not enough semaphore identifiers available, IHS will write this to error log:

    [Thu Aug 28 17:26:02 2003] [emerg] (28)No space left on device: Couldn't create accept lock

semsys:seminfo_semmns - SEMMNS - semaphores in system

The web server will only use one per web server instance for the accept mutex.

If there are not enough semaphores available, the symtom is the same as shown above under SEMMNI.

semsys:seminfo_semmnu - SEMMNU - undo structures in system

The web server will use one undo structure per child process for the accept mutex, so figure out how many child processes there can be in your configuration to determine how to adjust this.

Apache/IHS 2.0 with worker MPM: if MaxRequestsPerChild=0 and idle child cleanup is disabled, use MaxClients/ThreadsPerChild

If there are insufficient undo structures, the web server could fail at any time (perhaps far after initialization when a new child process is created to handle more load). The message will look like this:

    [Thu Aug 28 17:44:42 2003] [emerg] (28)No space left on device:
    apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.

Occasionally you may need to adjust the number of web server child processes to handle increased load. That requires a restart of the web server. But that may also require the number of undo structures to be adjusted, which requires a reboot of the machine. Thus, it is wise to overallocate the number of kernel undo structures to allow the number of child processes to be increased without rebooting the machine.

semsys:seminfo_semmsl - SEMMSL - max semaphores per id

One is sufficient for the web server accept mutex.

semsys:seminfo_semopm - SEMOPM - max operations per semop call

One is sufficient for the web server accept mutex.

semsys:seminfo_semume - SEMUME - max undo entries per process

One is sufficient for the web server accept mutex.

Further info

Solaris documentation on tunable parameters for SysV semaphores is at http://docs.sun.com/db/doc/806-7009/6jftnqsj6?a=view.