AIX sendfile/send_file issues

On AIX, the system call used to send static files with EnableSendfile On is send_file(). Files sent via send_file() can be cached by the kernel in the network buffer cache.

Problems

Default tuning of cache size too high

Default AIX tuning of the network buffer cache size is not appropriate on some AIX systems, and use of send_file() by IHS can result in large amounts of kernel memory devoted to the caching of static files. This may impact other system operations. Because very few applications utilize the network buffer cache, the use of send_file() by IBM HTTP Server may be the the trigger for a problem symptom if the network buffer cache limits are not appropriate.

How to address this issue if send_file() needs to be used:

  1. Display the old cache size, with the command no -o nbc_limit.

  2. Determine if a new value is appropriate. We recommend that the value be no larger than 128MB initially. Choose the current value or 128MB, whichever is smaller.

  3. If the new value is different, set it with no -o nbc_limit=new_value.

  4. Monitor actual network buffer cache memory usage with the command netstat -c during normal web server operation. If the memory usage is at or near the configured maximum value, increasing the limit may yield better performance. If memory usage is significantly smaller than the maximum value, we recommend reducing the maximum value to a value closer to actual usage so that the kernel memory usage can't increase significantly after this initial monitoring phase has completed.

Memory use increases with a larger number of unique files served.

Use the -p option of the no command to make a permanent change to the value of nbc_limit.

Delivery of old file contents

The network buffer cache may send old file contents even after the file has been updated by the user. This will result in the web browser receiving incorrect file contents.

If the file is now larger than the previously cached content, the client may receive no response at all, with the following logged in the main error log:

  [Wed Jun 20 08:58:24 2007] [info] [client 127.0.0.1] (22)Invalid argument: core_output_filter: writing data to the network

If the file is now smaller than the previously cached content, the client may receive a truncated version of the original contents.

The network buffer cache will check for updates to the file in order to send the most recent version, but only after a configurable interval has elapsed. This interval is controlled by the send_file_duration variable which can be displayed or set with the no command.

How to address this issue if send_file() needs to be used:

Change the network buffer cache to check for file updates before sending

Set the send_file_duration interval to 0; the network buffer cache will now check for file updates each time the file is sent. This can affect other applications on the system which utilize send_file(). The default value for send_file_duration is 300 (seconds), so without this configuration change the wrong data could be sent for up to five minutes after the file update.

Example:

  no -o send_file_duration=0

Disable the cache temporarily while making file updates

Disable the cache, make the file updates, then restore the cache, as follows:

  1. Display the old cache size, with the command no -o nbc_limit.

  2. Set the size of the network buffer cache to zero temporarily to clear the cache, with the command no -o nbc_limit=0.

  3. Update the static files.

  4. Restore the previous network buffer cache size, with the command no -o nbc_limit=old_value, where old_value is the value displayed in step 1 above.

The set of files to be served with send_file() can be customized in order to avoid needing to use the special update mechanism for infrequently-served files. Examples:

  # Example 1.  Sendfile is enabled globally, but we disable it
  #             for some rarely-served files in a particular
  #             directory.
  <Directory /www/site/www.example.com/infrequently_served/>
  EnableSendfile Off
  </Directory>

  # Example 2.  Sendfile is disabled globally, but we enable it
  #             for some often-served files.
  <FilesMatch "^(robots\.txt|index\.html|favicon\.ico)$">
  EnableSendfile On
  </FilesMatch>

Can't unmount filesystem

On some levels of AIX, filesystems containing files sent by IHS via send_file() can not be unmounted until the network buffer cache is emptied.

How to address this issue if send_file() needs to be used:

When unmounting the filesystem containing cached files,

  1. Display the old cache size, with the command no -o nbc_limit.

  2. Set the size of the network buffer cache to zero temporarily to clear the cache, with the command no -o nbc_limit=0.

  3. Unmount the filesystem.

  4. Restore the previous network buffer cache size, with the command no -o nbc_limit=old_value, where old_value is the value displayed in step 1 above.

Other documentation: