Threads in java.io.FileOutputStream.writeBytes

A thread dump (javacore) shows a lot of threads in a stack that looks like

"WebContainer : 89" daemon prio=10 tid=0x01683c58 runnable (0x73f7d000..0x73f7faf0)
  at java.io.FileOutputStream.writeBytes(Native Method)
  at java.io.FileOutputStream.write(FileOutputStream.java:260)
  at com.ibm.ejs.ras.WrappingFileOutputStream.write(WrappingFileOutputStream.java:364)
  - locked (0x97ff0230) (a com.ibm.ejs.ras.WrappingFileOutputStream)
  at java.io.PrintStream.write(PrintStream.java:412)

Strategy 1:  Ran out of disk space OR Slow file system I/O OR Anti-Virus Protection OR Active backup

Can be due to either running out of disk space on the file system or the file system I/O is slow (i.e. high latency connection to a SAN).

• Check if the file system is full.  If the file system is full then archive and delete unnecessary files.
• If the file system is slow then change the application configuration to point to a more robust file system.
• Anti-Virus protection may be aggressively scanning the file system providing limited access to all other applications to the file system.
• Active backup that is aggressively accessing the file system providing limited access to all other applications to the file system.

Monitor

• If the disk is highly utilized (for example, 80%), notify the appropriate system administrators.
• File system performance.  If aggressive disk usage is detected above your threshold, notify the appropriate system administrators.
  • Investigate re-architecting the environment so that not all the applications are pointed to the same file system.
  • If the problem is related to local disk speed replace local disks with faster disks.
    • If this is due to too many vertically deployed application servers consider expanding the infrastructure horizontally.
• If Anti-Virus protection is aggressively accessing the file system then reconfigure the process not to aggressively access the file system.
• If a backup is aggressively accessing the file system then either reconfigure the process not to aggressively access the file system or investigate using other disk replication techniques.

Caveats

May require restarting the application servers which may require an outage. 

Some of the recommended re-architecture/infrastructure can be quite extensive and time/labor consuming.  Plan appropriately.

Make sure to investigate a multiple cell infrastructure deployment to help improve availability.

Previous Section (Slow or Hung Application) | Next Section (Logging PMI Data) | Back to Table of Contents