IBM WebSphere Application Server Performance Cookbook

Troubleshooting OpenJ9 and IBM J9 Recipes

  1. Write verbosegc to rotating log files; for example, -Xverbosegclog:verbosegc.%Y%m%d.%H%M%S.%pid.log,5,51200
  2. On recent versions of IBM Java, enable Health Center to write to rotating log files; for example, -Xhealthcenter:level=headless -Dcom.ibm.java.diagnostics.healthcenter.headless.files.max.size=268435456 -Dcom.ibm.java.diagnostics.healthcenter.headless.files.to.keep=4
  3. Periodically monitor stderr (native_stderr.log in WAS Classic, console.log in WebSphere Liberty) for "JVM" messages, including those noting the production of javacores, heapdumps, core dumps, and snap dumps.
  4. Create a dedicated filesystem for JVM artifacts such as javacores, heapdumps, Snaps, and core dumps (so that if it fills up, the program directories are not affected) and use the -Xdump directory option to change the default directory of these artifacts; for example, -Xdump:directory=${SOME_DIRECTORY} and also set -Xdump:nofailover if there is any concern about filling up the temporary directory.
  5. In recent versions, a core dump is produced on the first OutOfMemoryError. Assuming core dumps are configured correctly to be untruncated (see the Troubleshooting Operating System Recipes), then the core dump is sufficient to investigate OutOfMemoryErrors (a PHD may always be extracted from a core) and you should disable heapdumps with -Xdump:heap:none
  6. Enable large object allocation tracking and monitor stderr for JVMDUMP039I messages; for example, -Xdump:stack:events=allocation,filter=#10m
  7. Consider setting the excessive garbage collection threshold (at which point the JVM is considered to be out of Java memory) to something more aggressive; for example, -Xgc:excessiveGCratio=80
  8. A well-tuned JVM is a better-behaving JVM, so also review the Java tuning recipes.
  9. Review the Troubleshooting Operating System Recipes.

Java OutOfMemoryError (OOM)

  1. If you have verbosegc enabled (you should), then review the verbosegc log:
    1. Review the allocation failure right before the OOM to see its size. The cause of the OOM may be an abnormally large allocation request.
    2. Review the pattern of heap usage to see if there are signs of a leak.
  2. By default, an OutOfMemoryError should produce a javacore.txt file. Review the javacore:
    1. Review the reason code for the OutOfMemoryError at the top of the javacore. For example:
      1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Java heap space" received
    2. Review the maximum heap size in the javacore. In general, if -Xmx is <= 512M, a sizing exercise may not have been done. For example:
      2CIUSERARG -Xmx3800m
    3. Search for the word "deadlock." If you find "Deadlock detected !!!" then investigate the cause of the deadlock. A deadlock often indirectly causes an OutOfMemory because the deadlocked threads and any threads waiting for a monitor owned by the deadlocked threads are hung indefinitely and this may hold a lot of memory on those threads or impede other processing that cleans up memory.
    4. In some cases, the thread that proximately causes the OOM is reported as the "Current thread." Review the stack for anything abnormal. For example:
      1XMCURTHDINFO Current thread
  3. Review the coredump or heapdump in the Eclipse Memory Analyzer Tool.

Additional Recipes