IBM WebSphere Application Server Performance Cookbook

J9 System Dump Recipe

The following are instructions to gather a reliable operating system (OS) process core dump of an OpenJ9 JVM (IBM Java, IBM Semeru Runtimes, etc.). Although the instructions are long and complicated, OS core dumps are one of those most powerful diagnostics and they are worth the time to understand and properly gather.

Note that an OS process core dump is referred to as a "system dump" in J9 documentation but this is the same as an operating system core dump of a single process (despite "system" being in the name). These names may be used interchangeably.

Also, an OS core dump should not be confused with a javacore despite both having "core" in the name. In contrast to an OS core dump, a javacore is lightweight and small whereas an OS core dump is heavyweight and big and they serve different purposes.

Always gather a system dump using a mechanism that uses the JVM system dump API with request=exclusive+prepwalk as shown below. Gathering a system dump using operating system utilities is not reliable (e.g. gcore, gencore, Windows Task Manager, z/OS manual console dump, etc.). For example, if you get unlucky and happen to grab it during a garbage collection, it will likely be essentially useless for Java heap analysis. Even if it's not taken during such a critical event, some thread stacks may not be walkable.

Before gathering an operating sytem core dump, review the security, disk and performance risks and ensure that the process and operating system are configured for it (e.g. Unix core and file ulimits, Linux kernel.core_pattern truncation settings, etc.).

After gathering a core dump, if it will be used for Java heap analysis with the Eclipse Memory Analyzer Tool, then no post-processing is needed (just ensure it's extracted and ends with the .dmp extension). If it will be used for native memory or crash analysis, run $JAVA/bin/jextract on it for IBM Java and $JAVA/bin/jpackcore for Semeru Java.

Examples requesting a system dump using the JVM system dump API with request=exclusive+prepwalk:

  1. For Semeru Java, replace $JAVA_HOME with the path to Java, and $PID with the process ID in: 
    $JAVA_HOME/bin/jcmd $PID Dump.system
  2. For IBM Java >= 8.0.6.25 that is a JDK (does not work with JREs), replace $JAVA_HOME twice with the path to Java, and $PID with the process ID in: 
    $JAVA_HOME/bin/java -Xbootclasspath/a:$JAVA_HOME/lib/tools.jar openj9.tools.attach.diagnostics.tools.Jcmd $PID Dump.system
  3. Starting with WebSphere Application Server traditional 8.5.5.17 and 9.0.5.2:
    1. Administrative Console } Troubleshooting } Java dumps and cores } Check the server(s) } System dump
    2. $WEBSPHERE/bin/wsadmin -lang jython and the command AdminControl.invoke(AdminControl.completeObjectName("type=JVM,process=$SERVER,*"), "generateSystemDump")
    3. On z/OS, MODIFY $JOB,JAVATDUMP
    4. On IBM i, WRKJVMJOB with Type=SYSTEM
  4. Starting with WebSphere Liberty 20.0.0.2: 
    $LIBERTY/bin/server javadump $SERVER --include=system
  5. For IBM Java >= 8.0.7.20 and Semeru >= 11.0.17.0 on non-Windows platforms, restart with: 
    -Xdump:system:events=user2,request=exclusive+prepwalk
    Then request the system dump with: 
    kill -USR2 $PID
  6. For IBM Java, use Java Surgery: 
    java -jar surgery.jar -pid $PID -command SystemDump
  7. Use code within the JVM that executes com.ibm.jvm.Dump.triggerDump("system:request=exclusive+prepwalk") using reflection
  8. Use -Xdump:java+system:events=user,request=exclusive+prepwalk to take one on kill -3/Ctrl+Break. Note that we do not recommend running with this option permanently because the default handler only produces javacores which are often used for performance investigations whereas a system dump causes its own significant performance overhead.
  9. Use -Xdump:system:defaults:request=exclusive+prepwalk to change the system dump default to request exclusive+prepwalk and then use some mechanism that requests a system dump within the JVM. Note that we do not recommend running with this option permanently because then investigating JVM crashes may be problematic.
  10. Use -Xdump:tool:events=user,request=exclusive+prepwalk,exec="gcore %pid" to execute a program (e.g. gcore) that requests the core dump on kill -3/Ctrl+Break. Note that we do not recommend running with this option permanently because the default handler only produces javacores which are often used for performance investigations whereas a system dump causes its own significant performance overhead.
  11. The IBM Java system Dump Agent can take a system dump on various events. For example, the following will create a system dump when the Example.bad method throws a NullPointerException: 
    -Xdump:system:events=throw,range=1..1,request=exclusive+prepwalk,filter=java/lang/NullPointerException#com/ibm/example/Example.bad
  12. The trace engine may be used to request a system dump on method entry and/or exit. The following example JVM argument produces a system dump when the Example.trigger() method is called: 
    -Xtrace:maximal=mt,trigger=method{com/ibm/example/Example.trigger,sysdump,,,1}