Troubleshooting Oracle (HotSpot) Java
jmap
jmap is an unsupported memory diagnostics tool shipped with the JDK: "This utility is unsupported and may or may not be available in future versions of the JDK." (http://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html).
histo
The `histo` option may be used to print a histogram of Java objects by class, including number of instances and number of bytes. The `live` option only counts reachable objects, although it does force a Full GC.
jinfo
jinfo is an unsupported tool shipped with the JDK which prints Java configuration of a live Java process or from a core dump: http://docs.oracle.com/javase/7/docs/technotes/tools/share/jinfo.html
Thread Dump
Oracle Java can produce a thread dump which is printed to stdout and details the activity of each thread.
Request Thread Dump
- On POSIX operating systems, by default, the command `kill -3 ${PID}` will request a thread dump.
HPROF Heapdumps
An HPROF heapdump contains the full Java heap object graph as well as Java object memory contents (for example, Strings, primitives, etc.). This is used for investigating OutOfMemoryErrors, tuning Java heap usage, etc.
By default, when a Java memory request cannot be fulfilled, an OutOfMemoryError is thrown, but an HPROF dump is not produced. Use -XX:+HeapDumpOnOutOfMemoryError to produce an HPROF dump in this condition: http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html. Starting with Java 6, by default (-XX:+UseGCOverheadLimit), when garbage collection is more than 98% of the processing time of the process (-XX:GCTimeLimit=98) and less than 2% of the heap is being recovered (-XX:GCHeapFreeLimit=2), an OutOfMemoryError is thrown with the details "GC overhead limit exceeded" (an HPROF dump is only produced with -XX:+HeapDumpOnOutOfMemoryError): http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html. HeapDumpOnOutOfMemoryError only produces a dump on the first OOM: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6280629
-XX:HeapDumpPath may be used to control where the dumps are written to: http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
Some HPROF options may be changed while the JVM is running using MBeans (e.g. jconsole).
-XX:OnOutOfMemoryError may be used to execute an operating system command on an OOM: http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
On recent versions of HotSpot Java, an HPROF heapdump also includes thread information which describes which Java objects are stack frame locals on which stacks (for example, you can see the actual SQL string for an executing database query). Available with Java 6 Update >= 14 and Java 7: https://bugs.eclipse.org/bugs/show_bug.cgi?id=268458
To analyze heapdumps, see the IBM Memory Analyzer Tool chapter.
Generating HPROF heapdumps
Additional methods of requesting heap dumps are documented in the Troubleshooting Operating Systems.
- Automatically produced on OOM with -XX:+HeapDumpOnOutOfMemoryError:
- Ctrl+Break or kill -3 with -XX:+HeapDumpOnCtrlBreak
- Pass the PID to the jmap tool. Note: The jmap tool is unsupported: "This utility is unsupported and may or may not be available in future versions of the JDK." (http://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html).
$ jmap -dump:format=b,file=heap.hprof ${PID} - Produce an operating system core dump (see the Troubleshooting Operating Systems chapter) and then extract the HPROF heapdump from the core dump:
$ jmap -dump:format=b,file=heap.hprof ${PATH_TO_JAVA} ${PATH_TO_CORE} - Use -XX:OnOutOfMemoryError (see below)
- Jconsole with HotSpotDiagnostic Mbean dumpHeap
- From within MAT: File > Acquire Heap Dump
Use -XX:OnOutOfMemoryError to Spawn jmap
#!/bin/sh
# Usage:
# 1. Create oom.sh with the contents of this script
# 2. Change paths in the "Variables" section if needed
# 3. chmod a+x oom.sh
# 4. Run java with the following argument, replacing $PATH with path to oom.sh
# -XX:OnOutOfMemoryError="/$PATH/oom.sh %p"
# Variables
LOCKFILE=/tmp/oomlock
OUT=/tmp/oomout.txt
NOW=`date +"%Y%m%d_%H%M%S"`
CURDIR=`pwd`
JAVA_HOME=/opt/IBM/WebSphere/AppServer/java/
# Execution
echo "OOM handler script started for PID $1 at $NOW in $CURDIR" >> $OUT
if [ ! -f $LOCKFILE ]; then
touch $LOCKFILE >> $OUT 2>&1
NOW=`date +"%Y%m%d_%H%M%S"`
echo "OOM handler requested hprof at $NOW" >> $OUT
FILENAME="heap_${NOW}_${1}.hprof"
$JAVA_HOME/bin/jmap -F -dump:format=b,file=$FILENAME $1 >> $OUT 2>&1
# /usr/bin/gcore -F -o core_$NOW.dmp $1 >> $OUT 2>&1
CODE=$?
echo "OOM handler returned with $CODE at $NOW" >> $OUT
rm -f $LOCKFILE >> $OUT 2>&1
fi
NOW=`date +"%Y%m%d_%H%M%S"`
echo "OOM handler finished at $NOW" >> $OUT
Use -XX:OnOutOfMemoryError to Spawn gcore
#!/bin/sh
# Usage:
# 1. Create oom.sh with the contents of this script
# 2. Change paths in the "Variables" section if needed
# 3. chmod a+x oom.sh
# 4. Run java with the following argument, replacing $PATH with path to oom.sh
# -XX:OnOutOfMemoryError="/$PATH/oom.sh %p"
# 5. After an OOM occurs, check /tmp/oomout.txt for output of the command
# 6. Run `jmap -dump:format=b,file=heap.hprof ${PATH_TO_JAVA} ${CORE}`
#
# Notes:
# OnOutOfMemoryError runs /usr/bin/sh -c $CMD synchronously, during which it
# appears it has a lock that prevents jmap to attach. Tried -F, but that
# generated an infinite loop and ran into other issues, so running gcore
# instead.
# Variables
LOCKFILE=/tmp/oomlock
OUT=/tmp/oomout.txt
NOW=`date +"%Y%m%d_%H%M%S"`
CURDIR=`pwd`
GCORE_PATH=/usr/bin/gcore
# Execution
echo "OOM handler script started for PID $1 at $NOW in $CURDIR" >> $OUT
if [ ! -f $LOCKFILE ]; then
touch $LOCKFILE >> $OUT 2>&1
NOW=`date +"%Y%m%d_%H%M%S"`
echo "OOM handler requested hprof at $NOW" >> $OUT
# $JAVA_HOME/bin/jmap -dump:format=b,file=heap_$1.hprof $1 >> $OUT 2>&1
$GCORE_PATH -F -o core_$NOW.dmp $1 >> $OUT 2>&1
CODE=$?
echo "OOM handler returned with $CODE at $NOW" >> $OUT
rm -f $LOCKFILE >> $OUT 2>&1
fi
NOW=`date +"%Y%m%d_%H%M%S"`
echo "OOM handler finished at $NOW" >> $OUT
Code to Request Diagnostics from within the JVM
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.lang.management.ManagementFactory;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.concurrent.atomic.AtomicInteger;
public class Play
{
public static void main(String... args) throws Throwable
{
System.out.println("Requesting core...");
tryGenerateCore();
}
public static void tryGenerateCore()
{
try
{
String requestedFileName = generateCore();
if (requestedFileName != null)
{
System.out.println("Started writing core dump to " + requestedFileName);
}
}
catch (Throwable t)
{
System.out.println("Error generating core: " + t.getLocalizedMessage());
t.printStackTrace();
}
}
private final static boolean ENABLE_REQUESTING_COREDUMPS = Boolean.getBoolean("ENABLE_REQUESTING_COREDUMPS");
private final static SimpleDateFormat DIAG_NAME_FORMAT = new SimpleDateFormat("yyyyMMdd.HHmmss");
private final static String CORE_PROGRAM_PATH = System.getProperty("CORE_PROGRAM_PATH", "/usr/bin/gcore");
private final static int MAX_CORE_DUMPS = Integer.getInteger("MAX_CORE_DUMPS", 1);
private static final AtomicInteger coreDumpsTaken = new AtomicInteger();
private static int coreDumpsRequested;
/**
* Disabled by default. Enable with -DENABLE_REQUESTING_COREDUMPS=true
* <p />
* Request a non-destructive core dump in a separate thread by spawning out
* to the gcore command. gcore will attach to and pause the process, dump
* all virtual memory (so the size will be about the size in ps VSZ) and
* then the process should continue. Unlike an OOM or using jmap to request
* an HPROF dump, requesting a core does not request a Full GC. Jmap can be
* used to extract an HPROF heapdump from the core:
* <p />
* <code>$ jmap -dump:format=b,file=heap.hprof ${PATH_TO_java} ${CORE}</code>
* <p />
* Whereas asking the JVM to generate a heapdump with jmap is a complex
* operation because the JVM has to walk all the data structures, the
* operating system generating a core is very simple: the OS just pauses the
* process and dumps out all of the virtual memory. The overhead of a core
* file is almost completely in writing the large amount of bytes to disk.
* There are some techniques to make this very fast. First, if there is
* sufficient filecache in RAM (i.e. a large amount of free RAM), then the
* OS will write the core to RAM and then asynchronously write to disk, thus
* making the pause quite fast. However, this can have some performance side
* effects. An alternative way to do this is to mount a RAMdisk and write
* the core to a RAMdisk.
* <p />
* Warning: ensure sufficient core, file and other ulimits. Also ensure
* sufficient disk space in the current working directory.
*
* @return null if -DMAX_CORE_DUMPS (default 1) has been reached or
* -DENABLE_REQUESTING_COREDUMPS=false; otherwise, the requested
* core file name.
* @throws IOException
* @throws InterruptedException
*/
public static synchronized String generateCore() throws IOException, InterruptedException
{
if (!ENABLE_REQUESTING_COREDUMPS || coreDumpsRequested++ >= MAX_CORE_DUMPS) { return null; }
CoreDumpThread coreDumpThread = new CoreDumpThread();
coreDumpThread.start();
return coreDumpThread.getRequestedFileName();
}
public static int getPID()
{
String name = ManagementFactory.getRuntimeMXBean().getName();
if (name != null)
{
int x = name.indexOf('@');
if (x != -1)
{
name = name.substring(0, x);
return Integer.parseInt(name);
}
}
throw new RuntimeException("Could not find PID");
}
static class CoreDumpThread extends Thread
{
private final int pid;
private final String requestedFileName;
private Throwable error;
public CoreDumpThread()
{
super("CoreDumpThread : " + coreDumpsTaken.get());
// Writing the core can take a while, so we'll prefer to block the
// JVM
setDaemon(false);
pid = getPID();
requestedFileName = "core." + DIAG_NAME_FORMAT.format(new Date()) + "." + pid + ".dmp";
}
@Override
public void run()
{
try
{
ProcessBuilder processBuilder = new ProcessBuilder(CORE_PROGRAM_PATH, "-o", requestedFileName, "" + pid);
processBuilder.redirectErrorStream(true);
Process process = processBuilder.start();
BufferedReader br = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
StringBuilder sb = new StringBuilder();
while ((line = br.readLine()) != null)
{
sb.append(line);
}
int exitValue = process.waitFor();
if (exitValue == 0)
{
coreDumpsTaken.incrementAndGet();
}
else
{
System.out.println("Error requesting core. Exit value " + exitValue + ". Output " + sb.toString());
}
}
catch (Throwable t)
{
error = t;
System.out.println("Error generating core: " + t.getLocalizedMessage());
t.printStackTrace();
}
}
public String getRequestedFileName()
{
return requestedFileName;
}
public Throwable getError()
{
return error;
}
}
}
Previous Section (Troubleshooting IBM Java) | Next Section (Troubleshooting WebSphere Application Server) | Back to Table of Contents