Troubleshooting HotSpot JVM
jcmd
jcmd is a tool for executing various diagnostic operations.
jmap
jmap is an unsupported tool for memory operations.
histo
The histo option may be used to print a histogram of
Java objects by class, including number of instances and number of
bytes. The live option only counts reachable objects,
although it does force a full GC first. Example:
$ jmap -histo 15078
num #instances #bytes class name
----------------------------------------------
1: 399 4747704 [I
2: 1565 151240 [C
3: 450 51456 java.lang.Class
4: 194 48144 [B
5: 1229 29496 java.lang.String ...jinfo
jinfo is an unsupported tool which prints Java configuration of a live Java process or from a core dump.
Thread Dump
HotSpot Java can produce a thread dump which details the activity of each thread. For example:
2020-01-01 01:23:45
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode):
"pool-1-thread-8402" prio=3 tid=0x000000010956f000 nid=0x3cff waiting on condition [0xfffffff7868fe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xfffffffe90fb54a0> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)...The "nid" is the hexadecimal native thread ID.
Request Thread Dump
- On POSIX operating systems, send the
SIGQUITsignal and replace$PIDand a text thread dump will be printed tostderr:kill -3 $PID - Use
jcmdand replace$PIDand a text thread dump is printed to thejcmdconsole:jcmd PID Thread.print
HPROF Heapdumps
An HPROF heapdump contains the full Java heap object graph as well as Java object memory contents (for example, Strings, primitives, etc.). This is used for investigating OutOfMemoryErrors, tuning Java heap usage, etc.
By default, when a Java memory request cannot be fulfilled, an
OutOfMemoryError is thrown, but an HPROF dump is not produced. Use -XX:+HeapDumpOnOutOfMemoryError
to produce an HPROF dump in this condition. Consider tuning
-XX:GCTimeLimit and -XX:GCHeapFreeLimit to
control when an OutOfMemoryError is thrown.
-XX:HeapDumpPath may be used to control where the dumps are
written to. -XX:OnOutOfMemoryError may be used to execute
an operating system command on an OOM.
To analyze heapdumps, see the Eclipse Memory Analyzer Tool chapter.
Generating HPROF heapdumps
Additional methods of requesting heap dumps are documented in the Troubleshooting WAS chapter.
- An HPROF heapdump is automatically produced on OOM after adding
-XX:+HeapDumpOnOutOfMemoryError(this must be explicitly configured and the JVM restarted as it's not enabled by default). Example error output instdout:java.lang.OutOfMemoryError: GC overhead limit exceeded Dumping heap to java_pid28537.hprof ... - Use
jcmdand replace$PID:jcmd $PID GC.heap_dump heapdump.hprof - Use the HotSpotDiagnostic.dumpHeap operation
- Use
jmapand replace$PID:jmap -dump:format=b,file=heapdump.hprof ${PID} - Produce an operating system core dump (see the Troubleshooting Operating
Systems chapter) and then extract the HPROF heapdump using
jmap:jmap -dump:format=b,file=heapdump.hprof ${PATH_TO_JAVA} ${PATH_TO_CORE} - Use -XX:OnOutOfMemoryError (see below)
- From within Eclipse Memory Analyzer Tool: File } Acquire Heap Dump
Use DTrace to Produce Stacks Calling certain Methods
The following DTrace script prints stacks when System.gc is called. The JVM must be started with -XX:+ExtendedDTraceProbes
Attach the DTrace script to a running PID: /usr/sbin/dtrace -qs methodtrace.d -p ${PID}
It is often the case that the overhead of such a DTrace script is very high and may be inappropriate for a production environment.
#pragma D option bufsize=128m
dtrace:::BEGIN
{
/* Java level tracing */
traceJava = 1; /* 1 = enable, 0 = disable */
classFilter = "java/lang/System"; /* e.g. "java/util" */
methodFilter = "gc"; /* e.g. "get" */
/* Initialise the per-thread indentation variable */
self->indent = 0;
}
hotspot$target:::method-entry
/
traceJava
&& (classFilter == "" || strstr(copyinstr(arg1, arg2), classFilter) != NULL)
&& (methodFilter == "" || strstr(copyinstr(arg3, arg4), methodFilter) != NULL)
/
{
self->indent += 2;
wt = walltimestamp;
printf("%Y.%09d: %d/%d:%*s-> %s.%s%s\n",
wt,
wt % 1000000000,
pid,
tid,
self->indent,
"",
copyinstr(arg1, arg2),
copyinstr(arg3, arg4),
copyinstr(arg5, arg6));
jstack(500, 8192);
}Example output:
2018 Jan 10 14:03:24.280004000: 18021/2:
-> java/lang/System.gc()V
libjvm.so`__1cNSharedRuntimeTdtrace_method_entry6FpnKJavaThread_pnNmethodOopDesc__i_+0x1ac
java/lang/System.c
SystemDemo.ai
0xfbc0021c
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x318
libjvm.so`jni_CallStaticVoidMethod+0x67c
libjli.so`JavaMain+0x740
libc.so.1`_lwp_startUse -XX:OnOutOfMemoryError to Spawn jmap
#!/bin/sh
# Usage:
# 1. Create oom.sh with the contents of this script
# 2. Change paths in the "Variables" section if needed
# 3. chmod a+x oom.sh
# 4. Run java with the following argument, replacing $PATH with path to oom.sh
# -XX:OnOutOfMemoryError="/$PATH/oom.sh %p"
# Variables
LOCKFILE=/tmp/oomlock
OUT=/tmp/oomout.txt
NOW=`date +"%Y%m%d_%H%M%S"`
CURDIR=`pwd`
JAVA_HOME=/opt/IBM/WebSphere/AppServer/java/
# Execution
echo "OOM handler script started for PID $1 at $NOW in $CURDIR" >> $OUT
if [ ! -f $LOCKFILE ]; then
touch $LOCKFILE >> $OUT 2>&1
NOW=`date +"%Y%m%d_%H%M%S"`
echo "OOM handler requested hprof at $NOW" >> $OUT
FILENAME="heap_${NOW}_${1}.hprof"
$JAVA_HOME/bin/jmap -F -dump:format=b,file=$FILENAME $1 >> $OUT 2>&1
# /usr/bin/gcore -F -o core_$NOW.dmp $1 >> $OUT 2>&1
CODE=$?
echo "OOM handler returned with $CODE at $NOW" >> $OUT
rm -f $LOCKFILE >> $OUT 2>&1
fi
NOW=`date +"%Y%m%d_%H%M%S"`
echo "OOM handler finished at $NOW" >> $OUTUse -XX:OnOutOfMemoryError to Spawn gcore
#!/bin/sh
# Usage:
# 1. Create oom.sh with the contents of this script
# 2. Change paths in the "Variables" section if needed
# 3. chmod a+x oom.sh
# 4. Run java with the following argument, replacing $PATH with path to oom.sh
# -XX:OnOutOfMemoryError="/$PATH/oom.sh %p"
# 5. After an OOM occurs, check /tmp/oomout.txt for output of the command
# 6. Run `jmap -dump:format=b,file=heap.hprof ${PATH_TO_JAVA} ${CORE}`
#
# Notes:
# OnOutOfMemoryError runs /usr/bin/sh -c $CMD synchronously, during which it
# appears it has a lock that prevents jmap to attach. Tried -F, but that
# generated an infinite loop and ran into other issues, so running gcore
# instead.
# Variables
LOCKFILE=/tmp/oomlock
OUT=/tmp/oomout.txt
NOW=`date +"%Y%m%d_%H%M%S"`
CURDIR=`pwd`
GCORE_PATH=/usr/bin/gcore
# Execution
echo "OOM handler script started for PID $1 at $NOW in $CURDIR" >> $OUT
if [ ! -f $LOCKFILE ]; then
touch $LOCKFILE >> $OUT 2>&1
NOW=`date +"%Y%m%d_%H%M%S"`
echo "OOM handler requested hprof at $NOW" >> $OUT
# $JAVA_HOME/bin/jmap -dump:format=b,file=heap_$1.hprof $1 >> $OUT 2>&1
$GCORE_PATH -F -o core_$NOW.dmp $1 >> $OUT 2>&1
CODE=$?
echo "OOM handler returned with $CODE at $NOW" >> $OUT
rm -f $LOCKFILE >> $OUT 2>&1
fi
NOW=`date +"%Y%m%d_%H%M%S"`
echo "OOM handler finished at $NOW" >> $OUTCode to Request Diagnostics from within the JVM
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.lang.management.ManagementFactory;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.concurrent.atomic.AtomicInteger;
public class Play
{
public static void main(String... args) throws Throwable
{
System.out.println("Requesting core...");
tryGenerateCore();
}
public static void tryGenerateCore()
{
try
{
String requestedFileName = generateCore();
if (requestedFileName != null)
{
System.out.println("Started writing core dump to " + requestedFileName);
}
}
catch (Throwable t)
{
System.out.println("Error generating core: " + t.getLocalizedMessage());
t.printStackTrace();
}
}
private final static boolean ENABLE_REQUESTING_COREDUMPS = Boolean.getBoolean("ENABLE_REQUESTING_COREDUMPS");
private final static SimpleDateFormat DIAG_NAME_FORMAT = new SimpleDateFormat("yyyyMMdd.HHmmss");
private final static String CORE_PROGRAM_PATH = System.getProperty("CORE_PROGRAM_PATH", "/usr/bin/gcore");
private final static int MAX_CORE_DUMPS = Integer.getInteger("MAX_CORE_DUMPS", 1);
private static final AtomicInteger coreDumpsTaken = new AtomicInteger();
private static int coreDumpsRequested;
/**
* Disabled by default. Enable with -DENABLE_REQUESTING_COREDUMPS=true
* <p />
* Request a non-destructive core dump in a separate thread by spawning out
* to the gcore command. gcore will attach to and pause the process, dump
* all virtual memory (so the size will be about the size in ps VSZ) and
* then the process should continue. Unlike an OOM or using jmap to request
* an HPROF dump, requesting a core does not request a Full GC. Jmap can be
* used to extract an HPROF heapdump from the core:
* <p />
* <code>$ jmap -dump:format=b,file=heap.hprof ${PATH_TO_java} ${CORE}</code>
* <p />
* Whereas asking the JVM to generate a heapdump with jmap is a complex
* operation because the JVM has to walk all the data structures, the
* operating system generating a core is very simple: the OS just pauses the
* process and dumps out all of the virtual memory. The overhead of a core
* file is almost completely in writing the large amount of bytes to disk.
* There are some techniques to make this very fast. First, if there is
* sufficient filecache in RAM (i.e. a large amount of free RAM), then the
* OS will write the core to RAM and then asynchronously write to disk, thus
* making the pause quite fast. However, this can have some performance side
* effects. An alternative way to do this is to mount a RAMdisk and write
* the core to a RAMdisk.
* <p />
* Warning: ensure sufficient core, file and other ulimits. Also ensure
* sufficient disk space in the current working directory.
*
* @return null if -DMAX_CORE_DUMPS (default 1) has been reached or
* -DENABLE_REQUESTING_COREDUMPS=false; otherwise, the requested
* core file name.
* @throws IOException
* @throws InterruptedException
*/
public static synchronized String generateCore() throws IOException, InterruptedException
{
if (!ENABLE_REQUESTING_COREDUMPS || coreDumpsRequested++ >= MAX_CORE_DUMPS) { return null; }
CoreDumpThread coreDumpThread = new CoreDumpThread();
coreDumpThread.start();
return coreDumpThread.getRequestedFileName();
}
public static int getPID()
{
String name = ManagementFactory.getRuntimeMXBean().getName();
if (name != null)
{
int x = name.indexOf('@');
if (x != -1)
{
name = name.substring(0, x);
return Integer.parseInt(name);
}
}
throw new RuntimeException("Could not find PID");
}
static class CoreDumpThread extends Thread
{
private final int pid;
private final String requestedFileName;
private Throwable error;
public CoreDumpThread()
{
super("CoreDumpThread : " + coreDumpsTaken.get());
// Writing the core can take a while, so we'll prefer to block the
// JVM
setDaemon(false);
pid = getPID();
requestedFileName = "core." + DIAG_NAME_FORMAT.format(new Date()) + "." + pid + ".dmp";
}
@Override
public void run()
{
try
{
ProcessBuilder processBuilder = new ProcessBuilder(CORE_PROGRAM_PATH, "-o", requestedFileName, "" + pid);
processBuilder.redirectErrorStream(true);
Process process = processBuilder.start();
BufferedReader br = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
StringBuilder sb = new StringBuilder();
while ((line = br.readLine()) != null)
{
sb.append(line);
}
int exitValue = process.waitFor();
if (exitValue == 0)
{
coreDumpsTaken.incrementAndGet();
}
else
{
System.out.println("Error requesting core. Exit value " + exitValue + ". Output " + sb.toString());
}
}
catch (Throwable t)
{
error = t;
System.out.println("Error generating core: " + t.getLocalizedMessage());
t.printStackTrace();
}
}
public String getRequestedFileName()
{
return requestedFileName;
}
public Throwable getError()
{
return error;
}
}
}