IBM WebSphere Application Server Performance Cookbook

Javacore Overhead

By default, when a Java Dump (javacore*.txt) is requested under normal conditions (unless something else already has exclusive access), the Java Dump waits for exclusive access meaning that all threads have paused themselves. Therefore, the javacore pauses the JVM for this time plus the time to produce the javacore itself and thus has a direct impact on performance. In general, the duration of this pause is less than a few hundred milliseconds.

Starting with OpenJ9 0.36.0 on non-Windows operating systems (IBM Java 8.0.8.0, IBM Semeru Runtimes 11.0.18.0 & 17.0.6.0) and OpenJ9 0.40.0 on Windows (IBM Java 8.0.8.10, IBM Semeru Runtimes 11.0.20.0 & 17.0.8.0), a line at the end of the javacore shows how long the JVM was paused. For example:

1TIDMPDURATION Approximate time to produce this dump: X ms

The overhead of javacores may be gauged in a performance test environment by adding up these 1TIDMPDURATION values and dividing by the sum of intervals between javacores.

If 1TIDMPDURATION is high, there may be various potential causes:

  1. There have been rare observed cases on Linux of javacores causing 20 second pauses with high system/kernel CPU time during the first 10 seconds. As observed with Linux perf On-CPU sampling with wall-clock times, the Signal Reporter thread consumes most of that CPU (with stack frames in various places such as _raw_spin_unlock_irqrestore, system_call_after_swapgs, __sigqueue, do_send_sig_info, __sched_yield, and __GI___getuid). There is a timeout of 20 seconds waiting for threads to quiesce and observed cases of unreliable Linux sigqueue. If you observe this behavior, please open a support case to help us understand this issue.

    There have been similar observed cases of javacores taking about 20 seconds to produce although with low system/kernel CPU time. Again, if you observe this behavior, please open a support case to help us understand this issue.

    One potential workaround for these long pauses may be to remove the request=preempt option from the javacore agents:

    -Xdump:java:defaults:request=exclusive

  2. If the disk is very slow, then it may take a long time to write the javacore file. Javacores are generally small (less than 10MB), so if writing is taking a long time, then the disk has serious performance issues. Use operating system tools to evaluate disk performance with a data gathering interval in the same magnitude as the excessive pause time (e.g. iostat, nfsiostat, etc.).