IBM WebSphere Application Server Performance Cookbook

Java Standard Edition (JSE)

Best Practices

  • Avoid the costs of object creation and manipulation by using primitive types for variables
  • Cache frequently-used objects to reduce the amount of garbage collection needed, and avoid the need to re-create the objects.
  • Group native operations to reduce the number of Java Native Interface (JNI) calls when possible.
  • Use synchronized methods only when necessary to limit the multitasking in the JVM and operating system.
  • Avoid invoking the garbage collector unless necessary. If you must invoke it, do so only during idle time or some noncritical phase.
  • Declare methods as final whenever possible. Final methods are handled better by the JVM.
  • Use the static final key word when creating constants in order to reduce the number of times the variables need to be initialized.
  • Avoid unnecessary "casts" and "instanceof" references, because casting in Java is done at run time.
  • Avoid the use of vectors whenever possible when an array will suffice.
  • Add and delete items from the end of the vector.
  • Avoid allocating objects within loops.
  • Use connection pools and cached-prepared statements for database access.
  • Minimize thread creation and destruction cycles.
  • Minimize the contention for shared resources.
  • Minimize the creation of short-lived objects.
  • Avoid remote method calls.
  • Use callbacks to avoid blocking remote method calls.
  • Avoid creating an object only used for accessing a method.
  • Keep synchronized methods out of loops.
  • Store string and char data as Unicode in the database.
  • Reorder the CLASSPATH so that the most frequently used libraries occur first.
  • Reduce synchronization
  • Keep application logging to a minimum or add log guards
  • Consider using work areas for passing around application state through JNDI: https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/welc6tech_wa_tun.html

Synchronization

"Problem determination... tools often report the class of the object on which contention is occurring. A uniquely named class for the object helps identify where in the application code those objects are being used." (http://www.ibm.com/developerworks/websphere/techjournal/1111_dawson/1111_dawson.html)

Applications that overuse the synchronized keyword or have one placed in a frequently used method can often result in poor application response times and/or application deadlocks. Applications should be written to be thread safe (http://www.ibm.com/developerworks/java/library/j-jtp09263/index.html).

Strings

Unicode

Practically, developers are most interested in Unicode code points which map unique numbers (commonly represented in Unicode with hexadecimal numbers such as U+000041 representing the character A) to logical characters. A font then maps a logical character to a glyph which is what is seen on a computer screen.

However, what's confusing in Java is that the char primitive type is not the same as a logical character. Before Java 5, they were the same since a char was internally represented using UTF-16 with 2 bytes for each character and this matched the original Unicode 1.0 standard which only defined up to 65,536 characters.

Java 5 and above support Unicode 4.0 which allows up to 1,112,064 characters. The first 65,536 characters (U+000000 to U+00FFFF) are referred to as the Basic Multilingual Plane (BMP) and are still represented with a single char value. The remaining characters (U+010000 to U+10FFFF) are called supplementary characters and represented with a pair of char values called a surrogate. A surrogate uses the special Unicode code point range of U+00D800 to U+00DFFF which was reserved for use in UTF-16. This range is split into two ranges: a high-surrogates range (U+00D800 to U+00DBFF) and a low-surrogates range (U+00DC00 to U+00DFFF). The first code unit comes from the high-surrogates range and the second code unit comes from the low-surrogates range.

Encoding and decoding surrogates is discussed in RFC 2781; however, the Java Character class provides all the necessary methods for interrogating and manipulating surrogates either using a char[] or an int.

Most modern file and wire encodings use UTF-8 which is a more compact encoding scheme. Java classes such as String, InputStreamReader, OutputStreamWriter, etc. transparently handle encoding and decoding to and from UTF-8.

java.lang.ThreadLocal

ThreadLocals are a powerful way to cache information without incurring cross thread contention and also ensuring thread safety of cached items. When using ThreadLocals in thread pools, consider ensuring that the thread pool minimum size is equal to the thread pool maximum size, so that ThreadLocals are not destroyed. (http://docs.oracle.com/javase/7/docs/api/java/lang/ThreadLocal.html)

Note that ThreadLocals may introduce classloader leaks if the ThreadLocal object (or an object it references) is loaded from an application classloader which is restarted without the JVM being restarted. In this case, the only way to clear ThreadLocals is to allow those threads to be destroyed or the ThreadLocal values to be updated to a class from the new classloader (this can be done with a module listener).

Migrating to Java 11

See the following links:

Printing timestamps

For debugging, as an alternative to java.util.logging or other logging mechanisms, you may use SimpleDateFormat to print the timestamps with milliseconds and the thread ID. Use ThreadLocal because SimpleDateFormat is not thread safe. For example:

import java.text.SimpleDateFormat;
import java.util.Date;

public final class CustomLogger {
    public static final ThreadLocal<SimpleDateFormat> DATE_FORMATTER = new ThreadLocal<SimpleDateFormat>() {
        @Override
        protected SimpleDateFormat initialValue() {
            return new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS Z");
        }
    };

    public static void log(String str) {
        System.out.println("[" + DATE_FORMATTER.get().format(new Date()) + " thr 0x"
                + Long.toHexString(Thread.currentThread().getId()) + "]: " + str);
    }

    public static void main(String... args) throws Throwable {
        log("Hello World");
    }
}

Example output:

[2021-03-08 09:42:02.543 -0800 thr 0x1]: Hello World

Speculative Tracing

Intermittent performance problems are one of the most difficult type of performance problems. If the issues can't be reproduced easily, then they may be difficult to capture much data for. Often, running the necessary tracing all the time may be too expensive (even with optimization technologies such as High Performance Extensible Logging or Binary Logging). If you've exhausted all other performance techniques (tracing to RAM, thread sampling, request metrics, PMI, and the like), then you may consider tracing instead.

The first stage in finding intermittent performance problems using tracing is to isolate the location(s) of the problems. This is the most iterative and time-consuming stage. One way to do this is to start with the outermost component that demonstrates the symptom and then slowly isolate down. For example, if all that's known is that HTTP requests are intermittently slow, then you may start with the servlet (or if it's proved to be outside that, the container itself). If lightweight tracing doesn't exist, then it must be added. Below is a Java example which speculatively monitors how long a method takes, and if the duration exceeds some configurable threshold, the speculative traces are dumped, thus allowing one iteration of isolation. This code is most efficient when run within fixed-sized thread pools.

private static final int SPECULATIVE_DURATION_THRESHOLD = Integer.parseInt(System.getProperty("SPECULATIVE_DURATION_THRESHOLD", "-1"));
private static final boolean SPECULATIVE_TRACE_ENABLED = SPECULATIVE_DURATION_THRESHOLD == -1 ? false : true;
private static ThreadLocal<ArrayList<String>> speculativeTraces = new ThreadLocal<ArrayList<String>>() {
  @Override
  protected ArrayList<String> initialValue() {
    return new ArrayList<String>(8);
  }
};

public void foo() {
  final long methodStartTime = SPECULATIVE_TRACE_ENABLED ? System.currentTimeMillis() : -1;
  final ArrayList<String> spec = SPECULATIVE_TRACE_ENABLED ? speculativeTraces.get() : null;
  if (SPECULATIVE_TRACE_ENABLED) {
    spec.clear();
    spec.add(methodStartTime + " started");
  }

  doWork1();

  if (SPECULATIVE_TRACE_ENABLED) {
    spec.add(System.currentTimeMillis() + " doWork1 finished");
  }

  doWork2();

  if (SPECULATIVE_TRACE_ENABLED) {
    spec.add(System.currentTimeMillis() + " doWork2 finished");
  }

  doWork3();

  if (SPECULATIVE_TRACE_ENABLED) {
    final long methodDuration = System.currentTimeMillis() - methodStartTime;
    if (methodDuration >= SPECULATIVE_DURATION_THRESHOLD) {
      System.out.println("Speculative tracing threshold (" + SPECULATIVE_DURATION_THRESHOLD + " ms) exceeded with a call of " + methodDuration + " ms");
      for (String speculativeTrace : spec) {
        System.out.println(speculativeTrace);
      }
      System.out.println("Speculative tracing set end at " + System.currentTimeMillis());
    }
  }
}

Sampled Timing Calls

ThreadLocals may be used to sample long method call execution times and print out sampled durations:

    private static final int SAMPLE_COUNTPERTHREAD_FOO = Integer.getInteger("samplecountperthread.foo", 1000);
    private static final int THRESHOLD_FOO = Integer.getInteger("threshold.foo", 0);
    private static final String SAMPLE_MESSAGE_FOO = "Sampled duration (threshold=" + THRESHOLD_FOO + ", rate="
                    + SAMPLE_COUNTPERTHREAD_FOO + " calls per thread) in ms of foo = ";

    private static final AtomicInteger calls_foo = new AtomicInteger(0);

    public void foo() {
        final boolean doSample = (calls_foo.incrementAndGet() % SAMPLE_COUNTPERTHREAD_FOO) == 0;
        final long startTime = doSample ? System.currentTimeMillis() : -1;

        doLongWork();

        if (doSample) {
            final long diff = System.currentTimeMillis() - startTime;
            if (diff >= THRESHOLD_FOO) {
                System.out.println(SAMPLE_MESSAGE_FOO + diff);
            }
        }
    }

    private void doLongWork() {
        // ...
    }

Always Timing Calls

As an alternative to sampling, you may add code to time all calls and then print details if they are above some threshold:

    private static final int THRESHOLD_FOO = Integer.getInteger("threshold.foo", 0);
    private static final String TIMING_MESSAGE_FOO = "Duration (threshold=" + THRESHOLD_FOO + ") in ms of foo = ";
    
    public void foo() {
        final long startTime = System.currentTimeMillis();
        
        doLongWork();
        
        final long diff = System.currentTimeMillis() - startTime;
        if (diff >= THRESHOLD_FOO) {
            System.out.println(TIMING_MESSAGE_FOO + diff);
        }
    }
    
    private void doLongWork() {
        // ...
    }

Request Heap Dump

    public static String requestHeapDump()
            throws IllegalAccessException, IllegalArgumentException, java.lang.reflect.InvocationTargetException,
            NoSuchMethodException, SecurityException, ClassNotFoundException,
            javax.management.InstanceNotFoundException, javax.management.MalformedObjectNameException,
            javax.management.ReflectionException, javax.management.MBeanException {
        String jvm = System.getProperty("java.vendor");
        if ("IBM Corporation".equals(jvm)) {
            return requestJ9SystemDump();
        } else if ("Eclipse Adoptium".equals(jvm)) {
            return requestHPROFDump(true);
        } else {
            throw new UnsupportedOperationException("Unknown JVM vendor " + jvm);
        }
    }

    public static String requestJ9SystemDump()
            throws IllegalAccessException, IllegalArgumentException, java.lang.reflect.InvocationTargetException,
            NoSuchMethodException, SecurityException, ClassNotFoundException {
        return (String) Class.forName("com.ibm.jvm.Dump").getMethod("triggerDump", new Class<?>[] { String.class })
                .invoke(null, new Object[] { "system:request=exclusive+prepwalk" });
    }

    public static String requestHPROFDump(boolean performFullGC)
            throws javax.management.InstanceNotFoundException, javax.management.MalformedObjectNameException,
            javax.management.ReflectionException, javax.management.MBeanException {
        String filename = "dump" + new java.text.SimpleDateFormat("yyyyMMdd.HHmmss").format(new java.util.Date())
                + ".hprof";
        java.lang.management.ManagementFactory.getPlatformMBeanServer().invoke(
                new javax.management.ObjectName("com.sun.management:type=HotSpotDiagnostic"), "dumpHeap",
                new Object[] { filename, performFullGC },
                new String[] { String.class.getName(), boolean.class.getName() });
        return new java.io.File(filename).getAbsolutePath();
    }

Request J9 Thread Dump

public class DiagnosticThreadDump {
    private static final boolean QUIET = Boolean.parseBoolean(System.getProperty("Diagnostics.Quiet", "false"));
    private static final Class<?> j9Dump;
    private static final java.lang.reflect.Method j9triggerDump;

    static {
        j9Dump = loadJ9Dump();
        j9triggerDump = loadJ9TriggerDump(j9Dump);
    }

    private static Class<?> loadJ9Dump() {
        try {
            return Class.forName("com.ibm.jvm.Dump");
        } catch (Throwable t) {
            if (!QUIET) {
                t.printStackTrace();
            }
            return null;
        }
    }

    private static java.lang.reflect.Method loadJ9TriggerDump(Class<?> c) {
        if (c != null) {
            try {
                return c.getMethod("triggerDump", new Class<?>[] { String.class });
            } catch (Throwable t) {
                if (!QUIET) {
                    t.printStackTrace();
                }
            }
        }
        return null;
    }

    public static String requestJ9ThreadDump() {
        try {
            return (String) j9triggerDump.invoke(null, new Object[] { "java:request=exclusive+prepwalk" });
        } catch (Throwable t) {
            if (QUIET) {
                return null;
            } else {
                throw new RuntimeException(t);
            }
        }
    }

    public static void main(String[] args) throws Throwable {
        requestJ9ThreadDump();
    }
}

Request J9 System Dump

public class DiagnosticSystemDump {
    private static final boolean QUIET = Boolean.parseBoolean(System.getProperty("Diagnostics.Quiet", "false"));
    private static final Class<?> j9Dump;
    private static final java.lang.reflect.Method j9triggerDump;

    static {
        j9Dump = loadJ9Dump();
        j9triggerDump = loadJ9TriggerDump(j9Dump);
    }

    private static Class<?> loadJ9Dump() {
        try {
            return Class.forName("com.ibm.jvm.Dump");
        } catch (Throwable t) {
            if (!QUIET) {
                t.printStackTrace();
            }
            return null;
        }
    }

    private static java.lang.reflect.Method loadJ9TriggerDump(Class<?> c) {
        if (c != null) {
            try {
                return c.getMethod("triggerDump", new Class<?>[] { String.class });
            } catch (Throwable t) {
                if (!QUIET) {
                    t.printStackTrace();
                }
            }
        }
        return null;
    }

    public static String requestJ9SystemDump() {
        try {
            return (String) j9triggerDump.invoke(null, new Object[] { "system:request=exclusive+prepwalk" });
        } catch (Throwable t) {
            if (QUIET) {
                return null;
            } else {
                throw new RuntimeException(t);
            }
        }
    }

    public static void main(String[] args) throws Throwable {
        requestJ9SystemDump();
    }
}

Requesting Thread Dumps, Heap Dumps, and System Dumps

The following example code shows how to request a thread dump (IBM Java only), heap dump or system dump:

/**
* These are handled in synchronized methods below.
*/
private static int threadDumpsTaken = 0, heapDumpsTaken = 0, coreDumpsTaken = 0;

private static final int maxThreadDumps = Integer.parseInt(System.getProperty("MAXTHREADDUMPS", "-1"));
private static final int maxHeapDumps = Integer.parseInt(System.getProperty("MAXHEAPDUMPS", "1"));
private static final int maxCoreDumps = Integer.parseInt(System.getProperty("MAXCOREDUMPS", "1"));

private static final boolean isIBMJava;
private static final Class<?> ibmDumpClass;
private static final java.lang.reflect.Method ibmJavacoreMethod;
private static final java.lang.reflect.Method ibmHeapDumpMethod;
private static final java.lang.reflect.Method ibmSystemDumpMethod;
private static final Class<?> hotSpotMXBeanClass;
private static final Object hotspotMXBean;
private static final java.lang.reflect.Method hotspotMXBeanDumpHeap;
private static final java.text.SimpleDateFormat hotspotDateFormat = new java.text.SimpleDateFormat("yyyyMMdd'T'HHmmss");

static {
  try {
    isIBMJava = isIBMJava();
    ibmDumpClass = isIBMJava ? Class.forName("com.ibm.jvm.Dump") : null;
    ibmHeapDumpMethod = isIBMJava ? ibmDumpClass.getMethod("HeapDump") : null;
    ibmJavacoreMethod = isIBMJava ? ibmDumpClass.getMethod("JavaDump") : null;
    ibmSystemDumpMethod = isIBMJava ? ibmDumpClass.getMethod("SystemDump") : null;
    hotSpotMXBeanClass = isIBMJava ? null : getHotSpotDiagnosticMXBeanClass();
    hotspotMXBean = isIBMJava ? null : getHotSpotDiagnosticMXBean();
    hotspotMXBeanDumpHeap = isIBMJava ? null : getHotSpotDiagnosticMXBeanDumpHeap();
  } catch (Throwable t) {
    throw new RuntimeException("Could not load Java dump classes", t);
  }
}

public static boolean isIBMJava() {
  try {
    // We could use System.getProperty, but that requires elevated permissions in some cases.
    Class.forName("com.ibm.jvm.Dump");
    return true;
  } catch (Throwable t) {
    return false;
  }
}

private static Class<?> getHotSpotDiagnosticMXBeanClass() throws ClassNotFoundException {
  return Class.forName("com.sun.management.HotSpotDiagnosticMXBean");
}

private static Object getHotSpotDiagnosticMXBean() throws ClassNotFoundException, java.io.IOException {
  javax.management.MBeanServer server = java.lang.management.ManagementFactory.getPlatformMBeanServer();
  return java.lang.management.ManagementFactory.newPlatformMXBeanProxy(server,
      "com.sun.management:type=HotSpotDiagnostic", hotSpotMXBeanClass);
}

private static java.lang.reflect.Method getHotSpotDiagnosticMXBeanDumpHeap() throws NoSuchMethodException, SecurityException {
  return hotSpotMXBeanClass.getMethod("dumpHeap", String.class, boolean.class);
}

public static synchronized void requestThreadDump() {
  if (maxThreadDumps == -1 || (maxThreadDumps > -1 && threadDumpsTaken++ < maxThreadDumps)) {
    try {
      ibmJavacoreMethod.invoke(ibmDumpClass);
    } catch (Throwable t) {
      throw new RuntimeException(t);
    }
  }
}

public static synchronized void requestHeapDump() {
  if (maxHeapDumps == -1 || (maxHeapDumps > -1 && heapDumpsTaken++ < maxHeapDumps)) {
    try {
      if (ibmHeapDumpMethod != null) {
        ibmHeapDumpMethod.invoke(ibmDumpClass);
      } else {
        requestHotSpotHPROF();
      }
    } catch (Throwable t) {
      throw new RuntimeException(t);
    }
  }
}

public static synchronized void requestCoreDump() {
  if (maxCoreDumps == -1 || (maxCoreDumps > -1 && coreDumpsTaken++ < maxCoreDumps)) {
    try {
      if (ibmSystemDumpMethod != null) {
        ibmSystemDumpMethod.invoke(ibmDumpClass);
      } else {
        requestHotSpotHPROF();
      }
    } catch (Throwable t) {
      throw new RuntimeException(t);
    }
  }
}

private static void requestHotSpotHPROF() throws IllegalAccessException, java.lang.reflect.InvocationTargetException {
  String fileName = "heap" + hotspotDateFormat.format(new java.util.Date()) + ".hprof";
  boolean live = true;
  hotspotMXBeanDumpHeap.invoke(hotspotMXBean, fileName, live);
}

java.util.logging

Example of how to use java.util.logging:

package com.test;

public class Foo {
  private static final java.util.logging.Logger LOG = java.util.logging.Logger.getLogger(Foo.class.getName());

  public void bar(String param1) {
    if (LOG.isLoggable(java.util.logging.Level.FINE)) {
      LOG.entering(Foo.class.getName(), "bar", param1);
    }

    // Do work...

    if (LOG.isLoggable(java.util.logging.Level.FINER)) {
      LOG.finer("Work step1 complete");
    }

    // Do work...

    if (LOG.isLoggable(java.util.logging.Level.FINE)) {
      LOG.entering(Foo.class.getName(), "bar");
    }
  }
}

For example, the logging or trace specification may control the logging of this class with com.test.Foo=all

Finalizers

"The Java service team recommends that applications avoid the use of finalizers if possible." (http://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.win.80.doc/diag/understanding/mm_gc_finalizers.html)

"It is not possible to predict when a finalizer is run... Because a finalized object might be garbage that is retained, a finalizer might not run at all." (http://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.win.80.doc/diag/understanding/mm_gc_finalizers_contract.html)

XML Parsers

One of the common misconceptions about writing XML applications is that creating a parser instance does not incur a large performance cost. On the contrary, creation of a parser instance involves creation, initialization, and setup of many objects that the parser needs and reuses for each subsequent XML document parsing. These initialization and setup operations are expensive.

In addition, creating a parser can be even more expensive if you are using the JAXP API. To obtain a parser with this API, you first need to retrieve a corresponding parser factory -- such as a SAXParserFactory -- and use it to create the parser. To retrieve a parser factory, JAXP uses a search mechanism that first looks up a ClassLoader (depending on the environment, this can be an expensive operation), and then attempts to locate a parser factory implementation that can be specified in the JAXP system property, the jaxp.property file, or by using the Jar Service Provider mechanism. The lookup using the Jar Service Provider mechanism can be particularly expensive as it may search through all the JARs on the classpath; this can perform even worse if the ClassLoader consulted does a search on the network.

Consequently, in order to achieve better performance, we strongly recommend that your application creates a parser once and then reuses this parser instance.

http://www.ibm.com/developerworks/library/x-perfap2/index.html#reuse

Apache HttpClient

This section has been moved to Apache HttpClient.

WeakReferences and SoftReferences

A typical use of the SoftReference class is for a memory-sensitive cache. The idea of a SoftReference is that you hold a reference to an object with the guarantee that all of your soft references will be cleared before the JVM reports an out-of-memory condition. The key point is that when the garbage collector runs, it may or may not free an object that is softly reachable. Whether the object is freed depends on the algorithm of the garbage collector as well as the amount of memory available while the collector is running.
The WeakReference class

A typical use of the WeakReference class is for canonicalized mappings. In addition, weak references are useful for objects that would otherwise live for a long time and are also inexpensive to re-create. The key point is that when the garbage collector runs, if it encounters a weakly reachable object, it will free the object the WeakReference refers to. Note, however, that it may take multiple runs of the garbage collector before it finds and frees a weakly reachable object.

http://www.ibm.com/developerworks/java/library/j-refs/

Logging

Always use a logger that can be dynamically modified at run time without having to restart the JVM.

Differentiate between Error logging (which should go to SystemOut.log) and Audit logging which has different requirements and should not be contaminating the SystemOut.log.

Use a fast disk for Audit logging.