IBM WebSphere Application Server Performance Cookbook

Java Recipes

  1. Review the Operating System recipe for your OS.
  2. Tune the maximum Java heap size (-Xmx or -XX:MaxRAMPercentage):
    1. Ensure that verbose garbage collection is enabled (which it is by default in recent versions of Liberty and tWAS) which generally has an overhead less than 0.5% and then use a tool such as the IBM Garbage Collection and Memory Visualizer (GCMV) and ensure that the proportion of time spent in garbage collection versus application processing time is less than 5% and ideally less than 1%.
    2. In general, a place to start is to set the maximum size to 43% larger than the maximum occupancy of the application, although the latter is largely a function of workload and thread pool size, so this is just a heuristic.
  3. Consider testing different garbage collector for the OpenJ9/IBM JVM and HotSpot JVM.
  4. Consider testing an increased maximum nursery size for generational collectors.
  5. Ensure there is no memory leak after global garbage collections with long running tests by reviewing verbosegc.
  6. If using a generational collector (which most modern default collectors are):
    1. Ensure tests run through full/tenured collections and ensure those pause times are not too long.
    2. Ensure that there is a sawtooth pattern in the heap usage after collection. Otherwise, the heap size may be too small or the nursery too big.
  7. Consider monitoring for pause times over one second and tune GC if found. Sometimes high pause times are acceptable.
  8. Use a profiler such as IBM Java Health Center or OpenJDK Mission Control with a particular focus on the profiling and lock contention analysis; otherwise, use periodic thread dumps to review JVM activity with the IBM Thread and Monitor Dump Analyzer tool.
  9. Object allocation failures for objects greater than 5MB should generally be investigated. Sometimes high allocation sizes are acceptable.
  10. If the node only uses IPv4 and does not use IPv6, then add the JVM parameters -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false
  11. Consider taking a system dump or HPROF heapdump during peak activity in a test environment and review it with the Eclipse Memory Analyzer Tool to see if there are any areas in the heap for optimization.
  12. Review the stderr and stdout logs for any errors, warnings, or high volumes of messages (e.g. OutOfMemoryErrors, etc.).
  13. If running multiple JVMs on the same machine, consider pinning JVMs to sets of processor cores and tuning -Xgcthreads/-XcompilationThreads (IBM/OpenJ9 JVM) or -XX:ParallelGCThreads (HotSpot JVM).
  14. In general, if memory usage is very flat and consistent, it may be optimal to fix -Xms = -Xmx. For widely varying heap usage, -Xms < -Xmx is generally recommended.
  15. If heavily using XML, consider explicitly configuring JAXP ServiceLoader properties to avoid unnecessary classloading activity.

For details, see the Java chapter and the chapter for your particular JVM.

OpenJ9 and IBM J9 JVMs Recipe

  1. In most cases, the default -Xgcpolicy:gencon garbage collection policy works best, with the key tuning being the maximum heap size (-Xmx or -XX:MaxRAMPercentage) and maximum nursery size (-Xmn).
  2. Upgrade to the latest version and fixpack as there is a history of making performance improvements and fixing issues or regressions over time.
  3. Take a javacore and review the Java arguments (UserArgs) and Environment Variables sections and remove any unnecessary debug options.
  4. Take a javacore and review if the JIT code cache is full or nearly full; if so, and there's available physical memory, test increasing it with -Xcodecachetotal384m -Xcodecache32m
  5. Take a javacore and review if the shared class cache is full or nearly full; if so, and there's available physical memory, consider increasing -Xscmx
  6. If using -Xgcpolicy:gencon and you want to reduce average nursery pause times at some throughput and CPU cost, consider concurrent scavenge.
  7. Consider setting -XX:+CompactStrings where available, applicable, and not already the default.
  8. Review the performance tuning topics in the OpenJ9 or IBM Java documentation.
  9. When running benchmarks or comparing performance to other JVMs, consider testing various benchmark ideas.
  10. If using IBM Semeru Runtimes:
    1. If JIT CPU or memory usage are a concern, consider using the remote JITServer on available platforms.
    2. For AIX and Linux, ensure OpenSSL is on the system path for maximum security performance.
    3. On z/OS, consider enabling IBM Java Health Center (-Xhealthcenter:level=headless) for post-mortem CPU and lock profiling data, although this has an overhead of about 2%.
    4. On z/OS, consider using the "pauseless" garbage collection option -Xgc:concurrentScavenge if using gencon and on recent software and hardware.
  11. If using IBM Java (does not apply to IBM Semeru Runtimes):
    1. Consider setting -XX:MaxDirectMemorySize to avoid some unnecessary full garbage collections.
    2. Consider using the IBMJCEPlus security provider that may offer large performance improvements in encryption. This is now the default except on z/OS since 8.0.7.0.
    3. If the node is using a static IP address that won't be changed while the JVM is running, use the JVM option -Dcom.ibm.cacheLocalHost=true.
    4. Consider enabling IBM Java Health Center (-Xhealthcenter:level=headless) for post-mortem CPU and lock profiling data, although this has an overhead of about 2%.

For details, see the OpenJ9 and IBM J9 JVMs chapter.

HotSpot JVM Recipe

  1. In most cases, the default -XX:+UseG1GC or -XX:+UseParallelOldGC garbage collection policies (depending on version) work best, with the key tuning being the maximum heap size (-Xmx).
  2. Set -XX:+HeapDumpOnOutOfMemoryError.
  3. Enable verbose garbage collection and use a tool such as the Garbage Collection and Memory Visualizer to confirm the proportion of time in stop-the-world garbage collection pauses is less than ~10% and ideally less than 1%.
    1. Check for long individual pause times (e.g. greater than 400ms or whatever response time expectations are)
    2. For G1GC, check for humongous allocations.
    3. Review the latest garbage collection tuning guidance.

For details, see the HotSpot JVM chapter.

Java Profilers Recipe

  1. In most cases, sampling profilers are used first and tracing profilers are only used for fine grained tuning or deep dive analysis.
  2. Analyze any methods that use more than 1% of the reported time in themselves.
  3. Analyze any methods that use more than 10% of the reported time in themselves and their children.
  4. Analyze any locks that have large contention rates, particularly those with long average hold times.

Enabling profilers: