Java Recipes

1. Review the Operating System recipe for your OS.
2. Tune the maximum Java heap size (-Xmx or -XX:MaxRAMPercentage):
   1. Enable verbose garbage collection (-verbose:gc) which prints statistics on garbage collection to files and generally has an overhead less than 1%. Use a tool such as the IBM Garbage Collection and Memory Visualizer to analyze the verbosegc output. The proportion of time spent in garbage collection versus application processing time should generally be less than 10% and ideally less than 1%.
   2. In general, a place to start is to set -Xmx to 43% larger than the maximum occupancy of the application, although the latter is largely a function of workload and thread pool size, so this is just a heuristic.
3. If the node only uses IPv4 and does not use IPv6, then add the JVM parameters -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false
4. Consider the particular type of garbage collector to use (see the comparison tables in the JVM sub-chapters).
5. Ensure there is no memory leak with long running tests.
6. If using a generational collector (which most modern default collectors are):
   1. Ensure tests run through full/tenured collections and ensure those pause times are not too long.
   2. Ensure that there is a sawtooth pattern in the heap usage after collection. Otherwise, the heap size may be too small or the nursery too big.
   3. A heuristic is that the sawtooth should drop about 25% of the heap size on full collections.
7. Total pause times over a few seconds should be routinely investigated.
8. Use a profiler such as IBM Java Health Center or OpenJDK Mission Control with a particular focus on the profiling and lock contention analysis. Otherwise, use periodic thread dumps to review JVM activity with the IBM Thread and Monitor Dump Analyzer tool.
9. If heavily using XML, consider explicitly configuring JAXP ServiceLoader properties to avoid unnecessary classloading activity.
10. Object allocation failures for objects greater than 5MB should generally be investigated.
11. Take a system dump or HPROF heapdump during peak activity and review it with the Eclipse Memory Analyzer Tool to see if there are any areas in the heap for optimization.
12. If using RMI (e.g. EJBs over RMI), then consider avoiding periodic full GCs from the RMI engine that are generally considered obsolete: -Dsun.rmi.dgc.client.gcInterval=3600000000000 -Dsun.rmi.dgc.server.gcInterval=3600000000000
13. Review the stderr and stdout logs for any errors, warnings, or high volumes of messages (e.g. OutOfMemoryErrors).
14. If running multiple JVMs on the same machine, consider pinning JVMs to sets of processor cores and tuning -Xgcthreads/-XcompilationThreads or -XX:ParallelGCThreads.
15. In general, if memory usage is very flat and consistent, it may be optimal to fix -Xms = -Xmx. For widely varying heap usage, -Xms < -Xmx is generally recommended. You may get the best of both worlds by setting -Xms to the lowest steady state memory usage, -Xmaxf1.0 to eliminate shrinkage, -Xminf to avoid compaction before expansion, and -Xmine to reduce expansions.
16. Request a thread dump and search its output for "deadlock" to ensure that no threads are deadlocked (thus reducing throughput).

For details, see the Java chapter and the chapter for your particular JVM.

OpenJ9 and IBM J9 JVMs Recipe

1. In most cases, the default -Xgcpolicy:gencon garbage collection policy works best, with the key tuning being the maximum heap size (-Xmx or -XX:MaxRAMPercentage) and maximum nursery size (-Xmn).
2. Upgrade to the latest version and fixpack as there is a history of making performance improvements and fixing issues or regressions over time.
3. Take a javacore and review the Java arguments (UserArgs) and Environment Variables sections and remove any unnecessary debug options.
4. Take a javacore and review if the JIT code cache is full or nearly full; if so, and there's available physical memory, test increasing it with -Xcodecachetotal384m -Xcodecache32m
5. Take a javacore and review if the shared class cache is full or nearly full; if so, and there's available physical memory, consider increasing -Xscmx
6. If using -Xgcpolicy:gencon and you want to reduce average nursery pause times at some throughput and CPU cost, consider concurrent scavenge.
7. Consider setting -XX:+CompactStrings where available, applicable, and not already the default.
8. Review the performance tuning topics in the OpenJ9 or IBM Java documentation.
9. When running benchmarks or comparing performance to other JVMs, consider testing various benchmark ideas.
10. If using IBM Semeru Runtimes:
    1. If JIT CPU or memory usage are a concern, consider using the remote JITServer on available platforms.
    2. For AIX and Linux, ensure OpenSSL is on the system path for maximum security performance.
    3. On z/OS, consider enabling IBM Java Health Center (-Xhealthcenter:level=headless) for post-mortem CPU and lock profiling data, although this has an overhead of about 2%.
    4. On z/OS, consider using the "pauseless" garbage collection option -Xgc:concurrentScavenge if using gencon and on recent software and hardware.
11. If using IBM Java (does not apply to IBM Semeru Runtimes):
    1. Consider using the IBMJCEPlus security provider that may offer large performance improvements in encryption. This is now the default except on z/OS since 8.0.7.0.
    2. If the node is using a static IP address that won't be changed while the JVM is running, use the JVM option -Dcom.ibm.cacheLocalHost=true.
    3. Consider enabling IBM Java Health Center (-Xhealthcenter:level=headless) for post-mortem CPU and lock profiling data, although this has an overhead of about 2%.

For details, see the OpenJ9 and IBM J9 JVMs chapter.

HotSpot JVM Recipe

1. In most cases, the default -XX:+UseG1GC or -XX:+UseParallelOldGC garbage collection policies (depending on version) work best, with the key tuning being the maximum heap size (-Xmx).
2. Set -XX:+HeapDumpOnOutOfMemoryError.
3. Enable verbose garbage collection and use a tool such as the Garbage Collection and Memory Visualizer to confirm the proportion of time in stop-the-world garbage collection pauses is less than ~10% and ideally less than 1%.
   1. Check for long individual pause times (e.g. greater than 400ms or whatever response time expectations are)
   2. For G1GC, check for humongous allocations.
   3. Review the latest garbage collection tuning guidance.

For details, see the HotSpot JVM chapter.

Java Profilers Recipe

1. In most cases, sampling profilers are used first and tracing profilers are only used for fine grained tuning or deep dive analysis.
2. Analyze any methods that use more than 1% of the reported time in themselves.
3. Analyze any methods that use more than 10% of the reported time in themselves and their children.
4. Analyze any locks that have large contention rates, particularly those with long average hold times.

Enabling profilers:

• J9 Health Center Enable at Startup
• J9 Health Center Enable at Startup of Limited Duration
• J9 Health Center Enable at Runtime
• J9 Health Center Enable at Runtime of Limited Duration
• J9 Health Center Enable at Runtime of Limited Duration on z/OS
• HotSpot Mission Control Enable at Startup

Previous Section (macOS Recipes) | Next Section (J9 Health Center Enable at Startup) | Back to Table of Contents