IBM WebSphere Application Server Performance Cookbook

Linux perf Recipe

Linux perf record gathers high frequency native stack samples of what's using the CPU. The overhead is generally in the range of a few % and is production ready although it's best to gauge the overhead in a test environment.

  1. Install perf if it's not installed. To check if it's installed: 
    perf version
  2. Prepare the Java process:
    1. For an IBM Java or IBM Semeru Runtimes JVM:
      1. For IBM Java >= 8.0.7.20 or IBM Semeru Runtimes >= v8.0.352 / 11.0.17.0 / 17.0.5.0, restart the Java process with -XX:+PerfTool
      2. For older versions of IBM Java or IBM Semeru Runtimes, restart the Java process with -Xjit:perfTool while making sure to combine with commas with any pre-existing -Xjit options
    2. For a HotSpot JVM:
      1. Restart the Java process with -XX:+UnlockDiagnosticVMOptions -XX:+PreserveFramePointer -XX:+ShowHiddenFrames
      2. For a HotSpot JVM >= Java 16, also add the option -XX:+DumpPerfMapAtExit which will create the valuable /tmp/perf-$PID.map file on graceful JVM exit. If you won't be able to easily do a graceful JVM exit, then use the next option instead.
      3. For a HotSpot JVM < Java 16:
        1. Also add the option -XX:-UseCodeCacheFlushing
        2. Compile perf-map-agent into some directory
  3. As root, during the performance problem, run one of the following commands. Change 60 to the number of seconds you want to gather data for:
    1. For IBM Java and IBM Semeru Runtimes running on any processor: 
      date +'%Y-%m-%d %H:%M:%S.%N %Z' &>> diag_starttimes_$(hostname).log; cat /proc/uptime &>> diag_starttimes_$(hostname).log; perf record --call-graph dwarf,65528 -F 99 -a -g -T -- sleep 60
    2. For IBM Java and IBM Semeru Runtimes running on top of an Intel processor that is Haswell or later (see cat /proc/cpuinfo and reference Intel Processor names), LBR may be lower overhead than DWARD although note that LBR has a limited stack depth: 
      date +'%Y-%m-%d %H:%M:%S.%N %Z' &>> diag_starttimes_$(hostname).log; cat /proc/uptime &>> diag_starttimes_$(hostname).log; perf record --call-graph lbr -F 99 -a -g -T -- sleep 60
    3. For a HotSpot JVM: 
      date +'%Y-%m-%d %H:%M:%S.%N %Z' &>> diag_starttimes_$(hostname).log; cat /proc/uptime &>> diag_starttimes_$(hostname).log; perf record --call-graph fp -F 99 -a -g -T -- sleep 60
  4. Wait for the above command to complete.
  5. If running a HotSpot JVM:
    1. For a HotSpot JVM >= Java 16, stop the JVM gracefully to produce the /tmp/perf-$PID.map file. If you can't do this, then, instead, follow the instructions above for a HotSpot JVM < Java 16 using perf-map-agent
    2. For a HotSpot JVM < Java 16, run bin/create-java-perf-map.sh $PID from the perf-map-agent directory to create the /tmp/perf-$PID.map file
  6. As root from the directory where perf record was run, execute the following commands: 
    perf script --header -I -f -F comm,cpu,pid,tid,time,event,ip,sym,dso,symoff > diag_perfscript_$(hostname)_$(date +%Y%m%d_%H%M%S_%N).stdout.txt 2> diag_perfscript_$(hostname)_$(date +%Y%m%d_%H%M%S_%N).stderr.txt
perf report -n --show-cpu-utilization -v --stdio > diag_perfreport_$(hostname)_$(date +%Y%m%d_%H%M%S_%N).stdout.txt 2> diag_perfreport_$(hostname)_$(date +%Y%m%d_%H%M%S_%N).stderr.txt
perf archive --all || perf archive
  7. Gather a thread dump. This is very low overhead with the process pausing for generally about 10ms to 100ms. 
    kill -3 $PID
  8. Optionally, for IBM Java and IBM Semeru Runtimes processes, gather an operating sytem core dump of the process if the security, disk and performance risks are acceptable (the process may pause for up to 30 seconds or more) and the process and operating system are configured for it (e.g. core and file ulimits, kernel.core_pattern truncation settings, etc.) using one of various mechanisms and then run jextract (IBM Java) or jpackcore (Semeru) on it; for example: 
    $JDK/bin/jpackcore core*.dmp
  9. As root from the directory where perf record was run, replace $LOGS_DIR with the location of the logs of the JVM (including stdout/stderr, verbosegc, server/application logs, etc.), $THREAD_DUMPS_DIR with the location where thread dumps were produced, and $OS_CORE_DUMPS_DIR if a core dump was produced: 
    tar czhvf diag_perfall_$(hostname)_$(date +%Y%m%d_%H%M%S).tar.gz diag* perf*bz2 diag_* /tmp/perf*map $LOGS_DIR $THREAD_DUMPS_DIR/javacore*.txt $OS_CORE_DUMPS_DIR/core*.dmp.zip
  10. Upload diag_perfall_*.tar.gz

If you want to do basic analysis of the perf output yourself:

  1. Top 10 CPU-using stack frames: 
    cat diag_perfscript*txt | awk 'go { go=0; print; } /cpu-clock:/ || /cycles:/ { go=1; }' | sort | uniq -c | sort -nr | head
  2. Create FlameGraphs:
    1. git clone https://github.com/brendangregg/FlameGraph
    2. cd FlameGraph
    3. cat diag_perfscript*txt | ./stackcollapse-perf.pl > out.perf-folded
    4. ./flamegraph.pl --width 1024 out.perf-folded > perf.svg
    5. ./flamegraph.pl --reverse --width 1024 out.perf-folded > perf-reverse.svg
    6. Open perf.svg and perf-reverse.svg in your browser

Notes:

  • For IBM Java and IBM Semeru Runtimes, if not all symbols are resolved, try again with the additional option -Xlp:codecache:pagesize=4k

For background, see Linux perf.