IBM WebSphere Application Server Performance Cookbook

HP-UX

HP-UX Recipe

  1. CPU core(s) should not be consistently saturated.
  2. Generally, physical memory should never be saturated and the operating system should not page memory out to disk.
  3. Input/Output interfaces such as network cards and disks should not be saturated, and should not have poor response times.
  4. TCP/IP and network tuning, whilst sometimes complicated to investigate, may have dramatic effects on performance.
  5. Operating system level statistics and optionally process level statistics should be periodically monitored and saved for historical analysis.
  6. Review operating system logs for any errors, warnings, or high volumes of messages.
  7. Review snapshots of process activity, and for the largest users of resources, review per thread activity.
  8. If the operating system is running in a virtualized guest, review the configuration and whether or not resource allotments are changing dynamically.
  9. If there is sufficient network capacity for the additional packets, consider reducing the default TCP keepalive timer (tcp_keepalive_interval) from 2 hours to a value less than intermediate device idle timeouts (e.g. firewalls).
  10. Test disabling delayed ACKs

Also review the general topics in the Operating Systems chapter.

General

Review some of the tuning recommendations in the following documentation pages:

  1. https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tprf_tunehp.html
  2. https://www.ibm.com/support/knowledgecenter/SSAW57_8.0.0/com.ibm.websphere.installation.nd.doc/info/ae/ae/tins_hpuxsetup.html
  3. https://h20392.www2.hpe.com/portal/swdepot/displayProductInfo.do?productNumber=HPJCONFIG

Check the BIOS to ensure highest speed:

  1. Power Management -> HP Power Profile -> Maximum Performance
  2. Power Management -> HP Power Regulator -> HP Static High Performance Mode
  3. Advanced Options -> Advanced Performance Tuning Options -> Memory Speed with 2 DIMMS per channel -> Maximum MHz

Consider installing the following generally useful software:

Query basic system information:

$ uname -a; model; machinfo; sysdef; swlist -l

Central Processing Unit (CPU)

Check if hyperthreading is enabled or disabled using machinfo and consider enabling/disabling it, if applicable:

Hyperthreading enabled:

$ machinfo
LCPU attribute is enabled...

Hyperthreading disabled:

$ machinfo
LCPU attribute is disabled...

Use the general performance MustGather (http://www-01.ibm.com/support/docview.wss?uid=swg21127574&aid=1):

$ hpux_performance.sh $PID

Use the top and vmstat commands for basic process monitoring.

Consider enabling sar for historical data (http://www.ibm.com/developerworks/aix/library/au-unix-perfmonsar.html).

The ptree command is a useful way to visualize the process tree.

For custom columns in ps:

UNIX95= ps -ef -o pid,pcpu,pri,pset

GlancePlus

GlancePlus (license required) is a very useful tool. To run it for a few minutes, use this hpux_glance.sh script: http://www-01.ibm.com/support/docview.wss?uid=swg21127574&aid=3

caliper

The caliper tool is a native sampling profiler (http://h20566.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=4268168&docId=emr_na-c04221975&docLocale=en_US). The simplest report is the flat profile:

/opt/caliper/bin/caliper fprof --process=all --attach $PID --duration 60 -o fprof.txt

System wide:

/opt/caliper/bin/caliper fprof -o fprofsystem.txt --ev all -w -e 30

Or

/opt/caliper/bin/caliper fprof --scope=kernel --duration=60 -o kernelfprof.txt

HPjmeter

HPjmeter is a powerful Java profiler: https://h20392.www2.hpe.com/portal/swdepot/displayProductInfo.do?productNumber=HPJMETER

$ /opt/hpjmeter/bin/javaGlanceAdviser.ksh $PID

"If you also collected GC information using the -Xverbosegc option, you can append the Glance data to the GC log file and then use HPjmeter to read the combined file."

jps

Use the jps tool to map Java server names to process IDs. Example:

$ /opt/IBM/WebSphere/AppServer/java/bin/jps -m
9326 WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer /opt/IBM/WebSphere/AppServer/profiles/node1/config cell1 node1 nodeagent
7113 WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer /opt/IBM/WebSphere/AppServer/profiles/dmgr1/config cell1 dmgr1 dmgr
6283 WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer /opt/IBM/WebSphere/AppServer/profiles/node1/config cell1 node1 server1

Or using caliper (on Itanium systems):

$ for i in `ps -elfx | grep java | grep -v grep | awk '{print $4}'`; \
do echo $i; /opt/caliper/bin/caliper fprof --process=root --attach $i --duration 1 | grep Invocation: ;done;

Physical Memory (RAM)

swapinfo: http://h20331.www2.hp.com/Hpsub/downloads/task_guide.pdf

Input/Output (I/O)

Use the bdf command to review disk utilization.

Networking

Update the TCP listen backlog to 511 by adding "ndd -set /dev/tcp tcp_conn_request_max 511" to /etc/rc.config.d/nddconf and running "ndd -c" (https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tprf_tunehp.html)

Update the TCP keepalive interval by adding "ndd -set /dev/tcp tcp_keepalive_interval 7200000" to /etc/rc.config.d/nddconf and running "ndd -c" (https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tprf_tunehp.html)

Update the TCP keepalive maximum probes by adding "ndd -set /dev/tcp tcp_keepalives_kill 1" to /etc/rc.config.d/nddconf and running "ndd -c" (https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tprf_tunehp.html)

Use the following command to print socket details: netstat -anf inet

Ping a remote host. In general, and particularly for LANs, ping times should be less than a few hundred milliseconds with little standard deviation.

$ ping -ns 10.20.30.1
PING 10.20.30.1 : 56 data bytes
64 bytes from 10.20.30.1: icmp_seq=0. time=77.9 ms
64 bytes from 10.20.30.1: icmp_seq=1. time=77.2 ms
64 bytes from 10.20.30.1: icmp_seq=2. time=78.3 ms
64 bytes from 10.20.30.1: icmp_seq=3. time=76.9 ms

nettl

Capture network packets using nettl (http://www-01.ibm.com/support/docview.wss?uid=swg21175744).

Start capturing all traffic:

# nettl -tn all -e all -f networktrace

Stop capturing all traffic:

# nettl -tf -e all

Profiling

The JVM on HP supports dynamically enabling and disabling low-overhead sampling profiling using the kill command:

  • Enable Profiling:
    $ kill -USR2 PID
  • Disable Profiling:
    $ kill -USR2 PID

The profiling will write information on each signal to native_stderr.log. For example:

First signal

eprof: starting profiling Tue Nov 20 14:05:02 2012
eprof: terminating profiling
eprof: cannot measure profiling intrusion

Second signal

eprof: writing profile data to /opt/IBM/WebSphere/AppServer/profiles/node1/java10760_75806.eprof
eprof: done.

Modifying Kernel Parameters

Review the following instructions to modify core HP-UX kernel parameters: http://www.ibm.com/support/knowledgecenter/SS7J6S_6.2.0/com.ibm.websphere.wesb620.doc/doc/tins_set_kernel_hpux.html

Running ndd -set will not maintain the parameters after rebooting. Instead, it is recommended to update the parameters in /etc/rc.config.d/nddconf and run "ndd -c" to load the values from this file and the values will also be picked up on reboot.

tusc

tusc is a system call tracer.

$ /usr/local/bin/tusc -f -C -o tusc_counts.txt $PID & sleep 30; kill -INT $!
$ /usr/local/bin/tusc -f -l -D -R -T "" -o tusc.txt $PID & sleep 30; kill -INT $!

Processor Sets

"The default processor set (0) always exists and may not be destroyed. All processes and processors at system init time start out in the system default processor set."

Therefore, you may want to "reserve" processor set 0 for background processes and non-application server JVMs, and only distribute the JVMs across the other processor sets. You should take into account the core, hyperthread, and L3 layout to avoid sharing processors from pset 0 with the JVM processor sets.

List CPU IDs and which processor set IDs they're bound to:

$ /usr/sbin/psrset -p
SPU   0          PSET  0
SPU   1          PSET  0
SPU   2          PSET  0
SPU   3          PSET  0

Create a processor set for a CPU ID:

$ /usr/sbin/psrset -c 1

Bind PID to processor set 1:

$ /usr/sbin/psrset -b 1 `cat /opt/IBM/WebSphere/AppServer/profiles/node1/logs/server1/*.pid`

Query processor sets for PIDs:

$ /usr/sbin/psrset -q `cat /opt/IBM/WebSphere/AppServer/profiles/node1/logs/server1/*.pid` `cat /opt/IBM/WebSphere/AppServer/profiles/node1/logs/server2/*.pid`
PID 28493 PSET 0
PID 25756 PSET 0

Automation

To assign processor sets automatically, you will need to modify the Java command line. This means that you will not be able to use the administrative console to start servers (you can still use it to stop servers)

  1. For each application server instance, run startServer.sh $NAME -script to generate its start script.
  2. Now you should have start_$JVMID.sh script for each JVM.
  3. Edit each start_...sh script and you should see an exec java line at the bottom. Update to redirect output:
    exec "/opt/IBM/WebSphere/AppServer/java/bin/java" $DEBUG "-XX:...
    Changes to:
    exec "/opt/IBM/WebSphere/AppServer/java/bin/java" $DEBUG "-XX:... >> /opt/IBM/WebSphere/AppServer/profiles/node1/logs/dynamiccluster1_node1/native_stdout.log 2>> /opt/IBM/WebSphere/AppServer/profiles/node1/logs/dynamiccluster1_node1/native_stderr.log &
  4. Start the JVM in the processor set with (each Nth JVM will have _N in the shell script name) -- replace 1 with the processor set ID:
    /usr/sbin/psrset -e 1 ./start_server1.sh

...