IBM WebSphere Application Server Performance Cookbook

Troubleshooting HP-UX

32-bit Native OutOfMemoryErrors

You may increase the user virtual address space from 2GB to 3GB (at the cost of less space to the kernel for things like network buffers) with:

chatr +q3p enable ${PATH_TO_JAVA}

You can check if this is enable with:

chatr ${PATH_TO_JAVA}
...
third quadrant private data space enabled

gdb/wdb

When the process is hung, attach to the PID, for example:

/opt/langtools/bin/gdb /opt/IBM/WebSphere/AppServer/java/bin/IA64W/java 24072

Then run thread apply all bt

HP-UX does not provide a tool (such as "ps") to print the full command line of a running program (no equivalent of Solaris /usr/ucb/ps). The -x parameter of ps only prints the first 1024 characters, which is often insufficient for Java programs:

Only a subset of the command line is saved by the kernel; as much of the command line will be displayed as is available... The value of DEFAULT_CMD_LINE_WIDTH should be between 64 and 1020.

You can attach to a process using gdb/wdb and print argc/argv. First, we attach to a process by passing in the location of java (which you can get from ps -elfx) followed by the PID (note that the process will be completely paused until you detach gdb):

$ /opt/langtools/bin/gdb /opt/IBM/WebSphere/AppServer/java/bin/IA64W/java 24072

__argc and __argv are global variables that we can access, so let's first see how many arguments there are:

(gdb) print __argc
$1 = 3

In this example, we have 3 arguments. Next, we know that argv is a pointer to a list of pointers, each with one of the program arguments, so we print that many addresses at the location of argv (i.e. replace 3 with your value of argc):

(gdb) x/3a __argv
0x9ffffffffffff950:     0x9ffffffffffff9e8      0x9ffffffffffffa19
0x9ffffffffffff960:     0x9ffffffffffffa24

Each of these addresses is a pointer to a null-terminated string, so we print each using the s option:

(gdb) x/s 0x9ffffffffffff9e8
0x9ffffffffffff9e8:      "/opt/IBM/WebSphere/AppServer/java/bin/IA64W/java"
(gdb) x/s 0x9ffffffffffffa19
0x9ffffffffffffa19:      "HelloWorld"
(gdb) x/s 0x9ffffffffffffa24
0x9ffffffffffffa24:      "testarg"

Don't forget to "detach" to continue the process.

Although the Java jps command with the -v parameter is no better, at least you can use jps -m to map PID to WAS server name.

If you are using an Itanium system, the following caliper command prints the full command line. This will have some overhead as it is gathering a flat profile of sampled process instructions for 1 second, but it is presumably more lightweight (and more user-friendly) than gdb:

/opt/caliper/bin/caliper fprof --process=root --attach $PID --duration 1 | grep Invocation:

And here's a one-line command that runs the above on all java PIDs:

for i in `ps -elfx | grep java | grep -v grep | awk '{print $4}'`; do echo $i; /opt/caliper/bin/caliper fprof --process=root --attach $i --duration 1 | grep Invocation: ;done;