IBM WebSphere Application Server Performance Cookbook

Troubleshooting Solaris

Mapping LWP ID to Java thread

It's often useful to map an LWP ID (for example, reported in prstat -L) to a Java thread. The pstack command may be used to print all native stack traces along with the LWP ID:

# prstat -mvLp 5598 5 2
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
  5598 root      78 2.8 0.1 0.0 1.7 1.3 7.0 8.7 135 502  3K   2 java/2
  5598 root      12 0.0 0.0 0.1 0.0  85 0.0 2.7  54  59 124   0 java/10...
# pstack 5598
5598:    /opt/IBM/WAS855/AppServer/java_1.7_32/bin/java Play
-----------------  lwp# 2 / thread# 2  --------------------
 fbc895a0 * *java/util/StringTokenizer.nextToken()Ljava/lang/String; [compiled] +74 (line 691)
 fbc895a0 * *javax/crypto/Cipher.a(Ljava/lang/String;)[Ljava/lang/String;+55
 fbca2d10 * *javax/crypto/Cipher.b(Ljava/lang/String;)Ljava/util/List; [compiled] +2
 fbc99494 * *javax/crypto/Cipher.getInstance(Ljava/lang/String;)Ljavax/crypto/Cipher; [compiled] +2
 fbcbc29c * *Play.main([Ljava/lang/String;)V [compiled] +61 (line 39)
 fbc0021c * StubRoutines (1)
 fe5b035c __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (fe07fed8, fe07fe70, e, 27800, f5cb0, d79a4fc8) + 3a0
 fe65be7c jni_CallStaticVoidMethod (27928, d79a4fc8, 21240, e, 27800, ff117e5c) + 678
 ff361bd8 JavaMain (fe66537c, 28e6c, 27928, ff387370, ff0f261c, fe65b804) + 740
 ff2c5238 _lwp_start (0, 0, 0, 0, 0, 0)...

Request core dump

  1. The gcore command pauses the process while the core is generated and then the process should continue. Replace ${PID} in the following example with the process ID. You must have permissions to the process (i.e. either run as the owner of the process or as root). The size of the core file will be the size of the virtual size of the process (ps VSZ). If there is sufficient free space in physical RAM and the filecache, the core file will be written to RAM and then asynchronously written out to the filesystem which can dramatically improve the speed of generating a core and reduce the time the process is paused. In general, core dumps compress very well (often up to 75%) for transfer. (http://docs.oracle.com/cd/E36784_01/html/E36870/gcore-1.html)
    $ gcore ${PID}

  2. If gcore does not start to write the core dump immediately, it may be hung waiting to acquire control of the process. If this does not succeed for some time, try instead with the -F option.

  3. If none of the other options work, you may crash the process which should process a core dump using one of:
    $ kill -6 ${PID}
    $ kill -11 ${PID}

Debug Symbols

"To compile optimized code for use with dbx, compile the source code with both the -O (uppercase letter O) and the -g options... The -g0 (zero) option turns on debugging and does not affect inlining of functions." (http://docs.oracle.com/cd/E19205-01/819-5257/gevhr/index.html)

Create separate debug files: http://docs.oracle.com/cd/E19205-01/819-5257/gevia/index.html