Troubleshooting WebSphere Liberty
Troubleshooting WebSphere Liberty Recipe
Review all warnings and errors in messages.log (or using binaryLog if binary logging is enabled) before and during the problem. A regular expression search is
" [W|E] ". One common type of warning is an FFDC warning which points to a matching file in the FFDC logs directory.If you're on Linux or use cygwin, use the following command:
find . -name "*messages*" -exec grep " [W|E] " {} \; | grep -v -e known_error
Review all JVM* messages in console.log before and during the problem. This may include things such as OutOfMemoryErrors. The filename of such artifacts includes a timestamp of the form YYYYMMDD. Review any other strange messages in console.log before and during the problem.
If verbose garbage collection is enabled, review verbosegc in console.log, or any verbosegc.log files (if using -Xverbosegclog or -Xloggc) in the IBM Garbage Collection and Memory Visualizer Tool and ensure that the proportion of time in garbage collection for a relevant period before and during the problem is less than 10%.
Review any javacore*.txt files in the IBM Thread and Monitor Dump Analyzer tool. Review the causes of the thread dump (e.g. user-generated, OutOfMemoryError, etc.) and review threads with large stacks and any monitor contention.
Review any heapdump*.phd and core*.dmp files in the IBM Memory Analyzer Tool.
Application Start and Stop
The CWWKZ0001I message is printed when an application starts. For example:
[10/27/20 16:56:31:644 UTC] 0000002b com.ibm.ws.app.manager.AppMessageHelper A CWWKZ0001I: Application daytrader started in 8.215 seconds.The CWWKZ0003I message is printed when an application is dynamically updated:
[10/27/20 18:13:04:882 UTC] 0000007f com.ibm.ws.app.manager.AppMessageHelper A CWWKZ0003I: The application daytrader updated in 1.305 seconds.The CWWKZ0009I message is printed when an application stops:
[10/27/20 18:11:54:519 UTC] 0000008a com.ibm.ws.app.manager.AppMessageHelper A CWWKZ0009I: The application daytrader has stopped successfully.Server Dumps
The server dump command requests various types of status information of a running server: http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/twlp_setup_dump_server.html
$LIBERTY/bin/server dump $SERVERBy default, the produced files will go to
$LIBERTY/usr/servers/$SERVER
Request Thread Dump
Additional methods of requesting thread dumps are documented in the Troubleshooting Java chapters.
$LIBERTY/bin/server javadump $SERVERRequest Heap Dump
Additional methods of requesting heap dumps are documented in the Troubleshooting Java chapters.
$LIBERTY/bin/server javadump $SERVER --include=heapRequest System Dump
Additional methods of requesting system dumps are documented in the Troubleshooting Operating Systems and Troubleshooting Java chapters.
Starting with Liberty 20.0.0.2,
this request exclusive+prepwalk.
$LIBERTY/bin/server javadump $SERVER --include=systemClient fails to Connect to Liberty Messaging Engine
The Liberty logs show the application has started normally and bound to the correct ports. However, the client application is getting the error.
1. Telnet to the IP port# fails.
2. netstat -an | grep LIST on the Liberty server shows
the port is bound to 127.0.0.1 (localhost).
com.ibm.ws.sib.jfapchannel.JFapConnectFailedException: CWSIJ0063E: A network connection to host name /192.168.2.234, port 9,126 cannot be established...
Strategy: Fix server.xml endpoints
Make sure the server.xml entries for the endpoints attribute host is set as:
host="*"Sending messages.log and trace.log to stdout on Linux
In some cases, it may be useful to send all WAS logs to stdout (i.e. console.log by default); for example, to intermingle JVM -Xtrace with WAS diagnostic trace for easier trace analysis:
Remove the <logging /> element from server.xml.
Add the following properties to bootstrap.properties (create this file in the same directory as server.xml if it doesn't exist), and replace the diagnostic trace specification with the desired WAS diagnostic trace:
com.ibm.ws.logging.log.directory=/dev/
com.ibm.ws.logging.message.file.name=stdout
com.ibm.ws.logging.max.file.size=0
com.ibm.ws.logging.max.files=0
com.ibm.ws.logging.trace.file.name=stdout
com.ibm.ws.logging.newLogsOnStart=false
com.ibm.ws.logging.trace.specification=*=info:com.ibm.ws.kernel.*=allAdd other configuration such as -Xtrace to jvm.options. For example:
-Xtrace:iprint=mt,methods={com/ibm/ws/kernel/launch/internal/FrameworkManager.launchFramework},trigger=method{com/ibm/ws/kernel/launch/internal/FrameworkManager.launchFramework,jstacktrace}
Start the server and now you should see intermingled trace in console.log; for example, the following shows both -Xtrace and WAS diagnostic trace for com/ibm/ws/kernel/launch/internal/FrameworkManager.launchFramework:
16:47:30.776*0x998700 mt.0 > com/ibm/ws/kernel/launch/internal/FrameworkManager.launchFramework(Lcom/ibm/ws/kernel/boot/BootstrapConfig;Lcom/ibm/wsspi/logprovider/LogProvider;)V bytecode method, this = 0xfffd6b68
16:47:30.777 0x998700 j9trc_aux.0 - jstacktrace:
16:47:30.777 0x998700 j9trc_aux.1 - [1] com.ibm.ws.kernel.launch.internal.FrameworkManager.launchFramework (FrameworkManager.java:198)
16:47:30.777 0x998700 j9trc_aux.1 - [2] com.ibm.ws.kernel.launch.internal.LauncherDelegateImpl.doFrameworkLaunch (LauncherDelegateImpl.java:114)
16:47:30.777 0x998700 j9trc_aux.1 - [3] com.ibm.ws.kernel.launch.internal.LauncherDelegateImpl.launchFramework (LauncherDelegateImpl.java:100)
16:47:30.777 0x998700 j9trc_aux.1 - [4] com.ibm.ws.kernel.boot.internal.KernelBootstrap.go (KernelBootstrap.java:213)
16:47:30.777 0x998700 j9trc_aux.1 - [5] com.ibm.ws.kernel.boot.Launcher.handleActions (Launcher.java:241)
16:47:30.777 0x998700 j9trc_aux.1 - [6] com.ibm.ws.kernel.boot.Launcher.createPlatform (Launcher.java:117)
16:47:30.777 0x998700 j9trc_aux.1 - [7] com.ibm.ws.kernel.boot.cmdline.EnvCheck.main (EnvCheck.java:59)
16:47:30.777 0x998700 j9trc_aux.1 - [8] com.ibm.ws.kernel.boot.cmdline.EnvCheck.main (EnvCheck.java:35)
[11/25/19 16:47:30:777 UTC] 00000001 id=fccc7204 com.ibm.ws.kernel.launch.internal.FrameworkManager > launchFramework Entry
Important notes:
- This configures WAS logging to unlimited size and disables cleaning up old WAS logs on startup, so you must properly manage disk usage of console.log (e.g. truncating with $(truncate -s 0 console.log) or copying off and zipping files or moving files to a different partition while running, if running out of space).
- Some of the file writes will be uncoordinated (e.g. WAS trace and JVM trace), so be careful in using any automated tooling in processing the files as some lines may be spliced together; instead, use human analysis.
- FFDC files may fail to be written because they will try to write to /dev/ffdc/*
- As an alternative to this procedure, consider writing a post-processing tool that combines multiple files together based on timestamp.
Alternatively, you may switch the log format to JSON which supports sending output to stdout (and messages may be disabled since trace contains messages); this has the benefit of not breaking FFDC:
com.ibm.ws.logging.message.format=json
com.ibm.ws.logging.message.source=
com.ibm.ws.logging.trace.file.name=stdout
com.ibm.ws.logging.max.file.size=0
com.ibm.ws.logging.max.files=0
com.ibm.ws.logging.newLogsOnStart=false
com.ibm.ws.logging.trace.specification=*=info:com.ibm.ws.kernel.*=all
Override Context Root
To override the context root of an EAR, use the following and match the module name to the module name in the EAR:
<enterpriseApplication location="my.ear">
<web-ext moduleName="" context-root=""/>
</enterpriseApplication>ws-javaagent.jar
ws-javaagent.jar is a Java
agent included as part of Liberty with the
-javaagent:$LIBERTY/bin/tools/ws-javaagent.jar option. It
implements Liberty's diagnostic trace feature by using byte code
injection (thus avoiding the performance overhead of log guards and
other things when trace is disabled). Thus, it is needed to diagnostic
issues using diagnostic trace.