IBM WebSphere Application Server Performance Cookbook

Liberty in Containers

Recipe

Review the Java in Containers recipes
Execute configure.sh as the last step in your Containerfile to make it fit-for-purpose and initialize the shared class cache.
Review the Configuring Security best practices
If using IBM or Semeru Java, mount a shared volume for the shared class cache in ${WLP_OUTPUT_DIR}/.classCache
Consider logging in JSON format for consumption by centralized logging.
If using IBM or Semeru Java and startup time is highly variable, review the potential impact of the maximum heap size on the shared class cache.
Unless an application uses 2 phase commit (2PC) and transaction logs are properly handled (e.g. in a database, peer recovery, on a shared disk, etc.), then consider setting the JVM option -Dcom.ibm.tx.jta.disable2PC=true so that an application doesn't accidentally use 2PC and risk unrecoverable transactions.
OpenShift:
1. Review the Application Monitoring options.
Review the Liberty recipe
Review the Java recipes
Review the Operating System Recipes

Container Images

WebSphere Liberty common images:
- WebSphere Liberty with IBM Semeru Runtimes most recent LTS release on UBI: FROM icr.io/appcafe/websphere-liberty
- WebSphere Liberty with IBM Semeru Runtimes Java 25 on UBI: FROM icr.io/appcafe/websphere-liberty:kernel-java25-openj9-ubi-minimal
- WebSphere Liberty with IBM Semeru Runtimes Java 21 on UBI: FROM icr.io/appcafe/websphere-liberty:kernel-java21-openj9-ubi-minimal
- WebSphere Liberty with IBM Semeru Runtimes Java 17 on UBI: FROM icr.io/appcafe/websphere-liberty:kernel-java17-openj9-ubi
- WebSphere Liberty with IBM Semeru Runtimes Java 11 on UBI: FROM icr.io/appcafe/websphere-liberty:kernel-java11-openj9-ubi
- WebSphere Liberty with IBM Semeru Runtimes Java 8 on UBI: FROM icr.io/appcafe/websphere-liberty:kernel-java8-openj9-ubi
- WebSphere Liberty with IBM Java 8 on UBI: FROM icr.io/appcafe/websphere-liberty:kernel-java8-ibmjava-ubi
- See all tags:
```
curl -s https://icr.io/v2/appcafe/websphere-liberty/tags/list | jq .tags
```
OpenLiberty common images:
- OpenLiberty with IBM Semeru Runtimes most recent LTS release on UBI: FROM icr.io/appcafe/open-liberty
- OpenLiberty with IBM Semeru Runtimes Java 25 on UBI: FROM icr.io/appcafe/open-liberty:kernel-slim-java25-openj9-ubi-minimal
- OpenLiberty with IBM Semeru Runtimes Java 21 on UBI: FROM icr.io/appcafe/open-liberty:kernel-slim-java21-openj9-ubi-minimal
- OpenLiberty with IBM Semeru Runtimes Java 17 on UBI: FROM icr.io/appcafe/open-liberty:kernel-slim-java17-openj9-ubi
- OpenLiberty with IBM Semeru Runtimes Java 11 on UBI: FROM icr.io/appcafe/open-liberty:kernel-slim-java11-openj9-ubi
- OpenLiberty with IBM Semeru Runtimes Java 8 on UBI: FROM icr.io/appcafe/open-liberty:kernel-slim-java8-openj9-ubi
- OpenLiberty with IBM Java 8 on UBI: FROM icr.io/appcafe/open-liberty:kernel-slim-java8-ibmjava-ubi
- See all tags:
```
curl -s https://icr.io/v2/appcafe/open-liberty/tags/list | jq .tags
```

Liberty Operator

The WebSphere Liberty Operator and Open Liberty Operator have similar features and conceptually mirror the difference between WebSphere Liberty and Open Liberty.

By default, the Liberty Operator installs into the entire cluster. This means that one instance of the operator manages all Liberty Operator Custom Resources across all namespaces. Alternatively, you may install unique instances of the Liberty Operator into specific namespaces.

The benefits of installing the operator once into all namespaces is that there's a single operator to upgrade and monitor and it's easier for teams to use the operator in new namespaces. The risks are that a problem or behavior change after an operator upgrade might impact more applications and memory usage of the operator might be significant if there are a lot of namespaces (up to 1 to 10MB per namespace) and the operator may crash with an insufficient memory limit.

Resources

Background

6 reasons why Open Liberty is an ideal choice for developing and deploying microservices

Environment

Liberty on Docker changes the directory of messages.log and FFDC to /logs using the LOG_DIR envar: https://github.com/WASdev/ci.docker/blob/46b7c0a/ga/latest/kernel/Dockerfile#L54

Liberty on Docker changes the current working directory of the Liberty process to /opt/ibm/wlp/output using the WLP_OUTPUT_DIR envar so this is where javacores, etc. will go: https://github.com/WASdev/ci.docker/blob/46b7c0a/ga/latest/kernel/Dockerfile#L55

To use externally configured JVM parameters (since jvm.options doesn't support variable substitution), consider using the JVM_ARGS envar or mount the jvm.options file using a ConfigMap.

Shared Class Cache in Containers

If using the J9 JVM, mount a shared volume for the shared class cache in ${WLP_OUTPUT_DIR}/.classCache. For WebSphere Liberty, that's /opt/ibm/wlp/output/${SERVER_NAME}/.classCache and for OpenLiberty, that's /opt/ol/wlp/output/${SERVER_NAME}/.classCache. If using the default server name of defaultServer and default ${WLP_OUTPUT_DIR}, these resolve to:

WebSphere Liberty: /opt/ibm/wlp/output/defaultServer/.classCache
OpenLiberty: /opt/ol/wlp/output/defaultServer/.classCache

Run Examples

Examples:

podman run --rm icr.io/appcafe/open-liberty:full-java8-ibmjava-ubi sh -c "printf '<server><featureManager><feature>jakartaee-9.1</feature><feature>microProfile-5.0</feature></featureManager></server>' > /config/configDropins/overrides/override.xml && sed -i '/<feature>.*<\/feature>/d' /config/server.xml && curl -L https://github.com/IBM/helloworldjsp/releases/download/0.1.20240212/helloworldjsp.war -so /config/dropins/helloworldjsp.war && (/opt/ol/wlp/bin/server run defaultServer &> /logs/console.log &) && sleep 2 && tail -99999f /logs/messages.log | grep -q CWWKF0011I && cat /logs/messages.log && /opt/ol/wlp/bin/server stop defaultServer"
podman run --rm icr.io/appcafe/open-liberty:full-java8-ibmjava-ubi sh -c "curl -L https://github.com/IBM/helloworldjsp/releases/download/0.1.20240212/helloworldjsp.war -so /config/dropins/helloworldjsp.war && echo '<?xml version=\"1.0\" encoding=\"UTF-8\"?><server><httpEndpoint id=\"defaultHttpEndpoint\" host=\"*\" httpPort=\"9080\" httpsPort=\"9443\"><accessLogging filepath=\"\${server.output.dir}/logs/access.log\" logFormat=\"%h %u %t "%r" %s %b %D %{R}W\" /></httpEndpoint></server>' > /config/configDropins/overrides/server.xml && (/opt/ol/wlp/bin/server run defaultServer &> /logs/console.log &) && sleep 2 && tail -99999f /logs/messages.log | grep -q CWWKF0011I && curl -sv --trace-time http://localhost:9080/helloworldjsp/helloworld && cat /opt/ol/wlp/output/defaultServer/logs/access.log && /opt/ol/wlp/bin/server stop defaultServer"
podman run --rm icr.io/appcafe/open-liberty:full-java8-ibmjava-ubi sh -c "curl -L https://github.com/IBM/helloworldjsp/releases/download/0.1.20240212/helloworldjsp.war -so /config/dropins/helloworldjsp.war && echo '<?xml version=\"1.0\" encoding=\"UTF-8\"?><server><httpEndpoint id=\"defaultHttpEndpoint\" host=\"*\" httpPort=\"9080\" httpsPort=\"9443\"><accessLogging filepath=\"\${server.output.dir}/logs/access.log\" logFormat=\"%h %u %t "%r" %s %b %D %{R}W "%{Host}i" "%{User-Agent}i"\" /></httpEndpoint></server>' > /config/configDropins/overrides/server.xml && printf 'com.ibm.ws.logging.console.format=json\ncom.ibm.ws.logging.console.log.level=info\ncom.ibm.ws.logging.console.source=message,trace,accessLog,ffdc,audit\ncom.ibm.ws.logging.json.access.log.fields=logFormat\n' > /config/bootstrap.properties && (/opt/ol/wlp/bin/server run defaultServer &> /logs/console.log &) && sleep 2 && tail -99999f /logs/messages.log | grep -q CWWKF0011I && curl -sv --trace-time http://localhost:9080/helloworldjsp/helloworld &>/dev/null && cat /opt/ol/wlp/output/defaultServer/logs/access.log && /opt/ol/wlp/bin/server stop defaultServer &>/dev/null && echo '* Access log:' && cat /opt/ol/wlp/output/defaultServer/logs/access.log && echo '* JSON access log:' && grep liberty_accesslog /logs/console.log"

Log Analysis Dashboards

Liberty provides a Grafana dashboard example for mpMetrics (source), Kibana dashboard example, Splunk OCP dashboard example, and mpFaultTolerance dashboard.

Containers

Examples

Splash-Only

Running a simple WebSphere Liberty container:

docker run --rm -p 80:9080 -p 443:9443 open-liberty:latest

Startup is complete when you see:

[AUDIT   ] CWWKF0011I: The server defaultServer is ready to run a smarter planet.

Access the default splash page at http://localhost/ or https://localhost/

Simple Web Application File

Mount the absolute path to a .war or .ear file from the host to /config/dropins/. For example, download https://raw.githubusercontent.com/kgibm/java_web_hello_world/master/builds/java_web_hello_world.ear and run:

docker run --rm -p 80:9080 -p 443:9443 -v $(pwd)/java_web_hello_world.ear:/config/dropins/java_web_hello_world.ear open-liberty:latest

The default context root is the basename of the .war or .ear file unless a specific context root has been configured. In this example, the context root is explicitly configured within the ear to / so access this example at http://localhost/

Adding server.xml Configuration

The base server.xml configuration may be specified with .xml files in /config/configDropins/defaults/

Overriding server.xml configuration may be specific with .xml files in /config/configDropins/overrides/

"The configuration that is specified in the configDropins/overrides directory takes precedence over the configuration in the server.xml file. Configuration specified in server.xml file takes precedence over configuration that is specified in the configDropins/defaults directory." https://www.ibm.com/support/knowledgecenter/en/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_setup_dropins.html