Revision History

This version was built on 2024-11-27T11:09:08.661-0600.

1.0.59 (October 2023)

• Due to time constraints, updates are being made without summaries in this revision history; however, the date above is always updated on a new build.

1.0.58 (January 2023)

• Migrate to local CSS and image resources.

1.0.57 (October 2022)

• Clean up the home page and move the full table of contents into the Full Table of Contents page.

1.0.56 (August 2022)

• Move the Recipes chapter to the top and update the introduction.
• Some of the anchors for the General sub-chapters may be broken.

1.0.55 (February 2021)

• Note that Liberty's JAX-RS client has changed to RESTEasy
• Add Linux iotop batch example
• Describe OOM killer task dump RSS to bytes calculation

1.0.54 (January 2021)

• Add Linux perf recipe
• Add Wireshark Time Sequence Graph (tcptrace) section
• Add Windows pktmon recipes for all ports and a single port
• Add Linux perf probe and bpftrace sections
• Add Linux recipe item to check for CPU cgroup throttling
• Add AIX Example Script to Deny/Allow Packets on a port
• Add notes on AIX DNS and ARP caches
• Add example HotSpot JIT code cache compilation exclusion
• Add OpenShift General Troubleshooting Recipe
• Update Health Center download instructions
• Update GCMV download link

1.0.53 (December 2021)

• Add note that IBMJCEPlus is enabled by default since IBM Java 8.0.7.0
• Revamp the Linux perf section including discussion of call stack walking with --call-graph dwarf and --call-graph lbr, frame pointer omission with J9, and perf and J9 with assembly annotated profiling of JITted code
• Add Linux eBPF section
• Add Linux KUTrace section
• Add WAS traditional Intelligent Management recipe
• Update Memory Analyzer Tool download link
• Add link to HotSpot async-profiler
• Add section on CPU registers such as the stack pointer and the call stack, the frame pointer, and call stack walking and frame pointer omission (FPO)
• Add note about the difference between Intel and AT&T assembly syntax
• Add section about assembly syntax
• Add IBM Operational Decision Manager (ODM) chapter
• Add IBM Business Automation Workflow (BAW) chapter

1.0.52 (November 2021)

• Clarify that IBM Support Assistant and GCMV are supported as-is.
• Add Java J9 in Containers recipe items for -XX:+ClassRelationshipVerifier and issues with tight container memory limits.
• Add note that the Liberty mpMetrics-x.x and microProfile-x.x features implicitly enable monitor-1.0 that has some overhead and some ways to tune this.
• Add Liberty JSF section and performance note on -Dorg.apache.myfaces.INITIALIZE_SKIP_JAR_FACES_CONFIG_SCAN=true
• Add migration performance note about SCC being cleared on fixpack upgrade.
• Highlight new %{remote}p HTTP access logging option in Liberty.
• Add discussion of Forcing Revalidation of HTTP Cached Responses.
• Add discussion of Apache HttpClient Keep-Alive.
• Add jdmpview example of figuring out what caused a core dump
• Add macOS zprint command for kernel memory.
• Add example to print approximate bytes free on macOS.

1.0.51 (July 2021)

• Note that TCP retransmits may also cause the socket to switch into 'slow start' mode which may affect subsequent packet performance/throughput.
• Note that IBM Java Health Center is currently not shipped with nor supported on OpenJ9 and IBM Java 11.
• Created J9 Native OutOfMemoryError Recipe including a link to a !belowthebar jdmpview query that helps with NOOMs
• Add Wireshark section on how to Visualize TCP Response Times.
• Add Wireshark section on how to Visualize HTTP Response Times.
• Update default J9 JIT compilation thread count.
• Note new J9 -XX:+GlobalLockReservation option for AIX and Linux on Power.
• Add z/OS netstat -A example.
• Add Http(s)URLConnection connect and read timeout options.
• Add Windows instructions on getting a core dump on a crash with "EXE has stopped working".
• Add example of how to create a custom jdmpview DDR command.
• Updates associated with OL App Stack Intro changes.

1.0.50 (April 2021)

• Add recipe items for WAS traditional and Liberty to avoid client keepalive socket churn for HTTP/1.0 and HTTP/1.1 by setting maximum keepalive requests per connection to unlimited.
• Add recipe items for WAS traditional and Liberty to increase the keepalive idle timeout for servers with incoming LAN HTTP traffic from clients using persistent TCP connection pools with keep alive (e.g. a reverse proxy like IHS/httpd or web service client).
• Add recipe items for WAS traditional and Liberty to minimize the number of application responses with HTTP codes 400, 402-417, or 500-505 to reduce keepalive socket churn.
• Add recipe item to Liberty to consider using local interface or no-interface equivalents if using non-@Asynchronous remote EJB interfaces in the application for EJBs available within the same JVM to avoid extra processing and thread usage.
• Note virtual and physical memory constraints as another factor for potentially explicitly tuning the Liberty executor thread pool
• Add AIX recipe item on disabling TCP delayed ACKs and monitoring for retransmissions
• Add discussion of AIX network dog threads that may help with network processing bottleneck with the default single-CPU interrupt processing model
• Add discussion of using AIX netstat -v to ensure that network switches are not sending PAUSE frames.
• Add IHS discussion of %X and %k to check incoming connection re-use
• Add Apache HttpClient page and discussions of tuning its connection pool and using a custom user token handler to maximize pool re-use in mutual authentication scenarios.
• Add discussion on determining bottlenecks
• Add WAS traditional Hung Thread Detection Recipe
• Add WebSphere Liberty HTTP Access Log Recipe
• Add AIX nmon recipe
• Add AIX vmstat recipe
• Add AIX perfpmr recipe
• Add AIX iptrace recipe
• Add discussion of the DB2 z/OS type 2 JDBC driver using under-the-2GB-bar native memory which can drive native OutOfMemoryErrors
• Add z/OS IPCS VSMDATA command for reviewing under the 2GB bar native memory usage
• Add Java ODR discussion of its use of the 'Default' thread pool
• Add J9 example of late attach diagnostic logging
• Add note about APAR IJ31667 for J9 -Xgc:classUnloadingKickoffThreshold
• Add discussion of Java Unicode surrogate representation
• Add macOS pstack-equivalent lldb example
• Move POSIX awk section to a dedicated awk page
• Rename IBM MAT page to Eclipse MAT
• Add MAT examples to run in headless mode
• Add odo example on creating an application
• Add discussion of Oracle AWR reports
• Add IBM InfoSphere Master Data Management including discussion of monitoring its ODBC database transaction responses

1.0.49 (March 2021)

• Add Linux instructions to disable delayed ACKs with quickack 1 and add to the Linux recipe
• Add OpenShift Review Logs Recipe
• Add discussion of HotSpot -Xlog:gc option
• Add IHS LogFormat time until first response bytes option with %^FB
• Add Wireshark discussion of sequence and acknoledgment numbers
• Add notes on k8s CPU limits causing throttling
• Add Linux discussion of tmpfs and RAM usage
• Add Linux tcpdump Recipe
• Add Linux tcpdump on a port Recipe
• Add Linux nmon recipe
• Add Linux vmstat Recipe
• Add %{R}W to WAS traditional and Liberty HTTP access log examples
• Add Add WAS traditional HTTP Access Log Recipe
• Add WAS traditional Dynamic Diagnostic Trace Recipe
• Add WAS traditional Diagnostic Trace from Startup Recipe
• Add WAS traditional Dynamic verbosegc Recipe
• Add WAS traditional verbosegc from Startup Recipe
• Add WAS traditional collector recipe
• Add Liberty Docker one liner
• Add discussion about macOS mds_stores high CPU usage
• Add example Java code using ThreadLocal SimpleDateFormat for debug traces
• Add Java example of running hello world in a container
• Update citation of the potential revenue impacts of performance tuning

1.0.48 (March 2021)

• Update links to performance tuning education topics
• Add more details about TCP congestion control and the congestion window. Add TCP buffer tuning item to Linux, AIX, and z/OS recipes. Add Linux slow start on idle tuning recipe item.
• Add Linux kernel symbol table details
• Add various recipes for OpenShift: Use Image Registry, Remote into a Container, Analyze a Pod, Analyze a Node, etc.
• Add note that Liberty doesn't work with Health Center dynamic enablement if some jndi related features are loaded
• Add discussion of Kubernetes CPU and memory resource requests and limits
• Move Troubleshooting Recipes to the top level
• Add WAS traditional notes on tuning database transaction log concurrency
• Add Linux notes on fio and NVMe SSDs
• Update the Dynacache recipe
• Add example code to take J9 dumps
• Add AIX details about compress and split
• Add z/OS discussion of system dumps, IPCS, XL C/C++ compiler, dbx, EzWAS, MCEVS1, and PLPSC
• Add z/OS uncompress and pax examples
• Add WAS traditional PasswordEncoder examples

1.0.47 (January 2021)

• Add warning about possible native OutOfMemoryErrors to Health Center usage and recipes.
• Add discussion about RAM corruption and ECC RAM.
• Add tWAS enableJDBCTiming for tracking long JDBC queries.
• Add OpenShift web console examples.
• Remove HotSpot Java's no-longer-valid option -XX:+HeapDumpOnCtrlBreak.
• Add details on J9 percolate-collect.
• Add J9 jdmpview commands to find min/max heaps.
• Update Linux Kernel Log description.
• Update the AIX recipe to highlight the importance of tuning Virtual Ethernet Adapter buffers and add an item for watching for network PAUSE frames.
• Add WAS Plugin discussion of IgnoreAffinityRequests.
• Add Dynacache flush to disk on stop details.
• Add Linux Pressure Stall Information discussion for CPU and memory
• Add discussion about AWS EFS PercentIOLimit

1.0.46 (November 2020) (14 major updates)

• Add Java J9 recipe item to check if the JIT code cache is full
• Add Apache CXF note about -Dorg.apache.cxf.JDKBugHacks.gcRequestLatency=true to avoid full GCs
• Add -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false to the Java recipes
• Add Java in Containers recipes
• Add Liberty in Containers recipe
• Add WebSphere Application Server traditional in Containers recipe
• Add awk tip to allow processing a lot of files from find using ARGV instead of xargs
• Add note that WAS servlet caching adds a CACHED_RESPONSE header if served from cache
• Add note about the WAS traditional WSVR0652W thread pool size warning
• Add J9 HealthCenter -Dcom.ibm.diagnostics.healthcenter.data.profiling=off option to disable profiling.
• Add Logging Custom PMI Data with Dynacache recipe
• Add note about getting accumulated CPU time of threads from Windows minidump through jdmpview
• Add Linux recipe item to change the CPU speed governors to performance if power consumption isn't a concern.
• Add lots of OpenShift okd examples.

1.0.45 (October 2020) (23 major updates)

• Suggest testing with disabling delayed ACKs to all OSes as generally recommended by John Nagle
• Add Linux EarlyOOM discussion
• Add tips on launching Eclipse
• Add J9 jdmpview examples of finding if a core was taken with exclusive access and which thread has exclusive access
• Note that zWLM with one service class does not overflow into discretionary
• Add Wireshark discussion about TCP streams
• Add details about -Dcom.ibm.CORBA.SocketWriteTimeout and -Dcom.ibm.CORBA.FragmentSize
• Add details on J9 gencon concurrent marking
• Add Troubleshooting macOS page including instructions on creating core dumps
• Add Example C Program that Crashes
• Add tWAS hung thread detection overhead discussion
• Add java.util.logging configuration section
• Add jdmpview commands for dumping byte codes of a method from a core dump
• Add zWAS WLMStatefulSession discussion
• Add Java patching discussion
• Add Java Swing page
• Add Java Enterprise Edition (JEE) page
• Add AIX tar/compress/gzip instructions
• Add Health Center recipes
• Update Windows perfmon instructions
• Add Windows PowerShell and wmic commands including an example of printing pooled paged- and non-paged bytes
• Add example Linux commands to induce network delay with tc
• Add Java options for connect and read timeouts of HttpURLConnection
• Add HotSpot Mission Control recipe

1.0.44 (September 2020) (41 major updates)

• Refresh Liberty and tWAS recipes.
• Add Liberty Web Response Cache discussion.
• Add MAT Dark Matter issue discussion.
• Fix HCL Commerce links.
• Add lots of Linux CPU, network, and disk tools.
• Add discussion of how peak throughput may drop when response times increase.
• Add Wireshark discussion of calculating packet loss and finding SYNs without SYN/ACKs.
• Update IBM Java ORB tuning discussion.
• Add general guidance on tuning timeouts.
• Add Traditional WAS recipe on logging TPV/PMI data.
• Add Linux section on installing kernel debug symbols for various distributions.
• Add general guidance on investigating Java lock contention.

1.0.43 (August 2020) (57 major updates)

• Update DB2 driver discussion of timerLevelForQueryTimeOut.
• Add discussion about Spring JMS cacheLevel=0.
• Add IHS discussion about mod_event and KeepAlive.
• Add discussion about Wireshark and decrypting TLS traffic with SSLKEYLOGFILE.
• Add undocumented HealthCenter com.ibm.java.diagnostics.healthcenter.headless.filename option.
• Update Linux kernel hang script.
• Add Linux basic troubleshooting script.
• Add Linux Sysrq discussion.
• Add note about Linux systemd killing nohupped processes.
• Add Bash command line movement examples.
• Add HealthCenter instructions on getting an HCD from a running process.
• Add discussion of SIB read ahead.
• Add discussion of Docker --pid=host.
• Add zWAS WLM HTTP request distribution discussion.
• Update discussion of WAS Plugin MaxConnections.
• Add link about JMS max batch size.
• Describe J9's native memory allocation header and footer eye catchers.
• Add advanced SIB ME tuning notes.
• Add discussion of HotSpot -XX:+PreserveFramePointer.
• Add example tshark usage including long DNS lookup times.
• Describe how to fix AIX nmon PoolIdle value of 0.
• Add JEE citation to JDBC shareable discussion.
• Add J9 Linux perf-hottest post-processor script.
• Add DB2 discussion of a WLM pressure valve and MALLOCOPTIONS.
• Add AIX Virtual Ethernet Adapter tuning.
• Add DB2 thread dump example.
• Add POSIX script to watch for some output and send a trigger line to a file.
• Add example JMS producer and MDB consumer.

1.0.42 (July 2020)

• Fix missing links to the Troubleshooting Java pages.
• Add section on tuning the Java XML JAXP ServiceLoader classloading.
• Add Maven sections on listing archetypes and creating a simple Liberty app using an archetype.
• Add examples on how to run Java code on application startup.
• Add tWAS CDI tuning options.
• Add awk tips on printing the current file, current line number, and printing to stderr.
• Add section describing differences between tWAS SIB and Liberty embedded messaging.
• Add Liberty section on JakaraEE.
• Add simple description of z/OS CP, zAAP, zIIP, and IFL processors.
• Add OpenShift command for dynamically forwarding a port to a pod.
• Add example Linux systemd Liberty service.

1.0.41 (June 2020)

• Add information on k8s sizing of clusters, masters, and spanning data centers
• Add discussion of tWAS on z/OS servant contraction
• Add tWAS idle tuning considerations paper
• Add macOS docs on XCode Instruments, log, sysdiagnose, Console, and Activity Monitor
• Add Resiliency page
• Add comment about how to deal with tWAS serialized startup
• Add tWAS Startup order option
• Add information on Java -Djavax.net.debug
• Add Liberty HTTP compression element
• Add Liberty consoleFormat=simple option
• Add Liberty startAfter attribute
• Add openssl dump TLS certificate example
• Add discussion about killing runaway threads
• Add discussion about the 'The server has decided to close this client connection.' warning
• Add OpenJ9 Internals Documentation
• Move cloud paks page to section under IBM Cloud
• Add Liberty section on how to investigate OSGi bundle startup times
• Add Liberty section on how to pass configuration
• Add glibc_malloc_info.py to investigate glibc malloc free lists
• Add discussion of SSLUtils.flushCloseDown and -DtimeoutValueInSSLClosingHandshake=0
• Add section on J9 -Xverify:none deprecation and -XX:+ClassRelationshipVerifier alternative

1.0.40 (June 2020)

• Internal format conversion.

1.0.39 (May 2020)

• Add details about Linux strace and ltrace with stacks.

1.0.38 (March 2020)

• Add details about TIME_WAIT processing in Linux.

1.0.37 (February 2020)

• Add discussion about the OpenJDK JCL HTTP(S) Client implementation.

1.0.36 (February 2020)

• Add sub-chapters on WAS/Java/IHS configuration/log analysis/healthcheck

1.0.35 (January 2020)

• Add Linux network analysis commands: iptraf-ng, nethogs, iftop, jnettop, trafshow, and speedtest-cli.

1.0.34 (November 2019)

• Add Linux ip, ss, and nstat commands.

1.0.34 (November 2019)

• Migrate blog content (since developerWorks blogs is being sunset) to various sections of this cookbook.

1.0.33 (October 2019)

• Remove link to developerWorks forum as that is being sunset.

1.0.32 (September 2019)

• Change chapter order.

1.0.31 (August 2019)

• Break up Java chapters into J9 and HotSpot
• Add details about TCP KeepAlive

1.0.30 (April 2019)

• Fix Java RMI Explicit GC default from 1 minute to 1 hour

1.0.29 (February 2019)

• Add Containers chapter

1.0.28 (January 2019)

• Add AIX native memory debug info

1.0.27 (November 2018)

• Add Linux eBPF example for native memory leaks

1.0.26 (November 2018)

• Add Linux lsof

1.0.25 (November 2018)

• Add Linux perf On-CPU stack sampling with wallclock timestamps

1.0.24 (October 2018)

• Add Traditional WAS page to discuss Startup

1.0.23 (October 2018)

• Update GCMV installation instructions

1.0.22 (October 2018)

• Added information on tuning DirectByteBuffer pool sizes for both editions of WAS
• Describe how to request exclusive access for a system dump on IBM Java to avoid dumps taken during GC

1.0.21 (May 2018)

• Added Mac section
• Added Linux Available memory reference

1.0.20 (March 2018)

• Added pureScale section

1.0.19 (January 2018)

• Add WAS traditional instructions on setting up ODR custom logging

1.0.18 (September 2017)

• Update MAT instructions with DTFJ Java 8 note.

1.0.17 (August 2017)

• Update Memory Analyzer download instructions.

1.0.16 (April 2017)

• Change IBM Java -Xverbosegclog recommendation

1.0.15 (April 2017)

• Add Linux swappiness, OOM killer, and swap recommendations

1.0.14 (January 2017)

• Change major tool instructions to use Eclipse instead of ISA

1.0.13 (June 2016)

• Fix broken Knowledge Center links.
• Change WAS Classic to WAS traditional

1.0.12 (February 2016)

• Various updates based on field work.

1.0.11 (December 2015)

• Fix error referencing META-INF/lib instead of META-INF/resources in WAS > WAS Classic > HTTP, in the section "ServletContext.getResource performance"

1.0.10 (December 2015)

• Change graphs from R to gnuplot

1.0.9 (December 2015)

• Update Solaris KSSL guidance.

1.0.8 (December 2015)

• Add more TPV/PMI screenshots.
• Add more Health Center screenshots.
• Add Liberty request timing and event logging details.

1.0.7 (December 2015)

• Update GCMV page.

1.0.6 (September 2015)

• Add troubleshooting recipes.

1.0.5 (August 2015)

• Rename WAS traditional Profile to WAS Classic

1.0.4 (May 2015)

• Remove unused Liberty idle tuning option.

1.0.3 (April 2015)

• Rename Java chapters.

1.0.2 (April 2015)

• Add example Solaris DTrace scripts and vmstat -p information.

1.0.1 (February 2015)

• Add February SPECj benchmark results.

1.0.0 (January 2015)

• First public version.

Previous Section (Other) | Next Section (Notices) | Back to Table of Contents