IBM WebSphere Application Server Performance Cookbook

WebSphere ESB

Processing Large Objects

Ensuring optimum performance is attained on systems processing large objects is an issue commonly faced by users of middle-ware software. In general, objects of 1M or more can be considered to be 'large' and require special attention, please review the following articles for awareness of considerations and tuning / application design advice:

Large Messages dW article: https://www.ibm.com/developerworks/library/ws-largemessaging/

Claim Check Pattern: http://www.ibm.com/developerworks/websphere/library/techarticles/1006_kharlamov/1006_kharlamov.html

Aggregation Design Patterns

There are several application design considerations that should be understood when developing Mediation Flows utilising aggregation design patterns in order to attain optimal performance and avoid unnecessary processing costs. The following article details these design considerations:

Aggregation dW article: http://www.ibm.com/developerworks/websphere/library/techarticles/1111_norris/1111_norris.html

Depending on whether there is a FanIn Mediation Primitive downstream of a FanOut Mediation Primitive alters the logic within the FanOut mediation Primitive. When using a FanOut Mediation Primitive without an associated FanIn Mediation Primitive an array of SMOs is created up-front before the output terminal is first fired. If there is an associated FanIn then each SMO is created as required instead of all ahead of time. If the SMO is large in size or a large number need to be created (for example, iterating on a large array of elements, firing a large number of times, or a large number of branches), then this can have a significant effect on memory overhead. For example, if you have a 1MB input message and you use a FanOut to iterate over an element that repeats 1000 times, transforms the message and passes on to a JMS queue (without a FanIn), then before the first output terminal fire on the FanOut, 1000 SMOs will be created each of ~1MB in size which would mean you would have a 1GB array allocated to the JVM Heap. You need to be aware of this behaviour when creating your application and tuning the size of the JVM Heap and application threadpools.

Asynchronous Invocation of Synchronous Services Design Patterns

In general, synchronous service invocations are recommended, because they have less processing overhead and provide better performance. In some cases however, asynchronous invocations can reduce the overall response time of the application and are preferred, such as in the simultaneous invocation of multiple long-running services. When invoking a synchronous service asynchronously, however, additional processing is incurred in the messaging layer of the product that needs to be understood and tuned appropriately. The following article details these considerations and processing logic:

Parallel Invocation of Synchronous Services dW article: http://www.ibm.com/developerworks/websphere/library/techarticles/1312_ross/1312_ross.html

NB. You do not need to switch to asynchronous invocations for JAX-WS Import Bindings to impose specific time-out settings - this can be done by applying an appropriate HTTP Policy configurable through the Administration Console (and can be exported to be brought into the development environment) which does not incur the overhead in the messaging layer of the product - as with the switch to asynchronous invocations.. There are some small caveats that need to be considered / understood - please see the "Defining a Timeout on Synchronous JAX-WS Imports" section below.

Shared Libraries

The default setting for libraries is share by copy - this means that each Mediation Module referencing a particular library retains its own copy, which can result in bloated and redundant memory usage. You may need to consider shared libraries as detailed in the following technote:

http://www-01.ibm.com/support/docview.wss?rs=2307&uid=swg21322617

Shared libraries can also benefit run-time performance through reduced serialisation in addition to reducing overall memory footprint - for instance in Lazy Parsing applications employing SCA Binding componentisation.

Parsing Modes

Don't mix parsing modes within a deployment. Moving between a Lazy Parsing module an Eager Parsing configured module through SCA Bindings causes increased overhead in processing costs that should be avoided.

Some scenarios will perform better in Eager Parsing mode (lightweight scenarios with small payloads), however, mediation modules which are more complex, or are processing larger payload workloads will typically benefit from Lazy Parsing and can exhibit significant performance improvements (dependant on application design).

Memory Analyzer Plugin

IBM Extensions for Memory Analyzer for WebSphere ESB is an extension for IBM Monitoring and Diagnostic Tools for Java™ -- Memory Analyzer, augmenting data structures and providing reports specific to WebSphere ESB. It significantly improves the effectiveness and efficiency of problem diagnosis and resolution, and provides a deeper understanding of your WebSphere ESB deployment. The following article and WebSphere Technical Exchange show you how to use the IBM Extensions for Memory Analyzer for WebSphere ESB to analyze operating system level dumps or portable heap dumps from a WebSphere Enterprise Service Bus solution:

IBM Extensions for Memory Analyzer for WebSphere ESB: http://www.ibm.com/developerworks/websphere/library/techarticles/1206_ross/1206_ross.html

WebSphere Technical Exchange on Memory Analyzer Plugin and APARs associated with memory management: http://www-01.ibm.com/support/docview.wss?uid=swg27036363

Comparative Transformation Technologies (XSLT vs. Business Object Map)

XSLT Mediation Primitives are designed for applications that have .XSL currently or that want to utilise specific XSLT function.

Business Object Mapper Primitives are designed for improved performance, but may require specific function to be coded manually within the map.

Business Object Maps have some reduced function out of the box, but much can be implemented in simple custom Java utilising the BO API. They provide improved performance, especially for larger message payloads, as they work at the API so do not need to be passed through a transformation engine which will produce bytes causing additional SMO construction and serialization / de-serialization costs.

In IBM Integration Designer V8.0 a new mediation primitive has been introduced to enable the developer to switch between the targeted run-time transformation technologies through a simple combo-box - previously a complete re-write of the transformation would be required within the new primitive if a customer wanted to switch between technologies for improved performance.

First Messing Response Time

The performance, specifically regarding response time, of the first message entering the system is often of high importance. Typically there is a trade off between artefact initialisation costs being associated with server start-up or first message processing. There are several techniques and product features that can be utilised to improve and control the system when response times are critical for first message processing.

Synthetic Messages

First message response times can be improved by priming the Mediation Module with a synthetic message:

Synthetic messages TechNote: http://www-01.ibm.com/support/docview.wss?uid=swg21589355

This may require the Mediation Module / Components to have an additional "no op" operation or flow path to process the synthetic message without affecting downstream systems, but will result in the vast majority of initialisation costs to have been met prior to the first "production" message entering the system.

XPath and XSL Pre-compilation

Pre-compilation of XSL and XPath was introduced in V7.5, these artefacts are now compiled at deploy time rather than on first message for Lazy Parsing Modules (the transformation engine usitilised for Eager Parsing does not have such a concept). This can provide substantial improvements to first message processing time for Lazy Parsing Mediation Flows. The improvement factor is dependent on the number of XSLT Mediation Primitives and XPath statements in the intitial path through the Mediation Flow, and the complexity of the XSL / XPath.

Pre-loading of Mediation Flows

The option to load Mediation Modules and associated artefacts and resources at server start up, opposed to on first message, was introduced in V7.5.1. A property was exposed that enables the user to use wildcards to select appropriate Mediation Modules and Components and define how many instances of the artefacts to load into the runtime:

http://www.ibm.com/support/knowledgecenter/en/SS7J6S_7.5.1/com.ibm.websphere.wesb.z.administering.doc/topics/cadm_medflows.html

Test cases used to evaluate this feature show that >50% improvement can be achieved in initial message processing times - although this is dependent on a number of factors, including the number of components in a project and the complexity of the message definitions. This not only builds the Message Flows but also many of the objects required to model the message structures during server start-up and applies to both Eager and Lazy Parsing Modules.

Associated APARS

Several APARs may be required relating to pre-compilation of XSL / XPath and pre-loading of Mediation Flows:

IC96060: EXTRANEOUS OR MISLEADING ERROR MESSAGES DURING MEDIATION FLOW PRE-LOADING
IC96845: MULTIPLE PROBLEMS CACHING XSL MAPS RESULTING IN SLOW RESPONSE TIMES AND UPDATES NOT BEING PICKED UP AFTER MODULE RESTART
IC95917: CACHE PRECOMPILED STYLESHEETS PER CONTEXT CLASSLOADER (http://www-01.ibm.com/support/docview.wss?uid=swg1IC95917)
IC96799: NULLPOINTEREXCEPTION DURING SERVER STARTUP WHEN PRELOAD VARIABLE IS SET (http://www-01.ibm.com/support/docview.wss?uid=swg1IC96799)
IC91519: POOR PERFORMANCE/ EXCESSIVE MEMORY USE OF BO MAPPINGS IN BPEL PROCESSES, OR WHEN MAPPINGS ARE APPLIED IN CUSTOM CODE (http://www-01.ibm.com/support/docview.wss?uid=swg1IC91519)

Restricting the Instances of Mediation Flows on the JVM Heap

For WebSpere ESB a Mediation Flow object is required for each concurrent thread executing a unique flow / operation in a Mediation Module. Due to product changes introduced in V7.0 the concepts differ depending on the version of the run-time and the version of the development environment used to generate the run-time artefacts.

V6 Run-time / Applications

EAR files generated prior to V7.0 utilise the EJB Container, whether they are deployed to a V6 or V7 run-time. Each Mediation Module "Application" is represented by a stateless session EJB - the number of EJBs created is controlled as follows:

1.Transport threadpool: Controls maximum concurrency in the system (ie. WebContainer threadpool)
2.Application EJB threadpool (default min=50, max=500): Each Application will create up to the maximum defined number of EJBs in a module-specific pool

If the min value for an EJB pool is set lower then we might free up memory as the pool contracts. The following APAR may be required:

http://www-01.ibm.com/support/docview.wss?uid=swg1IC76728.

V7 Run-time / Applications

With the exception of EAR files generated prior to V7.0 (but deployed to a V7 run-time) the number of Mediation Flows on the JVM Heap is controlled as follows:
1.Transport threadpool: Controls maximum concurrency in the system (ie. WebContainer threadpool)
2.JVM Managed: Weak / Soft references will clean up unused resources

The references that keep the Mediation Flow objects alive on the JVM Heap have been modified in V7 onwards to enable clean-up to occur when the JVM Heap is under stress. The following APARs may be required:

IC94803: ALLOW FOR GARBAGE COLLECTION OF CERTAIN REFERENCES (http://www-01.ibm.com/support/docview.wss?uid=swg1IC94803)

IC82189: ENABLE MEDIATION FLOWS TO BE GCD WHEN HEAP IS UNDER STRESS (http://www-01.ibm.com/support/docview.wss?uid=swg1IC82189)

Throttling Individual Applications

Often it is required to "throttle" individual applications that may be having an adverse effect on the system (to limit memory usage, or CPU consumption for instance). The concepts and methods differ depending on the version of the run-time and the version of the development environment used to generate the run-time artefacts.

You can throttle applications by tuning / restricting the appropriate threadpools on which they run. This typically has a global impact as many applications may be running on the same threadpool, however, it is possible to isolate applications (or groups of applications) to specific threadpools by creating new transport chains through which to invoke them.

First create a new threadpool, in the administrative console click "Servers > Server Types > WebSphere application servers > server_name > Thread pools", then click "New" and fill in the required details. Next create a new transport chain as detailed in the following article:

http://www-01.ibm.com/support/knowledgecenter/SSAW57_7.0.0/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/trun_chain_transport.html

The next step is to configure the transport chain that has just been created. In the administrative console navigate to the newly created transport chain, click TCP inbound channel and modify the Thread Pool setting to use your new threadpool.

NB. If you create a new web container transport chain, the initial value for the writeBufferSize attribute is 8192, which is too small for most web container transport chains. Navigate to the newly create transport chain, click Web container inbound channel, and specify 32768 (or appropriate value) in the Write buffer size field.

You may also need to configure the appropriate virtual host as described in the following article:

http://www-01.ibm.com/support/knowledgecenter/SSAW57_7.0.0/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/tcws_plugin_vhost.html

For you application to run on the new threadpool you must invoke it through the appropriate transport chain by using the port you specified on its creation.

V6 Generated Applications

EAR files generated prior to V7.0 utilise the EJB Container, whether they are deployed to a V6 or V7 run-time. Every EJB in WebSphere Application Server has an associated pool with default min, max value of 50,100.

These can be overridden globally or modified on a per Application basis by specifying a system property as follows:

-Dcom.ibm.websphere.ejbcontainer.poolSize=<J2EE-bean-name>=min,max

The J2EE-bean-name is formed by concatenating the application name (without the file extension), the module name (without the file extension) and the name of the bean ("ejb-name" as defined in the bean's deployment descriptor), using a # separator. For example, if you have an application called SMApp.ear that includes module PerfModule.jar, and module PerfModule.jar uses a bean named TunerBean, the J2EE name of the bean is specified as SMApp#PerfModule#TunerBean.

If the property is set correctly you should see a line similar to the following output in the system log on first invocation of an operation in the module:

[24/05/11 15:28:02:444 BST] 00000025 EJBMDOrchestr I CNTR0060I: (Min,Max) pool size is (5,100) for bean com.ibm.wsspi.sibx.mediation.flow.ejb.MediationFlowBean

Unfortunately, every WESB module will output a message with the same class name but the pool values will apply to individual beans.

For verification, a trace string of com.ibm.ejs.container.BeanMetaData=all will output the details of every bean on first invocation including the correct J2EE name needed above and the current pool settings for the EJB.

Reducing the min value of an EJB pool will mean that during quiet spells for a particular application (Mediation Module) the pool will be shrunk down to that minimum value and any associated mediation flow objects will be eligible for GC. The EJB pool is shrunk back down (in increments) to the minimum size after the pool has been inactive for a certain period of time. This can be configured from the admin console at "Application servers > server1 > EJB container", the setting is labelled "Inactive pool cleanup interval" and defaults to 30 seconds.

Defining a Timeout on Synchronous JAX-WS Imports

Synchronous JAX-WS Bindings do not offer a simple setting to modify the default time-out options, as is available with the asynchronous invocation. However, switching to asynchronous invocations introduces unnecessary overhead that can affect application performance. If you need to set binding / application specific time-out values for a synchronous JAX-WS invocation then this can be achieved by applying an appropriate policy set, and does not incur any additional overheads. To achieve this follow these steps:

1.Create a new policy set in the Administrative Console. Click Services > Policy Sets > Application policy sets > New
2.Add an HTTP Transport policy and configure the time-out values appropriately
3.Save and export the policy set from the Administrative Console
4.Import the policy set into the development environment (Import > WebServices > WebSphere Policy Sets)
5.Attach the policy set to the Import Binding (Properties > Binding > Policy Sets > Default policy set)

It should be noted that if a time-out occurs the exception propagated back to WebSphere ESB is not a modelled fault, thus the failure message is propagated to the fail terminal (the timeout terminal is just for handling time-outs for asynchronous invocations). The SMO failinfo section will appear as follows:

<failInfo>
<failureString>javax.xml.ws.WebServiceException: java.net.SocketTimeoutException: Async operation timed out</failureString> <origin>External Service</origin> </failInfo>

The reference to "Async operation timed out" just refers to the fact that it is using the Java Async IO API, nothing to do with the existing asynchronous SCA model.

Best Practices and Tuning Red Papers