Message Driven Beans (MDBs)

Activation Specifications versus Listener Ports

You can choose Activation Specifications or Listener Ports for handling MDBs. In general, we recommend using Activation Specifications because Listener Ports are stabilized:

• Listener Ports follow the principle of the Application-Service-Facility (ASF) part of the JMS specification and thus they poll for messages from the destinations. Activation Specifications on the other hand work on the principle of callbacks and notifications, so there is no polling involved.
• Listener Ports are not portable, whereas Activation Specifications are portable across JEE environments.
• Activation Specifications are better in performance due to the nature of callbacks: the moment a message is available on the destination the message provider notifies the consumers. For Listener Ports, the thread has to constantly poll for messages. Once a message is found, it has to spawn a new thread, pass the message reference to the other thread so that the other thread can actually do a get.

Sizing Thread Pools for Message Driven Beans

Whether you're using activation specifications or listener ports, tuning the relevant thread pools is a key aspect of MDB performance. Thread pools are configured at a server level, while the number and concurrency of the MDBs is configured independently. Therefore, if your maximum thread pool size is too small, messages may queue unnecessarily. Below are equations for the most common setups which define how to setup the relevant thread pool maximums to avoid queuing. The x=1..n items are all the items of that type processing MDBs for that server (which may be configured at a higher scope). The thread pool maximum size is not the only variable -- the JVM heap, OS resources, etc. also have to be able to support the configured concurrency.

Service Integration Bus (SIB):

WebSphere MQ Messaging Provider:

Listener Ports:

Activation Specifications

MDB concurrency is the primary tuning variable, along with the thread pool on which MDBs execute:

• SIBus MDB concurrency set with: Maximum Concurrent MDB invocations per endpoint. Updates to this value require a restart of the messaging cluster.
• WebSphere MQ Messaging Provider concurrency set with: Advanced Properties } Maximum Server Sessions or maxConcurrency

The batch size is how many messages are queued up in front of each MDB.

Pausing and Resuming Activation Specifications

Pause an Activation Specification using wsadmin -lang jython:

AdminControl.invoke(AdminControl.queryNames("*:type=J2CMessageEndpoint,ActivationSpec=jms/testactivationspec1,*"), "pause")
J2CA0524I: The Message Endpoint ... is deactivated.

Resume an Activation Specification using wsadmin -lang jython:

AdminControl.invoke(AdminControl.queryNames("*:type=J2CMessageEndpoint,ActivationSpec=jms/testactivationspec1,*"), "resume")
J2CA0523I: The Message Endpoint ... is activated.

Get status of an Activation Specification using wsadmin -lang jython:

GetStatus: AdminControl.invoke(AdminControl.queryNames("*:type=J2CMessageEndpoint,ActivationSpec=jms/testactivationspec1,*"), "getStatus")
1 (Active), 2 (Inactive), 3 (Stopped).

Service Integration Bus (SIB)

The Service Integration Bus is a pure Java JMS provider built into WAS:

• Bus: Group of messaging engines
• Bus Member: Hosts messaging engine
• Messaging Engine (ME): Handles destinations (queues, topics), connections, and messages
• ME Cluster Policy:
  • High availability: ME(s) will failover to other available cluster members
  • Scalability: Each cluster member runs an ME
  • Both: Each ME may failover to one other cluster member

SIB Thread Pools

JMS messages for Activation Specifications are processed on the SIBJMSRAThreadPool thread pool. Therefore, the sum of the maximum concurrent invocations per endpoint for all Activation Specifications should be less than or equal to the maximum size of the SIBJMSRAThreadPool thread pool.

JMS messages for Listener Ports are processed on the MessageListenerThreadPool thread pool. Therefore, the sum of the maximum sessions for all listener ports should be less than or equal to the maximum size of the MessageListenerThreadPool thread pool.

Network communication is received by the JS-ReceiveListenerDispatcher thread pool. Its maximum size is controlled by com.ibm.ws.sib.jfapchannel.MAX_CONCURRENT_DISPATCHES and defaults to 32.

Incoming messages are received by the SIBFAPInboundThreadPool threads. If such messages need to be persisted to a database in a messaging engine, that is done by the sib.PersistentDispatcher thread pool whose maximum size is controlled by sib.msgstore.jdbcWriteThreads.

General SIB Tuning Tips

• Tune the maximum concurrent endpoints and maximum batch sizes.
• Consider tuning the memory buffers
• From Best Practices for Large WebSphere Application Server Topologies

  There are several factors that affect SIBus performance. The more destinations there are hosted on a messaging engine, the longer it takes for the engine to start... If the same number of destinations are apportioned over more than one bus member, the startup time improves considerably. However, the drawback is that there are more network connections between the bus members, more overall resource usage, and that the configuration becomes more complex.

  If you have many disjoint destinations in your bus being used by different applications, consider creating different buses.

  You must tune the environment so that messages are consumed at a rate slightly higher than the rate that they are produced. If the producer produces messages at a faster rate, the messages will overload the bus.

SIB Configuration

SIB properties set in the administrative console take precedence over properties set in the sib.properties file.

Consider increasing various data buffer sizes.

If you are using durable subscriptions, explicitly set the activation specification configuration in the Administrative Console within the cluster that hosts the durable subscription home ME to Target type = Messaging engine name, Target Significance = Required, and Target = Durable subscription home messaging engine. Otherwise, remote GETs may occur in some situations (particularly failover) and they are pointless overhead (both CPU and memory) for durable subscriptions.

On z/OS, the control region adjunct (CRA) address space runs SIBus messaging engines and the MDBs run in the servants.

Message Reliability

The SIBus provides five different levels of reliability.

• Best effort non-persistent
• Express non-persistent
• Reliable non-persistent
• Reliable persistent
• Assured persistent

Persistent messages are always stored to some form of persistent data store, while non-persistent messages are generally stored in volatile memory. There is a trade-off between reliability of message delivery and the speed with which messages are delivered. The further the reliability level decreases, the faster messages can be processed.

Non-persistent message reliability may cause a significant throughput improvement (in one benchmark, 29%).

If you are using mediations and not using assured persistent messages, consider skipping the well formed check.

Message Store

Message store type:

• Local Derby database data store: With this option, a local, in-process Derby database is used to store the operational information and messages associated with the messaging engine. Although convenient for development purposes, this configuration uses valuable cycles and memory within the application server to manage the stored messages.
• File-based data store: (default) If the message engine is configured to use a file-based data store, operating information and messages are persisted to the file system instead of a database. This performs faster than the local Derby database and, when a fast disk such as a redundant array of independent disks (RAID) is used, can perform just as fast as a remote database. The test results shown below did not use a RAID device for the file-based data store and do not reflect this additional improvement.
• Remote database data store: In this configuration, a database residing on a remote system is configured to act as the message engine data store. This frees up cycles for the application server JVM process that were previously used to manage the Derby database or file-based stores, enabling a more performant, production level database server (such as IBM DB2 Enterprise Server) to be used. One technical advantage of using a database for the data store is that some J2EE applications can share JDBC connections to benefit from one-phase commit optimization. For more information see information on sharing connections to benefit from one-phase commit optimization. File store does not support this optimization.

Using a remote data store may cause a significant throughput improvement (in one benchmark, 55%).

Database Message Store

IBM DB2 tuning: "To get the best performance from messages in the 3 KB to 20 KB range, you should consider putting the SIBnnn tables into a tablespace with 32 KB pages and adjusting the column width of the VARCHAR column to 32032 bytes."

Data Tables

If statistics suggest a concurrency bottleneck on the SIBnnn tables for a data store, you might try to solve the problem by increasing the number of tables. Example wsadmin code to perform this:

engines = AdminTask.listSIBEngines('[-bus BUS_NAME]')
datastore = AdminConfig.list('SIBDatastore', engines)
AdminConfig.modify(datastore, [['permanentTables', '10'], ['temporaryTables', '10']])
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Relevant points:

1. Every destination (queue/topic) is represented as multiple streams internally based on various conditions. The permanent table will be allocated to different streams of various destinations during the creation of the messaging engine. So, if there are multiple permanent tables configured then there is a mechanism to allocate different tables to multiple streams and the information about the streams and table allocation would be stored in the SIB000 table.
2. Once the permanent table ID is set on a particular stream, it would use the same table to put all the data which uses that particular stream. So in a scenario where only a single destination is used by the application to put all the messages and all the messages have same priority and reliability, then only a single stream would be used and hence all the messages would be put to a single table.
3. However, if there are messages of different priorities and there are multiple destinations and the messages are being added concurrently, then multiple streams would be selected and the messages would be written to the SIB table depending on the table allocated to that stream.

If all of the messages are put to the same destination and have same priority and reliability set, then they are written to the same table. One simple way to check if multiple tables are indeed selected in your environment is to review the SIB000 table and look for the permanentId or permanent_id column.

If the messages/notifications produced are put to the same destination/queue in SIB and have the same priority and reliability, then all of these would go to only one of the permanent table. Currently there is no provision to distribute this evenly across all the tables. Since the decision to use a particular SIB table is made during queue creation, you can run some tests by deleting and recreating some of the queues and observe the behaviour.

Database Lock

After SIB has started, it checks the database to ensure a lock every 20 seconds. If this check fails due to a fatal database exception and jdbcFailoverOnDBConnectionLoss = true (default), then this will lead to the JVM shutting itself down (through an HAManager panic) to force a SIB failover to another JVM. If jdbcFailoverOnDBConnectionLoss = false, SIB will continue trying to get the lock every 20 seconds (during which there is a potential for data loss). If another highly available cluster member is running, the high availability manager will automatically start the messaging engine on another running server. During SIB startup, the properties jdbcInitialDatasourceWaitTimeout (default 15 minutes) and jdbcStaleConnectionRetryDelay (default 2 seconds) are used to retry errors during startup.

File Message Store

The file store log directory can be specified during the creation of an SIBus member using the -logDirectory option in the AdminTask.addSIBusMember command or via the administration console SIBus Member creation panels. This should be on fast disks.

Monitoring

There are various SIB wsadmin monitoring scripts.

The CWSID0016I message indicates the state of messaging engines. For example:

000000fe SibMessage    I   [:] CWSID0016I: Messaging engine ${NAME} is in state Starting.
000000fe SibMessage    I   [:] CWSID0016I: Messaging engine ${NAME} is in state Joining.
000000fe SibMessage    I   [:] CWSID0016I: Messaging engine ${NAME} is in state Joined.
000000fe SibMessage    I   [:] CWSID0016I: Messaging engine ${NAME} is in state Started.
000000fe SibMessage    I   [:] CWSID0016I: Messaging engine ${NAME} is in state Stopping.
000000fe SibMessage    I   [:] CWSID0016I: Messaging engine ${NAME} is in state Stopped.

Useful PMI Statistics

• Thread Pools
  • ActiveCount
• Enterprise Beans
  • MessageBackoutCount: The number of backed out messages that failed to be delivered to the onMessage method of the bean (applies to: message-driven beans).
  • MessageCount: The number of messages delivered to the onMessage method of the bean (applies to: message-driven beans).
  • MethodResponseTime: The average response time in milliseconds on the remote methods of the bean.
  • ActiveMethodCount: Average concurrently actively called methods.
  • MethodCallCount
• SIB Service } SIB Messaging Engines } * } Destinations } Queues
  • AvailableMessageCount: Number of messages available for consumption from this queue
  • AggregateMessageWaitTime: Total amount of time spent in the bus by messages consumed from this queue
  • UnavailableMessageCount: Number of messages on this queue but not available for consumption
  • TotalMessagesProducedCount: Total number of messages produced to this queue
  • LocalConsumerCount: Number of local consumers currently attached to this queue
  • LocalProducerCount: Number of local producers currently attached to this queue
  • LocalMessageWaitTime: Total amount of time spent on this queue by messages consumed from this queue
  • TotalMessagesConsumedCount: Total number of messages consumed from this queue
• SIB Service } SIB Messaging Engines } * } Storage Management } Data Store
  • JDBCTransactionTime: Total execution time of internal batches
  • PersistentDispatcherAvoidanceCount: Measures the number of operations on reliable persistent data dispatched for writing to the data store but whose writing was subsequently unnecessary.
• SIB Service } SIB Messaging Engines } * } Storage Management } File Store
  • FileStoreLogSpace: Space in bytes left in the file store log
  • FileStorePermanentObjectStoreSpace: Space in bytes left in the file store permanent store
• SIB Service } SIB Communications } Messaging Engines } Standard
  • MEStats.BufferedReadBytes
  • MEStats.BufferedWriteBytes
  • MEStats.MessageBytesRead
  • MEStats.MessageBytesWritten
• JCA Connection Factory
  • PoolSize
  • FreePoolSize
  • UseTime
  • WaitTime

Message Visibility/Message Gathering

Message visibility/message gathering may be used to consume messages from all available queue points of a destination. This may be useful when cluster members have different configurations or processing speeds; however, message visibility itself has a high performance overhead. In general, using a single, highly available, clustered destination will probably perform better in the case of differing node performance.

Read ahead

Read ahead will preemptively mark messages so that they can be flowed down a client connection and made available for a consumer to receive before explicitly requested. The default behaviour is for this to be used for non-durable and unshared durable subscribers (where there can only be a single consumer receiving a subscription message at any one time). The reason for this is that in both these cases, once messages are assigned for a particular subscription, there should only be one consumer at a time that can attempt to receive those messages. So, if messages are marked, but for some reason the consumer doesn't receive them (for example, either because work is rolled back or the consumer is closed before receiving all the messages) then those messages can be unmarked on the destination without affecting any other consumers. While read ahead can provide some performance benefit in simple scenarios, it can be disabled if problems arise in more complicated scenarios.

Administrative Console Monitoring

In recent versions of WAS, the administrative console provides basic monitoring on the Runtime tab of the Messaging Engine.

Publication point queue depth:

Subscription queue depth:

Service Integration Bus Destination Handler

The IBM Service Integration Bus Destination Handler tool is a tool that can view, move, copy, delete and restore messages.

Service Integration Bus Explorer

SIB Explorer is a tool to monitor SIB:

Service Integration Bus Performance

SIB Performance is a tool to monitor SIBus performance.

Advanced ME Tuning

Consider various performance tuning properties either through custom properties or the sib.properties file. Setting properties requires restarting the MEs. Potential values worth testing:

• sib.msgstore.cachedDataBufferSize=80000000
  • Default size is 40000000. With a larger cache, it is more likely a message can be delivered from memory, instead of being read from the persistence store.
• sib.msgstore.jdbcWriteMaxBytesPerBatch=8000000
• sib.msgstore.jdbcWriteMaxTasksPerBatch=128
• sib.msgstore.jdbcWriteThreads=16
• sib.msgstore.transactionSendLimit=200
• sib.msgstore.jdbcWriteRate=160

Lightweight Tracing

To investigate activation specification maxConcurrency (search for maxWorkCount in the client):

• Client: *=info:com.ibm.ws.sib.ra.inbound.impl.SibRaSingleProcessListener=all:com.ibm.ws.sib.ra.inbound.impl.SibRaDispatcher=all
• ME: *=info:com.ibm.ws.sib.comms.server.clientsupport.CATAsynchConsumer=all:com.ibm.ws.sib.comms.server.clientsupport.StaticCATConnection=all:com.ibm.ws.sib.processor.impl.ConsumerDispatcher=all

WebSphere MQ Messaging Provider

JMS messages are processed on the WMQJCAResourceAdapter thread pool. Therefore, the sum of the maximum concurrent invocations per endpoint for all Activation Specifications should be less than or equal to the maximum size of the WMQJCAResourceAdapter thread pool.

MDB Response Times

Request Metrics (covered earlier) can be used to track the response times of individual MDB transactions:

1. Ensure "Prepare Servers for Request metrics collection" is checked
2. Select "Custom" for "Components to be instrumented" and select "JMS" and any other relevant components
3. Set "Trace level" to "Hops"
4. Check "Standard Logs"

Simply save and synchronize the changes and request metrics is dynamically enabled.

Here is an example where a servlet calls a stateless session bean which puts a message on a queue. Then an Activation Specification reads from this service integration bus queue to parse the message and sleep one second. Here is the output in SystemOut.log:

[4/17/14 8:56:48:531 PDT] 00000028 SystemOut     O TestMDB received message
[4/17/14 8:56:48:529 PDT] 00000021 PmiRmArmWrapp I   PMRM0003I:  parent:ver=1,ip=127.0.0.1,time=1397749842692,pid=21621,reqid=8,event=1 - current:ver=1,ip=127.0.0.1,time=1397749842692,pid=21621,reqid=8,event=1 type=URI detail=/TestAdvancedWeb/Test elapsed=78
[4/17/14 8:56:49:563 PDT] 00000028 PmiRmArmWrapp I   PMRM0003I:  parent:ver=1,ip=127.0.0.1,time=1397749842692,pid=21621,reqid=8,event=1 - current:ver=1,ip=127.0.0.1,time=1397749842692,pid=21621,reqid=4097,event=1 type=JMS detail=queue1 elapsed=1034

The "type=JMS" line indicates that an MDB has finished processing, the detail field shows the WAS queue name, and the elapsed field shows it took 1034ms. Interestingly, I also had the Servlet component enabled in request metrics, and you can see that the "parent" of the JMS line is the "current" of the servlet line, which means the correlation crosses some boundaries and allows us to know that this message was processed from the same transaction as that particular servlet invocation. This is very useful for tracking asynchronous requests.

JMS Connections Explained

When an application obtains a J2EE Managed connection, which includes JMSConnections obtained from a (JCA managed) connection factory looked up in the J2EE app server's naming context, the J2C connection manager owns/manages the JMS Connection, what the application is given is a wrapper (an instance of the JMSConnectionHandle class which implements the JMS Connection interface). It is this connection handle that is 'closed' by the container/J2C - once it is closed further attempts to use the connection handle (for example to create a session) will fail with javax.jms.IllegalStateException: Connection closed

Once closed the connection handle cannot be re-opened.

When looking in the connection pool what you see are the JMS Connections that the J2C connection manager is managing (technically it actually manages javax.resource.spi.ManagedConnection objects which themselves are wrappers to the actual JMS Connection, and the connection handles). JMS Connections are not closed when the close is called (on the connection handle) but returned to the pool (for unshareable; for shareable, they are available for reuse until returned to the free pool when the transaction context ends).

The handle is closed in compliance with the J2EE connector architecture specification. The close of the handle is not part of the transaction context (JTA or LTC) ending but performed by the container in concert with J2C as part of application component instance lifecyle management as per the JCA specification. While it is absolutely correct that JMS Connections are required by the JMS specification to be thread safe and are non-Transactional, they are still managed connections in a J2EE app server environment.

In other words, managed JMS Connections can be reused under different LTCs; handles to the managed connections are closed, rendering them unusable, as part of the interaction between J2C and the container managing the lifecycle of the application component instance.

It is possible to cache the connection handle inside a stateful session bean; however, passivation would need to be accounted for, as would connection failure handling. This is generally discouraged since J2C is managing connections and it is generally a bad idea for two entities to attempt to manage the same resource - which effectively is what the app would be attempting to do by caching+reusing the connection. It is also worth noting that JMS connections themselves may not map one-to-one with the actual TCP connections and a large number of them may not pose a resource issue; for example, WMQ may multiplex a configurable number of multiple JMS connections and sessions down the same TCP connection though this will be JMS provider specific.

An alternative is to use J2SE JMS. Using this alternative means using a non JCA managed connection factory which will produce non-managed connections and non-managed sessions. Management (caching/reuse/threading/connection failure etc) of the connections/sessions etc is then the sole responsibility of the application. Any work performed against the sessions would not be enlisted with transactions (LTC or JTA) - they would behave just as they would in a J2SE environment.

Listener Ports

MDB concurrency is the primary tuning variable, along with the thread pool on which MDBs execute:

• MDB concurrency set with: Maximum Sessions

Pausing and Resuming Listener Ports

Stop a listener port using wsadmin -lang jython:

AdminControl.invoke(AdminControl.queryNames("*:type=ListenerPort,name=LPNAME,process=server1,*"), "stop")
WMSG0043I: MDB Listener... stopped...

Start a listener port using wsadmin -lang jython:

AdminControl.invoke(AdminControl.queryNames("*:type=ListenerPort,name=LPNAME,process=server1,*"), "start")
WMSG0042I: MDB Listener... started successfully...

Print if a listener port is started or not via wsadmin -lang jython:

AdminControl.getAttribute(AdminControl.queryNames("*:type=ListenerPort,name=LPNAME,process=server1,*"), "getStatus")
Returns true or false

Previous Section (EJBs) | Next Section (Web Services) | Back to Table of Contents