Connection pool hangs in createOrWaitForConnection
A thread waiting in
com/ibm/ejs/j2c/FreePool.createOrWaitForConnection is
waiting to get a connection from a connection pool. This may mean either
that there are more threads than the maximum connection pool size (and a
subset of them is using all connections in the pool), or, less likely,
that one or more threads are holding more than one connection per
thread.
More threads than the maximum connection pool size
Normally, connections are requested from managed thread pools such as WebContainer, ORB.thread.pool, WMQJCAResourceAdapter, etc. If the sum of the maximum sizes of the thread pools that may use connections is less than the maximum size of the connection pool, then, with sufficient concurrent workload, or if there is a backup in the backend resource (database, queue, etc.), then this can become a bottleneck.
A common solution is to either:
- Increase the maximum connection pool size to sum(max(connection-using thread pool size))+1 or,
- Reduce the maximum thread pool sizes so that their sum is less than or equal to the maximum connection pool size + 1.
Option 1 may put unexpected stress on the backend resource (database, queue, etc.) and/or unexpected stress (e.g. CPU, Java heap, etc.) on the application server because now more concurrent work can run. Option 2 may increase response times due to queuing.
Therefore, there is no easy answer as this involves a tuning exercise.
Note: the reason for +1 in the formulas above is to avoid the case of a deadlock due to threads requesting more than one connection (though this can still be a livelock; see next section).
Threads holding more than one connection per thread
It's common that one thread should only use one concurrent connection from a connection pool, though there may be cases where an application may legitimately use multiple concurrent connections from the pool, so it depends on the use case.
In either case (as temporary relief, or as expected behavior, respectively), the connection pool maximum may be set to a multiple of sum(max(connection-using thread pool size))+1 with the same caveats above about additional stress on the backend resource and/or application server.
Connection leaks
Connection leaks are normally not a problem because Java enterprise edition managed resources default to shareable connections which means that the application server will close the connection at the end of the transaction containment even if the application forgot to call close.
Connection pool maximum less than thread pools' maximums
A connection pool maximum may be consciously set to less than the sum of the thread pools' maximums. This may be done because the backend resource (e.g. database, queue, etc.) has limited capacity; however, it's generally recommended to queue work in front of the application server rather than within it:
In general, requests wait in the network in front of the web server, rather than waiting in WebSphere Application Server. This configuration only supports those requests that are ready for processing to enter the queuing network.
Setting the thread pool some amount larger may be acceptable so that queued work is available immediately as connections are freed:
Because there is work waiting to enter a component at each point upstream, no component in this system must wait for work to arrive. The bulk of the requests wait in the network, outside of WebSphere Application Server.