Dynamic Cache (Dynacache)
Dynacache Recipe
- If using memory-to-memory HTTP session replication, weigh whether the costs and complexity are better than simple sticky sessions with re-login, or consider using a linearly scalable external cache provider, or the Dynacache client/server replication model.
- Install and use the Cache Monitor sample application to watch cache hit rates and cache exhaustion.
- If using
SHARED_PUSHreplication, consider usingSHARED_PUSH_PULLto reduce replication volume.
General Dynacache Notes
"WebSphere Application Server's Dynacache provides a general in-memory caching service for objects and page fragments generated by the server. The DistributedMap and DistributedObjectCache interfaces can be used within an application to cache and share Java objects by storing references to these objects in the cache for later use. Servlet caching, on the other hand, enables servlet and JSP response fragments to be stored and managed by a customizable set of caching rules."
Caching the output of servlets, commands, and JavaServer Pages (JSP) improves application performance... [Dynacache] intercepts calls through a servlet service method or a command execute method, and either stores the output of the object to the cache or serves the content of the object from the dynamic cache... The dynamic cache service is enabled by default. (https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tdyn_dynamiccache.html)
For command caching to operate properly, you must enable servlet caching. (https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tdyn_cachecommands.html)
There is an option called "limit memory cache size" to constrain how much memory Dynacache will use: https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/udyn_rcachesettings.html
Dynamic Cache Replication - Data Replication Service (DRS)
[DRS] replicates data from the dynamic cache service across the consumers in a replication domain.
To create replication domains manually, click Environment > Replication domains in the administrative console.
To create a new replication domain automatically when you create a cluster, click Servers > Clusters > New in the administrative console.
Do not use the default value of a single replica for the Number of replicas for dynamic cache replication domains. Instead, use a full group replica for any replication domains that you configure for dynamic cache.
In the administrative console, click Servers > Server Types > WebSphere application servers > server_name > Container services > Dynamic cache service. To enable replication, select Enable cache replication. Choose a replication domain.
With replication, data is generated one time and copied or replicated to other servers in the cluster, saving time and resources. Cache replication can take on three forms:
PUSH - Send out new entries, both ID and data, and updates to those entries.
PULL - Requests data from other servers in the cluster when that data is not locally present. This mode of replication is not recommended.
PUSH/PULL - Sends out IDs for new entries, then, only requests from other servers in the cluster entries for IDs previously broadcast. The dynamic cache always sends out cache entry invalidations.
Specifically, for PUSH or PUSH/PULL, the dynamic cache broadcasts the update asynchronously, based on a timed interval rather than sending them immediately when they are created. Invalidations are sent immediately.
SHARED_PUSH policy means as an object is added to the cache, it is immediately replicated to other nodes which is expensive in terms of JVM memory usage. Instead, the SHARED_PUSH_PULL policy should be used. This means only the cache key is replicated to the other nodes, and if the object is required it is replicated on the first 'cache miss'. This is much more memory efficient at the expense of a longer response time on the first access to the cached object. As the object would only be required on failover, this would be a rare occurrence anyway. This change in caching policy should be reviewed by the application development team, and tested in a failover scenario.
The other replication mode is NOT_SHARED: "When you use the Not Shared setting, as cache entries are created, neither the cache content nor the cache IDs are propagated to other servants or servers in the replication domain. However, invalidations are propagated to other servants or servers." (https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/udyn_rcachesettings.html)
There are two major types of invalidations: implicit and explicit. Implicit invalidations occur when a cache entry times out (if it has a time out) or it gets pushed out of the cache by the Least Recently Used (LRU) algorithm if the cache is full (based on the maximum cache size). Explicit invalidations occur when someone calls the DistributedMap invalidate* methods (for example, on a user logout) or through the same thing on a dependency. In some cases, implicit invalidations are not necessary to propagate, such as in large WebSphere Portal clusters: http://www-10.lotus.com/ldd/portalwiki.nsf/dx/Tuning_a_cluster_environment_%28Tuning_Guide_6.1.x%29. There are two JVM custom properties that avoid these implicit invalidations: com.ibm.ws.cache.CacheConfig.filterTimeOutInvalidation=true for the timeout case and com.ibm.ws.cache.CacheConfig.filterLRUInvalidation=true for the case when a cache is full and an entry is pushed out.
Replication type: https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/udyn_scacheinstancesettings.html
Architecture
The main architectures are peer-to-peer (default) and client/server.
Potential Tuning
Advanced options may help with DRS and Dynacache performance:
-Dcom.ibm.ws.cache.CacheConfig.useServerClassLoader=true- Deserializes an InvalidationEvent using the system classloader first and then using application classloader which generally improves performance.
-Dcom.ibm.ws.cache.CacheConfig.filterLRUInvalidation=true- In general, if a cache item is evicted because of lack of space, it's generally not needed to invalidate the entry in other JVMs.
-Dcom.ibm.ws.cache.CacheConfig.filterTimeOutInvalidation=true- In general, entries will timeout independently across all the JVMs at approximately the same time, so it's not needed to explicitly send an invalidation event.
-Dcom.ibm.ws.cache.CacheConfig.filterInactivityInvalidation=true- In general, entries will timeout independently across all the JVMs at approximately the same time, so it's not needed to explicitly send an invalidation event.
-Dcom.ibm.ws.cache.CacheConfig.ignoreValueInInvalidationEvent=true
This tuning may be applied globally using the instructions under "configure the custom property globally across all configured cache instances" at http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/ae/rdyn_tunediskcache.html
ignoreValueInInvalidationEvent
Specifies whether the cache value of Invalidation event is ignored. If it is true, the cache value of Invalidation event is set to NULL when the code is returned to the caller.
propogateInvalidationsNotSharedValue
Default set to false, which provides the best performance. If it is set to true, Dynacache will send invalidations to peer members in the cluster on cache entry insertions and updates for a NOT_SHARED cache instance. This can cause a significant performance impact.
DRS Thread Pool
Consider tuning the DRS thread pool. Defaults:
- DRS_THREADPOOL_MINSIZE=40
- DRS_THREADPOOL_MAXSIZE-100
- DRS_THREADPOOL_ISGROWABLE=false
ws/WSSecureMap
The ws/WSSecureMap Dynacache is used for horizontal security attribute propagation (web inbound security attribute propagation).
System Dump or HPROF Heapdump Analysis
With the IBM Memory Analyzer Tool and the IBM Extensions for Memory Analyzer, use the Dynacache queries to get details of Dynacache in a system dump. The list of queries are:
For example, review the number of entries, cache size, and hit ratio:
A high number of misses could mean that the cache size is too small, there are many invalidations, there is a get-check-update pattern without warmup/pre-loading, etc.
Clearing Cache
The DynaCache MBean is a scriptable interface to interact with DynaCache caches at runtime.
For example, to clear a DynaCache cache instance on a server at runtime, use the following Jython wsadmin code:
AdminControl.invoke(AdminControl.completeObjectName("type=DynaCache,node=MYNODE,process=MYSERVER,*"), "clearCache", "MYCACHEINSTANCE")
The documentation for clearCache says it will "clear all cache entries for the named cache instance." This will clear both the in-memory cache entries as well as any cache entries offloaded to disk. You can confirm this by executing the "getAllCacheStatistics" command on the cache instance before and after the clearCache call.
Servlet Caching
Servlet caching may cause a significant throughput improvement (in one benchmark, 30-60%).
Use this task to define cacheable objects inside the cachespec.xml, found inside the web module WEB-INF or enterprise bean META-INF directory... [or] you can save a global cachespec.xml in the application server properties directory, but the recommended method is to place the cache configuration file with the deployment module.
In situations where there is a global cachespec.xml file in the application server properties directory, and a cachespec.xml file in an application, the entries in the two cachespec.xml files are merged. If there are conflicting entries in the two files, the entries in the cachespec.xml file that is in the application override the entries in the global cachespec.xml file for that application.
To cache an object, WebSphere Application Server must know how to generate unique IDs for different invocations of that object. The
<cache-id>element performs that task. Each cache entry can have multiple cache-ID rules that run in order until either a rule returns a cache-ID that is not empty or no more rules remain to run. If no cache-ID generation rules produce a valid cache ID, then the object is not cached.Use dependency ID elements to specify additional cache group identifiers that associate multiple cache entries to the same group identifier. The dependency ID is generated by concatenating the dependency ID base string with the values returned by its component elements. If a required component returns a null value, then the entire dependency ID is neither generated nor.
Invalidate other cache entries as a side effect of this object start, if relevant. You can define invalidation rules in exactly the same manner as dependency IDs... The invalidation ID is generated by concatenating the invalidation ID base string with the values returned by its component element. If a required component returns a null value, then the entire invalidation ID is not generated and no invalidation occurs. Multiple invalidation rules can exist per cache-entry. All invalidation rules run separately.
The dynamic cache reloads the updated file automatically. If you are caching static content and you are adding the cache policy to an application for the first time, you must restart the application. You do not need to restart the application server to activate the new cache policy.
When new versions of the cachespec.xml are detected, the old policies are replaced. Objects that cached through the old policy file are not automatically invalidated from the cache; they are either reused with the new policy or eliminated from the cache through its replacement algorithm.
Full cachespec.xml schema: https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/rdyn_cachespec.html
The <timeout> is specified in seconds. If a
timeout is not specified, then the cache entry does not expire. However,
in both cases, a cache entry may be evicted or invalidated either
explicitly, by invalidation rules, or by the Least Recently Used (LRU)
algorithm when the cache is full.
Before WAS 9, if servlet caching is enabled in an application server, all requests pass through Dynacache, even if there is no chance they will be cached. Starting in WAS 9, only requests that have a context root associated with a cachespec.xml flow through Dynacache. This means that using a global cachespec.xml in WAS/profiles/{PROFILE}/properties only works for applications that have a cachespec.xml in the WAR's WEB-INF. To "defer" to a global cachespec.xml, a "dummy" cachespec.xml may be placed in the application such as:
A cached response served from the servlet cache will also set a
CACHED_RESPONSE: true response header.
<?xml version="1.0" ?>
<!DOCTYPE cache SYSTEM "cachespec.dtd">
<cache>
<cache-entry>
<class>servlet</class>
<name>/dummy</name>
</cache-entry>
</cache>Example custom ID generators: https://github.com/kgibm/WASServletCachingIDGenerators
Servlet Caching Example
Suppose that a servlet manages a simple news site. This servlet uses the query parameter "action" to determine if the request views (query parameter "view") news or updates (query parameter "update") news (used by the administrator). Another query parameter "category" selects the news category. Suppose that this site supports an optional customized layout that is stored in the user's session using the attribute name "layout". Here are example URL requests to this servlet:
http://yourhost/yourwebapp/newscontroller?action=view&category=sports (Returns a news page for the sports category )
http://yourhost/yourwebapp/newscontroller?action=view&category=money (Returns a news page for the money category)
http://yourhost/yourwebapp/newscontroller?action=update&category=fashion (Allows the administrator to update news in the fashion category)Define the
<cache-entry>elements that are necessary to identify the servlet. In this case, the URI for the servlet is "newscontroller", so this is the cache-entry<name>element. Because this example caches a servlet or JavaServer Pages (JSP) file, the cache entry class is "servlet".Define cache ID generation rules. This servlet caches only when action=view, so one component of the cache ID is the parameter "action" when the value equals "view". The news category is also an essential part of the cache ID. The optional session attribute for the user's layout is included in the cache ID.
Define dependency ID rules. For this servlet, a dependency ID is added for the category. Later, when the category is invalidated due to an update event, all views of that news category are invalidated.
Define invalidation rules. Because a category dependency ID is already defined, define an invalidation rule to invalidate the category when action=update. To incorporate the conditional logic, add "ignore-value" components into the invalidation rule. These components do not add to the output of the invalidation ID, but only determine whether or not the invalidation ID creates and runs.
<cache-entry> <name>newscontroller</name> <class>servlet</class> <cache-id> <component id="action" type="parameter"> <value>view</value> <required>true</required> </component> <component id="category" type="parameter"> <required>true</required> </component> <component id="layout" type="session"> <required>false</required> </component> </cache-id> <dependency-id> <component id="category" type="parameter"> <required>true</required> </component> </dependency-id> <invalidation> <component id="action" type="parameter" ignore-value="true"> <value>update</value> <required>true</required> </component> <component id="category" type="parameter"> <required>true</required> </component> </invalidation> </cache-entry>
Servlet Caching by User
Be careful in building your cache ID if cached objects may be user-specific. In such a case, you can use some user-identifiable component for the cache ID such as the JSESSIONID:
<cache-entry> <class>servlet</class> <name>/forward.do</name> <cache-id> <property name="EdgeCacheable">true</property> <component id="type" type="parameter"> <required>true</required> <value>esiParentConsume</value> </component> <component id="JSESSIONID" type="cookie" /> <timeout>35</timeout> <priority>1</priority> </cache-id> </cache-entry>
cachespec.xml
Example to Disable Caching for a Servlet or JSP
<cache-entry>
<class>servlet</class>
<name>com.example.servlet.MyServlet.class</name>
<property name="do-not-cache">true</property>
</cache-entry>
<cache-entry>
<class>servlet</class>
<name>com.ibm._jsp._myjsp.class</name>
<property name="do-not-cache">true</property>
</cache-entry>Monitoring
"Use the administrative console to install the cache monitor application from the app_server_root/installableApps directory. The name of the application is CacheMonitor.ear... you can access the cache monitor using http://your_host_name:your_port_number/cachemonitor" (https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tdyn_servletmonitor.html).
Monitor the cache hit ratio: https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tdyn_cache_tune.html
If javacores or other indicators show a lot of JMX activity causing performance issues, use com.ibm.ws.management.connector.soap.logClientInfo=true
Example in dmgr SystemOut.log:
[8/20/13 11:31:19:624 SGT] 0000001d SOAPConnector I SOAP client info: Host/port= localhost/54242, authSubject.getPrincipals()=Another method to monitor Dynacache is using this flight recorder: https://github.com/kelapure/dynacache/blob/master/scripts/DynaCacheStatisticsCSV.py.readme.txt and https://github.com/kelapure/dynacache/blob/master/scripts/DynaCacheStatisticsCSV.py
baseCache
The baseCache is a built-in cache instance. Its replication in a
replication domain is disabled by default (enable under
$SERVER } Container Services } Dynamic cache service). Its
JNDI name is services/cache/distributedmap.
Missed replication
Missed replication may be caused by:
- Uninitialized caches which is usually resolved by
-Dcom.ibm.ws.cache.CacheConfig.createCacheAtServerStartup=true
Disk Offload
Disk offload allows paging cache to/from disk to allow a cache larger than the in-memory cache. The disk offload also includes a garbage collector.
Flush to Disk on Stop
Flush to disk graceful JVM stop may be enabled globally with
-Dcom.ibm.ws.cache.flushToDiskOnStop=true
Messages are printed when this is enabled for each cache and another shows how long it takes to write. For example:
[11/20/20 16:26:41:432 EST] 000000ac Cache I DYNA0060I: Flush to disk on stop is enabled for cache name "baseCache".
[11/20/20 16:30:51:639 EST] 000000ac Cache I DYNA0073I: Flush to disk on stop for cache name "baseCache" has completed. The statistics are: numOfEntriesFlushToDisk=1964 numOfBytesFlushToDisk=70984192 timeElapsedEntriesFlushToDisk=1645 numDepIdsInAuxTable=76253 numCacheIdsInDepIdAuxTable=703152 numTemplatesInAuxTable=10 numCacheIdsInTemplateAuxTable=121752 timeElapsedWriteAuxTables=248560