Querying the toplink cache under high-load

Question

We've had some interesting experiences with "querying" the TopLink Cache lately.
It was recently discovered that our "read a single object" method was incorrectly
setting query.checkCacheThenDB() for all ReadObjectQueries. This was brought to light
when we upgraded our production servers from 4 cores to 8. We immediatly started
experiencing very long response times under load.
We traced this down to the following stack: (TopLink version 10.1.3.1.0)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab08fd26d8> (a oracle.toplink.internal.helper.ConcurrencyManager)
at java.lang.Object.wait(Object.java:474)
at oracle.toplink.internal.helper.ConcurrencyManager.acquireReadLock(ConcurrencyManager.java:179)
- locked <0x00002aab08fd26d8> (a oracle.toplink.internal.helper.ConcurrencyManager)
at oracle.toplink.internal.helper.ConcurrencyManager.checkReadLock(ConcurrencyManager.java:167)
at oracle.toplink.internal.identitymaps.CacheKey.checkReadLock(CacheKey.java:122)
at oracle.toplink.internal.identitymaps.IdentityMapKeyEnumeration.nextElement(IdentityMapKeyEnumeration.java:31)
at oracle.toplink.internal.identitymaps.IdentityMapManager.getFromIdentityMap(IdentityMapManager.java:530)
at oracle.toplink.internal.queryframework.ExpressionQueryMechanism.checkCacheForObject(ExpressionQueryMechanism.java:412)
at oracle.toplink.queryframework.ReadObjectQuery.checkEarlyReturnImpl(ReadObjectQuery.java:223)
at oracle.toplink.queryframework.ObjectLevelReadQuery.checkEarlyReturn(ObjectLevelReadQuery.java:504)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:564)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:779)
at oracle.toplink.queryframework.ReadObjectQuery.execute(ReadObjectQuery.java:388)
We moved the query back to the default, query.checkByPrimaryKey() and this issue went away.
The bottleneck seemed to stem from the read lock on the CacheKey from IdenityMapKeyEnumeration
public Object nextElement() {
if (this.nextKey == null) {
throw new NoSuchElementException("IdentityMapKeyEnumeration nextElement");
// CR#... Must check the read lock to avoid
// returning half built objects.
this.nextKey.checkReadLock();
return this.nextKey;
We had many threads that were having contention while searching the cache for a particular query.
From the stack we know that the contention was limited to one class. We've since refactored that code
not to use a query in that code path.
Question:
Armed with this better knowledge of how TopLink queries the cache, we do have a few objects that we
frequently read by something other than the primary key. A natural key, but not the oid.
We have some other caching mechanisms in place (JBoss TreeCache) to help eliminate queries to the DB
for these objects. But the TreeCache also tries to acquire a read lock when accessing the cache.
Presumably a read lock over the network to the cluster.
Is there anything that can be done about the read lock on CacheKey when querying the cache in a high load
situation?

Answer 1

CheckCacheThenDatabase will check the entire cache for a match using a linear search. This can be efficient if the cache is very large. Typically it is more efficient to access the database if your cache is large and the field you are querying in the database is indexed in the table.
The cache concurrency was greatly improved in TopLink 11g/EclipseLink, so you may wish to try it out.
Supporting indexes in the TopLink/EclipseLink cache is something desirable (feel free to log the enhancement request on EclipseLink). You can simulate this to some degree using a named query and a query cache.
-- James : http://www.eclipselink.org