Hung Threads with multi thread processing

Question

We use TopLink v10.1.3.5
One of our applications uses multi-core processing and when multiple threads try to access lazily loaded 1-m relationships at the same time, the threads just hang forever. There are no errors but the thread dump says it’s waiting on a condition.
We do not use cache Synchronization.
The 1-m relationship is privately owned, uses Batch reading and Indirection.
The problem occurs intermittently and we cannot reproduce it at will.
Found this discussion, but since we are on 10.3.5, I think we should already have the patch?
Hung Threads (Toplink 10.1.3)
Can anyone please provide any help on this.
Thanks for the help.
Thread dump -
at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:167(Compiled Code))
at oracle/toplink/internal/helper/ConcurrencyManager.acquire(ConcurrencyManager.java:76(Compiled Code))
at oracle/toplink/internal/identitymaps/CacheKey.acquire(CacheKey.java:85(Compiled Code))
at oracle/toplink/internal/identitymaps/IdentityMap.acquireLock(IdentityMap.java:85(Compiled Code))
at oracle/toplink/internal/identitymaps/IdentityMapManager.acquireLock(IdentityMapManager.java:101(Compiled Code))
at oracle/toplink/internal/sessions/IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:68(Compiled Code))
at oracle/toplink/internal/sessions/IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:58(Compiled Code))
at oracle/toplink/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:502(Compiled Code))
at oracle/toplink/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:382(Compiled Code))
at oracle/toplink/mappings/OneToOneMapping.valueFromRow(OneToOneMapping.java:1020(Compiled Code))
at oracle/toplink/mappings/DatabaseMapping.readFromRowIntoObject(DatabaseMapping.java:1045(Compiled Code))
at oracle/toplink/internal/descriptors/ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:245(Compiled Code))
at oracle/toplink/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:564(Compiled Code))
at oracle/toplink/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:382(Compiled Code))
at oracle/toplink/internal/descriptors/ObjectBuilder.buildObjectsInto(ObjectBuilder.java:678(Compiled Code))
at oracle/toplink/internal/queryframework/DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:142(Compiled Code))
at oracle/toplink/queryframework/ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:483(Compiled Code))
at oracle/toplink/queryframework/ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:813(Compiled Code))
at oracle/toplink/queryframework/DatabaseQuery.execute(DatabaseQuery.java:620(Compiled Code))
at oracle/toplink/queryframework/ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:781(Compiled Code))
at oracle/toplink/queryframework/ReadAllQuery.execute(ReadAllQuery.java:451(Compiled Code))
at oracle/toplink/publicinterface/Session.internalExecuteQuery(Session.java:2089(Compiled Code))
at oracle/toplink/publicinterface/Session.executeQuery(Session.java:993(Compiled Code))
at oracle/toplink/internal/indirection/QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:62(Compiled Code))
at oracle/toplink/internal/indirection/QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:55(Compiled Code))
at oracle/toplink/internal/indirection/DatabaseValueHolder.getValue(DatabaseValueHolder.java:61(Compiled Code))
at oracle/toplink/indirection/IndirectList.buildDelegate(IndirectList.java:202(Compiled Code))
at oracle/toplink/indirection/IndirectList.getDelegate(IndirectList.java:359(Compiled Code))
at oracle/toplink/indirection/IndirectList.size(IndirectList.java:703(Compiled Code))

Answer 1

All I can tell from the stack shown is that you are triggering indirection on a collection to an object that has an eagerly fetched 1:1 mapping to an object that is locked in the cache.  The thread is waiting on the lock owner to finish building the object before it can be returned.
To tell what is going wrong, you will need to look at the complete thread dump to see what other threads might be building the object with its cache key locked.  For instance, a long running query might make it appear the system is hung when it is just a bottleneck in the application.  If you can, you should also try to call session.getIdentityMapAccessor().printIdentityMapLocks(); at the time of the problem as it will print off information on the locks such as the object involved and which threads hold them.  As for patches; 10.1.3 is an older stream but I am unaware of fixes that are not included in 10.1.3.5 that might resolve this - you should go through support to be sure.
Best Regards,
Chris