When put() does not return

Question

We're using Coherence 3.0/315. We've experienced a couple of failures in our system and tracked it down to sort of deadlock in our code. I say "sort of" because really the threads don't seem to be interlocked in a traditional deadlock, but rather it's just that one of the threads never returns from the NamedCache.put() call which is inside a synchronized method that other threads are blocked on. (I wouldn't think that the Coherence code would be calling back to our code (listeners, etc.) on the same thread, or we'd be deadlocking every time.) We've now fixed our own code's synchronization to avoid unnecessary synchronized declarations, which will probably clear up most of the problems. However, we're concerned that we don't understand why the put() calls failed to return; we may still need to call put() from a synchronized block, which would be deadly if it failed to return. We can fairly consistently reproduce the deadlock by putting 5000 objects to the cache (from a different client) and then pausing the program for several seconds in the debugger while it's handling the big spike of cache insert notifications. When we resume from the debugger, the program soon hits the deadlock where put() is not returning.
There were two different stack crawls showing a thread stuck beneath put().
One I've only seen once, and it was in Coherence code with a call to Thread.join(). Evidently, the joined thread never completed. Unfortunately I don't have a stack crawl for that case (nor did I note which thread was joined to).
The other case which I am easily able to reproduce and get a stack crawl for is the following. Ultimately it's in com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.drainOverflow(), evidently sleeping (repeatedly?) but never completing its work.
<tt>
drainOverflow():16, com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher
send():17, com.tangosol.coherence.component.util.daemon.queueProcessor.Service
poll():13, com.tangosol.coherence.component.util.daemon.queueProcessor.Service
poll():1, com.tangosol.coherence.component.util.daemon.queueProcessor.Service
requestUpdate():37, com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache
updateResourceOptimistically():11, com.tangosol.coherence.component.util.daemon.queueProcessor.service.replicatedCache.Optimistic
put():3, com.tangosol.coherence.component.util.daemon.queueProcessor.service.replicatedCache.Optimistic$CacheHandler
put():1, com.tangosol.coherence.component.util.SafeNamedCache
</tt>
I realize that in reproducing the situation we are artificially inducing a cache service restart. But I think we can expect such a glitch-and-restart on rare occasions in a production environment due to network failures, and I believe our code manages the restart correctly, but if NamedCache.put() fails to return during a cache service restart that will definitely cause us trouble at some point if we need to call put() from a synchronized block. Any thoughts on this?
Thanks,
Trygve

Answer 1

Thanks, Mark. I will try to gather thread dumps and send them. We will also try reproducing with Coherence 3.1.1. We're using several different systems in production and in reproducing the problem, so I can confirm it for these OS/JVM versions (i.e., we see it on both systems we've tried to reproduce it on, and these are not identical to the production system) :
- Windows 2000, Sun JDK 1.4.2 (production system)
- Windows XP Pro SP2, Sun JDK 1.5 (Tim's dev machine)
- Mac OS X 10.4.6, Apple JDK 1.5, with 1.4.2 set as default for apps (my dev machine)
--Trygve