System-wide cluster freeze

Question

Hello,
We're experiencing many system-wide freeze instances in which many cluster members reported the following errors. We had to shut down cluster members one by one until the remaining system could be freed up. Our customer had many tools to monitor the network health. The network seemed not an issue. We've increased lots of memory to the nodes but there was no correlation to the memory size. What could be other possible causes that Grid.poll() hang?
Thanks!
-Edwin
2013-05-29 16:06:39,510 ERROR [Logger@9215997 3.6.1.9             ] [Coherence:3] ( ) 2013-05-29 16:06:39.510/23816.450 Oracle Coherence GE 3.6.1.9 <Error> (thread=ReplicatedCache:QuartzReplicatedCache, member=1): Detected soft timeout) of {WrapperGuardable Guard{Daemon=ReplicatedCache:QuartzReplicatedCache:EventDispatcher} Service=ReplicatedCache{Name=QuartzReplicatedCache, State=(SERVICE_STARTED), Id=6, Version=3.0, OldestMemberId=73}}
2013-05-29 16:06:39,588 WARN [Logger@9215997 3.6.1.9             ] [Coherence:3] ( ) 2013-05-29 16:06:39.588/23816.528 Oracle Coherence GE 3.6.1.9 <Warning> (thread=Recovery Thread, member=1): Attempting recovery of Guard{Daemon=ReplicatedCache:QuartzReplicatedCache:EventDispatcher}
2013-05-29 16:06:39,588 ERROR [Logger@9215997 3.6.1.9             ] [Coherence:3] ( ) 2013-05-29 16:06:39.588/23816.528 Oracle Coherence GE 3.6.1.9 <Error> (thread=ReplicatedCache:QuartzReplicatedCache:EventDispatcher, member=1): This thread was interrupted while waiting for the results of a request:
Poll
PollId=211929, active
InitTimeMillis=1369836341758
Service=QuartzReplicatedCache (6)
RespondedMemberSet=[]
LeftMemberSet=[]
RemainingMemberSet=[73]
2013-05-29 16:06:39,588 ERROR [Logger@9215997 3.6.1.9             ] [Coherence:3] ( ) 2013-05-29 16:06:39.588/23816.528 Oracle Coherence GE 3.6.1.9 <Error> (thread=ReplicatedCache:QuartzReplicatedCache:EventDispatcher, member=1): An exception occurred while dispatching the following event:
DispatchEvent: MemberEvent{Member=47 LEFT com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ReplicatedCache}
2013-05-29 16:06:39,588 ERROR [Logger@9215997 3.6.1.9             ] [Coherence:3] ( ) 2013-05-29 16:06:39.588/23816.528 Oracle Coherence GE 3.6.1.9 <Error> (thread=ReplicatedCache:QuartzReplicatedCache:EventDispatcher, member=1): The following exception was caught by the event dispatcher:
2013-05-29 16:06:39,619 ERROR [Logger@9215997 3.6.1.9             ] [Coherence:3] ( ) 2013-05-29 16:06:39.588/23816.528 Oracle Coherence GE 3.6.1.9 <Error> (thread=ReplicatedCache:QuartzReplicatedCache:EventDispatcher, member=1):
(Wrapped: Interrupted while waiting for response from:[Member(Id=73, Timestamp=2013-05-29 05:38:20.434, Address=10.70.0.50:8090, MachineId=19506, Location=site:machine:TPTEXT50,process:3632, Role=StorageNodeDaemon)]) java.lang.InterruptedException
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.processPollInterrupt(Grid.CDB:40)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:60)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ReplicatedCache.requestLock(ReplicatedCache.CDB:42)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ReplicatedCache.lockResource(ReplicatedCache.CDB:47)
     at com.tangosol.coherence.component.util.CacheHandler.lock(CacheHandler.CDB:3)
     at com.tangosol.coherence.component.util.SafeNamedCache.lock(SafeNamedCache.CDB:1)
     at com.navis.framework.quartz.cluster.cache.coherence.CoherenceQuartzCache.removeObjectFromLockedCache(CoherenceQuartzCache.java:195)
     at com.navis.framework.quartz.cluster.cache.coherence.CoherenceQuartzCache.removeNode(CoherenceQuartzCache.java:136)
     at com.navis.framework.quartz.QuartzHelper.removeNode(QuartzHelper.java:219)
     at com.navis.framework.quartz.cluster.cache.coherence.QuartzNodeListener.memberLeft(QuartzNodeListener.java:54)
     at com.tangosol.net.MemberEvent.dispatch(MemberEvent.java:116)
     at com.tangosol.net.MemberEvent.dispatch(MemberEvent.java:87)
     at com.tangosol.coherence.component.util.SafeService.translateEvent(SafeService.CDB:28)
     at com.tangosol.coherence.component.util.SafeService.memberLeft(SafeService.CDB:1)
     at com.tangosol.net.MemberEvent.dispatch(MemberEvent.java:116)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DispatchEvent.run(Grid.CDB:7)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onNotify(Service.CDB:26)
     at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
     at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.InterruptedException
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:485)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:32)
     ... 17 more
Edited by: user10362631 on May 29, 2013 4:32 PM

Answer 1

This log indicated the process caused the problem.
(Wrapped: Interrupted while waiting for response from:[Member(Id=73, Timestamp=2013-05-29 05:38:20.434, Address=10.70.0.50:8090, MachineId=19506, Location=site:machine:TPTEXT50,process:3632, Role=StorageNodeDaemon)]) java.lang.InterruptedException
If you still have the log file from member 73, take a look at the log file. You might find service guardian triggered and thread dump within the log. The log message just before the thread dump section should tell you which thread caused the problem.