ValidatePolls error with a distributed lock

Question

Hello
Running 3.7.1.2
I have 24 storage enabled Coherence nodes. Every night a client thread acquires a lock for a cache and key (other client threads also try to acquire the same lock and key afterwards). The lock request will hit one of the server nodes as the cache is setup with a distributed scheme. The service responsible for the lock cache is configured with 0 worker threads. Question is, why do I get the message below in the log? The service thread is not deadlocked, nor blocked, when I analyze the thread dump in the morning. What is the function of the poll? Cheers Niklas
2012-09-06 03:14:08,413 ERROR [Coherence](Logger@9245523 3.7.1.2) ()()(): (member=67): validatePolls: Manual intervention is required to stop this node or the members that have not responded to this poll (gap size=4194420): Poll
PollId=2118256570, active
InitTimeMillis=1346893755440
Service=DistributedCache (2)
RespondedMemberSet=[]
LeftMemberSet=[]
RemainingMemberSet=[19]
Request=Message "LockRequest"
FromMember=Member(Id=67, Timestamp=2012-08-16 02:24:31.96, Address=10.24.0.169:8088, MachineId=65018, Location=site:X-site-prod,machine:XXXX1903,process:3028, Role=integration)
FromMessageId=2429713031
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=67
ToPollId=0
Poll=null
Packets
Service=PartitionedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=disabled}
ToMemberSet=MemberSet(Size=1
Member(Id=19, Timestamp=2012-08-16 02:22:19.38, Address=10.24.0.153:8088, MachineId=21782, Location=site:X-site-prod,machine:XXXX1877,process:12400, Role=coherence)
NotifySent=false
CacheName=com.xx.xxxx.lock
WaitTimeout=1346894205490, LeaseExpiration=9223372036854775807
} within array capacity=8388608, expanions=17, size=2122451010, window index=0, window size=51, first index=2122450959, last index=2122451009, optimistic gets=2127944257, place holder allocations=548, waits=0, waiting threads=0

Answer 1

Here are a few more details:
The client thread that acquires the lock will keep the lock for at least 2 hours whilst clearing and loading a number of caches. Meanwhile another thread tries to obtain the lock in 5 minute intervals with
while(!cache.lock("motherOfAllLocks", 5*60*1000))
     LOG.debug("Waited another five minutes");
In the log I typically find that the other thread has waited a number of five minute intervals before the validatePolls ERROR appears. Subsequently the validatePolls error will show up roughly every 4 minutes during half an hour before the "Waited another five minutes" appears. So in my case the lock(Object oKey, long cWait) method will wait a lot longer than cWait millis. Today I also noticed a cryptic warning message prior to the cryptic validatePolls error message:
2012-09-11 04:35:28,652 WARN [Coherence](Logger@9217551 3.7.1.2) ()()(): (member=66): PollArray window size has grown to 8388608 elements; first element[1520943] = com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$LockRequest$Poll:Poll
PollId=1520943, active
InitTimeMillis=1347330683572
Service=DistributedCache (2)
RespondedMemberSet=[]
LeftMemberSet=[]
RemainingMemberSet=[8]
Thanx for any help, cheers Niklas