Clear operation takes long time and gets interrupted in ThreadGate.doWait

Question

Hi,
We are running Coherence 3.5.3 cluster with 16 storage enabled nodes and 24 storage disabled nodes. We have about hundred of partitioned caches with NearCaches (invalidation strategy = PRESENT, size limit for different caches 60-200K) and backup count = 1. For each cache we have a notion of cache A and cache B. Every day either A or B is active and is used by business logic while the other one is inactive, not used and empty. Daily we load fresh data to inactive caches, mark them as active (switch business logic to work with fresh data from those caches), and clear all yesterday's data in those caches which are not used today.
So at the end of data load we execute NamedCache.clear() operation for each inactive cache from storage disabled node. From time to time, 1-2 times a week, the clear operation fails on one of 2 our biggest caches (one has 1.2M entries and another one has 350K entries). We did some investigations and found that NamedCache.clear operation fires many events within Coherence cluster to clear NearCaches so that operation is quite expensive. In some other simular posts there were suggestions to not use NamedCache.clear, but rather use NamedCache.destroy, however that doesn't work for us in current timelines. So we implemented simple retry logic that retries NamedCache.clear() operation up to 4 times with increasing delay between the attempts (1min, 2 min, 4 min).
However that didn't help. 3 out of those attempts failed with the same error on one storage enabled node and 1 out of those 4 attempts failed on another storage enabled node. In all cases a Coherence worker thread that is executing ClearRequest on storage enabled node got interrupted by Guardian after it reached its timeout while it was waiting on lock object at ThreadGate.doWait. Please see below:
Log from the node that calls NamedCache.clear()
Portable(com.tangosol.util.WrapperException): (Wrapped: Failed request execution for ProductDistributedCache service on Member(Id=26, Timestamp=2012-09-04 13:37:43.922, Address=32.83.113.116:10000, MachineId=3149, Location=machine:mac305,process:2
7091,member:mac305.instance1, Role=storage) (Wrapped: ThreadGate{State=GATE_CLOSING, ActiveCount=3, CloseCount=0, ClosingT
hread= Thread[ProductDistributedCacheWorker:1,5,ProductDistributedCache]}) null) null
Caused by:
Portable(java.lang.InterruptedException) ( << comment: this came form storage enabled node >> )
at java.lang.Object.wait(Native Method)
at com.tangosol.util.ThreadGate.doWait(ThreadGate.java:489)
at com.tangosol.util.ThreadGate.close(ThreadGate.java:239)
at com.tangosol.util.SegmentedConcurrentMap.lock(SegmentedConcurrentMap.java:180)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onClearRequest(DistributedCache.CDB:27)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ClearRequest.run(DistributedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:619)
Log from the that storage enabled node which threw an exception
Sat Sep 08 04:38:37 EDT 2012|**ERROR**| com.tangosol.coherence.component.util.logOutput.Log4j | 2012-09-08 04:38:37.720/31330
1.617 Oracle Coherence EE 3.5.3/465 <Error> (thread=DistributedCache:ProductDistributedCache, member=26): Attempting recovery
(due to soft timeout) of Guard{Daemon=ProductDistributedCacheWorker:1} |Client Details{sdpGrid:,ClientName:  ClientInstanceN
ame: ,ClientThreadName:  }| Logger@9259509 3.5.3/465
Sat Sep 08 04:38:37 EDT 2012|**WARN**| com.tangosol.coherence.component.util.logOutput.Log4j | 2012-09-08 04:38:37.720/313301
.617 Oracle Coherence EE 3.5.3/465 <Warning> (thread=Recovery Thread, member=26): A worker thread has been executing task: Message "ClearRequest"
FromMember=Member(Id=38, Timestamp=2012-09-07 10:12:27.402, Address=32.83.113.120:10000, MachineId=40810, Location=machine:
mac313,process:22837,member:mac313.instance1, Role=maintenance)
FromMessageId=5278229
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=1
ToPollId=0
Poll=null
Packets
[000]=Directed{PacketType=0x0DDF00D5, ToId=26, FromId=38, Direction=Incoming, ReceivedMillis=04:36:49.718, ToMemberSet=nu
ll, ServiceId=6, MessageType=1, FromMessageId=5278229, ToMessageId=337177, MessagePartCount=1, MessagePartIndex=0, NackInProg
ress=false, ResendScheduled=none, Timeout=none, PendingResendSkips=0, DeliveryState=unsent, Body=0x000D551F0085B8DF9FAECE8001
0101010204084080C001C1F80000000000000010000000000000000000000000000000000000000000000000, Body.length=57}
Service=DistributedCache{Name=ProductDistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, B
ackupCount=1, AssignedPartitions=16, BackupPartitions=16}
ToMemberSet=MemberSet(Size=1, BitSetCount=2
Member(Id=26, Timestamp=2012-09-04 13:37:43.922, Address=32.83.113.116:10000, MachineId=3149, Location=machine:mac305,process:27091,member:mac305.instance1, Role=storage)
NotifySent=false
} for 108002ms and appears to be stuck; attempting to interrupt: ProductDistributedCacheWorker:1 |Client Details{sdpGrid:,C
lientName: ClientInstanceName: ,ClientThreadName: }| Logger@9259509 3.5.3/465
I am looking for your help. Please let me know if you see what is the reason for the issue and how to address it.
Thank you

Answer 1

Today we had that issue again and I have gathered some more information.
Everything was the same as I described in the previous posts in this thread: first attempt to clear a cache failed and next 3 retries also failed. All 4 times 2 storage enabled nodes had that "... A worker thread has been executing task: Message "ClearRequest" ..." error message and got interrupted by Guardian.
However after that I had some time to do further experiments. Our App has cache management UI that allows to clear any cache. So I started repeatedly taking thread dumps on those 2 storage enabled nodes which failed to clear the cache and executed cache clear operation form that UI. One of storage enabled nodes successfully cleared its part, but the other still failed. It failed with completely same error.
So, I have a thread dump which I took while cache clear operation was in progress. It shows that a thread which is processing that ClearRequest is stuck waiting in ThreadGate.close method:
at java.lang.Object.wait(Native Method)
at com.tangosol.util.ThreadGate.doWait(ThreadGate.java:489)
at com.tangosol.util.ThreadGate.close(ThreadGate.java:239)
at com.tangosol.util.SegmentedConcurrentMap.lock(SegmentedConcurrentMap.java:180)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onClearRequest(DistributedCache.CDB:27)
at
All subsequents attempts to clear cache from cache management UI failed until we restarted that storage enabled node.
It looks like some thread left ThreadGate in a locked state, and any further attempts to apply a lock as part of ClearRequest message fail. May be it is known issue of Coherence 3.5.3?
Thanks