Unexplained Ownership conflict with Coherence 3.5.2

Question

Hi all,
We are using Coherence 3.5.2 quite happily for a while.
Now we've start using it for a new project.
Basically we have a cache with 36000 object updated concurrently through EntryProcessors.
All this is running in 8 application nodes (java 1.6).
After running for a while (93% of our objects updated), the entire cluster starts hanging with unexplained errors.
The first error is a Ownership conflict error.
I googled that error and saw that it has been resolved with a patch in 3.5.1. So normally it should be fixed also in 3.5.2.
So my question:
Under which exact conditions can we expect to face that error?
Many thanks
Harry,
logs:
11:35:52,251|WARN|Logger@9247854 3.5.2/463||||2010-05-06 11:35:52.250/1684.780 Oracle Coherence GE 3.5.2/463 <Warning> (thread=DistributedCache:UnlimitedDistributedAppScheme, member=1): 5> Ownership conflict for partition 129 with member 66 (49!=0)|Coherence
11:35:52,264|WARN|Logger@9247854 3.5.2/463||||2010-05-06 11:35:52.252/1684.782 Oracle Coherence GE 3.5.2/463 <Warning> (thread=DistributedCache:UnlimitedDistributedAppScheme, member=1): 5> Ownership conflict for partition 114 with member 54 (66!=0)|Coherence
11:35:52,381|WARN|Logger@9247854 3.5.2/463||||2010-05-06 11:35:52.380/1684.910 Oracle Coherence GE 3.5.2/463 <Warning> (thread=DistributedCache:UnlimitedDistributedAppScheme, member=1): 5> Ownership conflict for partition 234 with member 16 (54!=0)|Coherence
11:35:52,391|WARN|Logger@9247854 3.5.2/463||||2010-05-06 11:35:52.391/1684.921 Oracle Coherence GE 3.5.2/463 <Warning> (thread=DistributedCache:UnlimitedDistributedAppScheme, member=1): 5> Ownership conflict for partition 115 with member 42 (16!=0)|Coherence
11:35:52,403|WARN|Logger@9247854 3.5.2/463||||2010-05-06 11:35:52.391/1684.921 Oracle Coherence GE 3.5.2/463 <Warning> (thread=DistributedCache:UnlimitedDistributedAppScheme, member=1): 5> Ownership conflict for partition 116 with member 42 (16!=0)|Coherence
11:35:52,413|WARN|Logger@9247854 3.5.2/463||||2010-05-06 11:35:52.391/1684.921 Oracle Coherence GE 3.5.2/463 <Warning> (thread=DistributedCache:UnlimitedDistributedAppScheme, member=1): 5> Ownership conflict for partition 117 with member 42 (16!=0)|Coherence

Answer 1

Hi ggleyzer,
yes that's the manisfestation.
We start having those errors (Ownership conflict) for a short period and after that the DistributedCache service associated to our cache starts giving up (executing "InvokeFilterRequest" for more than 9 minutes).
Then the service is stopped and members are removed from the cluster ("Removed missing member" error).
But actually the Ownership conflict message is logged at WARN level.
Regards
Harry
InvokeFilterRequest log
================
WARN|Logger@9240084 3.5.2/463||||2010-05-06 11:43:35.437/1242.324 Oracle Coherence GE 3.5.2/463 <Warning> (thread=Recovery Thread, member=1): A worker thread has been executing task: Message "InvokeFilterRequest"
FromMember=Member(Id=11, Timestamp=2010-05-06 11:29:42.623, Address=**.*.**.***:8093, MachineId=16170, Location=site:xxxx,process:MyProcess_Client,member:MyProcess_Client_11, Role=MyProcess_Client runner)
FromMessageId=373341
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=29
ToPollId=0
Poll=null
Packets
[000]=Directed{PacketType=0x0DDF00D5, ToId=1, FromId=11, Direction=Incoming, ReceivedMillis=11:34:35.522, ToMemberSet=null, ServiceId=2, MessageType=29, FromMessageId=373341, ToMessageId=36817, MessagePartCount=1, MessagePartIndex=0, NackInProgress=false, ResendScheduled=none, Timeout=none, PendingResendSkips=0, DeliveryState=unsent, Body=0x000041BD01000B01286D29F507000B01286D29F5040097CC9FD28DCA8001010101020000000000000000000000000000000000000000000000007FFFFFC00000000000000000000000010A27636F6D2E74616E676F736F6C2E7574696C2E66696C7465722E436F6E7461696E7346696C7465720A2F636F6D2E74616E676F736F..., Body.length=1394}
Service=DistributedCache{Name=UnlimitedDistributedAppScheme, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=26, BackupPartitions=25}
ToMemberSet=MemberSet(Size=1, BitSetCount=2
Member(Id=1, Timestamp=2010-05-06 11:23:08.003, Address=**.*.**.***:8089, MachineId=16170, Location=xxx,process:MyProcess_Server,member:MyProcess_Server_12, Role=MyProcess_Server Engine)
NotifySent=false
} for 539882ms and appears to be stuck; attempting to interrupt: UnlimitedDistributedAppSchemeWorker:0|Coherence
2010-05-06 11:43:35,962|INFO|Logger@9240084 3.5.2/463||||2010-05-06 11:43:35.955/1242.842 Oracle Coherence GE 3.5.2/463 <Info> (thread=pool-1-thread-3, member=1): Restarting NamedCache: MyProcess-cache|Coherence
2010-05-06 11:43:35,962|INFO|Logger@9240084 3.5.2/463||||2010-05-06 11:43:35.955/1242.842 Oracle Coherence GE 3.5.2/463 <Info> (thread=pool-1-thread-3, member=1): Restarting Service: UnlimitedDistributedAppScheme|Coherence
Removed missing membe log
====================
ERROR|Logger@9242415 3.5.2/463||||2010-05-06 18:53:07.289/4414.752 Oracle Coherence GE 3.5.2/463 <Error> (thread=DistributedCache:UnlimitedDistributedAppScheme, member=6): validatePolls: Removed missing member Member(Id=63, Timestamp=2010-05-06 17:39:46.382, Address=**.***.***.**:8088, MachineId=40700, Location=machine:xxx,process:MyProcess_Server,member:MyProcess_Server_64, Role=MyProcess_Server Engine) from poll: Poll