PriorityTask ignored by version 3.6.0 but working with 3.5.3

Question

Hello,
The priority task is used for a few entry processors with Coherence 3.5.3 no longer work with 3.6.0.
I get a "DEBUG ... Interrupted PartitionedCache, Thread[DistributedCache:LoaderDistributedCache,5,Cluster] " 160 seconds after my processors started then a "ERROR ... Terminating guarded execution (due to hard timeout) of ..." 30 seconds after the DEBUG message.
To summarize, coherence breaks 190 seconds after my processors start.
Is there something I missunderstood about PriorityTask ?
Cyrille
Extract of the logs :
20:21:40.973 [Logger@4257704 3.6.0.0] INFO  Coherence - [Info] (thread=main, member=n/a): Loaded operational configuration from "jar:file:/ ... /WEB-INF/lib/coherence-3.6.0.0b17229.jar!/tangosol-coherence.xml"
20:21:41.004 [Logger@4257704 3.6.0.0] INFO  Coherence - [Info] (thread=main, member=n/a): Loaded operational overrides from "jar:file:/ ... /WEB-INF/lib/coherence-3.6.0.0b17229.jar!/tangosol-coherence-override-dev.xml"
20:21:41.019 [Logger@4257704 3.6.0.0] INFO  Coherence - [Info] (thread=main, member=n/a): Loaded operational overrides from "file:/ ... /WEB-INF/classes/tangosol-coherence-override.xml"
20:21:41.019 [Logger@4257704 3.6.0.0] INFO  Coherence - [Info] (thread=main, member=n/a): Loaded operational overrides from "file:/ ... /WEB-INF/classes/custom-mbeans.xml"
20:21:41.019 [Logger@4257704 3.6.0.0] DEBUG Coherence -
Oracle Coherence Version 3.6.0.0 Build 17229
Grid Edition: Development mode
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
20:21:41.113 [Logger@4257704 3.6.0.0] INFO  Coherence - [Info] (thread=main, member=n/a): Loaded Reporter configuration from "jar:file:/ ... /WEB-INF/lib/coherence-3.6.0.0b17229.jar!/reports/report-group.xml"
20:21:41.269 [Logger@4257704 3.6.0.0] INFO  Coherence - [Info] (thread=main, member=n/a): Loaded cache configuration from "file:/ ... /WEB-INF/classes/coherence-cache-config.xml"
20:21:46.488 [Logger@4257704 3.6.0.0] DEBUG Coherence - [D4] (thread=main, member=n/a): TCMP bound to /{my-ip}:8088 using SystemSocketProvider
20:21:50.051 [Logger@4257704 3.6.0.0] INFO  Coherence - [Info] (thread=Cluster, member=n/a): Created a new cluster "cluster-paris" with Member(Id=1, Timestamp=2010-07-29 20:21:46.504, Address={my-ip}:8088, MachineId=63254, Location=site:equities.net.intra,machine:MYDESKTOP,process:3620, Role=ApacheCatalinaStartupBootstrap, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=1) UID=0x0A1EE3160000012A1F704888F7161F98
20:21:50.067 [Logger@9228429 3.6.0.0] INFO  Coherence - [Info] (thread=main, member=n/a): Started cluster Name=cluster-paris
WellKnownAddressList(Size=1,
  WKA{Address={my-ip}, Port=8088}
MasterMemberSet
  ThisMember=Member(Id=1, Timestamp=2010-07-29 20:21:46.504, Address={my-ip}:8088, MachineId=63254, Location=site:equities.net.intra,machine:MYDESKTOP,process:3620, Role=ApacheCatalinaStartupBootstrap)
  OldestMember=Member(Id=1, Timestamp=2010-07-29 20:21:46.504, Address={my-ip}:8088, MachineId=63254, Location=site:equities.net.intra,machine:MYDESKTOP,process:3620, Role=ApacheCatalinaStartupBootstrap)
  ActualMemberSet=MemberSet(Size=1, BitSetCount=2
    Member(Id=1, Timestamp=2010-07-29 20:21:46.504, Address={my-ip}:8088, MachineId=63254, Location=site:equities.net.intra,machine:MYDESKTOP,process:3620, Role=ApacheCatalinaStartupBootstrap)
  RecycleMillis=1200000
  RecycleSet=MemberSet(Size=0, BitSetCount=0
TcpRing{Connections=[]}
IpMonitor{AddressListSize=0}
20:21:50.145 [Logger@9228429 3.6.0.0] DEBUG Coherence - [D5] (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
20:21:50.770 [Logger@9228429 3.6.0.0] INFO  Coherence - [Info] (thread=Cluster, member=1): Loaded POF configuration from "file:/ ... /WEB-INF/classes/pof-config.xml"
20:21:50.770 [Logger@9228429 3.6.0.0] INFO  Coherence - [Info] (thread=Cluster, member=1): Loaded included POF configuration from "jar:file:/ ... /WEB-INF/lib/coherence-3.6.0.0b17229.jar!/coherence-pof-config.xml"
20:21:51.160 [Logger@9228429 3.6.0.0] DEBUG Coherence - [D5] (thread=DistributedCache:MyObjectDistributedCache, member=1): Service MyObjectDistributedCache joined the cluster with senior service member 1
20:21:51.238 [Logger@9228429 3.6.0.0] DEBUG Coherence - [D5] (thread=DistributedCache:LoaderDistributedCache, member=1): Service LoaderDistributedCache joined the cluster with senior service member 1
20:21:51.457 [Logger@9228429 3.6.0.0] DEBUG Coherence - [D5] (thread=ReplicatedCache:ReferenceDataReplicatedCache, member=1): Service ReferenceDataReplicatedCache joined the cluster with senior service member 1
20:21:51.457 [Logger@9228429 3.6.0.0] INFO  Coherence - [Info] (thread=DefaultCacheServer, member=1):
Services
  ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.6, OldestMemberId=1}
  InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
  PartitionedCache{Name=MyObjectDistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}
  PartitionedCache{Name=LoaderDistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}
  ReplicatedCache{Name=ReferenceDataReplicatedCache, State=(SERVICE_STARTED), Id=4, Version=3.0, OldestMemberId=1}
Started DefaultCacheServer...
20:22:46.443 [Logger@9228429 3.6.0.0] INFO  Coherence - [Info] (thread=xxx (LoaderDistributedCacheWorker:2), member=1): Loaded POF configuration from "file:/ ... /WEB-INF/classes/pof-config.xml"
20:22:46.458 [Logger@9228429 3.6.0.0] INFO  Coherence - [Info] (thread=xxx (LoaderDistributedCacheWorker:2), member=1): Loaded included POF configuration from "jar:file:/ ... /WEB-INF/lib/coherence-3.6.0.0b17229.jar!/coherence-pof-config.xml"
2010-07-29 20:23:45.194 - LoaderDistributedCacheWorker:0 - > process(...), Value=null, State=})
2010-07-29 20:23:45.194 - LoaderDistributedCacheWorker:3 - > process(...), Value=null, State=})
2010-07-29 20:23:45.194 - LoaderDistributedCacheWorker:1 - > process(...), Value=null, State=})
2010-07-29 20:23:45.209 - LoaderDistributedCacheWorker:4 - > process(...), Value=null, State=})
2010-07-29 20:23:45.209 - LoaderDistributedCacheWorker:2 - > process(...), Value=null, State=})
2010-07-29 20:23:45.209 - LoaderDistributedCacheWorker:9 - > process(...), Value=null, State=})
2010-07-29 20:23:45.209 - LoaderDistributedCacheWorker:7 - > process(...), Value=null, State=})
2010-07-29 20:23:45.209 - LoaderDistributedCacheWorker:6 - > process(...), Value=null, State=})
2010-07-29 20:23:45.194 - LoaderDistributedCacheWorker:8 - > process(...), Value=null, State=})
2010-07-29 20:23:45.194 - LoaderDistributedCacheWorker:5 - > process(...), Value=null, State=})
20:26:25.806 [Logger@9228429 3.6.0.0] DEBUG Coherence - Interrupted PartitionedCache, Thread[DistributedCache:LoaderDistributedCache,5,Cluster]
20:26:25.806 [Logger@9228429 3.6.0.0] ERROR Coherence - [Error] (thread=Cluster, member=1): Attempting recovery (due to soft timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:LoaderDistributedCache} Service=PartitionedCache{Name=LoaderDistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}}
20:26:56.244 [Logger@9228429 3.6.0.0] ERROR Coherence - [Error] (thread=Cluster, member=1): Terminating guarded execution (due to hard timeout) of {WrapperGuardable Guard{Daemon=DistributedCache:LoaderDistributedCache} Service=PartitionedCache{Name=LoaderDistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}}
20:26:56.260 [Logger@9228429 3.6.0.0] ERROR Coherence - [Error] (thread=Termination Thread, member=1): Full Thread Dump
... thread dump
2010-07-29 20:26:56.26 - LoaderDistributedCacheWorker:5 - Interrupted during sleep
... other processors are interruptedCode of the TestCase processor
public class MockLoader implements EntryProcessor, PortableObject, PriorityTask {
  private static final long serialVersionUID = 1L;
  private String id;
  private long sleepDurationInMillis = 10 * 60 * 1000;
  public Object getCacheKey() {
    return MockLoader.class.getName() + "-" + id;
  public MockLoader() {
    super();
  public MockLoader(String id) {
    super();
    this.id = id;
  public Object process(Entry entry) {
    System.err.println(new Timestamp(System.currentTimeMillis()) + " - " + Thread.currentThread().getName() + " - > process(" + entry + ")");
    if (abandonned) {
      System.err.println(new Timestamp(System.currentTimeMillis()) + " - " + Thread.currentThread().getName() + " - skipped abandonned");
      return null;
    long sleptDuration = 0;
    while (sleptDuration <= sleepDurationInMillis) {
      if (Thread.interrupted()) {
        break;
      try {
        Thread.sleep(100);
      } catch (InterruptedException e) {
        System.err.println(new Timestamp(System.currentTimeMillis()) + " - " + Thread.currentThread().getName() + " - Interrupted during sleep");
      sleptDuration += 100;
    if (Thread.interrupted()) {
      System.err.println(new Timestamp(System.currentTimeMillis()) + " - " + Thread.currentThread().getName() + " - Abort interrupted task");
    System.err.println(new Timestamp(System.currentTimeMillis()) + " - " + Thread.currentThread().getName() + " - < process(" + entry + ")");
    return null;
  @SuppressWarnings("unchecked")
  public Map processAll(Set entries) {
    Map results = new HashMap(entries.size());
    for (Entry entry : (Set<Entry>) entries) {
      Object result = process(entry);
      results.put(entry.getKey(), result);
    return results;
  public void readExternal(PofReader in) throws IOException {
    this.id = in.readString(0);
    this.sleepDurationInMillis = in.readLong(1);
  public void writeExternal(PofWriter out) throws IOException {
    out.writeString(0, this.id);
    out.writeLong(1, this.sleepDurationInMillis);
  public long getExecutionTimeoutMillis() {
    return PriorityTask.TIMEOUT_DEFAULT;
  public long getRequestTimeoutMillis() {
    return PriorityTask.TIMEOUT_DEFAULT;
  public int getSchedulingPriority() {
    return PriorityTask.SCHEDULE_STANDARD;
  private boolean abandonned;
  public void runCanceled(boolean abandonned) {
    System.err.println("runCanceled(" + abandonned + ")");
    this.abandonned = abandonned;
}

Answer 1

Hi Cyrille,
This bug (COH-3466) was indeed fixed in 3.5.3 patch 7 and the fix will be included in the first patch for 3.6.
Regards,
Gene