Clear cache operation blows up the cluster

Question

Hi,
We are running a 9 node coherence 3.6 cluster and every night we have a java process that starts up, issues a cache.clear() command on each cache and terminates itself. This appears to be causing adverse effects on the cluster health. Right after this process enters and leaves, several nodes run out of memory and quits. See error below
2011-12-07 03:05:18.786/57714.576 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=1): Service guardian is 23184ms late, indicating that this JVM may be running slowly or experienced a long GC
2011-12-07 03:05:20.518/57716.308 Oracle Coherence GE 3.6.1.0 <Error> (thread=DistributedCache:OA-DistributedCache, member=1): Terminating PartitionedCache due to unhandled exception: java.lang.OutOfMemoryError
2011-12-07 03:05:20.518/57716.308 Oracle Coherence GE 3.6.1.0 <Error> (thread=DistributedCache:OA-DistributedCache, member=1):
java.lang.OutOfMemoryError: Java heap space
     at com.tangosol.coherence.component.net.memberSet.ActualMemberSet.setMember(ActualMemberSet.CDB:11)
     at com.tangosol.coherence.component.net.memberSet.ActualMemberSet.add(ActualMemberSet.CDB:6)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.getOwnershipMemberSet(PartitionedService.CDB:13)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.getOwnershipSenior(PartitionedService.CDB:10)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.checkDistribution(PartitionedService.CDB:71)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:15)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)
     at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
     at java.lang.Thread.run(Unknown Source)
This creates a ripple effect to all members and the cluster goes down after thsi happens. Can you please let me know if you have seen this or what might be causing this?
Sairam
Edited by: SKR on Dec 7, 2011 9:47 AM

Answer 1

It is a simple java application that iterates through a list of cache names and issues clear() on each of them.
      * This method permanently clears the cache
      * specified by name <cacheName> and returns the
      * number of rows cleared
      * @param cacheName
      * @return the number of objects cleared after this operation
     public static long clearCache(String cacheName){
          NamedCache cache = CacheFactory.getCache(cacheName);
          long size = cache.size();
          cache.clear();
          return size;
      * Main method with the arguments to use the utility
      * the first argumeent args[0] will specify the operation and the
      * second argument args[1] will specify the value required for that operation
      * @param args
     public static void main(String args[]){
          int argsCount = args.length;
          if (argsCount==2 && args[0].equalsIgnoreCase("-clearCache")){
               String[] tokens= args[1].split(",");
               for (int i = 0; i < tokens.length; i++) {
                    String cacheName = tokens;
                    long nbrOfRowsCleared = clearCache(cacheName);
                    if(nbrOfRowsCleared>0){
                         System.out.println("Cleared all objects from cache: "+ cacheName);
                         System.out.println("Number of objects cleared = "+ nbrOfRowsCleared);
                    }else{
                         System.out.println( "No objects found in cache "+ cacheName +". Please verify if the cache name is correct");     
          }else{
               System.err.println("Invalid arguments. Valid arguments are \n" +
                         "-clearCache <comma separated list of cache names>: Clears the contents of the caches specified by a comma separated list of cache names without spaces. Warning: Cache names are case sensitive" +
                         " \n For Eg: -clearCache Option,Equity,Account");
               return;
Edited by: SKR on Dec 7, 2011 12:08 PM