Generate Events based on Cache size

Question

Hi,
I have a use case where I need to generate events when the size of the cache has reached a specific value. The cache will be highly transactions (lot of inserts and deletes totaling to 6 million transactions per day). Also, I cannot limit the size of the cache as I don't want any data to be lost (trying to insert when cache size has reached its limit).
One of the ways I could do this is to use a MapListener to generate an event on every insert to the Cache, and then check for size each time to decide if the further processing needs to be done. With the nature of the traffic in the cache, the items in the cache are to be processed when the size threshold is reached and then deleted. But, I'm not sure how much of an overhead these events and listeners will have (with 6 million items going through the cache each day)
Is there any other more efficient way to do this? I have looked at Continuous Query cache which uses a filter based on a cache. But, I'm not able to figure out a way to get the size of the cache using a filter.
Any pointers are helpful
Thanks in advance
Regards
Vikas
Edited by: vikascv on Nov 17, 2009 8:48 AM

Answer 1

Hi Vikas,
Typically a size type calculation is the cumulative result of all storage enabled nodes, i.e. a Count aggregation. In your use case you want to be made aware of changes to an aggregation but as an aggregation is point in time of request opposed to continually being updated it does not fit your use case. One option is to create a MapEventTransformer that you register with the cache which increments a counters cache (on a separate service) using an EntryProcessor, i.e.
MapEventTransformer...
public class CounterMapEventTransformer implements MapEventTransformer, PortableObject
    String m_cacheName;
    public CounterMapEventTransformer()
    public CounterMapEventTransformer(final String cacheName)
        m_cacheName = cacheName;
    public MapEvent transform(MapEvent event)
        Long prevCount    = null;
        Long currentCount = null;
        switch( event.getId() )
            case( MapEvent.ENTRY_INSERTED ):
                // perform counter++
                currentCount = (Long) CacheFactory.getCache("counters").invoke(m_cacheName, new CounterOp(true));
                break;
            case( MapEvent.ENTRY_DELETED ):
                // perform counter--
                currentCount = (Long) CacheFactory.getCache("counters").invoke(m_cacheName, new CounterOp(false));
                break;
        MapEvent newEvent = new MapEvent(event.getMap(), event.getId(), event.getKey(), currentCount-1, currentCount);
        return newEvent;
    public void writeExternal(PofWriter out) throws IOException
        out.writeString(0, m_cacheName);
    public void readExternal(PofReader in) throws IOException
        m_cacheName = in.readString(0);
    }Incrementing EP...
public class CounterOp extends AbstractProcessor implements PortableObject
    boolean m_increment;
    public CounterOp()
    public CounterOp(boolean increment)
        m_increment = increment;
    @Override
    public Object process(Entry entry)
        Long newVal = null;
        if( !entry.isPresent() )
            entry.setValue(newVal = Long.valueOf(1));
        else
            newVal = ((Long)entry.getValue()).longValue() + (isIncrement() ? 1 : -1);
            entry.setValue( newVal );
        return newVal;
    protected boolean isIncrement()
        return m_increment;
    public void writeExternal(PofWriter out) throws IOException
        out.writeBoolean(0, m_increment);
    public void readExternal(PofReader in) throws IOException
        m_increment = in.readBoolean(0);
    }cache-config
     <!--
          | listener-cacheconfig.xml | | Copyright 2001-2009 by Oracle. All
          rights reserved. | | Oracle is a registered trademarks of Oracle
          Corporation and/or its | affiliates. | | This software is the
          confidential and proprietary information of Oracle | Corporation. You
          shall not disclose such confidential and proprietary | information and
          shall use it only in accordance with the terms of the | license
          agreement you entered into with Oracle. | | This notice may not be
          removed or altered.
     -->
<!DOCTYPE cache-config SYSTEM "cache-config.dtd">
<cache-config>
     <caching-scheme-mapping>
          <cache-mapping>
               <cache-name>dist-*</cache-name>
               <scheme-name>distributed-scheme</scheme-name>
          </cache-mapping>
          <cache-mapping>
               <cache-name>counters</cache-name>
               <scheme-name>counter-scheme</scheme-name>
          </cache-mapping>
     </caching-scheme-mapping>
     <caching-schemes>
          <!--
    Distributed caching schemes
    -->
          <distributed-scheme>
               <scheme-name>distributed-scheme</scheme-name>
               <service-name>DistributedCache</service-name>
               <serializer>
                    <class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
                    <init-params>
                         <init-param>
                              <param-type>string</param-type>
                              <param-value>pof-config.xml</param-value>
                         </init-param>
                    </init-params>
               </serializer>
               <backing-map-scheme>
                    <local-scheme />
                    <!--
                         <read-write-backing-map-scheme> <internal-cache-scheme>
                         <local-scheme /> </internal-cache-scheme> <cachestore-scheme>
                         <class-scheme>
                         <class-name>com.oracle.coherence.cachecounter.store.CacheCounterStore</class-name>
                         </class-scheme> </cachestore-scheme>
                         </read-write-backing-map-scheme>
                    -->
               </backing-map-scheme>
               <autostart>true</autostart>
          </distributed-scheme>
          <distributed-scheme>
               <scheme-name>counter-scheme</scheme-name>
               <service-name>CounterDistributedCache</service-name>
               <serializer>
                    <class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
                    <init-params>
                         <init-param>
                              <param-type>string</param-type>
                              <param-value>pof-config.xml</param-value>
                         </init-param>
                    </init-params>
               </serializer>
               <backing-map-scheme>
                    <local-scheme />
               </backing-map-scheme>
               <autostart>true</autostart>
          </distributed-scheme>
     </caching-schemes>
</cache-config>pof-config...
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE pof-config SYSTEM "pof-config.dtd">
<pof-config>     
   <user-type-list>          
      <!-- include all of the standard Coherence type -->
      <include>coherence-pof-config.xml</include>          
      <!-- our custom types (user types) 1000+ -->          
      <user-type>                 
         <type-id>1000</type-id>            
         <class-name>com.oracle.coherence.cachecounter.event.CounterMapEventTransformer</class-name>          
      </user-type>
      <user-type>                 
         <type-id>1001</type-id>            
         <class-name>com.oracle.coherence.cachecounter.processors.CounterOp</class-name>          
      </user-type>
   </user-type-list>
   <allow-interfaces>true</allow-interfaces>
   <allow-subclasses>true</allow-subclasses>
</pof-config>Test class
public class MapEventTransformerTest
    public static final String m_cacheName = "dist-test1";
    public static void setUp()
        System.setProperty("tangosol.coherence.cacheconfig", "counter-cacheconfig.xml");
        System.setProperty("tangosol.coherence.distributed.localstorage", "false");
    public static void loadVals()
        final NamedCache cache = CacheFactory.getCache(m_cacheName);
        final int MAX_SIZE = 100;
        for( int i=0; i<MAX_SIZE; i++ )
            cache.put(i, "msg "+i);
    public static void removeVals()
        final NamedCache cache = CacheFactory.getCache(m_cacheName);
        final int MAX_SIZE = 100;
        for( int i=0; i<MAX_SIZE; i++ )
            if( i%2==0 )
                cache.remove(i);
    public static void testTransformer()
        final NamedCache cache = CacheFactory.getCache(m_cacheName);
        cache.addMapListener(new CounterAwareListener(),
                new MapEventTransformerFilter(AlwaysFilter.INSTANCE, new CounterMapEventTransformer(m_cacheName)),
                false);
    public static void main(String[] args) throws IOException
        setUp();
        testTransformer();
        loadVals();
        removeVals();
        System.out.println("press a key to end...");
        System.in.read();
    public static class CounterAwareListener extends MultiplexingMapListener
        @Override
        protected void onMapEvent(MapEvent evt)
            System.out.format( "# of items in cache: %d\n", evt.getNewValue() );
    }Note: Events could be received out of order but for your use case where you are interested in a threshold you may not care! -- food for thought :)
--harvey