Fast online processing and heavy data crunching in cluster

Question

Hi,
I have a Coherence cluster and two types of operations I need to perform:
1. Lightweight online processing e.g. updating user profiles etc. This type of processing is invoked by web application and must be very fast, so that application is responsive for the users.
2. Heavy data crunching (data aggregation). This type of processing must occur on hourly basis.
Both types of operations work on the same data.
I would like to minimize the impact of data crunching on online processing. What is the best way of doing it?
Is is possible to "dedicate" part of servers to do the fast, online work and run heavy tasks on other part of cluster?
I can imagine using active-passive sort of architecture, where data crunching is done on passive cluster. Is there any other option, avoiding active-passive pattern?
Best regards,
Jarek

Answer 1

Hi Jarek,
It may be simpler to use different services to dedicate different servers to different tasks, for example:
cache-config.xml:
<?xml version="1.0"?>
<!DOCTYPE cache-config SYSTEM "cache-config.dtd">
<cache-config>
    <caching-scheme-mapping>
        <cache-mapping>
            <cache-name>processing</cache-name>
            <scheme-name>processing</scheme-name>
        </cache-mapping>
        <cache-mapping>
            <cache-name>aggregation</cache-name>
            <scheme-name>aggregation</scheme-name>
        </cache-mapping>
    </caching-scheme-mapping>
    <caching-schemes>
        <distributed-scheme>
            <scheme-name>processing</scheme-name>
            <local-storage system-property="processing.localstorage">false</local-storage>
            <service-name>processing</service-name>
            <backing-map-scheme>
                <read-write-backing-map-scheme>
                    <internal-cache-scheme>
                        <local-scheme/>
                    </internal-cache-scheme>
                    <cachestore-scheme>
                        <class-scheme>
                            <class-name>MyCacheStore</class-name>
                            <init-params>
                                <init-param>
                                    <param-type>string</param-type>
                                    <param-value>aggregation</param-value>
                                </init-param>
                            </init-params>
                        </class-scheme>
                    </cachestore-scheme>
                </read-write-backing-map-scheme>
            </backing-map-scheme>
            <autostart>true</autostart>
        </distributed-scheme>
        <distributed-scheme>
            <scheme-name>aggregation</scheme-name>
            <local-storage system-property="aggregation.localstorage">false</local-storage>
            <service-name>aggregation</service-name>
            <backing-map-scheme>
                <local-scheme/>
            </backing-map-scheme>
            <autostart>true</autostart>
        </distributed-scheme>
    </caching-schemes>
</cache-config>MyCacheStore.java:
import com.tangosol.net.cache.MapCacheStore;
import com.tangosol.net.CacheFactory;
* dimitri Feb 22, 2010 7:28:39 PM
public class MyCacheStore extends MapCacheStore
    public MyCacheStore(String sCacheName)
        super(CacheFactory.getCache(sCacheName));
    }and run servers dedicated to online processing with -Dprocessing.localstorage=true -Daggregation.locastorage=false and servers dedicated to aggregation with -Dprocessing.localstorage=false -Daggregation.localstorage=true.
Regards,
Dimitri