How to force write-behind store on cache node shutdown?

Hi,
I built a small pilot project based on Coherence and now I test it for failover. I found replication issues with Distributed cache in the following scenario:
- start cache node 1 (JVM instance 1);
- connect Extend client to it and get 1 object from cache (only 1 object in the cache - loaded by CacheStore from DB);
- change the object and put it back (I use EntryProcessor for this);
- start cache node 2 (JVM instance 2);
- stop cache instance 1 (write-behind store wasn't invoked yet: write-delay = 2m);
- load/change the same object on node 2; all changes done on node 1 are lost.
My expectation was that cache will replicate its data between nodes when new member joins cache cluster. The backup count = 1 by default, right?
What should I do in order to prevent such behavior? Is it possible to force write-behind store on cache node shutdown event?
Thanks, Denis.
My cache-config, just in case:
<cache-config>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>AccountCache</cache-name>
<scheme-name>account-distributed</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<distributed-scheme>
<scheme-name>account-distributed</scheme-name>
<service-name>DistributedCache</service-name>
<serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>account-pof-config.xml</param-value>
</init-param>
</init-params>
</serializer>
<backing-map-scheme>
<read-write-backing-map-scheme>
<scheme-name>AccountDatabaseScheme</scheme-name>
<internal-cache-scheme>
<local-scheme>
<!--scheme-ref>default-eviction</scheme-ref-->
<eviction-policy>LRU</eviction-policy>
<high-units>0</high-units>
<expiry-delay>30m</expiry-delay>
</local-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>com.roox.bss.cache.store.AccountCacheStore</class-name>
<init-params>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>dburl_</param-value>
</init-param>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>user</param-value>
</init-param>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>password</param-value>
</init-param>
</init-params>
</class-scheme>
</cachestore-scheme>
<write-delay>2m</write-delay>
<write-batch-factor>.5</write-batch-factor>
</read-write-backing-map-scheme>
</backing-map-scheme>
</distributed-scheme>
<proxy-scheme>
<service-name>ExtendTcpProxyService</service-name>
<thread-count>10</thread-count>
<acceptor-config>
<tcp-acceptor>
<local-address>
<address>localhost</address>
<port>9098</port>
<reuse-address>true</reuse-address>
<reusable>true</reusable>
</local-address>
</tcp-acceptor>
<serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>account-pof-config.xml</param-value>
</init-param>
</init-params>
</serializer>
</acceptor-config>
<autostart>true</autostart>
</proxy-scheme>
</caching-schemes>
</cache-config>

solved with autostart=true

Similar Messages

  • How to limit Write-Behind batch

    We have a scenario:
    we use read-write-backing-map-scheme having write-delay 60s.
    System insert a lot of data and then time comes to write data coherence find 40-50 k of unsaved record and pass them all to cachestore.
    Due to data volume or database busyness cashstore may work some time several seconds for instance.
    <read-write-backing-map-scheme>
    <scheme-name>TicketDatabaseScheme</scheme-name>
    <scheme-ref>DefaultDatabaseScheme</scheme-ref>
    <!--<write-delay>1m</write-delay>-->
    <cachestore-scheme>
    <class-scheme>
    <class-name>com.griddynamics.ticketon.app.dao.coherence.TicketCacheStore</class-name>
    </class-scheme>
    </cachestore-scheme>
    </read-write-backing-map-scheme>
    <read-write-backing-map-scheme>
    <scheme-name>DefaultDatabaseScheme</scheme-name>
    <internal-cache-scheme>
    <local-scheme>
    <scheme-ref>LocalScheme</scheme-ref>
    </local-scheme>
    </internal-cache-scheme>
    <write-delay>60s</write-delay>
    </read-write-backing-map-scheme>
    In the case we experience "Terminating guarded execution" followed by service termination.
    2010-02-11 09:26:52.223/511.457 Oracle Coherence GE 3.5.2/463 <Error> (thread=DistributedCache:TicketonCache, member=2): Terminating guarded execution (due to hard timeout 1924ms ago) of Daemon{Thread="Thread[WriteBehindThread:CacheStoreWrapper(com.griddynamics.ticketon.app.dao.coherence.TicketCacheStore),5,WriteBehindThread:CacheStoreWrapper(com.griddynamics.ticketon.app.dao.coherence.TicketCacheStore)]", State=Running}
    2010-02-11 09:26:52.225/511.459 Oracle Coherence GE 3.5.2/463 <Error> (thread=Termination Thread, member=2): Write-behind thread timed out; stopping the cache service
    2010-02-11 09:26:52.226/511.460 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache:TicketonCache, member=2): Service TicketonCache left the cluster
    INFO 09:26:52,227 [http--80-22$27432016 DaoCoherenceImpl] - PROFILE_doCreatetickets putAll 200 tickets time 3444 time per 10 objects 172
    INFO 09:26:52,227 [http--80-22$27432016 DaoCoherenceImpl] - PROFILE event and 1000 tickets ctreated time 9668
    Broadcast Message from root (msglog) on ip-10-226-137-172 Thu Feb 11 09:57:18...ets putAll 200 tickets time 3365 time per 10 objects 168
    2010-02-11 09:26:52.228/511.462 Oracle Coherence GE 3.5.2/463 <Info> (thread=httTHE SYSTEM ip-10-226-137-172 IS BEING SHUT DOWN NOW ! ! !et
    Log off now or risk your files being damagedence GE 3.5.2/463 <Info> (thread=http--80-27$25787595, member=2): Restarting Service: TicketonCache
    INFO 09:26:52,229 [http--80-20$15974570 DaoCoherenceImpl] - PROFILE_doCreatetickets putAll 200 tickets time 3447 time per 10 objects 172
    INFO 09:26:52,289 [http--80-22$26935588 BackingBeanSuper] - request HttpRequest[22]
    [09:26:53.446] {http--80-35$24027494} java.lang.RuntimeException: Failed to start Service "TicketonCache" (ServiceState=SERVICE_STOPPED)
    [09:26:53.446] {http--80-35$24027494} at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
    Resume: coherence crashes in case far from fatal.
    Questions
    1. May i limit the size of a batch passed to cachestore?
    2. Is it possible to configure the timeout "(due to hard timeout 1924ms ago)"
    3. Is it possible to handle like this some way to prevent self killing of coherence cluster.

    Thank you Mark you are vary helpfull
    Did you mean that (lower) by "bundle strategy"?
    <cachestore-scheme>
    <class-scheme>
    <class-name>com.griddynamics.ticketon.app.dao.coherence.TicketCacheStore</class-name>
    </class-scheme>
    <operation-bundling>
    <bundle-config>
    <operation-name>store</operation-name>
    <preferred-size>5000</preferred-size>
    <auto-adjust>true</auto-adjust>
    </bundle-config>
    </operation-bundling>
    </cachestore-scheme>
    And if yes is it looks sense?
    I mean by this, "send records to TicketCacheStore by 5000 per call " am i right?
    I dropped delay to 10s and set factor to 0.5
    Not coherece send me 5-20k records and cachestore handle whis successfuly.
    But! By diferent means it may work longer sometimes, some lock in database for instance.
    I want to find durable solution for the case, not only lower a chance i meet one.
    Issuing heartbeat from cachestore looks best for me now.
    I find that default guardian timeout is 65s and it is not looks as good idea to make it higher.

  • Mac: How to force Dw to store site (locally) where I want it?

    I am new to Dw CS5. I run a Mac mini, small internal HD. I store ALL my data and files on a large RAID 0 external Firewire HD. Dw CS5 seems to insist on storing a new site on the internal HD, While I accept that storing the site on my internal HD would work, I find it inconvenient having data on the internal drive for several reasons.
    I have tried to force Dw to store the new dreamweaver site folder in my chosen folder "Web Sites" on my external drive but it complains that " The Spry assets folder is not inside this site. Please select a valid folder in your local site root."
    Perhaps I am missing something here but I can't see why I can't have the local data on an external drive in a folder of my choice.
    Any suggestions and comments please.
    I used Dw3 years ago but this is a different beast!

    Well, you already have created content before defining a site. You have created a page. That's fine.
    Make sure it is sitting in a folder. If it is not already in a folder with nothing but site files in it, make a folder on your Firewire drive and put your file(s) into it.
    Then, when you define the site, make that your local root folder in the site definition.
    I know what you mean...the beginning seems pretty confusing. But it's pretty simple.
    You need to have a folder that will contain all of your site files. Any site files you have created before actually defining the site need to go into it.
    That root folder is the "universe" that will be uploaded to your server eventually, and maintained locally on your Firewire drive. When you place images on your pages, you will put copies of them into, for instance, an "images" folder in your site while linking them to your pages. Dreamweaver will do a great job of keeping track of where files are within that universe.
    A second folder...outside the root folder...that I find useful is one I call "working," where I store my original content files that I have collected for the site...images that may or may not have been optimized yet, text content, and so forth. I don't keep them inside my site root folder because they will not be on the web, but I like to keep them handy.
    Because I have more than one client, I will put both the site root folder and the working folder into a client folder, to keep my client material organized. But that is my system. Except for the need for a root folder for your site, you will develop your own system of organizing your materials.
    To be a little more clear, here is the organization I am describing:
    Computer
    Firewire drive
         Client A's folder
              Site folder
                   images folder
                   pages
              "Working" folder
         Client B's folder
              Site folder
                   images folder
                   pages
              "Working" folder
    etc.
    I think you'll do fine with the software...I know you're eager to get on to working on the site!
    Beth

  • Can a db slowdown with write-behind cause a slowdown in cache operations?

    If we have a coherence cluster, and one cache configured with write-behind is having trouble writing to the db (ie, it's slow), and we keep adding objects to the cache that exceed the ability of the db to consume them; will flow-control kick in and cause the writes to the cache to block/slow-down? Ie, the classic producer-consumer problem, where we are adding objects to the cache, faster than the cachestore can consume them.
    What happens in this case? Will flow-control kick in and block writes to the cache? Will an internal buffer just keep growing? Are there any knobs to tweak this behavior (eg, in the case of spikes, where temporarily the producer is producing faster than the consumer can consume for a brief period of time, but then things go back to normal)?

    user9222505 wrote:
    I believe we discovered that the same thread pool is used for all requests to the cache, including gets, puts and calls into the cachestore. So if the writes are slow within the cachestore, then it uses up all of the threads and slows everything down.Hi,
    This is not really correct.
    If a cache in a service is configured to use write-behind then a separate thread for that service is started, which deals with write-behind store and storeAll operations.
    The remove operations need to be handled synchronously to avoid corruption of the data-set in the scenario of reading a entry from the cache immediately after removing it (if it were not synchronously deleted from the backing storage, then reading it back could give an incorrect non-null value). Therefore remove operations are handled synchronously on the service / worker thread, and not delayed on the write-behind thread.
    Gets are also synchronously handled, so they again are served on the service / worker thread.
    So if the puts are slow and wait too much, that may delay other puts but should not contend with other threads. If the puts are computation intensive, then obviously they hinder other threads because of consumption of the same CPU resource, and not simply because they execute.
    Best regards,
    Robert

  • Write behind db errors

    Hi, so we have a system where we end up with failures from the WriteBehindThread for various reasons. It seems that the write is re-attempted over and over, hundreds of thousands of times per day, over several days.
    Is there any way to stop this from happening; after a "while", or after a set number of attempts?
    We have write-requeue-threshold set to 20, but i'm not entirely sure how this setting is applied. Eg, we have N storage nodes, so will each storage node keep up to 20 writes to retry per cache-store? And if we do have 20 that fail, but no more fail, those 20 will just keep getting retried over and over indefinitely?
    Thx.

    Failed write-behind store operations are not retried by default. Requeuing is enabled by setting <write-requeue-threshold> to a value greater than 0.
    The <write-requeue-threshold> value is the maximum number of entries allowed to exist in the queue upon failure. If zero, write-behind requeuing
    is disabled.
    Any idea as to why the store operations fail?
    /Mark
    Oracle Coherence

  • TTL specified in put operation doesn't always work when using write-behind

    I'm using a distributed cache with a write-behind cache store (see the config below). I found that when I do something like myCache.put(key, value, ttl), the entry survives the specified ttl. I tried doing the same with a distributed cache with a write-through cachestore and there everything does happen correctly.
    Is this sort of operation not permitted in caches containing a write-behind cachestore? If not wouldn't it be better to throw an UnsupportedOperationException.
    I created a small test to simulate this. I added values to the cache with a TTL of 1 to 10 seconds and found that the 10 second entries stayed in the cache.
    Configuration used:
    <?xml version="1.0"?>
    <!DOCTYPE cache-config SYSTEM "cache-config.dtd">
    <cache-config>
         <caching-scheme-mapping>
              <cache-mapping>
                   <cache-name>TTL_TEST</cache-name>
                   <scheme-name>testScheme</scheme-name>
              </cache-mapping>
         </caching-scheme-mapping>
         <caching-schemes>
              <distributed-scheme>
                   <scheme-name>testScheme</scheme-name>
                   <service-name>testService</service-name>
                   <backing-map-scheme>
                        <read-write-backing-map-scheme>
                             <internal-cache-scheme>
                                  <local-scheme>
                                       <service-name>testBackLocalService</service-name>
                                  </local-scheme>
                             </internal-cache-scheme>
                             <cachestore-scheme>
                                  <class-scheme>
                                       <scheme-name>testBackStore</scheme-name>
                                       <class-name>TTLTestServer$TestCacheStore</class-name>
                                  </class-scheme>
                             </cachestore-scheme>
                             <write-delay>3s</write-delay>
                        </read-write-backing-map-scheme>
                   </backing-map-scheme>
                   <local-storage>true</local-storage>
                   <autostart>true</autostart>
              </distributed-scheme>
         </caching-schemes>
    </cache-config>Code of test:
    import java.util.Collection;
    import java.util.List;
    import java.util.Map;
    import java.util.concurrent.Callable;
    import java.util.concurrent.ExecutionException;
    import java.util.concurrent.ExecutorService;
    import java.util.concurrent.Executors;
    import java.util.concurrent.Future;
    import org.joda.time.DateTime;
    import org.joda.time.Duration;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    import org.springframework.util.StopWatch;
    import org.testng.annotations.BeforeClass;
    import org.testng.annotations.Test;
    import com.google.common.collect.Lists;
    import com.tangosol.net.CacheFactory;
    import com.tangosol.net.NamedCache;
    import com.tangosol.net.cache.CacheStore;
    @Test
    public class TTLTestServer
         private static final int RETRIES = 5;
         private static final Logger logger = LoggerFactory.getLogger( TTLTestServer.class );
         private NamedCache m_cache;
          * List of Time-To-Lives in seconds to check
         private final List<Integer> m_listOfTTLs = Lists.newArrayList(1, 3, 5, 10);
          * Test is done in separate threads to speed up the test
         private final  ExecutorService m_executorService = Executors.newCachedThreadPool();
         @BeforeClass
         public void setup()
              logger.info("Getting the cache");
              m_cache =  CacheFactory.getCache("TTL_TEST");
         public static class TestCacheStore implements CacheStore
              public void erase(Object arg0)
              public void eraseAll(Collection arg0)
              public void store(Object arg0, Object arg1)
              public void storeAll(Map arg0)
              public Object load(Object arg0)
              {return null;}
              public Map loadAll(Collection arg0)
              {return null;}
         public void testTTL() throws InterruptedException, ExecutionException
              logger.info("Starting TTL test");
              List<Future<StopWatch>> futures = Lists.newArrayList();
              for (final Integer ttl : m_listOfTTLs)
                   futures.add(m_executorService.submit(new Callable()
                        public Object call() throws Exception
                             StopWatch stopWatch= new StopWatch("TTL=" + ttl);
                             for (int retry = 0; retry < RETRIES; retry++)
                                  logger.info("Adding a value in cache for TTL={} in try={}", ttl, retry+1);
                                  stopWatch.start("Retry="+retry);
                                  m_cache.put(ttl, null, ttl*1000);
                                  waitUntilNotInCacheAnymore(ttl, retry);
                                  stopWatch.stop();
                             return stopWatch;
                        private void waitUntilNotInCacheAnymore(final Integer ttl, final int currentTry) throws InterruptedException
                             DateTime startTime = new DateTime();
                             long maxMillisToWait = ttl*2*1000;     //wait max 2 times the time of the ttl
                             while(m_cache.containsKey(ttl) )
                                  Duration timeTaken = new Duration(startTime, new DateTime());
                                  if(timeTaken.getMillis() > maxMillisToWait)
                                       throw new RuntimeException("Already waiting " + timeTaken + " for ttl=" + ttl + " and retry=" +  currentTry);
                                  Thread.sleep(1000);
              logger.info("Waiting until all futures are finished");
              m_executorService.shutdown();
              logger.info("Getting results from futures");
              for (Future<StopWatch> future : futures)
                   StopWatch sw = future.get();
                   logger.info(sw.prettyPrint());
    }Failure message:
    FAILED: testTTL
    java.util.concurrent.ExecutionException: java.lang.RuntimeException: Already waiting PT20.031S for ttl=10 and retry=0
         at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
         at java.util.concurrent.FutureTask.get(Unknown Source)
         at TTLTestServer.testTTL(TTLTestServer.java:159)
    Caused by: java.lang.RuntimeException: Already waiting PT20.031S for ttl=10 and retry=0
         at TTLTestServer$1.waitUntilNotInCacheAnymore(TTLTestServer.java:139)
         at TTLTestServer$1.call(TTLTestServer.java:122)
         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
         at java.util.concurrent.FutureTask.run(Unknown Source)
         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
         at java.lang.Thread.run(Unknown Source)I'm using Coherence 3.4.2.
    Best regards
    Jan

    Hi, still no luck. However, I noticed that setting the write-delay value of the write-behind store to 0s or 1s, solved the problem. It only starts to given me "the node has already been removed" excpetions once the write-delay value is 2s or higher.
    You can find the coherence-cache-config.xml below:
    <?xml version="1.0"?>
    <!DOCTYPE cache-config SYSTEM "cache-config.dtd">
    <cache-config>
         <caching-scheme-mapping>
              <cache-mapping>
                   <cache-name>TTL_TEST</cache-name>
                   <scheme-name>testScheme</scheme-name>
              </cache-mapping>
         </caching-scheme-mapping>
         <caching-schemes>
              <distributed-scheme>
                   <scheme-name>testScheme</scheme-name>
                   <service-name>testService</service-name>
                   <backing-map-scheme>
                        <read-write-backing-map-scheme>
                             <internal-cache-scheme>
                                  <local-scheme>
                                       <service-name>testBackLocalService</service-name>
                                  </local-scheme>
                             </internal-cache-scheme>
                             <cachestore-scheme>
                                  <class-scheme>
                                       <scheme-name>testBackStore</scheme-name>
                                       <class-name>TTLTestServer$TestCacheStore</class-name>
                                  </class-scheme>
                             </cachestore-scheme>
                             <write-delay>2s</write-delay>
                        </read-write-backing-map-scheme>
                   </backing-map-scheme>
                   <local-storage>true</local-storage>
                   <autostart>true</autostart>
              </distributed-scheme>
         </caching-schemes>
    </cache-config>You can find the test program below:
    import java.util.ArrayList;
    import java.util.Collection;
    import java.util.List;
    import java.util.Map;
    import org.joda.time.DateTime;
    import org.joda.time.Duration;
    import org.springframework.util.StopWatch;
    import com.tangosol.net.CacheFactory;
    import com.tangosol.net.NamedCache;
    import com.tangosol.net.cache.CacheStore;
    public class TTLTestServer
         private static final int RETRIES = 5;
         private NamedCache m_cache;
          * List of Time-To-Lives in seconds to check
         private final List<Integer> m_listOfTTLs = new ArrayList<Integer>();
          * @param args
          * @throws Exception
         public static void main( String[] args ) throws Exception
              new TTLTestServer().test();
          * Empty CacheStore
          * @author jbe
         public static class TestCacheStore implements CacheStore
              public void erase(Object arg0)
              @SuppressWarnings ( "unchecked" )
              public void eraseAll(Collection arg0)
              public void store(Object arg0, Object arg1)
              @SuppressWarnings ( "unchecked" )
              public void storeAll(Map arg0)
              public Object load(Object arg0)
              {return null;}
              @SuppressWarnings ( "unchecked" )
              public Map loadAll(Collection arg0)
              {return null;}
          * Sets up and executes the test setting values in a cache with a given time-to-live value and waiting for the value to disappear.
          * @throws Exception
         private void test() throws Exception
              System.out.println(new DateTime() + " - Setting up TTL test");
              m_cache =  CacheFactory.getCache("TTL_TEST");
              m_listOfTTLs.add( 1 );
              m_listOfTTLs.add( 3 );
              m_listOfTTLs.add( 5 );
              m_listOfTTLs.add( 10);
              System.out.println(new DateTime() + " - Starting TTL test");
              for (final Integer ttl : m_listOfTTLs)
                   StopWatch sw = doTest(ttl);
                   System.out.println(sw.prettyPrint());
          * Adds a value to the cache with the time-to-live as given by the ttl parameter and waits until it's removed from the cache.
          * Repeats this {@link #RETRIES} times
          * @param ttl
          * @return
          * @throws Exception
         private StopWatch doTest(Integer ttl) throws Exception
              StopWatch stopWatch= new StopWatch("TTL=" + ttl);
              for (int retry = 0; retry < RETRIES; retry++)
                   System.out.println(new DateTime() + " - Adding a value in cache for TTL=" + ttl + " in try= " + (retry+1));
                   stopWatch.start("Retry="+retry);
                   m_cache.put(ttl, null, ttl*1000);
                   waitUntilNotInCacheAnymore(ttl, retry);
                   stopWatch.stop();
              return stopWatch;
          * Wait until the value for the given ttl is not in the cache anymore
          * @param ttl
          * @param currentTry
          * @throws InterruptedException
         private void waitUntilNotInCacheAnymore(final Integer ttl, final int currentTry) throws InterruptedException
              DateTime startTime = new DateTime();
              long maxMillisToWait = ttl*2*1000;     //wait max 2 times the time of the ttl
              while(m_cache.containsKey(ttl) )
                   Duration timeTaken = new Duration(startTime, new DateTime());
                   if(timeTaken.getMillis() > maxMillisToWait)
                        throw new RuntimeException("Already waiting " + timeTaken + " for ttl=" + ttl + " and retry=" +  currentTry);
                   Thread.sleep(1000);
    }You can find the output below:
    2009-12-03T11:50:04.584+01:00 - Setting up TTL test
    2009-12-03 11:50:04.803/0.250 Oracle Coherence 3.5.2/463p2 <Info> (thread=main, member=n/a): Loaded operational configuration from resource "jar:file:/C:/Temp/coherence3.5.2/coherence-java-v3.5.2b463-p1_2/coherence/lib/coherence.jar!/tangosol-coherence.xml"
    2009-12-03 11:50:04.803/0.250 Oracle Coherence 3.5.2/463p2 <Info> (thread=main, member=n/a): Loaded operational overrides from resource "jar:file:/C:/Temp/coherence3.5.2/coherence-java-v3.5.2b463-p1_2/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
    2009-12-03 11:50:04.803/0.250 Oracle Coherence 3.5.2/463p2 <D5> (thread=main, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
    2009-12-03 11:50:04.803/0.250 Oracle Coherence 3.5.2/463p2 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
    Oracle Coherence Version 3.5.2/463p2
    Grid Edition: Development mode
    Copyright (c) 2000, 2009, Oracle and/or its affiliates. All rights reserved.
    2009-12-03 11:50:04.943/0.390 Oracle Coherence GE 3.5.2/463p2 <Info> (thread=main, member=n/a): Loaded cache configuration from "file:/C:/jb/workspace3.5/TTLTest/target/classes/coherence-cache-config.xml"
    2009-12-03 11:50:05.318/0.765 Oracle Coherence GE 3.5.2/463p2 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
    2009-12-03 11:50:08.568/4.015 Oracle Coherence GE 3.5.2/463p2 <Info> (thread=Cluster, member=n/a): Created a new cluster "cluster:0xD3FB" with Member(Id=1, Timestamp=2009-12-03 11:50:05.193, Address=172.16.44.32:8088, MachineId=36896, Location=process:11848, Role=TTLTestServerTTLTestServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) UID=0xAC102C20000001255429380990201F98
    2009-12-03 11:50:08.584/4.031 Oracle Coherence GE 3.5.2/463p2 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
    2009-12-03 11:50:08.756/4.203 Oracle Coherence GE 3.5.2/463p2 <D5> (thread=DistributedCache:testService, member=1): Service testService joined the cluster with senior service member 1
    2009-12-03T11:50:08.803+01:00 - Starting TTL test
    2009-12-03T11:50:08.818+01:00 - Adding a value in cache for TTL=1 in try= 1
    2009-12-03T11:50:09.818+01:00 - Adding a value in cache for TTL=1 in try= 2
    Exception in thread "main" (Wrapped: Failed request execution for testService service on Member(Id=1, Timestamp=2009-12-03 11:50:05.193, Address=172.16.44.32:8088, MachineId=36896, Location=process:11848, Role=TTLTestServerTTLTestServer)) java.lang.IllegalStateException: the node has already been removed
         at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.tagException(Grid.CDB:36)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onContainsKeyRequest(DistributedCache.CDB:41)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ContainsKeyRequest.run(DistributedCache.CDB:1)
         at com.tangosol.coherence.component.net.message.requestMessage.DistributedCacheKeyRequest.onReceived(DistributedCacheKeyRequest.CDB:12)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:9)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:136)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onNotify(DistributedCache.CDB:3)
         at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
         at java.lang.Thread.run(Thread.java:619)
    Caused by: java.lang.IllegalStateException: the node has already been removed
         at com.tangosol.util.AbstractSparseArray$Crawler.remove(AbstractSparseArray.java:1274)
         at com.tangosol.net.cache.OldCache.evict(OldCache.java:580)
         at com.tangosol.net.cache.OldCache.containsKey(OldCache.java:171)
         at com.tangosol.net.cache.ReadWriteBackingMap.containsKey(ReadWriteBackingMap.java:597)
         at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache.onContainsKeyRequest(DistributedCache.CDB:25)
         ... 7 more
    2009-12-03 11:50:10.834/6.281 Oracle Coherence GE 3.5.2/463p2 <D4> (thread=ShutdownHook, member=1): ShutdownHook: stopping cluster node
    2009-12-03 11:50:10.834/6.281 Oracle Coherence GE 3.5.2/463p2 <D5> (thread=Cluster, member=1): Service Cluster left the clusterBest regards
    Jan

  • What triggers a write-through/write-behind of entry processor changes?

    What triggers a write-through/write-behind when a cache entry is modified by a custom entry processor (subclass of AbstractProcessor)? Is it simply the call to Entry.setValue() that triggers this or are entry modifications detected in some other way?
    The reason I am asking is that in our Coherence cluster it looks like some entry modifications through entry processors have not triggered a write-behind, and I see no logical pattern as to which ones have and which ones haven't except that some specific entries are just never written to our database. We see from our logs that our implementation of the CacheStore.store() method is not called in these cases, and we also see that the cache entry has been modified successfully.
    We are using Coherence 3.3 on a three machine cluster with 8 nodes on each machine, accessed from clients through a TCP extend proxy.
    Regards,
    Mikael Carlstedt
    mBlox Inc
    Edited by: user3849225 on 16-Sep-2010 04:57

    Hi Mikael
    Calling setEntry() will result in a call to the CacheStore.store() method unless the value you are setting is the same as the existing entry value. If you are using writebehind then storeAll() will be called instead of store() if there are multiple entries waiting to be stored. Writebehind will also coelesce entries so that only the last entry for a given key will be stored.
    What patch level are you using?
    Paul
    Edited by: pmackin on Sep 17, 2010 12:08 AM

  • Thread pool configuration for write-behind cache store operation?

    Hi,
    Does Coherence have a thread pool configuration for the Coherence CacheStore operation?
    Or the CacheStore implementation needs to do that?
    We're using write-behind and want to use multiple threads to speed up the store operation (storeAll()...)
    Thanks in advance for your help.

    user621063 wrote:
    Hi,
    Does Coherence have a thread pool configuration for the Coherence CacheStore operation?
    Or the CacheStore implementation needs to do that?
    We're using write-behind and want to use multiple threads to speed up the store operation (storeAll()...)
    Thanks in advance for your help.Hi,
    read/write-through operations are carried out on the worker thread (so if you configured a thread-pool for the service the same thread-pool will be used for the cache-store operation).
    for write-behind/read-ahead operations, there is a single dedicated thread per cache above whatever thread-pool is configured, except for remove operations which are synchronous and still carried out on the worker thread (see above).
    All above is of course per storage node.
    Best regards,
    Robert

  • Write behind cache, DB down, when should the system stop taking new data in

    Hello:
    We are trying to use Coherence for our custom ESB, which is brokering payloads of various size between consumer and provider applications.
    Before Coherence, stopping our DB meant organization-wide outage for critically important business services.
    Since we have at least 40G of RAM in production environment, we believe that our app
    can use Coherence write-behind option for tolerating at least several hours worth of DB outage.
    We are currently using a near cache backed by distributed cache in write-behind mode.
    9 business service JVMs (storage enabled=false) use 30 storage enabled JVMs.
    IMPORTANT: We need to create an automated alerting facility determining when
    amount of unsaved data reaches critical level since DB goes down. This alert should help us decide when our application stops accepting inbound traffic.
    It is hard to use QueueSize parameter for that because our payload memory footprint can vary from 1KB to 3MB.
    We do not expire any entries in order to enable support queries against the cache during DB outage.
    Our experiments with trying various flavors of overflow-scheme resulted in OutOfMemoryError, therefore
    we decided to implement RAM-only cache as a first step.
    <near-scheme>
    <scheme-name>message_payload_scheme</scheme-name>
    <front-scheme>
    <local-scheme>
    <scheme-ref>limited_entities_front_scheme</scheme-ref>
    <high-units>100</high-units>
    </local-scheme>
    </front-scheme>
    <back-scheme>
    <distributed-scheme>
    <backing-map-scheme>
    <read-write-backing-map-scheme>
    <internal-cache-scheme>
    <local-scheme>
    <scheme-ref>limited_bytes_scheme</scheme-ref>
    <high-units>199229440</high-units>
    </local-scheme>
    </internal-cache-scheme>
    <cachestore-scheme>
    <class-scheme>
    <class-name>com.comp.MessagePayloadStore</class-name>
    </class-scheme>
    </cachestore-scheme>
    <read-only>false</read-only>
    <write-delay-seconds>3</write-delay-seconds>
    <write-requeue-threshold>2147483646</write-requeue-threshold>
    </read-write-backing-map-scheme>
    </backing-map-scheme>
    <autostart>true</autostart>
    </distributed-scheme>
    </back-scheme>
    </near-scheme>
    <local-scheme>
    <scheme-name>limited_entities_front_scheme</scheme-name>
    <eviction-policy>LRU</eviction-policy>
    <unit-calculator>FIXED</unit-calculator>
    </local-scheme>
    <local-scheme>
    <scheme-name>limited_bytes_scheme</scheme-name>
    <eviction-policy>HYBRID</eviction-policy>
    <unit-calculator>BINARY</unit-calculator>
    </local-scheme>

    Good info ... I feel like I need to restate my original question along with a couple of new questions caused by the discussion above.
    Q1. Does Coherence evict 'dirty', or 'queued', or 'unsaved' objects for cache configuration provided above?
    The answer should be 'NO', otherwise Coherence is unsafe to use as a system of record,
    it should not just drop unsaved information on the floor.
    Q2. What happens to the front tier of the near+partitioned write behind cache described above when amount of unsaved data exceeds max cache capacity defined via high-units?
    I would expect that map.put starts throwing exceptions: cache storage is full, so it should not accept more data
    Q3. How can I determine a moment when amount of dirty data in bytes(!), not in objects, hits 85% of
    max allowed cache capasity configured in bytes (using high-units param and BINARY calculator).
    'DirtyUnits' counter can probably be built with some lower-level Coherence API. Can we use
    this API?
    Please, understand, that we purchased Coherence for reliability, for making our
    system independent from short DB outages, for keeping our business services up
    and running when DBA need some time for admin operations like rebuilding an index.
    Performance benefits are secondary and are not as obvious for our system which
    uses primary keys only and has a well-tuned co-located Oracle back-end.
    We simply cannot put Coherence to production unless we prove that Coherence
    can reliably hold the data and give us information about approaching crisis
    (the cache full of unsaved data).
    If possible, forward this message to Cameron Purdy,
    who was presenting Coherence to our team several moths ago.
    Thanks,
    Vasili Smaliak
    Applications Architect, Enterprise App Integration
    GMAC ResCap
    [email protected]

  • Cache write-behind complete check

    Is there a surefire way to check a cache that has a store persisting objects to the database and write-behind set to 2 seconds, has persisted all objects put into the cache?
    We have tried using JMX, querying the Cache's QueueSize and waiting until it reaches 0. It turns out that when putting objects into the write-behind cache, the write-behind queue is not necessarily non-zero immediately after the put(s). e.g. QueueSize may be 0, even if objects still need to be persisted.
    For our nightly integration tests we need to clear out the cache, but want to make sure we do not call NamedCache.clear() on a cache that still has objects that need to be persisted.
    Any ideas?

    Hi Rob,
    The problem may actually be the timeliness of updates to the QueueSize JMX attribute as we're using the MBeanConnector to obtain information on our cache members (and providers). Assuming that objects actually make it to the write-behind queue during the cache put call and certain tests need to be sure these objects are persisted, instead of doing the accounting approach discussed previously, I found a forum thread on ReadWriteBackingMap flush calls.
    To get access to the ReadWriteBackingMap.flush() I created a small test today using a subclass of ReadWriteBackingMap that registers the backingmap with our CashStore implementation:
        protected void configureCacheStore(CacheStore store, boolean readOnly) {
            super.configureCacheStore(store, readOnly);
            if (store instanceof CacheLoaderWriterProvider) {
                ((CacheLoaderWriterProvider)store).registerBackingMap(this);
        }Our cachstore (CacheLoaderWriterProvider) in turn exposes a call to the registered map's flush method as a JMX operation.
    Whenever we need to be sure the write-behind queue is empty during our tests, we'll call this JMX operation.
    Best Regards,
    Marcel.

  • Write-Behind Caching and Multiple Puts

    What happens when two consecutive puts are performed on the write-behind cache for the same key? Will CacheStore's store() or storeAll() be invoked once for every put() or only once for the last put() (the one which overrode the previous cached values)?

    Hi Denis,
         If you use write-behind, there will be no unnesessary database updates - only the last put() will result in database update.
         Regards,
         Dimitri

  • Write-Behind Caching and Limited Internal Cache Size

    Let's say I have a write-behind cache and configure its internal cache to be of a fixed limited size, e.g. 10000 units. What would happen if more than 10000 units are added to the write-behind cache within the write-delay period? Would my CacheStore's storeAll() get all of the added values or would some of the values be missed because of the internal cache size limitation?

    Hi Denis,     >
         > If an entry is removed while it is still in the
         > write-behind queue, it will be removed from the queue
         > and CacheStore.store(oKey, oValue) will be invoked
         > immediately.
         >
         > Regards,
         > Dimitri
         Dimitri,
         Just to confirm, that I understand it right if there is a queued update to a key which is then remove()-ed from the cache, then the following happens:
         First CacheStore.store(key, queuedUpdateValue) is invoked.
         Afterwards CacheStore.erase(key) is invoked.
         Both synchronously to the remove() call.
         I expected only erase will be invoked.
         BR,
         Robert

  • Write-Behind Caching and Old Values

    Is there a way to access the old value cached in the write-behind cache for the same key from the CacheStore's store() or storeAll() method?

    I have a business POJO with three parts: partA,     > partB, partC inside. Each of these three parts is
         > persisted by a separate SQL. So, every time I persist
         > my POJO, up to 3 SQLs may be executed.
         I understand.
         > When a change happens in my POJO, it goes onto the
         > write-behind queue. In my CacheStore.store() or
         > CacheStore.storeAll() I would like to be able to make
         > an intelligent decision about which of the three
         > parts: partA, partB or partC has actually changed and
         > only run the SQL updates for the changed parts. This
         > would allow me to avoid massive amounts of
         > unnecessary SQL updates for the parts that did not
         > change.
         Right. Keep in mind that there are two conditions that you must be aware of:
         1) Multiple updates could have occurred to the object, meaning that the database update would have to "roll up" the results of multiple changes to the object.
         2) Some or all of the updates could have already occurred to the database. This may be a little trickier to understand, but it reflects the possible machine failure conditions that occurred while a write-behind was in progress.
         Although the latter are unlikely, they should be accounted for, and of course they are harder to test for with certainty. As a result, the updates to the information (the CacheStore implementation) must be built in an "idempotent" manner, i.e. allowing it to be executed more than once with no additional side-effects.
         > If I had access to the POJO stored under the same key
         > before the new value was put in cache, I could use
         > equals() on each of the three parts to find out
         > exactly which one of them changed.
         While this is true, you would need to compare the "known previous database state" version, not just the "old" version.
         > Of course, if this functionality is not available, I
         > would have to create dirty flags for each of the
         > three POJO parts. But I can't really clear my POJO's
         > flags and recache the POJO from within the store() or
         > storeAll(), right?
         Yes, but remember that those flags are "could be dirty" flags, because of the above failure modes that I described.
         Peace,
         Cameron Purdy
         Tangosol Coherence: The Java Data Grid

  • Local Cache with write-behind backing map

    Hi there,
    I am a Coherence newb, so I hope my question isn't too naive. I have been experimenting with Coherence using a write-behind JPA backing map, but I have only been able to make it work with a distributed cache. Because of my specific database RAC architecture, I need to ensure that entries written to the database from a given WLS node are restricted to a specific RAC node. In my view, using a local cache rather than a distributed cache should solve my problem, although if there is a way of configuring my cache so this works I'd appreciate the info.
    So, the short form of the question: can I back a local cache with a write-behind JPA map?
    Cheers,
    Ron

    Hi Ron,
    The configuration for <local-scheme> allows you to add a cache store but you cannot use write-behind, only write-through.
    Presumably you do not want the data to be shared by the different WLS nodes, i.e. if one node puts data in the cache and that is eventually written to a RAC node, that data cannot be seen in the cache by other WLS nodes. Or do you want all the WLS nodes to share the data but just write it to different RAC nodes?
    If you use a local-scheme then the data will only be local to that WLS node and not shared.
    I can think of a possible way to do what you want but it depends on the answer to the above question.
    JK

  • Error removing object from cache with write behind

    We have a cache with a DB for a backing store. The cache has a write-behind delay of about 10 seconds.
    We see an error when we:
    - Write new object to the cache
    - Remove object from cache before it gets written to cachestore (because we're still within the 10 secs and the object has not made it to the db yet).
    At first i was thinking "coherence should know if the object is in the db or not, and do the right thing", but i guess that's not the case?

    Hi Ron,
    The configuration for <local-scheme> allows you to add a cache store but you cannot use write-behind, only write-through.
    Presumably you do not want the data to be shared by the different WLS nodes, i.e. if one node puts data in the cache and that is eventually written to a RAC node, that data cannot be seen in the cache by other WLS nodes. Or do you want all the WLS nodes to share the data but just write it to different RAC nodes?
    If you use a local-scheme then the data will only be local to that WLS node and not shared.
    I can think of a possible way to do what you want but it depends on the answer to the above question.
    JK

Maybe you are looking for

  • HP Laserjet 4 Usb adapter problem

    I recently switched my HP laserjet 4 from a parallel connection to a parallel to usb connector. The drivers for both installed properly however the printer does not seem to print is there a specific port I need to use or is there another issue? print

  • Using Lightroom 3 with OS Maverick

    I have just bought a new computer that is at OS 10.9 Maverick and likely will need to be upgrade to 10.10 Yosemite.  I have stayed with LR3 because I have not needed the upgrades. I have been told by another LR user that LR5 has a cloud service where

  • How to view what session is causing heavy load

    Hi, currently I was tasked to view what is wrong with the production server, there's some processing that is causing heavy usage. May I know how should I go about investigating this? thanks more precisely which machine, which sid, what query is causi

  • WHY THERE's NO "REPORTING SERVICES"?

    Hi, I'm trying to post a question about Reporting Services 2008 R2 displaying data from an Oracle DB, but SSRS is not in the "Forum Category" dropdown, and if I select "SQL Server" from this dropdown, the next dropdown "Forum" doesn't display SSRS ei

  • Python2-pyqt depends on python3

    The package python2-pyqt depends on pyqt, which itself requires python3. Is it a normal situation, or should I fill in a bug report ? Tondu