Write-behind queue resilience

For our project, our coherence cache is the trusted store. We plan on using a DB backing store, so that should we need to shutdown the cache cluster, or lose it for reasons outside our control we have a copy to reload from. We can't afford to lose any data from the cache or lose any writes through to the backing store.
Ideally, we would like to make use of write-behind caching for the obvious performance benefits. However, I've got some concerns with this strategy that I wasn't able to find the answer to in the user guide.
1. Can we configure the size of the write behind queue? If events are coming in faster than we can write to the DB, then the queue will grow. At some point the queue will exhaust system resources I assume.
2. If the primary is lost before the store operation is performed, will the backup partition assume responsibility?
3. Is there a safe way I can shutdown cache nodes? I.e. once we initiate the shutdown, the cache node will not accept any more puts, but will wait for the write behind queue to flush.

Hi,
It looks like you are using com.oracle.coherence.handson.DBCacheStore.
http://download.oracle.com/docs/cd/E13924_01/coh.340/e14135/cohjdev.htm
If so, then I think that you could simply change the following method in the DBCacheStore ...
public Connection getConnection()
        if (m_con == null || m_con.isClosed())
            configureConnection();
        return m_con;
        }Thanks,
Tom

Similar Messages

Read-through/write-behind and queued deletes (and updates)

Hi,
If I am changing the state of objects in a cache and using the write-behind and read-through mechanism what happens when I have deleted or updated an object in the cache but the change has not yet been committed to the database?.
If I delete and object in the cache and the delete DB operation is being queued and during this time try and perform a get against the key for the object is the value read through from the database or is it ignored since the database delete is pending?
For updates I presume that the value in the cache will be used - as the value exists in the cache and a read-through from the database will not be triggered.
Can you clarify the behavior of Coherence under these circumstances, particularly that of a pending delete.
Thanks,
Dave

Hi Dave,
If I am changing the state of objects in a cache and
using the write-behind and read-through mechanism
what happens when I have deleted or updated an object
in the cache but the change has not yet been
committed to the database?.
If I delete and object in the cache and the delete DB
operation is being queued and during this time try
and perform a get against the key for the object is
the value read through from the database or is it
ignored since the database delete is pending?I seem to remember a forum post mentioning that the removes from a write-behind cache are performed synchronously (they are done as part of the backingMap.remove(key)) operation so even if there are were pending updates in the write-behind queue. If I remember correctly, then the above mentioned problem cannot happen.
>
For updates I presume that the value in the cache
will be used - as the value exists in the cache and a
read-through from the database will not be
triggered.
Exactly.
Best regards,
Robert

Write-Behind batch behavior in EP partition level transactions

Hi,
We use EntryProcessors to perform updates on multiple entities stored in the same cache partition. According to the documentation, Coherence handles all the updates in a "sandbox" and then commits them atomically to the cache backing map.
The question is, when using write-behind, does Coherence guarantee that all entries updated in the same "partition level transaction" will be present in the same "storeAll" operation?
Again, according to the documentation, the write-behind thread behavior is the following:
The thread waits for a queued entry to become ripe.
When an entry becomes ripe, the thread dequeues all ripe and soft-ripe entries in the queue.
The thread then writes all ripe and soft-ripe entries either via store() (if there is only the single ripe entry) or storeAll() (if there are multiple ripe/soft-ripe entries).
The thread then repeats (1).
If all entries updated in the same partition level transaction become ripe or soft-ripe at the same instant they will all be present in the storeAll operation. If they do not become ripe/soft-ripe in the same instant, they may not be all present.
So, it all depends on the behavior of the commit of the partition level transaction, if all entries get the same update timestamp, they will all become ripe at the same time.
Does anyone know what is the behavior we can expect regarding this issue?
Thanks.

Hi,
That comment is still correct for 12.1 and 3.7.1.10.
I've checked Coherence APIs and the ReadWriteBackingMap behavior, and although partition level transactions are atomic, the updated entries will be added one by one to the write behind queue. In each added entry coherence uses current time to calculate when each entry will become ripe, so, there is no guarantee that all entries in the same partition level transaction will become ripe at the same time.
This leads me to another question.
We have a use case where we want to split a large entity we are storing in coherence into several smaller fragments. We use EntryProcessors and partition level transactions to guarantee atomicity in operations that need to update more than one fragment of the same entity. This guarantees that all fragments of the same entity are fully consistent. The cached fragments are then persisted into database using write-behind.
The problem now is how to guarantee that all fragments are fully consistent in the database. If we just relly on coherence write-behind mecanism we will have eventual consistency in DB, but in case of multi-server failure the entity may become inconsistent in database, which is a risk we wouldnt like to take.
Is there any other option/pattern that would allow us to either store all updates done on the entity or no update at all?
Probably if in the EntryProcessor we identify which entities were updated and if we place them in another persistency queue as a whole, we will be able to achieve this, but this is a kind of tricky workaround that we wouldnt like to use.
Thanks.

Write behind exception and recovery

Hi all,
I am working on write behind part in equity trading system. I know that cache store operation will eventually be thrown away if no of retry exceed write-requeue-threshold. However, this is not acceptable as DB must sync with caches at least at day end. For some more complicated caches, we use cache store implementation and Hiberate for simple cache. I am thinking to capture the sql statements that failed during the day and finally at day end, manually fix issues (egDB issue or others) then have them executed.
Questions:
1. Is this a good approach for handling the scenario? If yes, any way I can capture the statements and write to file for running in SQL plus for example in case of Hiberate?
2. Is there any out of box mechanism in Coherence for recovering write-behind queues in case of WHOLE cluster fail (not node fail).
Henry

922963 wrote:
Hi all,
I am working on write behind part in equity trading system. I know that cache store operation will eventually be thrown away if no of retry exceed write-requeue-threshold. However, this is not acceptable as DB must sync with caches at least at day end. For some more complicated caches, we use cache store implementation and Hiberate for simple cache. I am thinking to capture the sql statements that failed during the day and finally at day end, manually fix issues (egDB issue or others) then have them executed.
Questions:
1. Is this a good approach for handling the scenario? If yes, any way I can capture the statements and write to file for running in SQL plus for example in case of Hiberate?Hi Henry,
There are a few caveats you need to care about but in general it is possible.
Not necessarily SQLs but serialized entries would probably be simpler to work with when you try to restore them.
Also, you have to be aware that Coherence may fail to write an entry to the DB but at retry it may try to write a new entry. If it succeeds, you have to be able to figure that out that the earlier failure must not be reexecuted.
In effect, you should have per-entry versioning in the database and you should check versions of the entity in the database upon writing both from the cache store and also from your end-of-day retry logic.
2. Is there any out of box mechanism in Coherence for recovering write-behind queues in case of WHOLE cluster fail (not node fail).
No, nothing like that comes out-of-the-box, if you lost a partition, you lost your write-behind-enqueued entries, too. You could log your failed writes to disk though as you indicated above.
Best regards,
Robert

Cache write-behind complete check

Is there a surefire way to check a cache that has a store persisting objects to the database and write-behind set to 2 seconds, has persisted all objects put into the cache?
We have tried using JMX, querying the Cache's QueueSize and waiting until it reaches 0. It turns out that when putting objects into the write-behind cache, the write-behind queue is not necessarily non-zero immediately after the put(s). e.g. QueueSize may be 0, even if objects still need to be persisted.
For our nightly integration tests we need to clear out the cache, but want to make sure we do not call NamedCache.clear() on a cache that still has objects that need to be persisted.
Any ideas?

Hi Rob,
The problem may actually be the timeliness of updates to the QueueSize JMX attribute as we're using the MBeanConnector to obtain information on our cache members (and providers). Assuming that objects actually make it to the write-behind queue during the cache put call and certain tests need to be sure these objects are persisted, instead of doing the accounting approach discussed previously, I found a forum thread on ReadWriteBackingMap flush calls.
To get access to the ReadWriteBackingMap.flush() I created a small test today using a subclass of ReadWriteBackingMap that registers the backingmap with our CashStore implementation:
    protected void configureCacheStore(CacheStore store, boolean readOnly) {
        super.configureCacheStore(store, readOnly);
        if (store instanceof CacheLoaderWriterProvider) {
            ((CacheLoaderWriterProvider)store).registerBackingMap(this);
    }Our cachstore (CacheLoaderWriterProvider) in turn exposes a call to the registered map's flush method as a JMX operation.
Whenever we need to be sure the write-behind queue is empty during our tests, we'll call this JMX operation.
Best Regards,
Marcel.

Write-Behind Caching and Limited Internal Cache Size

Let's say I have a write-behind cache and configure its internal cache to be of a fixed limited size, e.g. 10000 units. What would happen if more than 10000 units are added to the write-behind cache within the write-delay period? Would my CacheStore's storeAll() get all of the added values or would some of the values be missed because of the internal cache size limitation?

Hi Denis,     >
     > If an entry is removed while it is still in the
     > write-behind queue, it will be removed from the queue
     > and CacheStore.store(oKey, oValue) will be invoked
     > immediately.
     >
     > Regards,
     > Dimitri
     Dimitri,
     Just to confirm, that I understand it right if there is a queued update to a key which is then remove()-ed from the cache, then the following happens:
     First CacheStore.store(key, queuedUpdateValue) is invoked.
     Afterwards CacheStore.erase(key) is invoked.
     Both synchronously to the remove() call.
     I expected only erase will be invoked.
     BR,
     Robert

Write-Behind Caching and Old Values

Is there a way to access the old value cached in the write-behind cache for the same key from the CacheStore's store() or storeAll() method?

I have a business POJO with three parts: partA,     > partB, partC inside. Each of these three parts is
     > persisted by a separate SQL. So, every time I persist
     > my POJO, up to 3 SQLs may be executed.
     I understand.
     > When a change happens in my POJO, it goes onto the
     > write-behind queue. In my CacheStore.store() or
     > CacheStore.storeAll() I would like to be able to make
     > an intelligent decision about which of the three
     > parts: partA, partB or partC has actually changed and
     > only run the SQL updates for the changed parts. This
     > would allow me to avoid massive amounts of
     > unnecessary SQL updates for the parts that did not
     > change.
     Right. Keep in mind that there are two conditions that you must be aware of:
     1) Multiple updates could have occurred to the object, meaning that the database update would have to "roll up" the results of multiple changes to the object.
     2) Some or all of the updates could have already occurred to the database. This may be a little trickier to understand, but it reflects the possible machine failure conditions that occurred while a write-behind was in progress.
     Although the latter are unlikely, they should be accounted for, and of course they are harder to test for with certainty. As a result, the updates to the information (the CacheStore implementation) must be built in an "idempotent" manner, i.e. allowing it to be executed more than once with no additional side-effects.
     > If I had access to the POJO stored under the same key
     > before the new value was put in cache, I could use
     > equals() on each of the three parts to find out
     > exactly which one of them changed.
     While this is true, you would need to compare the "known previous database state" version, not just the "old" version.
     > Of course, if this functionality is not available, I
     > would have to create dirty flags for each of the
     > three POJO parts. But I can't really clear my POJO's
     > flags and recache the POJO from within the store() or
     > storeAll(), right?
     Yes, but remember that those flags are "could be dirty" flags, because of the above failure modes that I described.
     Peace,
     Cameron Purdy
     Tangosol Coherence: The Java Data Grid

Shutting down a write-behind cluster

Hi,
What is the best way of shutting down a cluster with caches that are write-behind enabled? Is there a safe way to make sure the write-behind queue is cleared before taking the nodes down? Will CacheFactory.shutDown() take care of this? Also is it possible shutdown gracefully without the nodes redistributing the data? If data gets redistributed, the last node I am shutting down would run out of memory. Shoud I destroy each cache before shutting down? Pls advise.
Regards,
Sairam

Hi Sairam,
first you have to block producers.
Afterwards you can ensure that the write-behind queue is written out on all nodes by (optionally calling the flush method on the read-write backing map on each node e.g. from an Invocation and then) waiting for JMX showing that the write-behind queue is empty on each node.
Once there are no dirty entries in the wb queue, then you can just shut down consumers (anyone who reads from the cache).
After ensuring that no one wants to read from the cache, there is no point in having the data in the cache, therefore you can clear the caches with NamedCache.clear().
After this who cares that there is a redistribution of entry partitions.
No need to destroy() the caches, clear()-ing them is enough, except if you have backing map listeners which would hold up the execution of the clear() method.
Best regards,
Robert

Write behind cache, DB down, when should the system stop taking new data in

Hello:
We are trying to use Coherence for our custom ESB, which is brokering payloads of various size between consumer and provider applications.
Before Coherence, stopping our DB meant organization-wide outage for critically important business services.
Since we have at least 40G of RAM in production environment, we believe that our app
can use Coherence write-behind option for tolerating at least several hours worth of DB outage.
We are currently using a near cache backed by distributed cache in write-behind mode.
9 business service JVMs (storage enabled=false) use 30 storage enabled JVMs.
IMPORTANT: We need to create an automated alerting facility determining when
amount of unsaved data reaches critical level since DB goes down. This alert should help us decide when our application stops accepting inbound traffic.
It is hard to use QueueSize parameter for that because our payload memory footprint can vary from 1KB to 3MB.
We do not expire any entries in order to enable support queries against the cache during DB outage.
Our experiments with trying various flavors of overflow-scheme resulted in OutOfMemoryError, therefore
we decided to implement RAM-only cache as a first step.
<near-scheme>
<scheme-name>message_payload_scheme</scheme-name>
<front-scheme>
<local-scheme>
<scheme-ref>limited_entities_front_scheme</scheme-ref>
<high-units>100</high-units>
</local-scheme>
</front-scheme>
<back-scheme>
<distributed-scheme>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<local-scheme>
<scheme-ref>limited_bytes_scheme</scheme-ref>
<high-units>199229440</high-units>
</local-scheme>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>com.comp.MessagePayloadStore</class-name>
</class-scheme>
</cachestore-scheme>
<read-only>false</read-only>
<write-delay-seconds>3</write-delay-seconds>
<write-requeue-threshold>2147483646</write-requeue-threshold>
</read-write-backing-map-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
</back-scheme>
</near-scheme>
<local-scheme>
<scheme-name>limited_entities_front_scheme</scheme-name>
<eviction-policy>LRU</eviction-policy>
<unit-calculator>FIXED</unit-calculator>
</local-scheme>
<local-scheme>
<scheme-name>limited_bytes_scheme</scheme-name>
<eviction-policy>HYBRID</eviction-policy>
<unit-calculator>BINARY</unit-calculator>
</local-scheme>

Good info ... I feel like I need to restate my original question along with a couple of new questions caused by the discussion above.
Q1. Does Coherence evict 'dirty', or 'queued', or 'unsaved' objects for cache configuration provided above?
The answer should be 'NO', otherwise Coherence is unsafe to use as a system of record,
it should not just drop unsaved information on the floor.
Q2. What happens to the front tier of the near+partitioned write behind cache described above when amount of unsaved data exceeds max cache capacity defined via high-units?
I would expect that map.put starts throwing exceptions: cache storage is full, so it should not accept more data
Q3. How can I determine a moment when amount of dirty data in bytes(!), not in objects, hits 85% of
max allowed cache capasity configured in bytes (using high-units param and BINARY calculator).
'DirtyUnits' counter can probably be built with some lower-level Coherence API. Can we use
this API?
Please, understand, that we purchased Coherence for reliability, for making our
system independent from short DB outages, for keeping our business services up
and running when DBA need some time for admin operations like rebuilding an index.
Performance benefits are secondary and are not as obvious for our system which
uses primary keys only and has a well-tuned co-located Oracle back-end.
We simply cannot put Coherence to production unless we prove that Coherence
can reliably hold the data and give us information about approaching crisis
(the cache full of unsaved data).
If possible, forward this message to Cameron Purdy,
who was presenting Coherence to our team several moths ago.
Thanks,
Vasili Smaliak
Applications Architect, Enterprise App Integration
GMAC ResCap
[email protected]

Write-Behind, Expiration, and SQL Exceptions.

Hi Chaps,
If a cache with write-behind enabled has problems writing to the DB I understand that Coherence will re-queue the objects and write them when the DB is available.
The problem I have is that (after a DB failure) I don't see them being written - I can see these items in the cache but not in the DB, even several hours after the outage. (Items that were added to the cache after the outage are being written).
Is there anything the cachestore methods (specifically store() ) need to do with regards to exceptions to ensure that these items are re-qeueued?
Next question is: I was also wondering how is this managed with regards to expiry?
We have our own expiry routine which removes items from the cache that are older than 24 hours (this was from before we could expire objects by specifying the timeout in the put() method call, which I am intending to switch to).
If an item has not been written to the DB due to an outage and is then expired (by our own routine or by Coherence) is it then lost forever, or will it remain in the queue? (seeing as the queue holds references I am guessing not but though I'd check).
Thanks,
Randal.

Jon,
I have a question related to this...If you remember a few weeks back, I stumbled upon the problem of the "version-persistent" map for the versioned-backing-map-scheme does not accept putAll operations. The workaround until you guys implement it, was to override the putAll method of the cacheStore and throw and unsupported operation exception (to force individual puts).
Well, although this workaround works, I am getting tons and tons of:
2006-04-06 17:18:27.347 Tangosol Coherence 3.1/339 <Warning> (thread=WriteBehindThread:MyCacheStore, member=1): The CacheStore "MyCacheStore@46b9979b" does not support storeAll().
2006-04-06 17:18:27.348 Tangosol Coherence 3.1/339 <Error> (thread=WriteBehindThread:MyCacheStore, member=1): Failed to store keys="[16, 18, 21, 26, 5, 13, 14, 25, 17, 15, 23, 19, 2, 6, 9, 7]":
java.lang.UnsupportedOperationException
at ...MyCacheStore.storeAll(MyCacheStore.java:126)
at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.storeAll(ReadWriteBackingMap.java:3820)
at com.tangosol.net.cache.ReadWriteBackingMap$WriteThread.run(ReadWriteBackingMap.java:3538)
at com.tangosol.util.Daemon$1.run(Daemon.java:63)
2006-04-06 17:18:27.349 Tangosol Coherence 3.1/339 <Warning> (thread=WriteBehindThread:MyCacheStore, member=1): Requeued store for key="16"
2006-04-06 17:18:27.349 Tangosol Coherence 3.1/339 <Warning> (thread=WriteBehindThread:MyCacheStore, member=1): Requeued store for key="18"
2006-04-06 17:18:27.350 Tangosol Coherence 3.1/339 <Warning> (thread=WriteBehindThread:MyCacheStore, member=1): Requeued store for key="21"
2006-04-06 17:18:27.351 Tangosol Coherence 3.1/339 <Warning> (thread=WriteBehindThread:MyCacheStore, member=1): Requeued store for key="26"
the first OperationNotSupported is expected, but I'm not sure what the requeued warnings are all about. These are not failures to the DB...it is something else. (mind you that this happens when trying to load a lot of data into the map.)
1- Is this requeuing related or the same as in failed DB stores?
2- Is it possible to "lose" stores if I don't configure the write-requeue-threshold with very, very high values? I must ensure I don't lose anything.
In a related note, in some circumstances, I need to ensure that the "write queue" is flushed or cleared. For example, I may want to force a flush of all pending stores (and wait/block until that's done).
I have looked into it and I don't seem to know how to do it. I can read the write-queue length, but I believe that this is not very accurate...since my tests seem to indicate that the write-behind thread may take the entries to store off the write-queue and then deal with them in parallel (which means that there are still entries althought the write-queue size is 0). Also, there are some calls from the cache store that, at first, seem to give some access to the write thread (potentially allowing me to contact the thread to tell him to flush or discard any pending stores)...but I believe that all of the functions are protected...but there may be other ways..
I guess my second batch of questions are:
1- How can I effectively force a flush (or clear) of the pending stores. Such that there is no single store pending in any queue (visible or invisible to the programmer).
2- What is the role of re-queuing in these situations? where is the queue sitting, the thread? the cache store? who's responsible of retrying that, and when?...I would like to flush those entries too.
A quick explanation of the operation of the write thread would also be very appreciated.
Thanks!
Josep M.

Handling Database failure in Write-Behind

Hi,
In the link mentioned below it is mentioned that "The application is somewhat insulated from database failures: the Write-Behind feature can be configured in such a way that a write failure will result in the object being re-queued for write"
http://coherence.oracle.com/display/COH35UG/Read-Through,+Write-Through,+Write-Behind+and+Refresh-Ahead+Caching
I wanted know how write behind can be configured so as to insulate it from database failures. How can it be configured so that in case of db failure object is re-queded for write
Thanks,
Sudhir

Requeuing can be enabled by setting the write-requeue-threshold to the maximum number of expected entries
that will exist in the queue when it is time to write the data to the database.
A complete example can be found here: http://middlewaremagic.com/weblogic/?p=5954
Look for the cache configuration section.

Switching from write through to write behind automatically

Hi,
We are considering a Coherence solution to protect a customer facing application from outages due to database failures. This is for a financial company and the monetary value of each transaction is large and we want to provide 100% guarantee against data loss while not incurring any outages. We want to provide a write-through persistence to the database through Coherence which can switch to a write-behind automatically at runtime if the database persistence fails. Is this doable automatically and would it solve the problem I am trying to solve without losing any inflight transactions? Are there any real customer cases that were successful in achieving this using Coherence?
Thanks
Sairam
Edited by: SKR on Feb 16, 2012 3:14 PM
Edited by: SKR on Feb 16, 2012 3:15 PM

SKR wrote:
Jonathan.Knight wrote:
Hi Sairam
I know you can change the write-delay in JMX for a cache using write-behind but I pretty certauin you cannot make a write-through cache suddenly become a write-behind cache.
I'm not sure why you think changing from write-through to write-behind will allow you to guarantee 100% no data loss - do you mean no loss of updates to the DB or no loss of data in the cache cluster? There are certainly scenarios that can occur where you can loose data from either the cluster or the DB that write-through or write-behind will not save you from. Presumably you want to use write-behind to allow for the DB to go down, although you will still need to configure Coherence to properly retry failed write-behind calls CacheStore behaviour on failure. What happens to your data if you are using write-behind and you loose a partition from you cluster (i.e. you loose a physical machine or two or more JVMs in a short space of time) - you have data loss - you cannot guarantee against this you can only mitigate it and have a recovery policy/procedure.
JKJK,
Thanks for your reply. I must have explained the scenario better. What we are trying to do is to have our transactions commit to the database synchronously using write-through, so that during normal operation, the data will be committed, persisted and durable in the database. But our RW database becomes a single point of failure and if some problem occurs to the database during the peak load time, we run the risk of an outage till we fix the database problem or failover to the standby (We don't have RAC architecture or automatic failover and the manual switchover takes about 10 - 15 mins minimum). We want to avoid this by providing a cache-only operation mode during such a failure, where the customers can continue to transact and the writes will get queued in the cache. I do understand that losing both the database and the cache or losing the primary and the backup in the cache would result in a data loss. But I am assuming such a dual failure is rare.
We do not want to run write-behind all the time but only during the database failure window. From what you mentioned, it seems the runtime switching from write-through to write-behind is not available as an option.
SairamHi Sairam,
I would suggest that you configure write-behind to have a fairly short write-delay, and you only return a confirmation to the client
- either after the write-behind succeeded (you can use a backing map listener to listen for the removal of the decoration which meant that the entry was dirty)
- or if the database went down (noticeable from the failure), then it is up to you whether you send a confirmation which also mentions that it is not persisted to disk yet, or not at all
Best regards,
Robert

Write behind db errors

Hi, so we have a system where we end up with failures from the WriteBehindThread for various reasons. It seems that the write is re-attempted over and over, hundreds of thousands of times per day, over several days.
Is there any way to stop this from happening; after a "while", or after a set number of attempts?
We have write-requeue-threshold set to 20, but i'm not entirely sure how this setting is applied. Eg, we have N storage nodes, so will each storage node keep up to 20 writes to retry per cache-store? And if we do have 20 that fail, but no more fail, those 20 will just keep getting retried over and over indefinitely?
Thx.

Failed write-behind store operations are not retried by default. Requeuing is enabled by setting <write-requeue-threshold> to a value greater than 0.
The <write-requeue-threshold> value is the maximum number of entries allowed to exist in the queue upon failure. If zero, write-behind requeuing
is disabled.
Any idea as to why the store operations fail?
/Mark
Oracle Coherence

Preventing write-coalescing with write-behind

Dear all,
I am very interested in the write-behind feature but I would like to disable the write-coalescing optimization to see each.
Is there any way to do this ?
I feel the fact that CacheStore.storeAll(Map entries) works on cache-key/cache-value makes this impossible to have several cache-value for the same key :-( Not coalescing the consecutive changes on a given entry would require to have a method like CacheStore.storeAll(List<Change>) with Change holding the modification (insert/update/delete), the key and the value.
Thanks,
Cyrille
Cyrille Le Clerc
[email protected]
http://blog.xebia.fr

Thanks for the feedback Robert,
I implemented this "batch processing without coalescing" thanks a "command queue" colocated with my data.
Sample : perform async processing on each modification without coalescence of MyEntity identified by MyEntityKey store in "my-entity-cache".
In my agent/entry-processor, I simultaneously modify my data "MyEntity" and put an entry MyEntityCommand (stored in "my-entity-command-cache"), MyEntityCommand holds enough information to do my async processing. This processing is done asynchronously by MyEntityCommandcacheStore.
MyEntityCommand is associated with a key MyEntityCommandKey which is composed of MyIdentityKey+sequence-number, MyEntityCommandKey has a KeyAssociation with MyEntityKey to ensure colocation.
Benefits :
* There is no coalescence because MyEntityCommandKey contains a unique sequence number.
* The overweight of this "command queue" is limited because the command object only contains that limited piece of data I need for my async processing (foreign key on the entity + few things) and the queue is self purging thanks to a <expiry-delay> on "my-entity-command-cache".
Here is a configuration extract
<distributed-scheme>
<scheme-name>entity-partitionned</scheme-name>
<service-name>EntityDistributedCache</service-name>
<serializer>
</serializer>
<backing-map-scheme>
    <local-scheme>
      <scheme-ref>unlimited-backing-map</scheme-ref>
    </local-scheme>
</backing-map-scheme>
<thread-count>10</thread-count>
<autostart>true</autostart>
</distributed-scheme>
<distributed-scheme>
<scheme-name>entity-command-partitionned</scheme-name>
<service-name>EntityDistributedCache</service-name>
<serializer>
</serializer>
<backing-map-scheme>
    <read-write-backing-map-scheme>
      <cachestore-scheme>
        <class-scheme>
          <class-name>... EntityCommandStore</class-name>
        </class-scheme>
      </cachestore-scheme>
      <internal-cache-scheme>
        <local-scheme>
          <expiry-delay>30s</expiry-delay>
        </local-scheme>
      </internal-cache-scheme>
      <write-delay>10s</write-delay>
      <write-batch-factor>0.5</write-batch-factor>
    </read-write-backing-map-scheme>
</backing-map-scheme>
<thread-count>10</thread-count>
<autostart>true</autostart>
</distributed-scheme>
{code}
Please note that I had to play with <write-batch-factor> to increase the batching factor (ie the number of entries in each CacheStore.store()/storeAll() invocation). Under very high write load, <write-batch-factor> default value of 0 gave me an average of 1.2 entry per CacheStore.store()/storeAll() invocation. My first try of a 0.5 <write-batch-factor> largely increased this average to probably hundreds (my stats are done on an underlying layer, I don't have the exact average).
Cyrille
Cyrille Le Clerc
[email protected]
http://blog.xebia.fr

How to limit Write-Behind batch

We have a scenario:
we use read-write-backing-map-scheme having write-delay 60s.
System insert a lot of data and then time comes to write data coherence find 40-50 k of unsaved record and pass them all to cachestore.
Due to data volume or database busyness cashstore may work some time several seconds for instance.
<read-write-backing-map-scheme>
<scheme-name>TicketDatabaseScheme</scheme-name>
<scheme-ref>DefaultDatabaseScheme</scheme-ref>

<cachestore-scheme>
<class-scheme>
<class-name>com.griddynamics.ticketon.app.dao.coherence.TicketCacheStore</class-name>
</class-scheme>
</cachestore-scheme>
</read-write-backing-map-scheme>
<read-write-backing-map-scheme>
<scheme-name>DefaultDatabaseScheme</scheme-name>
<internal-cache-scheme>
<local-scheme>
<scheme-ref>LocalScheme</scheme-ref>
</local-scheme>
</internal-cache-scheme>
<write-delay>60s</write-delay>
</read-write-backing-map-scheme>
In the case we experience "Terminating guarded execution" followed by service termination.
2010-02-11 09:26:52.223/511.457 Oracle Coherence GE 3.5.2/463 <Error> (thread=DistributedCache:TicketonCache, member=2): Terminating guarded execution (due to hard timeout 1924ms ago) of Daemon{Thread="Thread[WriteBehindThread:CacheStoreWrapper(com.griddynamics.ticketon.app.dao.coherence.TicketCacheStore),5,WriteBehindThread:CacheStoreWrapper(com.griddynamics.ticketon.app.dao.coherence.TicketCacheStore)]", State=Running}
2010-02-11 09:26:52.225/511.459 Oracle Coherence GE 3.5.2/463 <Error> (thread=Termination Thread, member=2): Write-behind thread timed out; stopping the cache service
2010-02-11 09:26:52.226/511.460 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache:TicketonCache, member=2): Service TicketonCache left the cluster
INFO 09:26:52,227 [http--80-22$27432016 DaoCoherenceImpl] - PROFILE_doCreatetickets putAll 200 tickets time 3444 time per 10 objects 172
INFO 09:26:52,227 [http--80-22$27432016 DaoCoherenceImpl] - PROFILE event and 1000 tickets ctreated time 9668
Broadcast Message from root (msglog) on ip-10-226-137-172 Thu Feb 11 09:57:18...ets putAll 200 tickets time 3365 time per 10 objects 168
2010-02-11 09:26:52.228/511.462 Oracle Coherence GE 3.5.2/463 <Info> (thread=httTHE SYSTEM ip-10-226-137-172 IS BEING SHUT DOWN NOW ! ! !et
Log off now or risk your files being damagedence GE 3.5.2/463 <Info> (thread=http--80-27$25787595, member=2): Restarting Service: TicketonCache
INFO 09:26:52,229 [http--80-20$15974570 DaoCoherenceImpl] - PROFILE_doCreatetickets putAll 200 tickets time 3447 time per 10 objects 172
INFO 09:26:52,289 [http--80-22$26935588 BackingBeanSuper] - request HttpRequest[22]
[09:26:53.446] {http--80-35$24027494} java.lang.RuntimeException: Failed to start Service "TicketonCache" (ServiceState=SERVICE_STOPPED)
[09:26:53.446] {http--80-35$24027494} at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
Resume: coherence crashes in case far from fatal.
Questions
1. May i limit the size of a batch passed to cachestore?
2. Is it possible to configure the timeout "(due to hard timeout 1924ms ago)"
3. Is it possible to handle like this some way to prevent self killing of coherence cluster.

Thank you Mark you are vary helpfull
Did you mean that (lower) by "bundle strategy"?
<cachestore-scheme>
<class-scheme>
<class-name>com.griddynamics.ticketon.app.dao.coherence.TicketCacheStore</class-name>
</class-scheme>
<operation-bundling>
<bundle-config>
<operation-name>store</operation-name>
<preferred-size>5000</preferred-size>
<auto-adjust>true</auto-adjust>
</bundle-config>
</operation-bundling>
</cachestore-scheme>
And if yes is it looks sense?
I mean by this, "send records to TicketCacheStore by 5000 per call " am i right?
I dropped delay to 10s and set factor to 0.5
Not coherece send me 5-20k records and cachestore handle whis successfuly.
But! By diferent means it may work longer sometimes, some lock in database for instance.
I want to find durable solution for the case, not only lower a chance i meet one.
Issuing heartbeat from cachestore looks best for me now.
I find that default guardian timeout is 65s and it is not looks as good idea to make it higher.

Write-behind queue resilience

Similar Messages

Maybe you are looking for