Advice needed on design issue

Question

Hi
     I have the following problem at hand for which I have been looking at tangosol coherence lately :
     I have a huge deal dataset (~500,000 deals which are scattered across several tables in a database) and I have a process that would need to pull out the entire dataset for processing (several times a day). The database was too slow, so we designed a simple cache that writes to files on disk and keeps the file cache and database in sync.
     The file cache (each deal detail is written to a file, ~5 gigs in size on disk) is now proving to be too slow and we are looking at in-memory caches to speed up the data retrieval process. The deal dataset is not highly dynamic (typically 400-500 updates take place per day) but it is accessed by several consumers.
     We are looking at tangosol coherence to alleviate our problems. I am looking for a setup which will allow me to query an in-memory cache for data and if something bad happens and the in-memory cache goes down, I could fall back upon the file cache on disk.
     Is there a caching scheme that would allow me to use the existing file system cache as a backup (instead of replicating in memory?). We are a bit apprehensive about removing the file cache altogether. Also could you advise on what caching scheme would work best for such a setup? Any help would be highly appreciated.
     Thanks much
     Sachin

Answer 1

Hi Sachin,
     with the Coherence write-through/write-behind feature you can have an arbitrary persistent storage for your cached data (so you can write to a file instead of a DB).
     However, if you want to use files for such persitent storage, then you should use a network filesystem which is visible the same way on each storage node, which again might be slower. This is required because write-through is usable only with partitioned caches and in that case each node stores only a part of the entire dataset to the disk.
     However, you should also try Coherence with write-through/write-behind to the disk.
     Also, you should leverage all possible optimizations Coherence provides to you, e.g. if you had problems with slow data transfer speeds, then you can reduce the amount of transferred data with ExternalizableLite, speed up queries with indexes, use aggregations to possibly transfer only parts of the data entries, use a near cache for the most frequently used entries when retrieving them by key, etc.
     However it will be a very rare occasion that Coherence would go down without external intervention.
     Please be aware though, that to query Coherence caches to return the same kind of data as originally, the entire data-set should be cached.
     Just my 2 cents, of course...
     Best regards,
     Robert