Cache of a File System

Question

Would appreciate it if someone could validate my thought process:
I'm looking to put together a large distributed cache and use the file system (shared by all cluster nodes) as long-term storage. My approach has been to: use a distributed scheme with POF serialization, and to create a custom cache store in order to write-behind to Berkeley db
In order to speed up the process of seeding/bulk-loading the cache at start-up I thought that it would be a good idea to partition the data and to write to different files/directories based on my own fairly primitive partitioning strategy, and then to load the data similar to this example: [Performing Distributed Bulk Loading|http://coherence.oracle.com/display/COH35UG/Pre-Loading+the+Cache#Pre-LoadingtheCache-PerformingDistributedBulkLoading]
I later realized that coherence also provides a bdb-store-manager.
My question is:
1. Does the overall approach make sense? I'd like to optimize this as much as possible so if you can think of a better/faster way then please let me know
2. Can I achieve the same by just using the bdb-store-manager and not implementing my own Cache Store?
Thanks,
Kevin

Answer 1

user10766240 wrote:
Would appreciate it if someone could validate my thought process:
I'm looking to put together a large distributed cache and use the file system (shared by all cluster nodes) as long-term storage. My approach has been to: use a distributed scheme with POF serialization, and to create a custom cache store in order to write-behind to Berkeley db
You have to have a separate Berkeley DB for each partition they need to be "mounted" and "unmounted" when partitions move between nodes.
You also have to write a partition listener handling the PARTITION_TRANSFER_* events with "mounting" and "unmounting" the per-partition Berkeley DB instance appropriately as the partition is transferred.
In order to speed up the process of seeding/bulk-loading the cache at start-up I thought that it would be a good idea to partition the data and to write to different files/directories based on my own fairly primitive partitioning strategy, and then to load the data similar to this example: [Performing Distributed Bulk Loading|http://coherence.oracle.com/display/COH35UG/Pre-Loading+the+Cache#Pre-LoadingtheCache-PerformingDistributedBulkLoading]
It should be possible to adapt that example to your needs, e.g. get the keys for all entries in the same partition should be fairly easy by going to the appropriate per-partition BDB instance.
I later realized that coherence also provides a bdb-store-manager.
My question is:
1. Does the overall approach make sense? I'd like to optimize this as much as possible so if you can think of a better/faster way then please let me knowYes.
2. Can I achieve the same by just using the bdb-store-manager and not implementing my own Cache Store?
Thanks,
KevinNo. The BDB store manager is not able to initialize the cache from a non-empty BDB database. It recreates an empty BDB database on restart, thereby you would lose all your data before you could have preloaded it, even if you could configure the BDB store manager to access your data files.
Best regards,
Robert