Querying large amounts of data

Question

Suppose I want to store an amount of data that is too large to store in memory only (serveral GB's or TB's). The data needs to be highly available and fault tolerant and a user would need to query the data based on some criteria. Coherence would fit these requirements perfectly except for the fact that you would need to keep all data in memory before being able to query it using the filter API.
As I see it, the only possible way to make this happen is to store all this data in a database via a write-behind schema and query the db instead of the cache. This would make the db the single source of data instead of the cache. However, this would also seriously diminish the usefullness of the cache as you would now need to make sure the db is replicated and fault tolerant. You wouldn't even need the cache anymore as it's only purpose would be to pass the data to the db via the write-behind schema.
Is there another way to query the data not using a db query but using the coherence filter API and still storing the data somewhere outside the JVM's memory?
Best regards
Jan

Answer 1

user10601659 wrote:
Suppose I want to store an amount of data that is too large to store in memory only (serveral GB's or TB's). The data needs to be highly available and fault tolerant and a user would need to query the data based on some criteria. Coherence would fit these requirements perfectly except for the fact that you would need to keep all data in memory before being able to query it using the filter API.
As I see it, the only possible way to make this happen is to store all this data in a database via a write-behind schema and query the db instead of the cache. This would make the db the single source of data instead of the cache. However, this would also seriously diminish the usefullness of the cache as you would now need to make sure the db is replicated and fault tolerant. You wouldn't even need the cache anymore as it's only purpose would be to pass the data to the db via the write-behind schema.
Is there another way to query the data not using a db query but using the coherence filter API and still storing the data somewhere outside the JVM's memory?
Best regards
JanHi Jan,
three things come to my mind regarding your post:
1. Write-behind is not usable if you want the DB to be the system of record. With write-behind the cache is the system of record and not the backing storage. Will the data change? If yes, how frequently? What system changes the data?
2. To query the cache with such high amount of data with the Filter API you practically have to have all your data indexed as otherwise query response times would be too high if you needed to deserialize even a subset of your data. But indexes always have to reside on the heap. So the question is how complex your queries would be? The more complex the queries, the larger amount of indexes you would likely need.
3. You can query only a single cache with a Coherence filter. Would this be sufficient for your querying needs?
Best regards,
Robert