How to get total number of result count for particular key on cluster

Question

Hi-
My application requirement is client side require only limited number of data for 'Search Key' form total records found in cluster. Also i need 'total number of result count' for that key present on the custer.
To get subset of record i'm using IndexAwarefilter and returning only limited set each individual node. though i get total number of records present on the individual node, it is not possible to return this count to client form IndexAwarefilter (filter return only Binary set).
Is there anyway i can get this number (total result size) on client side without returning whole chunk of data?
Thanks in advance.
Prashant

Answer 1

user11100190 wrote:
Hi,
Thanks for suggesting a soultion, it works well.
But apart from the count (cardinality), the client also expects the actual results. In this case, it seems that the filter will be executed twice (once for counting, then once again for generating actual resultset)
Actually, we need to perform the paging. In order to achieve paging in efficient manner we need that filter returns only the PAGESIZE records and it also returns the total 'count' that meets the criteria.
If you want to do paging, you can use the LimitFilter class.
If you want to have paging AND total number of results, then at the moment you have to use two passes if you want to use out-of-the-box features because LimitFilter does not return the total number of results (which by the way may change between two page retrieval).
What we currently do is, the filter puts the total count in a static variable and but returns only the first N records. The aggregator then clubs these info into a single list and returns to the client. (The List returned by aggregator contains a special entry representing the count).
This is not really a good idea because if you have more than one user doing this operation then you will have problems storing more than one values in a single static variable and you used a cache service with a thread-pool (thread-count set to larger than one).
We assume that the aggregator will execute immediately after the filter on the same node, this way aggregator will always read the count set by the filter.
You can't assume this if you have multiple client threads doing the same kind of filtering operation and you have a thread-pool configured for the cache service.
Please tell us if our approach will always work, and whether it will be efficient as compared to using Count class which requires executing filter twice.
No it won't if you used a thread-pool. Also, it might happen that Coherence will execute the filtering and the aggregation from the same client thread multiple times on the same node if some partitions were newly moved to the node which already executed the filtering+aggregation once. I don't know anything which would even prevent this being executed on a separate thread concurrently.
The following solution may be working, but I can't fully recommend it as it may leak memory depending on how exactly the filtering and aggregation is implemented (if it is possible that a filtering pass is done but the corresponding aggregation is not executed on the node because of some partitions moved away).
At sending the cache.aggregate(Filter, EntryAggregator) call you should specify a unique key for each such filtering operation to both the filter and the aggregator.
On the storage node you should have a static HashMap.
The filter should do the following two steps while being synchronized on the HashMap.
1. Ensure that a ConcurrentLinkedQueue object exists in a HashMap keyed by that unique key, and
2. Enqueue the total number count you want to pass to the aggregator into that queue.
The parallel aggregator should do the following two steps while being synchronized on the HashMap.
1. Dequeue a single element from the queue, and return it as a partial total count.
2. If the queue is now empty, then remove it from the HashMap.
The parallel aggregator should return the popped number as a partial total count as part of the partial result.
The client side of the parallel aware aggregator should sum the total counts in the partial result.
Since the enqueueing and dequeueing may be interleaved from multiple threads, it may be possible that the partial total count returned in a result does not correspond to the data in the partial result, so you should not base anything on that assumption.
Once again, that approach may leak memory based on how Coherence is internally implemented, so I can't recommend this approach but it may work.
Another thought is that since returning entire cached values from an aggregation is more expensive than filtering (you have to deserialize and reserialize objects), you may still be better off by running a separate count and filter pass from the client, since for that you may not need to deserialize entries at all, so the cost on the server may be lower.
Best regards,
Robert