Problem with effectiveness of filters

Question

I have at several occasions run into cases were the built in "query optimizer" don’t apply filters in the best order from a performance point of view when dealing with multiple filters joined together using AND-filters.
As a work-around to the problem I have sub classed the built-in filters adding an extra constructor that allow setting the effectiveness as a hard coded integer value that (if specified) is returned instead of the calculated value. This way I was able to force an AND filter to pick the filter I know (from application knowledge) is most likely the most efficient first.
Would it be possible to get this kind of constructor added to the standard filter classes?
I would also like to know how a few things about how indexes are used by Coherence:
1.     How does the various built in filters calculate there effectiveness? Do coherence for instance keep some statistics about how many unique values an index contain or how does it decide what index that is more effective than another?
2.     Can Coherence use more than one index at the same time (i.e. merge indexes)? From my experience it seems like only one index is used fully, other indexes are only used to avoid de-serialization when performing a linear search of the hits from the first index. As far as I know only the most advanced RDBM are able to use more than one index (instead of only performing linear search from the result of one index) so I am not really surprised if Coherence do the same...
3.     Does making an index sorted in any way improve the performance if the index is used as the second or third index applied in a query (one could envision that some sort of binary search could be used instead of linear search of the hits from the first index used if a sorted index is available)?
/Magnus

Answer 1

MagnusE wrote:
I have at several occasions run into cases were the built in "query optimizer" don’t apply filters in the best order from a performance point of view when dealing with multiple filters joined together using AND-filters.
As a work-around to the problem I have sub classed the built-in filters adding an extra constructor that allow setting the effectiveness as a hard coded integer value that (if specified) is returned instead of the calculated value. This way I was able to force an AND filter to pick the filter I know (from application knowledge) is most likely the most efficient first.
Would it be possible to get this kind of constructor added to the standard filter classes?
You can simply create a wrapper filter which implements IndexAwareFilter and wraps another IndexAwareFilter. The calculateEffectiveness should be implemented as you want it, and you can delegate applyIndex() and evaluateEntry() to the wrapped filter.
MagnusE wrote:
I would also like to know how a few things about how indexes are used by Coherence:
1.     How does the various built in filters calculate there effectiveness?I believe they return the number of reverse index accesses as effectiveness if the index exists, and maybe the size of the candidate set if it does not exist.
MagnusE wrote:
Do coherence for instance keep some statistics about how many unique values an index contain or how does it decide what index that is more effective than another?Since the reverse index is just a map between extracted values and a collection of keys, practically this information is available from MapIndex.getIndexContents().size(), if I correctly understand your question, where you can get the MapIndex for an extractor with mapIndexes.get(extractor).
MagnusE wrote:
2.     Can Coherence use more than one index at the same time (i.e. merge indexes)? From my experience it seems like only one index is used fully, other indexes are only used to avoid de-serialization when performing a linear search of the hits from the first index. As far as I know only the most advanced RDBM are able to use more than one index (instead of only performing linear search from the result of one index) so I am not really surprised if Coherence do the same...Yes, any number of indexes can be used, but the stock filters use only a single index (as you can specify a single extractor to the filter). Of course if you form a logical expression between two filters using different extractors, indexes to both extractors will be used if they exist. The applyIndex() and calculateEffectiveness() methods receive all indexes in the mapIndexes parameter, so your custom index-aware filter can use any number of existing indexes at the same time.
MagnusE wrote:
3.     Does making an index sorted in any way improve the performance if the index is used as the second or third index applied in a query (one could envision that some sort of binary search could be used instead of linear search of the hits from the first index used if a sorted index is available)?If you need range queries on the extracted value, sorted index can help a great deal as you don't have to iterate all the keys in the reverse index. This is independent of what other filters you use in your logical expression, the evaluation of the subexpression on the sorted index will still be more efficient than if the index was unsorted.
Best regards,
Robert