Bloom Filter

Question

I'd like to implement a distributed bloom filter using Coherence.
Bloom filter: it is an array of bits (true/false). The size of the array m is determined using the number of keys n we would like to encode. For example, for a given n=16 keys, the size of the bit array is 16*(some-optimality-factor) = 128.
The filter uses k hash functions. We hash each given key using our k hash functions h1,h2,..,hk into the bit array locations i1,i2,...,ik where each i is between 0 and m-1.
The reason I am using the bloom filter is to save space. Say keys are Long, then instead of using n*64 bits, I'd like to use m bits.(less than n*64 bits)
How can I implement this structure efficiently using Coherence?
Message was edited by:
bulut

Answer 1

Let me take a shot a rephrasing/simplifying the question:
There's two important functions that need to be taken care of when implementing a distributed bloom filter:
1- There is the need to distribute a (potentially) very large bit vector.
2- There is the need to apply several accesses (i.e., check several bits) in order to get a single response. The response is a yes/no (i.e., the logical bitand of the bits accesses).
Now suggestions:
a) To solve 1 one, the first thing that comes to mind is to break the bit vector into smaller bit vectors that are keyed by a "bit block" index. For example, if my bitvector contains 800 bits, I may break it up in blocks of 80bits. In that case I could have a distributed map with 10 entries each of them containing an bit vector of 80bits (ie., each entry can contain a byte array of size 10) and the key of each entry can correspond to the offset of its first bit. Now, the question that bulut asked is: Is there a better way to represent a bit vector within a distributed map?
b) in connection to the previous question comes the fact of how to index the bits...if adopting the block partitioning strategy that I just describe, what's the most efficient way to index the bits (and not just the keys that refer to bitblocks)?
c) Finally there's the multiple access question. Since additions and lookups to the filter (i.e., bitarray) are done in groups, it would be highly beneficial to perform them in parallel.
An addition to the filter corresponds to setting k bits in the array in one shot. This would map into probably very distant bits in the array (therefore it does not map to change a single bitblock). That probably calls for using either the invocable map or aggregator functionality...although I'm not much familiar with it so I'm not sure.
In terms of a lookup, there's also an interesting difference from normal access. And that is that given a set of bit offsets, we want to get a yes/no answer (corresponding to the logical and of the values existing in the array). I guess this also has a strong aggregator smell...
Well I hope I clarified more than anything else.
Cheers,
Josep M.