How to efficiently get hold of N randomly selected keys out of X?

Question

I am writing some benchmark programs for Coherence caches and would need a way to as efficently as possible get hold of a "fairly random" selection of N keys from the cache out of the total X keys (where X >> N).
With "fairly random" I am in particular looking for a way to get keys somewhat uniformly spread over all partitions.
First i tried picking the first N keys obtained when iterating the key-set but they all seemed to come from just a few partitions (perhaps iteration is performed partition by partition?). Next I tried some algortihms I have used in the past to pick random keys from in memory-map (where one basically iterate and if a randomly generated value between 0.0 and 1.0 exceed a percentage calculatwed as the factor N/X the key is included. This does work (one may occasionally need to iterate the map more than once) but is VERY slow since iterating the key-set of a large cache remotly is kind of expensive.
All sugestions are warmly appreciated - the best would be solutions that do not require anything to be added to the pof-config (i.e. not rely on custome aggregators or invocables etc) - this because I would liek to be able to easilly use my benchmark as part of several applications without having to modify there configuration files!
/Magnus
Edited by: MagnusE on Aug 7, 2009 10:21 AM

Answer 1

Hi Magnus,
you can go the other way round:
With key-association you can ensure that your request goes to the partition you want it to go. You just need to generate a reverse mapping array between partition ids and integer associated key values which are mapped to that partititon.
You can use this method for it:
public static int[] generateReverseAssociatedKeysForService(DistributedCacheService service) {
     KeyPartitioningStrategy keyPartitioningStrategy = service.getKeyPartitioningStrategy();
     int partitionCount = service.getPartitionCount();
     int[] reverseKeys = new int[partitionCount];
     int i=1;
     while (partitionCount > 0) {
          Integer associatedKey = new Integer(i);
          int partitionId = keyPartitioningStrategy.getKeyPartition(associatedKey);
          if (reverseKeys[partitionId] == 0) {
               reverseKeys[partitionId] = i;
               --partitionCount;
          ++i;
     return reverseKeys;
}After this, you generate an equal amount of entries for each partitions with composite keys which implement key association in a way that it returns an integer which maps to your designated partition id to serve as your data. Alternatively you can sort an existing key-set into per-partition keysets.
After this, you just generate evenly spread random numbers between 0 and (partitionCount - 1) (both inclusive) which random number you use as an index to the reverse map to get an associated key value. Then you can generate another random number if you have multiple keys within a partition, or choose a key from the per-partition keyset for that partition if you have a set of existing keys.
This way you get keys which are as evenly spread between partitions as your random generator generating your associated key indexes is spread.
Best regards,
Robert
Edited by: robvarga on Aug 7, 2009 11:55 AM
Added some more ideas and method implementation.