What's the best strategy for storing URLs as keys?

Question

I need to create an index that maps URLs to 8-byte longs. I know that I can just use StringBinding and LongBinding to create my entries, however I'm wondering if there's anything that I could do to make more efficient use of memory. I need quick lookups, but I also need to store a lot of data (600M URLs and growing). The average length of a URL is about 90 characters.
Since I don't know what BDB actually keeps in memory at runtime, I'm wondering what advantage there is (if any) from using hashes for the keys instead of the full URLs. I'd have to store the URL in the data to distinguish hash collisions. There's obviously a trade-off between the size of the hash output and the number of collisions. I can't imagine that the lookup time would suffer that much if I have to iterate through all of the records that hash to a particular key, but I'd like to know if you have a reason to contradict that assumption.
In summary, I'd like to know what BDB would prefer from a performance standpoint (both size and speed):
+ smallest key/data records (store URL as key, long as data)
+ smaller keys/larger data/minimal collisions (store 160-bit hash of URL as key, URL and long as data)
+ even smaller keys/same data/more collisions (store 64-bit hash of URL as key, URL and long as data)
The key size would also be much more consistent if I use a hash function. I don't know if BDB cares about that.
I know that every situation is different and that I could just run my own performance tests (and I will), but it takes a few days to load the data into the index, so I'd like to call on the experience here to avoid making any really bad decisions and save me some time.
Thanks,
-Justin

Answer 1

Hi Justin,
I know you're not looking for an "it all depends" answer, but it does depend largely on how much of the data set you're accessing fits into the JE cache. If you're not sure about this and can't measure it by running your app and looking at the EnvironmentStats, please run the DbCacheSize utility. There's an FAQ about it, and several OTN threads.
If the large majority of the data set you access frequently fits into the cache, you'll get maximum performance with the first option you mention (no hashing), since you don't have to filter out hash collisions.
If I/O to read from disk is a big factor (because the JE cache isn't large enough), then you may be able to reduce this problem by trying to fit more into the JE cache.
Reducing the key size by using a hash is one way to fit more data into cache, since the keys are stored multiple times in the Btree internal nodes. However, unless the records with the same hash are likely to have been read recently, reading the colliding records will result in more random reads, and that will probably outweigh the advantage. JE stores each record separately on disk in write order, so records with the same hash key will not be stored in "pages" together -- JE has no pages.
So instead, you may want to try configuring key prefixing, if the URLs are likely to have common prefixes. See DatabaseConfig.setKeyPrefixing.
If that isn't sufficient, you could use a hash key and store all the records with that hash in a single record. In other words, you could store multiple logical records per JE record. This is quite a bit more work on your part, since you have to handle the packing and unpacking, as well as deletions, etc. For that reason I don't recommend it, but we have had some users do this successfully.
--mark