Working with Large Blobs in BerkeleyDB

Question

Hi Everyone,
I'm trying to use BerkeleyDB as a simple key/value store for large (~50 MB) blobs of data.
Before I get too deep into this, I wanted to see if anyone had any advice as to the best way to configure Berkeley to do this efficiently. Some of the areas I could imagine being important are...
1.) Whether or not to use a BTree index.
2.) How cache and page sizes should be configured.
3.) If the blobs are very large but divisible, are there any best practices around partitioning?
Thanks,
Caleb

Answer 1

Hi,
That's an interesting question. When Berkeley DB stores data items they are broken up into "page size" pieces, and stored on a number of different pages. The best advice is to configure large page sizes when dealing with larger data items, that will result in us splitting the data into less pieces, thus reducing the cost of re-assembling the data to service read requests.
Do you really need to store this actual data in a database file? Would it be practical for you to just keep metadata and reference information in the database, and store the actual blob data on disk? That configuration will likely deliver far superior performance - though you do loose the ACID guarantees on the blob data.
Answers to your specific questions:
1.) Whether or not to use a BTree index.
I would use a btree database (index) for your situation.
2.) How cache and page sizes should be configured.
A cache that can hold the entire data set, and the largest possible page size (64k) would be my recommended configuration.
3.) If the blobs are very large but divisible, are there any best practices around partitioning?
I'd say that you are best to split the blobs into smaller data items in that scenario. The smaller the better :)
Regards,
Alex Gorrod
Oracle Berkeley DB