Questions for Berkeley DB gurus regarding performance

Question

Dear BDB Gurus,
We are building an application that is about to store millions of pairs (hash/data). We are planning to use Berkeley DB as holder of the pairs hash/data and are planning to store thousands of millions of records.
What would be the best file system on Linux for such large BDB?
We would like to distribute the load. A machine will have multiple mount points (separate drives/LUNs) and we would like to get the highest parallelism possible. We believe that using multiple instances of BDB on each mount point would give us the best performance (we are dividing the content, or we could use the new feature from 4.8 - partitioning). Is this correct? Is there anything special we must be aware of, because preliminary results show that if we have more than ~100 BDB files (databases) per mount point the performance drops dramatically. On the other hand if we have smaller number of BDB files, that means each BDB will grow very big. What are the biggest size that we could grow the BDB files to (file size and number of records)? Our mount LUNs/drives are approximately 2TB in size. The average data size is between 4kB and 8kB (let's take 6kB for simplicity)while the hash is 160 bits (SHA-1). If we want to fill up the 2TB with records (each being 6kB) that would be about 333 million records. What would be the best way to store such large number of records?
Should we preallocate the size of the BDB file or let it grow automatically?
What is the maximum number of records that we could store in each BDB? The point is not to grow beyond reasonable number (file size and number of records).
What can we do to make the reads and writes very fast - is there anything special that we could fine tune the performance (we already tried different page sizes)?
The hash is SHA-1 and all the keys and database instances are evenly distributed due to the nature of the hashing.
We create DB_Env structure for each mounted data disc. Here is an extract from the code:
Create DB_ENV
ret = db_env_create(&envp, 0);
if (ret != 0) {
ERR("StorageRoot: Error creating env handle: %s\n", db_strerror(ret));
envp = NULL;
return NULL;
/* we can optimize this parameter in DB_CONFIG file */
ret = envp->set_cachesize(envp, 0, _1M * 64, 0);
if (ret != 0) {
ERR("StorageRoot: DB->set_cachesize failed: %s.\n", db_strerror(ret));
return NULL;
/* we can optimize this parameter in DB_CONFIG file */
ret = envp->mutex_set_max(envp, 100000);
if (ret != 0) {
ERR("StorageRoot: DB->mutex_set_max failed: %s.\n", db_strerror(ret));
return NULL;
envp->set_errpfx (envp, "store");
envp->set_errcall (envp, bdb_err_callback);
envp->set_msgcall (envp, bdb_msg_callback);
envp->set_data_dir(envp, sr->path);
/* Open the environment. */
env_flags = DB_CREATE | DB_RECOVER | DB_INIT_TXN | DB_THREAD | DB_INIT_MPOOL;
ret = envp->open(envp,
env_home, /* env home directory */
env_flags, /* Open flags */
0); /* File mode (default) */
if (ret != 0) {
ERR("StorageRoot: Environment open failed: %s", db_strerror(ret));
return NULL;
Create DB
/* Initialize the DB handle */
ret = db_create(&dbp, envp, 0);
if (ret != 0) {
ERR("Store: store_open -- %s\n", db_strerror(ret));
store_destroy(handle);
return NULL;
/* Point to the memory malloc'd by db_create() */
handle->dbp = dbp;
/* Set the open flags */
open_flags = DB_CREATE | DB_THREAD; /* Allow database creation */
/* Set the page size to 8K */
dbp->set_pagesize(dbp, _8K);
/* Now open the database */
ret = dbp->open(dbp, /* Pointer to the database */
NULL, /* Txn pointer */
handle->db_name, /* File name */
NULL, /* Logical db name */
DB_HASH, /* Database type (using hash) */
open_flags, /* Open flags */
0); /* File mode. Using defaults */
if (ret != 0) {
dbp->err(dbp, ret, "Database '%s' open failed.", handle->db_name);
ERR("Store: store_open -- Database '%s' open failed with code %d (%s).
\n",
handle->db_name, ret, db_strerror(ret));
store_destroy(handle);
return NULL;
Your help is greatly appreciated. Thanks in advance.
Alex

Answer 1

Hi Alex,
We are building an application that is about to store millions of pairs (hash/data). We are planning to use Berkeley DB as holder of the pairs hash/data and are planning to store thousands of millions of records.
What would be the best file system on Linux for such large BDB?On Linux you should use the ext3 file system. Please make sure that you use the default level of journaling when mounting the file system, that is, the ordered data mode:
[http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/build_unix_linux.html]
[http://batleth.sapienti-sat.org/projects/FAQs/ext3-faq.html]
We would like to distribute the load. A machine will have multiple mount points (separate drives/LUNs) and we would like to get the highest parallelism possible. We believe that using multiple instances of BDB on each mount point would give us the best performance (we are dividing the content, or we could use the new feature from 4.8 - partitioning). Is this correct? Correct. This will help improve disk I/O.
Is there anything special we must be aware of, because preliminary results show that if we have more than ~100 BDB files (databases) per mount point the performance drops dramatically. On the other hand if we have smaller number of BDB files, that means each BDB will grow very big. What are the biggest size that we could grow the BDB files to (file size and number of records)? Our mount LUNs/drives are approximately 2TB in size.Please review the following for details on limitations on the database size and number of records that can be stored:
[http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/am_misc_dbsizes.html]
[http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/am_misc_diskspace.html]
The average data size is between 4kB and 8kB (let's take 6kB for simplicity)while the hash is 160 bits (SHA-1). If we want to fill up the 2TB with records (each being 6kB) that would be about 333 million records. What would be the best way to store such large number of records?
Should we preallocate the size of the BDB file or let it grow automatically?
If you have access to My Oracle Support (ex Metalink) you can look into two interesting KM articles I wrote a while ago: Note 463507.1 and Note 463613.1.
Tuning the hash configuration values (page fill factor, database hash function, hash table size) is only useful if it doesn't dramatically alter the caching behavior of your application (i.e. it doesn't require that you always run in the cache).
Also, pre-sorting the keys will likely boost insert performance. If you know in advance the approximate number of elements that will be stored in the hash database (specified via DB->set_h_nelem ) and the estimated number of elements per page (specified via DB->set_h_ffactor), then you can estimate the number of buckets that will reside in the hash table. The number of buckets is determined using the formula:
no_of_keys / no_of_items_per_page
and the hash fill factor is determined using the formula:
*(pagesize - 32) / (average_key_size + average_data_size + 8)*
The no_of_items_per_page can be calculated based on the average record size and the useful-bytes-per-page. See the Hash specific formulas here: [http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/am_misc_diskspace.html]
To pre-sort the keys the appropriate hash function on each of those keys needs to be called (your own hash function set via DB->set_h_hash, or the default hash function __ham_func5, in BDB_SOURCE_TREE/hash/hash_func.c) and then take the result modulo no_of_buckets (the estimated number of buckets). Thus, a hash bucket where the key will be stored is identified. Sorting the keys this way should result in best-case insertion. If the initial number of elements in the hash table is not specified, then the hash table will grow dynamically and, although the items are sorted, a slowdown may be observed, due to frequent rehashing.
What is the maximum number of records that we could store in each BDB? The point is not to grow beyond reasonable number (file size and number of records).See the links on database size limits and disk space requirements.
What can we do to make the reads and writes very fast - is there anything special that we could fine tune the performance (we already tried different page sizes)?See the suggestions from the [Berkeley DB Programmer's Reference Guide|http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/index.html] on tuning:
[http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/am_misc_tune.html]
[http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/transapp_tune.html]
[http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/transapp_throughput.html]
Regards,
Andrei