Berkeley DB Storage related questions

Question

Hi, I am new to BDB.
We recently migrated our internal DB into BDB (we are using the C++ version). Here are some of the details:
Data is saved using hash table (key has a fixed size of 16 bytes, but data size varies - from 20 bytes to a few kbs)
We have an index on each record (the index has fixed size of 4 bytes). Index using btree.
We do about 200 read, 200 write (update or insert), 200 delete every second. Update will always change the index.
Over time, the number of records in the DB only increases slightly (total is about 2 million records).
In the last month or two we have observed the following storage change:
When the internal DB was converted into BDB, the data was about 8GB, index about 1GB. The actual data (key, data, index) we put into the BDB counts about 50% of the actual storage (the other 50% is BDB's overhead/other data structure, etc). Now, the data becomes about 9GB (slightly more than our data grows rate), but the index db size has grown to 9GB as well. Since the number of records are pretty much the same, the index DB has a lot of waste.
My questions:
Are there any parameters (eg fill factors) that could affect the storage usage? We just used the defaults.
Over time, how to defragment the DB without taking the DB off line?
Thanks in advance for your help.

Answer 1

Hi John,
So you are using the C++ API. What is the BDB release you are using (e.g. 5.1.19, 5.0.26 etc), on what OS, filesystem, and what is the underlying filesystem's I/O block size? (to get the block size, use something like "dumpe2fs /dev/hdN | grep 'Block size'" on Linux, "fsutil fsinfo ntfsinfo D:" on Windows -- look for "Bytes Per Cluster", "df -g /part_mount_point" on Solaris)
Note that Btree and Hash databases are grow-only, that is, space freed by deleting items from a Btree or Hash database is never returned to the filesystem, although it is reused where possible; see the Disk space requirements section in the Berkeley DB Reference Guide.
You can use compaction to reclaim disk space and shrink the databases' sizes; see the Db::compact() method. If you need clarification on the compact() method let me know; also, there are a lot of useful threads on the forum dealing with compaction, just do a search on "compact database".
Data is saved using hash table (key has a fixed size of 16 bytes, but data size varies - from 20 bytes to a few kbs)More precisely, to how many "few" Kbs?
the data becomes about 9GB (slightly more than our data grows rate), but the index db size has grown to 9GB as wellCould you check that the secondary key creator callback function is correct? It's a little bit surprising to see the secondary index database growing from 1GB to 9GB.
Are there any parameters (eg fill factors) that could affect the storage usage? We just used the defaults.Yes, the most important is the database page size which will influence the fill factor; see Selecting a page size.
As your data item size varies to a few Kbs your database probably has a lot of overflow pages (these pages store overflow items which are too large to be stored entirely onto a normal leaf page - an overflow item will be stored onto multiple overflow pages; retrieving such items is more expensive as the entire set of pages needs to be read into cache).
Also, for specific configuration for each of the two access methods see Btree access method specific configuration and Hash access method specific configuration.
Use the db_stat utility with the -d option and provide the database statistics for each of the databases (primary and secondary).
Over time, how to defragment the DB without taking the DB off line?Use compaction as initially mentioned; compaction is done online, that is, you do not have to close the database (take it offline).
Regards,
Andrei