Document size performance considerations

Question

We are trying to determine the performance implications of different approaches to document storage with BDB. Most of the XML we need to store will contain anywhere between 5000 to 20000 nodes at about 6MB per XML block. Best practices or document size analysis/data breakdown isn't a topic that is well explained on the docs websites. We are working primarily in a single-threaded environment.
1. Does performance degrade considerably when BDB containers hold documents upwards of 500MB? 2GB? or should documents generally be small? (Assuming Node type storage)
2. Does anyone know of or have any best practices for data breakdown and storage within BDB?
And completely unrelated:
3. Are Environments completely portable when fully moved to different systems?
Many thanks.

Answer 1

Hi Taka,
1. Does performance degrade considerably when BDB
containers hold documents upwards of 500MB? 2GB? or
should documents generally be small? (Assuming Node
type storage)Documents being written need to be parsed (larger docs take longer), inserted into the database and the appropriate indices updated (the more indices, the longer it takes). I think that the best thing to do is to build a prototype of your application, populate a database and benchmark the performance.
3. Are Environments completely portable when fully moved to different systems?There are two issues with copying or moving databases: database page log sequence numbers (LSNs), and database file identification strings.
Because database pages contain references to the database environment log records (LSNs), databases cannot be copied or moved from one transactional database environment to another without first clearing the LSNs. Note that this is not a concern for non-transactional database environments and applications, and can be ignored if the database is not being used transactionally. Specifically, databases created and written non-transactionally (for example, as part of a bulk load procedure), can be copied or moved into a transactional database environment without resetting the LSNs. The database's LSNs may be reset in one of three ways: the application can call the DB_ENV->lsn_reset method to reset the LSNs in place, or a system administrator can reset the LSNs in place using the -r option to the db_load utility, or by dumping and reloading the database (using the db_dump and db_load utilities).
Because system file identification information (for example, filenames, device and inode numbers, volume and file IDs, and so on) are not necessarily unique or maintained across system reboots, each Berkeley DB database file contains a unique 20-byte file identification bytestring. When multiple processes or threads open the same database file in Berkeley DB, it is this bytestring that is used to ensure the same underlying pages are updated in the database environment cache, no matter which Berkeley DB handle is used for the operation.
The database file identification string is not a concern when moving databases, and databases may be moved or renamed without resetting the identification string. However, when copying a database, you must ensure there are never two databases with the same file identification bytestring in the same cache at the same time. Copying databases is further complicated because Berkeley DB caches do not discard cached database pages when database handles are closed. Cached pages are only discarded when the database is removed by calling the DB_ENV->remove or DB->remove methods.
Bogdan Coman