Scalability issues w/rt large documents

Question

I use the following abbreviations in my query.
BDB = Berkeley DB
BDBXML = Berkeley DB XML
After reviewing the docs, I have two major gating issues about BDBXML's handling of large XML documents. By large, I mean documents that are can be between 2MB or several 10s of MB, particularly w/ frequent repetition of the keys.
(A) XML documents tend be very verbose w/ the repetition of keys. RDBMSes avoid that verbosity by encoding the keys into table column names. We want to use an embedded DB to eliminate the dependency on an RDBMS. However, this verbosity of XML documents could kill us b/c our XML "documents" are going to be relatively large w/ a lot of repetition of keys. The key question here is, will our BDBXML documents use a correspondingly large amounts of storage? Or does BDBXML optimize storage utilization somehow via such a mechanism as internment of repeated keys' strings?
(B) Depending on the answer to (A), this particular issue may not be a concern. The BDB docs strongly warns programmers to minimize overflow-page utilization; and gives detailed advice for setting the page-size to an appropriate value, starting w/ the block-size of the underlying filesystem then increasing the value to optimize etc., etc., the core idea being to minimize use 'overflow pages'. however, for documents on the order of megabytes, utilization of overflow pages is unavoidable w/o an encoding scheme that breaks the document into smaller parts. based on the BDBXML docs, it is not clear to me that BDBXML optimizes use of large XML documents. So - my question is, how does BDBXML handle large XML documents so as to avoid the performance degradation that accompanies overflow pages?
-rlehr
Software Architect
Cadence Design Systems, Inc.

Answer 1

George,
Thank you very much for clarifying those aspects.
RE (A) - Although query speed concerns me, the "bloated" database files currently concerns me more. "Bloated" databases have been a concern for us previously. We might be able to compensate for it in our app's current rev. however, that is still uncertain. the issues therefore remain.
Speed also concerns since, after the index search, the data still needs to be loaded and decoded from string to native data. however, that is still uncertain. the issues therefore remain.
If I understand correctly, BDB XML treats the XML documents as a hierarchical structure of nodes that are either
* simple string/string key/value pairs - two categories for these, attributes and elements
* string/node key/value pairs
This reduces essentially to a hierarchical structure of string/string key/value pairs. IOW, a lot of string/string key/value pairs are being written to storage and indexed.
Two aspects of this problem have me considering using bare BDB instead of BDB XML.
(A) to eliminate the add'l overhead that accompanies
[1] the add'l IO that accompanies plain-text vs native data
[2] the add'l CPU ops for converting between string and native data
(B) our application's data is pretty much well structured. we KNOW what we will be storing. so that aspect of the XML storage has little value for us. That being said, BDB XML offers a potential simplicity that would be valuable.
In one sense, this is uninteresting for you. OTOH, that summarizes my concerns w/rt BDB XML vs BDB or another solution, to which you can respond with add'l insight, having worked w/ BDB XML much more extensively. I am interested in your perspective.
Again, thanks. I look forward to your reply.
-rlehr