Database much larger than expected when using secondary databases

Question

Hi Guys,
When I load data into my database it is much larger than I expect it to be when I turn on secondary indexes. I am using the Base API.
I am persisting (using TupleBindings) the following data type:
A Key of size ~ *80 Bytes*
A Value consisting of 4 longs ~ 4*8= *32 Bytes*
I am persisting ~ *280k* of such records
I therefore expect ballpark 280k * (80+32) Bytes ~ *31M* of data to be persisted. I actually see ~ *40M* - which is fine.
Now, when I add 4 secondary database indexes - on the 4 long values - I would expect to see approximately an additional 4 * 32 * 280k Bytes -> ~ *35M*
This would bring the total amount of data to (40M + 35M) ~ *75M*
(although I would expect less given that many of the secondary keys are duplicates)
What I am seeing however is *153M*
Given that no other data is persisted that could account for this, is this what you would expect to see?
Is there any way to account for the extra unexpected 75M of data?
Thanks,
Joel
Edited by: JoelH on 10-Feb-2010 10:59
Edited by: JoelH on 10-Feb-2010 10:59

Answer 1

Hi Joel,
Now, when I add 4 secondary database indexes - on the 4 long values - I would expect to see approximately an additional 4 * 32 * 280k Bytes -> ~ 35MA secondary index consists of a regular JE database containing key-value pairs, where the key of the pair is the secondary key and the value of the pair is the primary key. Your primary key is 80 bytes, so this would be 4 * (4 + 80) * 280k Bytes -> ~ 91M.
The remaining space is taken by per-record overhead for every primary and secondary record, plus the index of keys itself. There are (280k * 5) records total. Duplicates do not take less space.
I assume this is an insert-only test, and therefore the log should contain little obsolete data. To be sure, please run the DbSpace utility.
Does this answer your question?
--mark