Problem about duplicate use

Question

Hi,All
I have used je with
dbConfig.setSortedDuplicates(true);
Then I stored my data just using
db.put(key, value);
The problem is that data bloated seriously.My key of every record is int，and the value is about 100 bytes.I have put about 3,000,000 records,but the data size is about *3GB*.
I have read the doc and found that:
If a key value has duplicate data values,then the BIN references a duplicate internal node (DIN), which is the root of a tree that stores all the duplicate data
items. Duplicate trees are structured like database trees, with INs replaced by DINs and BINs replaced with DBINs. Thus, a database is potentially a tree of
trees.
As BIN stores key and LSN of its childs nodes ,the DBIN stores values and LSN of ite dup nodes.
In one word,Je stores values at DBIN,which means the size of our data is doubled(Maybe more).It is a nightmare.
How could I do when faced this?
Thanks
yun

Answer 1

Hi Yun,
We are aware that the per-record overhead for duplicates can be high and we are intending to improve this in a future release. In the meantime, I may be able to help you work around the problem.
First, post your EnvironmentConfig settings (and je.properties, if any). If your cache size is too small, this could also be causing a high per-record overhead.
Second, post your schema -- the types of records, relationships between records, key sizes, data sizes, etc. How many records per key (on average) are you inserting in a duplicates database? With a small number of records per key, the per-record overhead in a duplicates database is quite high.
It may be that you can meet your needs without using duplicates. For example, in your duplicates database, if the data size is fairly small, then rather than using a duplicates database you can use a non-duplicates with a two-part key and empty data. The first part of the new key is the key you are using now, and the second part is the data you are using now. This approach has proven to be very effective. I can explain further if this approach turns out to be something that would work for your application.
--mark