Need a way to speed up the initial population of the entire BDB

Question

We are transitioning an existing production system from flat file based DB to BDB. On the day we switch to BDB, we have to populate the BDB for more than 2 Million users. Once we populate, BDB serves the data vary adequately.
However, the initial population is very slow. For some users with large set of keys, it takes up to 30 minutes. Just the thought of server keeling over creating the initail DB for all these users is keeping me awake at nights :-) I was wondering if there are any known tricks I can use to speed up this initial population.
One thing that comes to mind is, disable BDB caching as we don't need to cache any entries during population. Is this possible? Profiler reports that BDB is spending the maximum time in eviction (Specifically xxx.tree.BIN.getChildEvictionType method).
Does use of transaction or no transactions affect this use case?
I have all the objects to put in DB in-memory. Is there some batch-put I can do to speed the operation?
Highly appreciate any response.
Cheers!

Answer 1

Hi,
I will try deferredWrite mode next. Good, I think it's worth trying. The DeferredWrite mode was originally added as a way to speed up database loading.
Is it possible for me to avoid opening secondary
database until after the full population of the
primary database? Yes. See the SecondaryConfig.setAllowPopulate method. You can use this you load your database as follows:
1) Open the primary DB (openDatabase) but do not open any secondary DBs (openSecondaryDatabase).
2) Load all records into the primary DB.
3) Create a SecondaryConfig and call setAllowPopulate(true). Use this config object to call openSecondaryDatabase.
Note that when opening a secondary DB with AllowPopulate, the secondary DB will be populated only if it is initially empty or non-existent. It is populate by reading the entire primary DB and writing secondary key records as necessary.
Also, even though we started out with entire load
in-memory before population, we had to abandon that
approach because of enormous memory constraint it put
on the system. However data is being generated from
walking a directory structure. Is there any thing I
can do there (depth first vs breadth first search)
that will help BDB do the inserts faster?Presorting the input records by primary key will provide a significant performance increase. If you can't presort them all, it will still help a lot to presort a subset at a time. In other words, trying loading them into memory in batches, where the size of the batch is determined by available memory. After loading a batch into memory, presort it, and then insert the records in sorted order into the JE database.
Thanks as always. You guys are doing such a great job
answering questions and providing support.You're welcome! (on my and Linda's behalf)
--mark