Build a dictionary with bulk records

Question

I have 100 million records where key and value are both 64-bit integers. I need to read the records from a flat file and build a dictionary in DB format so that I can load it for random access. During building, DB runs very slow when the cache is used up (I tried both BTree and Hash). I don't need to change the records once the dictionary is built, so is there anyway to optimize for this situation? Thanks!
Message was edited by:
user583103

Answer 1

Hi,
Since you'll access the key/data pairs in random order I assume you haven't defined your own Btree comparison function (you don't need the keys to be in order).
When using the Hash access method you should ensure that your hash function works well; also, you should carefully set the fill factor and the hash table size:
http://www.oracle.com/technology/documentation/berkeley-db/db/ref/am_conf/h_hash.html
http://www.oracle.com/technology/documentation/berkeley-db/db/ref/am_conf/h_ffactor.html
http://www.oracle.com/technology/documentation/berkeley-db/db/ref/am_conf/h_nelem.html
To determine if your cache is sized properly and to get statistics on your database use the db_stat utility like this:
#db_stat -d <db_file>
#db_stat -m -h <env_home>
http://www.oracle.com/technology/documentation/berkeley-db/db/utility/db_stat.html
Try both access methods, and post here information on how you've configured in each situation, and the output of db_stat.
Now, when you load the data, that is, reading it from the flat file and writing it to the BDB database you won't need any durability guarantees, as the process can be restarted in case of a failure, and the load will start from scratch. So, I suggest not writting the records transactionally. After you have completed loading the data you can take a backup of the database, and open the database within an environment configured with transactions and logging in place.
Nevertheless, if you need a certain degree of certainty in the face of a crash when loading the data you could configure your transactional environment with DB_TXN_NOSYNC or DB_TXN_WRITE_NOSYNC. More information on transaction tuning and throughput is here:
http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/tune.html
http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/throughput.html
Regards,
Andrei