Best Access Method and Estimated Performance

Question

Dear folks,
I'm new to BDB. Although I have read all the documentation and the DS Presentation from Margo, I would like to ask your help to tunne my db layout.
I have to index 1 million key/data pairs in wich key is a logical number and data is 256 lenght (64 int array).
The size of the database is around 200MB. After the initial seed of data, the data will be READ ONLY. No need for concurrent access. One C program will acess the DB (single threaded).
I'm running on a Dual Xeon 3.2 GHz with 4 gb RAM.
I've choosen to use the QUEUE ACCESS METHOD (since I'm using logical numbers as key and a fixed data size).
I need to randomly access 140.000 keys in the DB.
As I've read from the documentation this would be accomplished in less than a second.
I would like to ask you what would be the expected retrieval time (number of keys/second) for my case (millions of 256 bytes data, with default configuration options) and if QUEUE is the best choice.
From my implementation test the c program takes a considerable amount of time the first time it queries (dpb->get) the database for 140.000 keys (is this the so called COLD START?). The successive times I run the program querying the DB with different 140.000 random keys it becames faster for each new run (is this because I'm warming up the cache?).
I'm using default configuration values for all tunning params: cache, mmapsize, etc...
Would it be possible to load the whole DB in memory before querying it with dpb->get? Would it make the 140.000 querying process faster?
I've tryied to iterate trough every record with a cursor and it takes around 5 seconds the first time and then gets faster on the successive runs. I've also tryied to iterate through every page to "warm up" the process before querying the 140.000 keys but did not have a great improvement.
My goal is to query 140.000 random keys from the DB as fastest as possible. (Be it changing by choosing a more appropriate access method, be it for tunning the config parameters).
I appreciate your help.
Maurice S.

Answer 1

Thank you Michael for your reply!!!
I've seen the thread regarding the strategy of openning the mpool file and iterating through the pages.
I've implemented that sucessfully (that reduced the total time required to perform the "get's" and also make this time a CONSTANT of around 0.5 seconds).
My implementation works the following way:
1) First I create the DB_ENV pointer
1.A) All my dbs are in a folder called "dbs". Therefore I call dbenv->set_data_dir appropriatelly
1.B) dbenv->set_mp_mmapsize(dbenv,300*1024*1024); (each DB has 183 MB)
1.C) I set the cache size to the smallest possible value dbenv->set_cachesize(dbenv, 0, 20 *1024, 0); [as I've read through the forum about similar cases]
1.D) Finally dbenv->open(dbenv, "db",DB_CREATE | DB_RDONLY| DB_INIT_MPOOL, 0))
2) Then I call the db (wich has 700.000 records; with for 4 bytes for each key, and 260 bytes for each data)
2.A) db_create(&dbs[c], dbenv, 0))
2.B) dbs[c]->set_re_len(dbs[c],260)
2.C) dbs[c]->set_pagesize(dbs[c], 64*1024))
2.D) dbs[c]->open(dbs[c],NULL, database, NULL, DB_QUEUE, DB_RDONLY, 0664))
Then I do the "mpool file iteration":
.....all the code from that thread.....
DB_MPOOLFILE* mfp =NULL;
mfp = dbs[c]->get_mpf(dbs[c]);
db_pgno_t lastPageNum;
void* pageAddr;
mfp->get(mfp, &lastPageNum, NULL, DB_MPOOL_LAST, &pageAddr);
mfp->put(mfp, pageAddr, DB_PRIORITY_UNCHANGED, 0);
db_pgno_t pageNum=0;
for(pageNum=0; pageNum<=lastPageNum; pageNum++)
mfp->get(mfp,&pageNum, NULL, 0, &pageAddr);
mfp->put(mfp,pageAddr, DB_PRIORITY_UNCHANGED,0);
And then I get 140.000 random records:
DBT key, data;
for(k=0;k<140000;k++)
db_recno_t recno=rand()%700000;
memset(&key, 0, sizeof(DBT));
memset(&data, 0, sizeof(DBT));
key.data = &recno;
key.size = sizeof(recno);
int dc[65];
data.data = dc;
data.ulen = sizeof(dc);
data.flags = DB_DBT_USERMEM;
dbp->get(dbp, NULL, &key, &data, 0);
It takes around 0.5 seconds to perform all 140.000 random GET's.
I have 20 DB's like the one described above.
Changing the code above in order to use another one of the 20 DB's results in the same EXECUTION TIME (around 0.5 seconds).
Is this the performance that you would expect from BDB? 280.000 random accessess per second in a 700.000 records QUEUE DB with 260 bytes data field?
Is there any other parameters I could tunne to improve performace?
Would BTREE acess method perform better ? (I could index the record numbers as strings: "00000001", "00000002", ..... )
I do appreciate your attention.
Hope my post can be a source for help to other BDB users, as the forum has been to me.
Best rgs,
Maurice S.