Extremely Poor Read Performance

Question

Hey guys,
For a work project, I have been instructed to use a Berkeley DB as our data storage mechanism. Admittedly, I know little about BDB, but I've been learning more in the past day as I am reading up on it. I'm hoping, though, that even if no one can help me with the problem I am having, they can at least tell me if what I am seeing is typical/expected, or definitely wrong.
Here's what I got:
- Parent table A - Has 0 or 1 key for table B, and 0 or 1 key for table C
- Table B
- Table C
For purpose of discussion, let's ignore table C as it is logically the same as Table B.
Table B has 25 million rows, keyed by a 34-36 digit string, and a payload of 500-1000 bytes.
Table A has 26 million rows, 25 million of which reference the 25 million rows in Table B.
My question is not on the merits of why the data is structured the way it is, but rather about the performance I am seeing, so please refrain from questions such as "why is your data structured that way - can you structure it another way?" I know I can do that - again I just want to know what other people are experiencing for performance.
Anyway, what's happening is this - my program runs a cursor on Table A to get all records. As it gets each record in Table A, it retrieves the referenced records in Table B. So, the cursor on table A represents sequential key access. The subsequent retrievals from Table B represent "random" retrievals - i.e. the key may be anywhere in the index, and is not at all related to the previous retrieval.
Cruising the cursor on Table A, I am seeing performance of about 100,000 records per 2 seconds. However, when I add in the retrievals from Table B, performance stoops all the way down to 100,000 records per 1000 seconds, or better put 100 per second. At this rate, it will take nearly 70 hours to traverse my entire data set.
My question is, am I simply running into a fundamental hardware issue in that I am doing random retrievals from Table B, and I cannot expect to see better performance than 100 retrievals per second due to all of the disk reads? Being that the DB is 20 GB in size, I cannot cache the entire table in memory, so does that mean that reading the data in this fashion is simply not feasible?
If it isn't feasible, does anyone have a suggestion on a different way to read the data, without changing the table relationship as it currently stands? Considering Table B has a reverse reference to table A, I've considered putting a secondary index on table B so that instead of doing random seeks into table B, I can run a cursor on the secondary index of table B at the same time I run the cursor on table A. Then, for each record in table A that has a reference to table B, the first record in the cursor for table B should be the one I need. However, reading about secondary indexes, it looks like all a secondary index does is give a reference to the key to the table. Thus, my concern is that running a cursor on the secondary index of table B will really be no different than randomly selecting the records from table B, as it will be doing just that in the background anyway. Am I understanding this correctly, or would a secondary index really help in my case?
Thank you for your time.
-Brett

Answer 1

Hi Brett,
Is the sorting order the same between the two databases, A and B, that is, are the keys ordered in the same way? For example, to key N in database A, key N in database B is referred.
I would guess not, because you mention the "randomness" in retrieving from B when doing the cursor sequential traversal of A, and the 34-36 digit keys in B are probably randomly generated.
With B as a secondary database, associated to A as the primary database, it would make sense having a cursor on secondary database B to iterate, if you expect that the same ordering of keys in A (as mentioned in the beginning of this post). For example, you would use DBcursor->get to iterate in the secondary database B, or DBcursor->pget if you also want to retrieve the key from the primary database A: DBcursor->get(), DBcursor->pget()
Basically secondary indexes allow for accessing records in a database (primary db) based on a different piece of information other than the primary key:
Secondary indexes
So, when you iterate with a cursor in B you would retrieve the data from A (and in addition the key from A) in the order given by the keys (secondary keys) in B.
However, a secondary database does not seem to me feasible in your case. You seem to have about 1 mil records in primary db A for which you would not have to generate a secondary key, so you would have to return DB_DONOTINDEX from the secondary callback: DB->associate()
(it may be difficult to account exactly for the records in A for which you do not want to generate secondary keys)
Also, the secondary key, the 34-36 digit string, would have to somehow be derived from the primary key and data in A.
If the ordering is not similar (in the sense explained at the beginning of the post) between A and B, then having the secondary index does not too much value, other than simplifying retrieval from A in queries where the query criteria involves the 34-36 digit string.
Back to your current way of structuring data, there are some suggestion that could improve retrieval times:
- try using the latest Berkeley DB release, 5.1.19: Berkeley DB Release History
- try configuring a page size for the databases A and B equal to that of the filesystem's block size: Selecting a page size
- try to avoid the creation of overflow items and pages by properly sizing the page size -- you can inspect database statistics using db_stat -d: db_stat
- try increasing the cache size to a larger value: Selecting a cache size
- if there's a single process accessing the environment, try to back the environment shared region files in per-process private memory (using DB_PRIVATE flag in the DB_ENV->open() call);
- try performing read operations outside of transactions, that is, do not use transactional cursors.
For reference, review these sections in the Berkeley DB Reference Guide:
Access method tuning
Transaction tuning
Regards,
Andrei