Optimizing cursor read performance

Question

I frequently iterate through an 8gb table with a cursor (using the C api on a btree table), and find that i/o read performance can be quite poor on my production system, with iostat showing 5mb/second read speeds.
However, when I db_dump / db_load the table, read performance goes up significantly (60mb/second on my imac, which I develop on).
When times, I'm able to iterate over my 8gb (1.5 million record) table in 140 seconds after dump/load. Before the dump/load, it took about 20 minutes. File size is not much reduced after the dump/load.
My suspicion is that the production database, that has constant read/writes to it, is fragmented, and thus perhaps the data is not optimized for btree cursor scanning of the entire table.
So... my question is...
What should I do to maintain the highest level of read performance on a large database? Will DB->compact provide me with the same performance level that a full dump/load will?
I could periodically down my web site do do a dump/load, but with 12gb of data, that would be a significant downtime. I'm also nervous about dump/load in a logging environment (ie, not losing any data in the log but not yet in the tables)
Another question, that I suspect the answer is "no" to, is: is there a way to read records out of a table in written-to-disk order, so as to optimize disk drive read head performance? i.e., avoiding asking the disk drive head to move all over the table as the cursor iterates.
-john

Answer 1

db_dump (the utility itself), when executed with standard options, calls db->dump(). So what one has to look for is what __db_dump() itself is doing:
1048         /*
1049          * Get a cursor and step through the database, printing out each
1050          * key/data pair.
1051          */
1052         if ((ret = __db_cursor(dbp, NULL, &dbcp, 0)) != 0)
1053                 return (ret);
1054
1055         memset(&key, 0, sizeof(key));
1056         memset(&data, 0, sizeof(data));
1057         if ((ret = __os_malloc(dbenv, 1024 * 1024, &data.data)) != 0)
1058                 goto err;
1059         data.ulen = 1024 * 1024;
1060         data.flags = DB_DBT_USERMEM;
1061         is_recno = (dbp->type == DB_RECNO || dbp->type == DB_QUEUE);
1062         keyflag = is_recno ? keyflag : 1;
1063         if (is_recno) {
1064                 keyret.data = &recno
1065                 keyret.size = sizeof(recno);
1066         }
1067
1068 retry: while ((ret =
1069             __dbc_get(dbcp, &key, &data, DB_NEXT | DB_MULTIPLE_KEY)) == 0) {
1070                 DB_MULTIPLE_INIT(pointer, &data);
1071                 for (;;) {
1072                         if (is_recno)
1073                                 DB_MULTIPLE_RECNO_NEXT(pointer, &data,
1074                                     recno, dataret.data, dataret.size);
1075                         else
1076                                 DB_MULTIPLE_KEY_NEXT(pointer,
1077                                     &data, keyret.data,
1078                                     keyret.size, dataret.data, dataret.size);As you can see, it's reading through the database and using the DB_MULTIPLE interface to retrieve multiple keys at once. Now this in itself, while performance oriented, may not explain a magnitude 10 difference in the particular pattern you're seeing. What may explain such an immediate difference is the simple fact that after having scanned through the database within your own application, you've already primed both the database cache and the OS' own filesystem cache.
What you're experiencing is not pointing towards any kind of fragmentation but rather showing the positive benefits of caching.
Since it would be atypical (making an educated assumption) for your application to do a full database traversal in standard use - what you're seeing with a 20 minute full traverse on cold cache is not really accurate in what your application will see anyways.
What you should do to maintain the highest level of read performance is quite simple:
1. Locality of reference (http://en.wikipedia.org/wiki/Locality_of_reference).
In the context of BDB this simply means choosing the appropriate key and (if necessary) btree comparison function that will place related data near or on the same page as other related data. Almost every application has locality of reference. Even if there is an initial page location and load hit in efficiency there will most likely follow an opportunity to take advantage of LoR from that point on. Ask yourself: is your application making the most of LoR based on the data and usage patterns it has available to it?
2. I/O costs. It may be quite possible to simply store less actual data by reducing any redundancy of data within the data itself. Is there a common trend for repetition of information between row to row? Here's a simplistic example:
key:data (year|make|occupation|location)
Sam:1982 Ford|Office Manager|Los Angeles
John:1992 Honda|Mechanic|San Francisco
Billie:1993 Honda|Pilot|Los Angeles
Susie:1989 Dodge|Office Manager|Detroit
Jimbo:2000 Ford|Policeman|San Francisco
Dale:1962 Dodge|Mechanic|New YorkAlmost every "column" there can be collapsed into an atomic integer (uint8_t, uint16_t, uint32_t, etc.) type which will significantly cut down on overall costs of storing the row data itself. For instance, applied to the above data, even if we were using uint32_t for each specific column (when some could utilize uint16_t or even uint8_t), you're looking at 12-16 bytes and the cost savings increase as more repetition is present among rows beforehand - as after they will share integer ids for duplicate "columns" between rows. It's not free per se, as it will transfer costs to CPU when one has to expand the integer id back to byte string (when actually making use of the data) and it will shift I/O to another "resolution" database - albeit the majority of this I/O you would be doing anyway. However, since one is storing less data overall we will be able to take more advantage of the cache that is available to us and definitely amortize the costs of decomposition/composition of integer keys to byte strings. The actual CPU cost of calling additional functions is ridiculously small compared to waiting on I/O. If you're comfortable with the additional complexity and your data has high levels of redundancy, then you should explore the possibilities.
In addition, another thing to explore is the possibility of database normalization (http://en.wikipedia.org/wiki/Database_normalization) for your data.
The common theme here, along with both normalization and the previous method, is to eliminate any significant sources of redundancy. If you don't have any source of redundancy and actually have 12GB of unrelated, unique (from row to row) data, then the only thing you can do is appropriately plan the database such that you'll be exploiting LoR - and if that offers no advantage - last resort: throw more cache and disk heads at it.
All that aside, there is a stronger question evident in your first statement:
<i>I frequently iterate through an 8gb table with a cursor</i>
and that question is <b>why</b>? This should <b>not</b> be necessary for a typical application to do.