Why does performance decrease as cache size increases?

Question

Hi,
We have some very serious performance problems with our
database. I have been trying to help by tuning the cache size,
but the results are the opposite of what I expect.
To create new databases with my data set, it takes about
8200 seconds with a 32 Meg cache. Performance gets worse
as the cache size increases, even though the cache hit rate
improves!
I'd appreciate any insight as to why this is happening.
32 Meg does not seem like such a large cache that it would
strain some system limitation.
Here are some stats from db_stat -m
Specified a 128 Meg cache size - test took 16076 seconds
160MB 1KB 900B Total cache size
1 Number of caches
1 Maximum number of caches
160MB 8KB Pool individual cache size
0 Maximum memory-mapped file size
0 Maximum open file descriptors
0 Maximum sequential buffer writes
0 Sleep after writing maximum sequential buffers
0 Requested pages mapped into the process' address space
34M Requested pages found in the cache (93%)
2405253 Requested pages not found in the cache
36084 Pages created in the cache
2400631 Pages read into the cache
9056561 Pages written from the cache to the backing file
2394135 Clean pages forced from the cache
2461 Dirty pages forced from the cache
0 Dirty pages written by trickle-sync thread
40048 Current total page count
40048 Current clean page count
0 Current dirty page count
16381 Number of hash buckets used for page location
39M Total number of times hash chains searched for a page (39021639)
11 The longest hash chain searched for a page
85M Total number of hash chain entries checked for page (85570570)
Specified a 64 Meg cache size - test took 10694 seconds
80MB 1KB 900B Total cache size
1 Number of caches
1 Maximum number of caches
80MB 8KB Pool individual cache size
0 Maximum memory-mapped file size
0 Maximum open file descriptors
0 Maximum sequential buffer writes
0 Sleep after writing maximum sequential buffers
0 Requested pages mapped into the process' address space
31M Requested pages found in the cache (83%)
6070891 Requested pages not found in the cache
36104 Pages created in the cache
6066249 Pages read into the cache
9063432 Pages written from the cache to the backing file
5963647 Clean pages forced from the cache
118611 Dirty pages forced from the cache
0 Dirty pages written by trickle-sync thread
20024 Current total page count
20024 Current clean page count
0 Current dirty page count
8191 Number of hash buckets used for page location
42M Total number of times hash chains searched for a page (42687277)
12 The longest hash chain searched for a page
98M Total number of hash chain entries checked for page (98696325)
Specified a 32 Meg cache size - test took 8231 seconds
40MB 1KB 900B Total cache size
1 Number of caches
1 Maximum number of caches
40MB 8KB Pool individual cache size
0 Maximum memory-mapped file size
0 Maximum open file descriptors
0 Maximum sequential buffer writes
0 Sleep after writing maximum sequential buffers
0 Requested pages mapped into the process' address space
26M Requested pages found in the cache (70%)
10M Requested pages not found in the cache (10812846)
35981 Pages created in the cache
10M Pages read into the cache (10808327)
9200273 Pages written from the cache to the backing file
9335574 Clean pages forced from the cache
1498651 Dirty pages forced from the cache
0 Dirty pages written by trickle-sync thread
10012 Current total page count
10012 Current clean page count
0 Current dirty page count
4099 Number of hash buckets used for page location
47M Total number of times hash chains searched for a page (47429232)
13 The longest hash chain searched for a page
118M Total number of hash chain entries checked for page (118218066)
vmstat says that a few minutes into the test, the box is
spending 80-90% of its time in iowait. That worsens as
the test continues.
System and test info follows
We have 10 databases (in 10 files) sharing a database
environment. We are using a hash table since we expect
data accesses to be pretty much random.
We are using the default cache type: a memory mapped file.
Using DB_PRIVATE did not improve performance.
The database environment created with these flags:
DB_CREATE | DB_THREAD | DB_INIT_CDB | DB_INIT_MPOOL
The databases are opened with only the DB_CREATE flag.
There is only one process accessing the db. In my tests,
only one thread access the db, doing only writes.
We do not use transactions.
My data set is about 550 Meg of plain ASCII text data.
13 million inserts and 2 million deletes. Key size is
32 bytes, data size is 4 bytes.
BDB 4.6.21 on linux.
1 Gig of RAM
Filesystem = ext3 page size = 4K
The test system is not doing anything else while I am
testing.

Answer 1

Surprising result: I tried closing the db handles with DB_NOSYNC and performance
got worse. Using a 32 Meg cache, it took about twice as long to run my test:
15800 seconds using DB->close(DB_NOSYNC) vs 8200 seconds using DB->close(0).
Here is some data from db_stat -m when using DB_NOSYNC:
40MB 1KB 900B Total cache size
1 Number of caches
1 Maximum number of caches
40MB 8KB Pool individual cache size
0 Maximum memory-mapped file size
0 Maximum open file descriptors
0 Maximum sequential buffer writes
0 Sleep after writing maximum sequential buffers
0 Requested pages mapped into the process' address space
26M Requested pages found in the cache (70%)
10M Requested pages not found in the cache (10811882)
44864 Pages created in the cache
10M Pages read into the cache (10798480)
7380761 Pages written from the cache to the backing file
3452500 Clean pages forced from the cache
7380761 Dirty pages forced from the cache
0 Dirty pages written by trickle-sync thread
10012 Current total page count
5001 Current clean page count
5011 Current dirty page count
4099 Number of hash buckets used for page location
47M Total number of times hash chains searched for a page (47428268)
13 The longest hash chain searched for a page
118M Total number of hash chain entries checked for page (118169805)
It looks like not flushing the cache regularly is forcing a lot more
dirty pages (and fewer clean pages) from the cache. Forcing a
dirty page out is slower than forcing a clean page out, of course.
Is this result reasonable?
I suppose I could try to sync less often than I have been, but more often
than never to see if that makes any difference.
When I close or sync one db handle, I assume it flushes only that portion
of the dbenv's cache, not the entire cache, right? Is there an API I can
call that would sync the entire dbenv cache (besides closing the dbenv)?
Are there any other suggestions?
Thanks,
Eric