Cache / performance question...

Question

Hello,
I've been running tests against an all-write load to try and understand limitations and BDB internals, and am wondering what causes BDB to begin 'forcing' pages from cache even though clean pages still exist, checkpoints are being regularly run, and trickling is being used? As an all-write load (in a testing environment) continues to operate, I notice performance begins to suffer and cannot recover at roughly the time db_stat begins reporting forced writes of both clean and dirty pages. For example, a write load of 5000 txn/sec degrades to 1000/sec and below after ~500K transactions, and I start to see things like the below in db_stat:
3219 Clean pages forced from the cache
7271 Dirty pages forced from the cache
183004 Dirty pages written by trickle-sync thread
Even when the cache has clean pages, as the database grows over time, I imagine that new keys may be written to "older" pages. This particular example uses DB_HASH. In this case, although the cache has a clean page, a given key "475000" may be written to an old page already on disk. This would, as far as I understand, require that page to be re-read into the cache to complete the write. However, monitoring vmstat does not show me any inbound IO activity whatsoever during the time that the performance begins to drop. I am therefore not certain that old pages are being read from disk, and confused as to why write performance begins to suffer so greatly when the cache fills up.
I'm trying this without DB_TXN_NOSYNC/etc - my environment favors steady performance over bursty speed, so I don't want to forward-load anything that might trigger a large number of writes later whenever possible.
I have sample source and testing results that I can attach, but I'm first trying to understand if I'm not understanding something basic before I post too much information. Thanks! :)
Edited by: user10542315 on Jan 1, 2009 7:22 PM

Answer 1

Hello,
Although I'm not an engineer on the product, and I'm sure one will chime in to help you tune your tests, I thought I'd ask you if you've seen the following white paper http://www.oracle.com/technology/products/berkeley-db/pdf/berkeley-db-perf.pdf ? It's a bit dated, but it is trying to do some of the same things you are doing. We're focusing a lot of resources on performance given the radical shifts in the ratio of cores, hyper-threads, memory, disk-speed, and flash-storage devices. All of these have changed the underlying assumptions one makes when building a database engine ("Will we be memory, or CPU bound? How big a bottle-neck will I/O be? How can we avoid that? Can we max out all the cores/threads of all resources when running purely in memory?"). There are a lot of interesting areas to research and improve. We're doing some of that work in 4.8, more later. I can't be specific, but we're trying our best to keep DB at the bleeding edge of hardware capabilities.
-greg
Gregory Burd - Product Manager - Oracle Berkeley DB - [email protected]