MVCC, freezer files and cache overflow

Question

Hi,
I had an idea. The docs say: "If the cache becomes full of page copies before the old copies can be discarded, additional I/O will occur as pages are written to temporary "freezer" files. This can substantially reduce throughput, and should be avoided if possible by configuring a large cache and keeping snapshot isolation transactions short."
How about allocating a new cache file (of the same size as the original) in the env instead of creating hundreds (or more) freezer files when the cache is overflowing ? One can have multiple cache files already by specifying the cache in multiple segments. What's particularly irksome about these freezer files is that they stick around, even past checkpoints and restarts.
Could this work ?
Andi..

Answer 1

Hi Michael,
What you just said makes a lot of sense. DB_READ_UNCOMMITTED and snapshot isolation don't make much sense used together. Using DB_READ_UNCOMMITTED everywhere, I should not be having many deadlocks or blockages. But, until now, I was getting lots of them unless I was using MVCC. The reason is that I found several places where I wasn't using the DB_READ_UNCOMMITTED bit with getting cursors or getting data. As a matter of fact, because of an egregious bug of mine not propagating the DB_READ_UNCOMMITTED bit properly, many data accesses were still wrongly using the default - maximum - isolation level.
To find these places, I set an artificially low txn timeout and traced back to the bogus calls from the DB_LOCK_DEADLOCK exceptions I'd receive. I believe I found them all now. :)
Reading data that is not yet committed is not a problem for us except in one specific case. DB_READ_UNCOMMITTED should be completely safe as the Chandler repository is, in a way, duplicating in a persistent way what MVCC also does, that is, keeping multiple versions of data around and presenting multiple versions of data to different transactions.
The only place where a higher isolation level is needed - DB_READ_COMMITTED - is in the small database containing the latest version number for the Chandler repository. Using DB_READ_UNCOMMITTED everywhere except in this version db and opening this and this database only with DB_MULTIVERSION solves all the problems I was trying to solve with using MVCC everywhere earlier. Because most transactions I create span multiple databases, I am still opening all read-only transactions with DB_TXN_SNAPSHOT but it should only have an effect when accessing this one lone db opened with DB_MULTIVERSION, correct ?
This is a major breakthrough for me. Commit throughput in a large mail import operation (done in chunks) is now three times what it was before.
It is also much harder, but however still possible, to get freezer files, with the usual dramatic worsening of performance. Maybe 64mb cache is too small. I'm going to double that and watch it over time. Even though the 'version' db is tiny, it is a btree with one record per version plus two records that are being updated every time a new version is written, I still managed to get, over time, up to two thousand freezer files.
This has been a productive exercise as I had very wrongly assumed that I was using DB_READ_UNCOMMITTED consistently already.
Regards,
Andi..