DB btree/hash performance

Question

I tested performance of some in-memory BDB containers with DB_BTREE/DB_HASH access. Performance of C++ std::map-based containers (with the simplest synchronization using a critical section) are ~ 30-40% higher as compared with these in-memory databases. The tests are done on a 1-CPU Windows XP computer running 1 writer and 4 readers. I used CDB Berkeley subsystem.
Is there any way to make BDB faster? Even if I run 4 readers only (and do not run a writer), the performance will not be much higher (~15%-20).
Thank you

Answer 1

Hi Michael,
Are you comparing the performance of Berkeley DB with some STL implementation?Surely, comparing the performance of BDB with standard C++ containers might be useless, if someone tried to answer the questions: “what is faster?“, or “is it possible to make BDB as fast as STL containers?“. Obviously, the answer would be: BDB and STL are for different purposes; so there is no sense to compare them. But behavior of the standard containers (along with simplest synchronization routines) is very well known; and comparison of BDB with a standard container on a particular PC could give not a bad relative estimation of the BDB performance. Such a relative estimation may be better than something like “performance is 10000 requests per second”, because the test results will be different on different PCs, with different CPUs, with different … many things. Maybe I am wrong. Please, correct me if so.
If so, is there some reason that you can't just use STL?I told about in-memory BDB just in order to show that performance of the tested BDBs did not relate to disk IO. But in fact, I need non-in-memory BDB in my application. The data must be on a disk. That is why I cannot just use STL. In fact, data will come to some STL containers first, and then data will be queued in order to update a BDB storage (non in-memory). When the application restarts, the data will be re-loaded from the persistent BDB storage.
What operations does your workload perform? What are your concurrency
requirements and performance goals? The application has a BDB storage (DB_BTREE/DB_HASH) with some 5000 keys. After the application has loaded this “map” at startup, only one “writer” thread may very rarely add/delete entries from the “map”. The impact of this process on the performance should be close to 0.
The same “writer” thread intensively updates values of the map (for given keys). The writer thread may issue ~ 1-10 updates per ms. The biggest problem is that the number of updates may be much higher in case of data spikes (up to 30 updates per ms and higher).
Also, the application has 4-5 “reader” threads, which relatively rarely (once per 100-500 ms) access the “map” reading the values “by key“.
The BDB write/read access should have “write favor” - readers may perform “uncommitted” reading, preferably not blocking the “writer“ thread.
Can you give some absolute numbers for the results you are seeing rather than >>percentages? Here is a typical output of the test application running on a 1-CPU Windows XP PC:
STL: std::map-based container (critical section synchronization):
== consumer statistics ==
consumer (0), duration sec.(0.454418), iterations done(1000000)
consumer (1), duration sec.(0.494315), iterations done(1000000)
consumer (2), duration sec.(0.343084), iterations done(1000000)
consumer (3), duration sec.(0.109292), iterations done(1000000)
== producer statistics ==
producer (0), duration sec.(0.297399), iterations done(1000000)
BDB:
== consumer statistics ==
consumer (0), duration sec.(24.055119), iterations done(1000000)
consumer (1), duration sec.(27.551431), iterations done(1000000)
consumer (2), duration sec.(28.282188), iterations done(1000000)
consumer (3), duration sec.(27.537262), iterations done(1000000)
== producer statistics ==
producer (0), duration sec.(30.986818), iterations done(1000000)
Thank you