Latency increase with large number of databases

Question

Hello,
I'm testing how BDB performs on random updates as the number of databases increases. I'm seeing increase in latency as the number of databases goes up. and strace shows that more time is spent on futex() when there are more databases. Is this an expected behavior?
Thanks!
--Michi
Set Up:
<li> BDB version: db-5.1.19
<li> Operating system: RHEL4
<li> Number of records: 2.5M
<li> Record key size: 32B
<li> Record value size: 4KB
<li> Page size: 32KB
<li> Access method: BTREE
<li> Records are inserted into a database based on the hash of the key.
<li> BDB is accessed via RPC server
<li> RPC server thread pool size: 64. All the threads share the same env and db handles.
<li> Number of RPC client process: 64
<li> Each client does random updates. Throughput is throttled to be about 250 requests/sec, which means each process does about 4 requests/sec.
Flags used for env and db:
ENV flag DB_THREAD | DB_RECOVER | DB_CREATE | DB_READ_COMMITTED | DB_INIT_TXN |
         DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL
DB flag DB_AUTO_COMMIT | DB_CREATE | DB_THREADDB_CONFIG
mutex_set_max 2000000
set_cachesize 1 0 1
set_lk_max_objects 2000
set_lk_max_locks 10000
set_lk_max_lockers 10000
set_lg_regionmax 8388608
set_lk_detect DB_LOCK_DEFAULT
set_thread_count 20000
set_lg_max 1073741824 Result
Number of databases: 256
Average latency: 52 milliseconds
strace output
% time     seconds  usecs/call     calls    errors syscall
64.74  121.199645        1032    117398           recvfrom
17.21   32.217921         487     66104      9335 futex
  8.76   16.409370        2724      6024           pread
  2.47    4.630256         813      5694           fdatasync
  2.19    4.102371         105     39162           sendto
Number of databases: 512
Average latency: 123 milliseconds
strace output
% time     seconds  usecs/call     calls    errors syscall
58.33  121.384230         933    130158     20045 futex
23.06   47.977408         383    125146           recvfrom
  4.41    9.167276        1425      6431           pread
  3.11    6.472676         155     41716           sendto
  3.08    6.414824         366     17506           sched_yield
Number of databases: 1024
Average latency: 133 milliseconds
strace output
% time     seconds  usecs/call     calls    errors syscall
70.46  102.981390        1077     95656     14884 futex
  9.88   14.435594         169     85256           recvfrom
  5.48    8.008787        1825      4389           pread
  3.13    4.576880         368     12436           sched_yield
  2.98    4.351077         153     28415           sendto
...

Answer 1

Hi James,
Presently, we never evict a DB after it's opened or encountered during recovery. Each DB takes about 2,000 bytes. So if you have 16K DBs you need approximately 32MB of memory, assuming all of them could be opened or recovered during a process lifetime. Unfortunately, even closing them does not cause them to be evicted. If you encounter them during recovery, this will also pull them into memory.
We have an FAQ entry on this:
http://www.oracle.com/technology/products/berkeley-db/faq/je_faq.html#37
So you will need a bigger cache size. If you are encountering this during recovery, then you could try a more frequent checkpoint interval.
I hope this is useful.
Regards,
Charles Lamb