Bug report & possible patch: Wrong memory allocation when using BerkeleyDB in concurrent processes

Question

When using the BerkeleyDB shared environment in parallel processes, the processes get "out of memory" error, even when there is plenty of free memory available. This results in possible database corruption.
Typical use case when this bug manifests is when BerkeleyDB is used by rpm, which is installing an rpm package into custom location, or calls another rpm instance during the installation process.
The bug seems to originate in the env/env_region.c file: (version of the file from BDB 4.7.25, although the culprit code is the same in newer versions too):
330     /*
331      * Allocate room for REGION structures plus overhead.
332      *
333      * XXX
334      * Overhead is so high because encryption passwds, replication vote
335      * arrays and the thread control block table are all stored in the
336      * base environment region.  This is a bug, at the least replication
337      * should have its own region.
338      *
339      * Allocate space for thread info blocks.  Max is only advisory,
340      * so we allocate 25% more.
341      */
342     memset(&tregion, 0, sizeof(tregion));
343     nregions = __memp_max_regions(env) + 10;
344     size = nregions * sizeof(REGION);
345     size += dbenv->passwd_len;
346     size += (dbenv->thr_max + dbenv->thr_max / 4) *
347         __env_alloc_size(sizeof(DB_THREAD_INFO));
348     size += env->thr_nbucket * __env_alloc_size(sizeof(DB_HASHTAB));
349     size += 16 * 1024;
350     tregion.size = size;
Usage from the rpm's perspective:
The line 346 calculates how much memory we need for structures DB_THREAD_INFO. We allocate structure DB_THREAD_INFO for every process calling db4 library. We don't deallocate these structures but when number of processes is greater than dbenv->thr_max then we try to reuse some structure for process that is already dead (or doesn't use db4 no longer). But we have DB_THREAD_INFOs in hash buckets and we can reuse DB_THREAD_INFO only if it is in the same hash bucket as new DB_TREAD_INFO. So line 346 should contain:
346     size += env->thr_nbucket * (dbenv->thr_max + dbenv->thr_max / 4) *
347         __env_alloc_size(sizeof(DB_THREAD_INFO));
Why we don't encounter this problem earlier? There are some magic reserves as you can see on line 349 and some other additional space is created by alligning to blocks. But if we have two processes running at the same time and these processes end up in the same hash bucket and we repeat this proces many times to fill all hash buckets with two DB_THREAD_INFOs then we have 2 * env->thr_nbucket(37) = 74 DB_THREAD_INFOs, which is much more than dbenv->thr_max(8) + dbenv->thr_max(8) / 4 = 10 and plus allocation from dbc_put, we are out of memory.
And how we will create two processes that end up in the same hash bucket. We can start one process (rpm -i) and then in scriptlet we start many processes (rpm -q ...) in loop and one of them will be in the same hash bucket as the first process (rpm -i).
I would like to know your opinion on this issue, and if the proposed fix would be acceptable.
Thanks in advance for answers.

Answer 1

The attached patch for db-4.7 makes two changes:
  it allows enough for each bucket to have the configured number of threads, and
  it initializes env->thr_nbuckets, which previously had not been initialized.
Please let us know how it works for you.
Regards,
Charles