BerkeleyDB cache size and Solaris

Question

I am having problems trying to scale up an application that uses BerkelyDB-4.4.20 on Sun Sparc servers running Solaris 8 and 9.
The application has 11 primary databases and 7 secondary databases.
In different instances of the application, the size of the largest pimary database
ranges only from 2MB to 10MB, but those will grow rapidly over the
course of the semester.
The servers have 4-8 GB of RAM and 12-20 GBytes of swap.
Succinctly, when the primary databases are small, the application runs as expected.
But as the primary databases grow, the following, counterintuitive phenomenon
occurs. With modest cache sizes, the application starts up, but throws
std::exceptions of "not enough space" when it attempts to delete records
via a cursor. The application also crashes randomly returning
RUN_RECOVERY. But when the cache size is increased, the application
will not even start up; instead, it fails and throws std::exceptions which say there
is insufficient space to open the primary databases.
Here is some data from a server that has 4GB RAM with 2.8 GBytes free
(according to "top") when the data was collected:
DB_CONFIG............db_stat -m.................................Result
set_cachesize........Pool......Ind. Cache
0 67108864 1.........80 MB.......8 KB................Starts but crashes and can't delete by
.....................................................................cursor because of insufficient space
0 134217728 1.......160 MB......8 KB.................Same as case above
0 268435456 1........320 MB.....8 KB.................Doesn't start and says there is
......................................................................not enough space to open a primary
......................................................................database.
0 536870912 1.........512 MB...16 KB.................Doesn't start and says there is
......................................................................not enough space to open a primary
......................................................................database (although it mentions a
......................................................................different primary database than before.
1 073741884 1........1GB 70MB....36 KB............Doesn't start and says there is
......................................................................not enough space to open a primary
......................................................................database (although it mentions a
......................................................................different pimary database than
......................................................................previously).
2 147483648 1.........2GB 140MB...672 KB........Doesn't start and says there is
......................................................................not enough space to open a primary
......................................................................database (although it mentions a
......................................................................different pimary database than
......................................................................previously).
I should also mention that the application is written in Perl and uses
the Sleepycat::Db Perl module to interface with the BerkeleyDB C++ API.
Any help on how to interpret this data and, if the problem is the
interface with Solaris, how to tweak that, will be greatly appreciated.
Sincerely,
Bill Wheeler, Department of Mathematics, Indiana University, Bloomington.

Answer 1

Having found answers to my questions, I think I should document them here.
1. On the matter of the error message "not enough space", this message
apparently orginates from Solaris. When a process (e.g., an Apache child)
requests additional (virtual) memory (via either brk or mmap) such that the
total (virtual) memory allocated to the process would exceed the system limit
(set by the setrlimit command), then the Solaris kernel rejects the request
and returns the error ENOMEM . Somewhat cryptically, the text for this error
is "not enough space" (in contrast, for instance, to "not enough virtual
memory").
Apparently, when the BerkeleyDB cache size is set too large, a process
(e.g., an Apache child) that attempts to open the environment and databases
may request a total memory allocation that exceeds the system limit.
Then Solaris will reject the request and return the ENOMEM error.
Within Solaris, the only solutions are apparently
(i) to decrease the cache size or
(ii) to increase the system limit via the setrlimit command.
2. On the matter of the DB_RUNRECOVERY errors, the cause appears
to have been the use of the DB_TXN_NOWAIT flag in combination with
code that was mishandling some of the resulting, complex situations.
Sincerely,
Bill Wheeler