Opening a db environment and preventing corruption?

Question

Greetings,
Sorry for the long post, but I’m having some problems and I’m not sure where to even begin. I also have a deadline looming and many people looking to me for fixes.
I’m working on a project and we’re trying to use the BDB for looking up data (read only tables). Things run fine when testing one process, but in production there will need to be 20 or more processes, and we cannot keep any of them running very long due to environment file corruption. Also, the processes do have some known memory leaks and crash occasionally, this problem cannot be corrected at this time and must be worked around (currently the processes are all terminated and restarted nightly).
1. How should the environment be set up / opened per process? Currently each process has code that calculates the environment cache size (based on calculations from the docs) and opens the databases (currently 15 .db files). Is this the correct method? Should every process perform all the environment settings, or should an existing environment be checked for first somehow? Every process has code that calls:
db_env_create()
set_cachesize(<size calculated to be large enough for all .db files>)
open(DB_CREATE | DB_THREAD | DB_INIT_MPOOL)
Each database is opened with DB_RDONLY | DB_THREAD options.
2. The environment cache size is calculated to be large enough to hold all the .db files in memory (currently about 1.08G), will that RAM be shared between processes, or does each process need the physical RAM available?
3. I have noticed that processes compiled with debugging symbols modify the environment such that when you try to run a program compiled with optimizations (and no debugging symbols), the program will hang trying to open the environment. Is this to be expected?
4. Once the environment files are created (currently any process that starts will create the __db.* files if they do not exist), it is possible to mark them read-only in an attempt to prevent corruption? We are only using the database files for read-only access.
Once the environment files are corrupt, there are generally two types of problems we encounter. One is when a process is trying to start and fails because the environment or a database cannot be opened:
In bdbdb_open(), DB->open(bdb_ru_rule.db) failed: Not enough space
unable to allocate memory for mutex; resize mutex region
The other problem is that a currently running process will just hang. This is a stack trace:
(dbx) where
current thread: t@1
[1] ___lwp_mutex_timedlock(0xf5a36028, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfe241314
[2] __db_pthread_mutex_lock(0x10bd08, 0x2d798, 0x2118000, 0x1c8, 0x10c190, 0x0), at 0xfdd11fac
[3] __db_tas_mutex_lock(0xf6de0000, 0x2d798, 0xf5a36050, 0x4000, 0xafc00000, 0xb02806c0), at 0xfdd11d14
[4] __memp_fget(0x0, 0xffbfd4d0, 0x0, 0x34036000, 0xced56bc0, 0x0), at 0xfde3f7ec
[5] __bam_get_root(0x10ccb8, 0x0, 0x1, 0x0, 0xffbfd5b8, 0x0), at 0xfdd2c318
[6] __bam_search(0x10ccb8, 0x0, 0x10ce24, 0x10ce24, 0x1, 0x0), at 0xfdd2c8c4
[7] __bamc_search(0x10ccb8, 0x0, 0xffbfd8e8, 0x1a, 0xffbfd6a4, 0x13), at 0xfdd1a420
[8] __bamc_get(0x10ccb8, 0xffbfd8e8, 0xffbfd8cc, 0x1a, 0xffbfd758, 0xfdd15b08), at 0xfdd16104
[9] __dbc_get(0x10ccb8, 0xffbfd8e8, 0xffbfd8cc, 0x1a, 0x14, 0x0), at 0xfdde145c
[10] __db_get(0x10c900, 0x50000000, 0xffbfd8e8, 0xffbfd8cc, 0x1a, 0x10c938), at 0xfddee648
[11] __db_get_pp(0x10bd08, 0x0, 0xffbfd8e8, 0xffbfd8cc, 0x8000000, 0x10c880), at 0xfddee38c
=>[12] bdb_lookup(i_bdb = 1, vp_key = 0xffbfd9a0, vp_data = 0xffbfd98c), line 334 in "bdb_base.c"
Any insights into these problems would be greatly appreciated!
Thanks,
Matthew

Answer 1

Hi Matthew,
I will try to answer your questions.
1. How should the environment be set up / opened
per process?
As documented in the flags section here:
http://www.oracle.com/technology/documentation/berkele
y-db/db/api_c/env_open.html
You should open the process you want to create the
environment with all of the configuration settings.
Then each subsequent environment open should be done
with no flags set.
Is there a way to detect if the environment is already opened so the first process that starts can open the environment with flags?
What about the settings done with function calls between env->create() and env->open(), like setting the cache size? Should those be skipped as well if another process already has the environment open?
What if the __db* files already exist, then does the first process still need to open the environment with flags set, or are the environment's settings stored in the __db* files directly?
3. I have noticed that processes compiled with
debugging symbols modify the environment such that
when you try to run a program compiled with
optimizations (and no debugging symbols), the program
will hang trying to open the environment. Is this to
be expected?
There is a slight distinction here. There are two
different flags (called slightly different things
depending on your platform).
Environments are not compatible if they have
different diagnostic settings. Diagnostic mode
(--enable-diagnostic using configure, or DIAGNOSTIC
preprocessor define in a Visual Studio project).
You can do a debug build without diagnostics enabled.
On systems using configure, just build without
--enable-diagnostic. On Windows, alter the Debug
project build files to remove the DIAGNOSTIC
pre-processor define.
Sorry for not being more clear, I was referring to the application I'm writing, not compiling the BDB. For example, if I compile with -g (Solaris CC) and run the process, then recompile with -xO4 (and remove -g), then the process will not be able to open the environment and will either terminate with an error or hang in the mutex lock.
>
5. In bdbdb_open(), DB->open(bdb_ru_rule.db)
failed: Not enough space
unable to allocate memory for mutex; resize mutex
region
If you are encountering this problem, you should read
the reference guide section on configuring Locking.
http://www.oracle.com/technology/documentation/berkele
y-db/db/ref/lock/max.html
6. [2] __db_pthread_mutex_lock(0x10bd08, 0x2d798,
0x2118000, 0x1c8, 0x10c190, 0x0), at 0xfdd11fac
Are you running deadlock detection? Is it possible
that a process has died while holding locks open on
the database?
I currently do not have deadlock detection enabled for two reasons:
1. All the databases are opened as read-only, so why is any locking happeneing? In the C-API docs for env->open(), the DB_INIT_LOCK description specifically says: "If all threads are accessing the database(s) read-only, locking is unnecessary."
2. When I find processes that have failed waiting on some mutex, running db_stat does not indicate that any processes are deadlocked and running db_deadlock hangs.
Thanks,
Matthew