Bdb deadlocks when opening database

Question

When I call _dbp->open(), the library hangs.  I am attempting to test how BDB works during system crashes, in order to code my application appropriately, but these hangs are difficult to work with.
I am running multiple processes all reading and writing to a single database. I then kill off the processes one at a time, until one of the terminations hangs all the other processes reading or writing to that database. I understand the database may at that time be in a bad state and may need to be re-opened with the DB_RECOVER flag, but I don't get any kind of an error message on the other processes reading or writing to the database. They just hang. (it looks like they are waiting for the lock of the terminated process to be closed).
I have code to check for a DB_RUNRECOVERY error number returned as part of the exception thrown during a db->open() operation, so I can then can db->open() again with the DB_RECOVER option, but db->open() never returns.
I have tried setting the transaction timeout values and lock timeout values, but nothing seems to be working.
I have tried working with the DB_REGISTER option, but would prefer not to limit each process to a single environment. I'd like to hide the BDB environment entirely from the caller and simply require the database name.
Any assistance would be appreciated,
Todd

Answer 1

I do not have the option of launching a separate process to monitor the database. Unfortunately, I have resigned to the fact I must provide a separate thread to test for failures and deadlocks.
As a result, I have created a static process-wide "EnvironmentManager" class to maintain a collection of Environments and Databases opened by the current process. This ensures the process will only have a single instance of any Environment open. As a result, I should be able to open my environment(s) with DB_REGISTER and DB_RECOVER.
I also created a monitor thread. This thread does the following:
foreach open environment
envp->lock_detect(0,DB_LOCK_DEFAULT,NULL);
envp->failchk(0);
If a DbException with errno set to DB_RUNRECORY occurs, I do the following:
     Close each database opened within the faulted environment
     Close the environment
     Re-Open the environment
     Re-Open each database
I would like to cycle through any open transactions, locks and cursors to close them, but it appears I do not have access to them.
My test engine runs in an infinite loop doing the following:
- Open Environment and Database
- Write 200 records
- Read 200 records
- Get a count of records
- Close the database and environment
To test the failures, I launch multiple instances of the test engine then kill the processes one at a time. Obviously my production application with do a normal shut-down under normal conditions, but I need to handle time when an application crashes unexpectedly. I understand problems will occur, my goal is simply to get the system to throw exceptions that I can recover from. I need to resolve all dead-lock issues that can hang my application and all unexpected application-aborts that occur as a result of DB_ASSERT failures.
The problems I am now encountering after implementing all of this:
1) After a process has crashed and I attempt to ‘recover’ from within another running process, I am getting an exception regarding an open file handle to __db.register.
PANIC: fatal region error detected; run recovery
File handles still open at environment close
Open file handle: C:\__db.register
assert failure: ..\mutex\mut_win32.c/174: "__ip != NULL && __ip->dbth_state != THREAD_OUT"
2) If I try to run the same application multiple times quickly by double clicking on the same executable, I get the following exception:
DB_REGISTER limits processes to one open DB_ENV handle per environment
DbEnv::open: Invalid argument
Both of these issues appear to be with the DB_REGISTER option being specified when I open the environment, but things get much murkier if I remove that option.
If I do not specify DB_REGISTER and DB_RECOVER on the command line, it is possible for my call to DbEnv->Open() to hang indefinitely. Otherwise, I would call Open() once and if it failed, call it again with DB_RECOVER.
Please let me know how to proceed. How have other people handled running Berkeley DB across multiple processes?
Regards,
Todd