Db recovery in a multiprocess environment

Question

Hi,
We have a application consisting of two tomcat servers using Berkeley db and hence the application is simultaneously accessed by two or more(I have a few utilities too) processes. I run one of these instances with primary flag and othe instance and all utilities are run as secondary instances. Only primary instance opens the environment with run_recover 'true', others dont specify this flag. While shutting down the instances, I try to close all open db handles along with environment handle for that instance.
There are following scenarios:
A) When I am shutting down the any of the tomcats, sometimes there are few open transactions and cursors. These cause some database handle close to fail and as a result, db recovery is initiated for almost every start.
My questions for scenario A) are:
1. There should be some way in which I can bring down the environment and still don't need to recover. In other words, assuming my environment has open cursors and transaction(over which I don't have any control as these are user-initiated). How do I properly close it. This question can be broken down in two parts:
a) how do i figure out or get to the open transactions and cursors.
b) then I can either wait for some time for them to complete or I can close cursors and abort/commit the transcations.
B) Sometimes one of these processes may crash, for ex: secondary tomcat going out of memory while some of the transactions are open. Can I somehow prevent the primary instance from being corrupted in the sense that it can still use the db?
Thanks and Regards,
Shishir

Answer 1

Hello,
From your description I believe that the "Architecting
Transactional Data Store applications" section of the documentation at:
http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/app.html
will provide the information you need.
If the application is organized as a group of related processes,
the order in which threads of control are created must be
controlled to serialize database environment recovery. Directly
from the documentation:
This architecture requires that threads of control be monitored.
If any thread of control exits with open Berkeley DB handles,
the application may call the DB_ENV->failchk method to detect lost
mutexes and locks and determine if the application can continue.
If the application does not call DB_ENV->failchk, or DB_ENV->failchk returns that the database environment can no longer be used,
the application must behave as if there has been a system
failure, performing recovery and creating a new database environment.
Once these actions have been taken, other threads of control can
be continued (as long as all existing Berkeley DB handles are
first discarded), or restarted.
If the application is organized as a group of unrelated processes,
some choices for monitoring them are detailed. One solution
(directly from the documentation) is to log a thread of
control ID when a new Berkeley DB handle is opened. For example,
an initial "watcher" process could run recovery on the database environment and then create a sentinel file. Any "worker" process
wanting to use the environment would check for the sentinel file.
If the sentinel file does not exist, the worker would fail or
wait for the sentinel file to be created. Once the sentinel file
exists, the worker would register its process ID with the watcher
(via shared memory, IPC or some other registry mechanism), and
then the worker would open its DB_ENV handles and proceed. When
the worker finishes using the environment, it would unregister
its process ID with the watcher. The watcher periodically checks
to ensure that no worker has failed while using the environment.
If a worker fails while using the environment, the watcher
removes the sentinel file, kills all of the workers currently
using the environment, runs recovery on the environment, and
finally creates a new sentinel file.
The documentation also describes the use of DB_REGISTER in this
case and the use of a "watcher" process.
Thanks,
Sandra