Database open (recovery) taking too long

Question

Hi,
Ive been using your awesome BerkeleyDB Java Edition for a couple of years, and have been very happy with it.
I am currently facing an issue with trying to open the database after a disk-full issue (which resulted in the database being unable to write, and hence not closed properly).
While recovery seems to be operating, it has been taking an inordinate amount of time - 16 hours so far. My database has data of around 200GB, which inflated to over 450GB during deletion of entries, hence gobbling up all free space on disk.
My questions are:
* Should i continue to wait for recovery?
* Is there any chance that recovery is looping?
* Is there an easier way (DBDump?) to extract data from the database without having to perform recovery?
Some other information that may help:
* The recovery has decreased the size of the last significant file, and created 3 new files since it started running.
* I have been monitoring the open files (using lsof), and they change every now and then to other files, though a good amount of its time is spent near the end of the database.
Thus, i feel like recovery is running normally, just taking too long. Please let me know your opinion.
A few other things i should mention regarding my issue:
* The database was, till yesterday, running on bdb java 3.3.75. After running several hours of recovery, i upgraded to 4.1.10 (since i read about a possible recovery looping bug in one of the versions)
* Once 4.1.10 started recovery, it spat out errors regarding the last 2 files. Only on deleting those 2 files (the last being 0 bytes, the 2nd-last being about 5k) did the recovery start. Note that the older 3.3.75's recovery never complained about those files. I can post the errors here if relevant.
* Some of the jdb files (about 500 files out of the 47,000 files that make up the database) are 100 MB files, since i had experimented with larger sized files for a few days, then reverted the setting.
Would any of these above affect a successful recovery?
My setup is:
OS:Linux CentOS 5.2, 64-bit, kernel 2.6.18-92.el5
JVM: Sun Java 1.6.0_20, 64-bit
Memory: 16 GB RAM, of which 8 GB is allocated to the java process (-Xmx8000M -Xms8000M)
BDB cache set to use 6GB RAM (envconfig.setCacheSize(6000000000))
Only the BDB basic API is being used (Environment, database, cursors). We do not use DPL, or HA features.
Awaiting your kind response,
Sushant A

Answer 1

Hi Sushant,
* Should i continue to wait for recovery?* Is there any chance that recovery is looping?>
I'm not aware of a bug that would cause recovery to loop, however, you may want to take thread dumps to see if it is progressing. It isn't easy to tell, however, since each phase of recovery is in fact a loop. What you can tell easily from the thread dumps is whether recovery is blocked (completely stopped) for some reason. I don't know of a bug that would cause this, but it's something I would check for.
Assuming it is not blocked, I suggest that you leave recovery running, and additionally (in parallel) try to obtain some information about your log. While recovery is running you can run the DbPrintLog utility, which does not itself run recovery. I suggest running the following command, which will tell us in general what your log looks like and in particular how far apart the checkpoints are:
java -jar je-x.y.z.jar DbPrintLog -h <envHome> -S > <output>Please post the output.
If checkpoints are not running in your application for some reason, or they are running very infrequently, this can cause VERY long recoveries. Unfortunately, you may have such a problem in your app and not be aware of it, until you crash and have to recover. To guard against this sort of thing in the future, you should keep an eye on the checkpoint frequency. EnvironmentStats.getNCheckpoints and getEndOfLog can together be used to tell how much log is written between checkpoints. We will also be able to see this from the DbPrintLog -S output.
* Is there an easier way (DBDump?) to extract data from the database without having to perform recovery?DbDump normally runs recovery. DbDump with the -r or -R option does not run recovery, but has other drawbacks. With -r, a large amount of memory may be necessary to dump an accurate representation of your data set. If this fails because you run out of memory, -R can be used, but this will dump multiple versions of each record and it will be up to you to interpret the output.
If regular recovery does not succeed, then DbDump -r is the next thing to try.
Would any of these above affect a successful recovery?No, I don't believe so.
--mark