Help! exception in Evictor.java, db becomes unsable after that

Question

Hi,
I am using BDBJE inside a project. The application runs normally at the beginning (BDB being used at all time). It seems when the heap max is hit (1G in my case) and garbage collection starts to run more often, then the following db error seems to surface , after which any db operation gives the same error.
I am running Sun jdk 1.5 and uses deferred write on the db. The application creates hundreds of dbs , then insert/delete entries from each db, and also delete a db entirely when that db becomes empty.
I can re-produce this fairly easily using the application.
Thanks for any help!
java.io.IOException: (JE 3.0.12) Database blogads.com id=404 IN type=BIN/2 id=3335404 not expected on INList
     at com.sleepycat.je.evictor.Evictor.selectIN(Evictor.java:469)
     at com.sleepycat.je.evictor.Evictor.evictBatch(Evictor.java:330)
     at com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:242)
     at com.sleepycat.je.evictor.Evictor.doCriticalEviction(Evictor.java:266)
     at com.sleepycat.je.dbi.CursorImpl.close(CursorImpl.java:666)
     at com.sleepycat.je.Cursor.close(Cursor.java:243)
     at com.sleepycat.je.Database.putInternal(Database.java:571)
     at com.sleepycat.je.Database.putNoOverwrite(Database.java:530)

Answer 1

Hello all,
I wanted to let everyone know about the resolution to this problem, in case you encounter it.
I'd like to thank the poster, user535685, for reporting this problem and working with us to test the fix.
The problem occurs under the following conditions:
+ A DeferredWrite Database is used (DatabaseConfig.setDeferredWrite(true) is called).
+ Some information is stored in the Database.
+ The Database is closed without ever calling Database.sync method.
After closing the Database, the exception reported in this thread may occur while performing other activities, causing the Environment to become unusable.
If you encounter this problem, or if you use DeferredWrite databases and you would like to avoid this problem, please send me email and I'll send you a fix. My email is mark.hayes at the obvious dot com.
I've also included a diff below that can be used to patch the 3.2 source code to apply the fix.
Index: src/com/sleepycat/je/dbi/DatabaseImpl.java
===================================================================
RCS file: /a/CVSROOT/je/src/com/sleepycat/je/dbi/DatabaseImpl.java,v
retrieving revision 1.157
diff -c -r1.157 DatabaseImpl.java
*** src/com/sleepycat/je/dbi/DatabaseImpl.java     13 Dec 2006 18:55:34 -0000     1.157
--- src/com/sleepycat/je/dbi/DatabaseImpl.java     12 Jan 2007 02:47:05 -0000
*** 543,604 ****
               * out the root.
              long rootLsn = tree.getRootLsn();
-             if (rootLsn == DbLsn.NULL_LSN) {
!                  * There's nothing in this database. (It might be the abort of
!                  * a truncation, where we are trying to clean up the new, blank
!                  * database. Do delete the MapLN.
!                 envImpl.getDbMapTree().deleteMapLN(id);
!             } else {
!                 UtilizationTracker snapshot = new UtilizationTracker(envImpl);
!                  * Start by recording the lsn of the root IN as obsolete.  A
!                  * zero size is passed for the last parameter because it is too
!                  * expensive to fetch the node.
                  snapshot.countObsoleteNodeInexact
                      (rootLsn, LogEntryType.LOG_IN, 0);
-                 /* Fetch LNs to count LN sizes only if so configured. */
-                 boolean fetchLNSize =
-                     envImpl.getCleaner().getFetchObsoleteSize();
-                 /* Use the tree walker to visit every child lsn in the tree. */
-                 ObsoleteProcessor obsoleteProcessor =
-                     new ObsoleteProcessor(snapshot);
-                 SortedLSNTreeWalker walker = new ObsoleteTreeWalker
-                     (this, rootLsn, fetchLNSize, obsoleteProcessor);
-                  * Delete MapLN before the walk. Note that the processing of
-                  * the naming tree means this MapLN is never actually
-                  * accessible from the current tree, but deleting the MapLN
-                  * will do two things:
-                  * (a) mark it properly obsolete
-                  * (b) null out the database tree, leaving the INList the only
-                  * reference to the INs.
-                 envImpl.getDbMapTree().deleteMapLN(id);
-                  * At this point, it's possible for the evictor to find an IN
-                  * for this database on the INList. It should be ignored.
-                 walker.walk();
-                  * Count obsolete nodes for a deleted database at transaction
-                  * end time.  Write out the modified file summaries for
-                  * recovery.
-                 envImpl.getUtilizationProfile().countAndLogSummaries
-                     (snapshot.getTrackedFiles());
          } finally {
              deleteState = DELETED;
--- 543,596 ----
               * out the root.
              long rootLsn = tree.getRootLsn();
!              * Use a snapshot tracker that is accumulated under the log write
!              * latch when we're doing counting.  Start by recording the LSN of
!              * the root IN as obsolete.  A zero size is passed for the last
!              * parameter because it is too expensive to fetch the node.
!             UtilizationTracker snapshot = new UtilizationTracker(envImpl);
!             if (rootLsn != DbLsn.NULL_LSN) {
                  snapshot.countObsoleteNodeInexact
                      (rootLsn, LogEntryType.LOG_IN, 0);
+
+             /* Fetch LNs to count LN sizes only if so configured. */
+             boolean fetchLNSize =
+                 envImpl.getCleaner().getFetchObsoleteSize();
+
+             /* Use the tree walker to visit every child lsn in the tree. */
+             ObsoleteProcessor obsoleteProcessor =
+                 new ObsoleteProcessor(snapshot);
+             SortedLSNTreeWalker walker = new ObsoleteTreeWalker
+                 (this, rootLsn, fetchLNSize, obsoleteProcessor);
+
+             /*
+              * Delete MapLN before the walk. Note that the processing of
+              * the naming tree means this MapLN is never actually
+              * accessible from the current tree, but deleting the MapLN
+              * will do two things:
+              * (a) mark it properly obsolete
+              * (b) null out the database tree, leaving the INList the only
+              * reference to the INs.
+              */
+             envImpl.getDbMapTree().deleteMapLN(id);
+
+             /*
+              * At this point, it's possible for the evictor to find an IN
+              * for this database on the INList. It should be ignored.
+              */
+             walker.walk();
+
+             /*
+              * Count obsolete nodes for a deleted database at transaction
+              * end time.  Write out the modified file summaries for
+              * recovery.
+              */
+             envImpl.getUtilizationProfile().countAndLogSummaries
+                 (snapshot.getTrackedFiles());
          } finally {
              deleteState = DELETED;
          }Mark