Serious lock problem - transaction has active cursors

Question

Hi,
i use BDB C via the Java-API (4.6.21, developing with WindowsXP). Currently i am having serious problems with LOCK_NOT_GRANTED_EXCEPTIONS.
My application is a queue implementation. One push objects into the database, another one reads them and a third one deletes entries with an outdated TTL (time-to-live). All of these processes are done by multiple dedicated threads what leads to highly concurrent situations.
The TTL cleaning uses a tx-cursor which deletes entries that are too old. The inserter simply uses put(). The reader uses a cursor too, which starts with a pointer (via getSearchKey) or at the beginning of the database (if no start could be found, the pointer is then outdated as it was deleted by TTL cleaning run).
The isolation level is at READ_COMMITTED. Sometimes i use RMW locks for reading and updating status information.
So i implemented a lock detection which retries at least the inserts if a lock fails. The reads can fail and report their exceptions to the outside world. This seems to work fine. Sometimes i get some LOCK_NOT_GRANTED_EXCEPTIONS and the recovery should work. But it works only sometimes.
BDB-C complains with "transaction has active cursors" (see txn.c - line 1186). All subsequent calls to rollback or commit or any other method of BDB-C handles fails then with a RunRecoveryException. I am absolutely sure, i have no open cursors, as i close them implicitly before tx.commit() or tx.abort(). I wrote a reference counter to ensure that and it shows always a count of 0 open cursors as expected.
The whole process is of course quite heavy stress for the DB - i am actually read, write, delete and checkpoint the db at the same position at the same time.
I am tending to introduce MVCC to avoid those locking issues. As i understood concurrent updates of the same db page with multiversion transactions may also cause DeadLockExceptions. Can i use RMW locks to avoid concurrent updates on the same pages with MVCC?
Some hints?
TIA
Dirk

Answer 1

Hi,
I have created a patch for Berkeley DB 4.6.21, which outputs information about all cursor open and close operations. This should help you see which cursor is not being closed.
If you apply the patch below to your Berkeley DB tree, and rebuild.
Then add setVerbose(VerboseConfig.CURSOR, true), to your EnvironmentConfig initialization.
It will send log messages to the output stream configured via EnvironmentConfig.setMessageStream(). The default is System.out.
There will likely be a lot of output.
Please let me know if this helps you resolve the problem. Otherwise, ask more questions ;)
Regards,
Alex Gorrod
diff -rc db-4.6.21.orig/build_windows/db.h db-4.6.21/build_windows/db.h
*** db-4.6.21.orig/build_windows/db.h     Fri Sep 28 01:32:08 2007
--- db-4.6.21/build_windows/db.h     Mon May 12 17:29:52 2008
*** 60,66 ****
  #define     DB_VERSION_MAJOR     4
  #define     DB_VERSION_MINOR     6
  #define     DB_VERSION_PATCH     21
! #define     DB_VERSION_STRING     "Berkeley DB 4.6.21: (September 27, 2007)"
--- 60,66 ----
  #define     DB_VERSION_MAJOR     4
  #define     DB_VERSION_MINOR     6
  #define     DB_VERSION_PATCH     21
! #define     DB_VERSION_STRING     "Berkeley DB 4.6.21: (May 12, 2008)"
*** 2133,2138 ****
--- 2133,2139 ----
  #define     DB_VERB_REGISTER     0x0010     /* Dump waits-for table. */
  #define     DB_VERB_REPLICATION     0x0020     /* Replication information. */
  #define     DB_VERB_WAITSFOR     0x0040     /* Dump waits-for table. */
+ #define     DB_VERB_CURSOR          0x0080     /* Cursor open/close information. */
       u_int32_t      verbose;     /* Verbose output. */
       void          app_private;     / Application-private handle. */
diff -rc db-4.6.21.orig/db/db_am.c db-4.6.21/db/db_am.c
*** db-4.6.21.orig/db/db_am.c     Thu Jun 14 04:21:30 2007
--- db-4.6.21/db/db_am.c     Mon May 12 17:32:08 2008
*** 283,288 ****
--- 283,291 ----
       TAILQ_INSERT_TAIL(&dbp->active_queue, dbc, links);
       F_SET(dbc, DBC_ACTIVE);
       MUTEX_UNLOCK(dbenv, dbp->mutex);
+      if (FLD_ISSET(dbenv->verbose, DB_VERB_CURSOR))
+           __db_msg(dbenv,
+               "Cursor (%p) opened, associated txn: %p", dbc, dbc->txn);
       *dbcp = dbc;
       return (0);
diff -rc db-4.6.21.orig/db/db_cam.c db-4.6.21/db/db_cam.c
*** db-4.6.21.orig/db/db_cam.c     Tue Jun  5 21:46:24 2007
--- db-4.6.21/db/db_cam.c     Mon May 12 17:32:25 2008
*** 71,76 ****
--- 71,79 ----
       opd = cp->opd;
       ret = 0;
+      if (FLD_ISSET(dbenv->verbose, DB_VERB_CURSOR))
+           __db_msg(dbenv,
+               "Cursor (%p) closed, associated txn: %p", dbc, dbc->txn);
        * Remove the cursor(s) from the active queue.  We may be closing two
        * cursors at once here, a top-level one and a lower-level, off-page
diff -rc db-4.6.21.orig/dbinc/db.in db-4.6.21/dbinc/db.in
*** db-4.6.21.orig/dbinc/db.in     Fri Jun 29 00:23:35 2007
--- db-4.6.21/dbinc/db.in     Mon May 12 17:23:20 2008
*** 2106,2111 ****
--- 2106,2112 ----
  #define     DB_VERB_REGISTER     0x0010     /* Dump waits-for table. */
  #define     DB_VERB_REPLICATION     0x0020     /* Replication information. */
  #define     DB_VERB_WAITSFOR     0x0040     /* Dump waits-for table. */
+ #define     DB_VERB_CURSOR          0x0080     /* Cursor open/close information. */
       u_int32_t      verbose;     /* Verbose output. */
       void          app_private;     / Application-private handle. */
diff -rc db-4.6.21.orig/dist/pubdef.in db-4.6.21/dist/pubdef.in
*** db-4.6.21.orig/dist/pubdef.in     Fri Jul  6 10:22:52 2007
--- db-4.6.21/dist/pubdef.in     Mon May 12 17:21:48 2008
*** 432,437 ****
--- 432,438 ----
  DB_USERCOPY_SETDATA          * I N
  DB_USE_ENVIRON               D I J
  DB_USE_ENVIRON_ROOT          D I J
+ DB_VERB_CURSOR               D I J
  DB_VERB_DEADLOCK          D I J
  DB_VERB_FILEOPS               D I J
  DB_VERB_FILEOPS_ALL          D I J
diff -rc db-4.6.21.orig/env/env_method.c db-4.6.21/env/env_method.c
*** db-4.6.21.orig/env/env_method.c     Fri May 18 01:15:11 2007
--- db-4.6.21/env/env_method.c     Mon May 12 17:28:57 2008
*** 1070,1075 ****
--- 1070,1076 ----
       int *onoffp;
       switch (which) {
+      case DB_VERB_CURSOR:
       case DB_VERB_DEADLOCK:
       case DB_VERB_FILEOPS:
       case DB_VERB_FILEOPS_ALL:
*** 1098,1103 ****
--- 1099,1105 ----
       int on;
       switch (which) {
+      case DB_VERB_CURSOR:
       case DB_VERB_DEADLOCK:
       case DB_VERB_FILEOPS:
       case DB_VERB_FILEOPS_ALL:
diff -rc db-4.6.21.orig/java/src/com/sleepycat/db/EnvironmentConfig.java db-4.6.21/java/src/com/sleepycat/db/EnvironmentConfig.java
*** db-4.6.21.orig/java/src/com/sleepycat/db/EnvironmentConfig.java     Fri Jul  6 10:22:54 2007
--- db-4.6.21/java/src/com/sleepycat/db/EnvironmentConfig.java     Mon May 12 17:25:22 2008
*** 114,119 ****
--- 114,120 ----
      private boolean yieldCPU = false;
      /* Verbose Flags */
+     private boolean verboseCursor = false;
      private boolean verboseDeadlock = false;
      private boolean verboseFileops = false;
      private boolean verboseFileopsAll = false;
*** 846,851 ****
--- 847,855 ----
      public void setVerbose(final VerboseConfig flag, boolean enable) {
          int iflag = flag.getInternalFlag();
          switch (iflag) {
+         case DbConstants.DB_VERB_CURSOR:
+             verboseCursor = enable;
+             break;
          case DbConstants.DB_VERB_DEADLOCK:
              verboseDeadlock = enable;
              break;
*** 876,881 ****
--- 880,887 ----
      public boolean getVerbose(final VerboseConfig flag) {
          int iflag = flag.getInternalFlag();
          switch (iflag) {
+         case DbConstants.DB_VERB_CURSOR:
+             return verboseCursor;
          case DbConstants.DB_VERB_DEADLOCK:
              return verboseDeadlock;
          case DbConstants.DB_VERB_FILEOPS:
*** 1151,1156 ****
--- 1157,1167 ----
              dbenv.set_flags(offFlags, false);
          /* Verbose flags */
+         if (verboseCursor && !oldConfig.verboseCursor)
+             dbenv.set_verbose(DbConstants.DB_VERB_CURSOR, true);
+         if (!verboseCursor && oldConfig.verboseCursor)
+             dbenv.set_verbose(DbConstants.DB_VERB_CURSOR, false);
+
          if (verboseDeadlock && !oldConfig.verboseDeadlock)
              dbenv.set_verbose(DbConstants.DB_VERB_DEADLOCK, true);
          if (!verboseDeadlock && oldConfig.verboseDeadlock)
*** 1342,1347 ****
--- 1353,1359 ----
          yieldCPU = ((envFlags & DbConstants.DB_YIELDCPU) != 0);
          /* Verbose flags */
+         verboseCursor = dbenv.get_verbose(DbConstants.DB_VERB_CURSOR);
          verboseDeadlock = dbenv.get_verbose(DbConstants.DB_VERB_DEADLOCK);
          verboseFileops = dbenv.get_verbose(DbConstants.DB_VERB_FILEOPS);
          verboseFileopsAll = dbenv.get_verbose(DbConstants.DB_VERB_FILEOPS_ALL);
diff -rc db-4.6.21.orig/java/src/com/sleepycat/db/VerboseConfig.java db-4.6.21/java/src/com/sleepycat/db/VerboseConfig.java
*** db-4.6.21.orig/java/src/com/sleepycat/db/VerboseConfig.java     Fri Jul  6 10:22:54 2007
--- db-4.6.21/java/src/com/sleepycat/db/VerboseConfig.java     Mon May 12 17:28:07 2008
*** 12,17 ****
--- 12,19 ----
  import com.sleepycat.db.internal.DbEnv;
  public final class VerboseConfig {
+     public static final VerboseConfig CURSOR =
+         new VerboseConfig("CURSOR", DbConstants.DB_VERB_CURSOR);
      public static final VerboseConfig DEADLOCK =
          new VerboseConfig("DEADLOCK", DbConstants.DB_VERB_DEADLOCK);
      public static final VerboseConfig FILEOPS =
diff -rc db-4.6.21.orig/java/src/com/sleepycat/db/internal/DbConstants.java db-4.6.21/java/src/com/sleepycat/db/internal/DbConstants.java
*** db-4.6.21.orig/java/src/com/sleepycat/db/internal/DbConstants.java     Fri Sep 28 01:32:07 2007
--- db-4.6.21/java/src/com/sleepycat/db/internal/DbConstants.java     Mon May 12 17:29:17 2008
*** 194,199 ****
--- 194,200 ----
      int DB_UPGRADE = 0x0000001;
      int DB_USE_ENVIRON = 0x0004000;
      int DB_USE_ENVIRON_ROOT = 0x0008000;
+     int DB_VERB_CURSOR = 0x0080;
      int DB_VERB_DEADLOCK = 0x0001;
      int DB_VERB_FILEOPS = 0x0002;
      int DB_VERB_FILEOPS_ALL = 0x0004;