Cursors throwing Unexpected lock status: 6

Question

I am using a Berkeley DB instance and have a multithreaded clients each performing various types of operations on the database. In general there are three types of operations being performed, reads, writes (which use RMW locks), and (non-transactional) cursors. After some time my application dumps the following:
Unexpected lock status: 6
PANIC: Invalid argument
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__os_stack+0x2b) [0xb7ef085f]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__os_abort+0x1d) [0xb7eecbb1]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__env_panic+0xa8) [0xb7e80b8e]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__lock_get_internal+0x1855) [0xb7e2e925]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__lock_vec+0x167) [0xb7e2bd3c]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__db_lget+0x45c) [0xb7e8bfe2]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so [0xb7d850a5]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so [0xb7d816ce]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__dbc_get+0x598) [0xb7e76a29]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__dbc_get_pp+0x11f) [0xb7e870ec]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(_ZN3Dbc3getEP3DbtS1_j+0x3f) [0xb7d71feb]
...(non-berkeley db layers)...
/lib/tls/i686/cmov/libpthread.so.0 [0xb7a2350f]
/lib/tls/i686/cmov/libc.so.6(clone+0x5e) [0xb7b20a0e]
Aborted
In studying the core dumps and doing some testing this behavior only occurs when cursors are used, when those operations are removed the error no longer manifests itself so either the cursor is the issue, or how the cursors are interacting with other operations. Below is a sample of cursor code being run:
void run_cursor()
Dbc *curp = NULL;
int ret;
try
int i = 0;
Dbt key(&i,sizeof(int));
Dbt data(buffer, 0);
data.set_flags(DB_DBT_USERMEM);
data.set_ulen(MAX_BUFFER_SIZE);
db->cursor(NULL, &curp, 0);
while ((ret = curp->get(&key, &data, DB_NEXT)) == 0) {
//keep a list of all the keys in the db
keys.push_back(i);
if (ret != DB_NOTFOUND) {
cerr << "Cursor Error" << endl;
curp->close();
catch (DbException &ex) {
cout << ex.what() << endl;
if( curp != NULL )
curp->close();
I have tried wrapping the cursor in a transaction and it seems to worsen the issue (it happens more frequently). In all methods I am printing error messages when exceptions are thrown by the database (e.g. DbExceptions) and the program prints no errors so I do not believe it is a side effect of some other error. Here is the relevant parts of the case where is RMW lock is used:
void update()
DbTxn *txn = NULL;
try
env->txn_begin(NULL, &txn, 0);
int id;
Dbt key( &id, sizeof(int) );
Dbt value(buffer, 0);
value.set_flags(DB_DBT_USERMEM);
value.set_ulen(MAX_BUFFER_SIZE);
int err = db->get(txn, &key, &value, DB_RMW);
if( err == DB_NOTFOUND )
cerr << "Error trying to read key" << endl;
txn->abort(0);
return;
//code to modify the value here
db->put(txn, &key, &value, 0);
txn->commit(0);
} catch (DbDeadlockException &de) {
cout << de.what() << endl;
if( txn != NULL )
txn->abort();
Just in case it helps, here is the output of db_stat -CA on the database after it has crashed:
Default locking region information:
12     Last allocated locker ID
0x7fffffff     Current maximum unused locker ID
9     Number of lock modes
5000     Maximum number of locks possible
5000     Maximum number of lockers possible
1000     Maximum number of lock objects possible
1     Number of lock object partitions
6     Number of current locks
9     Maximum number of locks at any one time
4     Maximum number of locks in any one bucket
0     Maximum number of locks stolen by for an empty partition
0     Maximum number of locks stolen for any one partition
11     Number of current lockers
13     Maximum number of lockers at any one time
3     Number of current lock objects
9     Maximum number of lock objects at any one time
2     Maximum number of lock objects in any one bucket
0     Maximum number of objects stolen by for an empty partition
0     Maximum number of objects stolen for any one partition
3786734     Total number of locks requested
3786664     Total number of locks released
0     Total number of locks upgraded
1     Total number of locks downgraded
7089     Lock requests not available due to conflicts, for which we waited
61     Lock requests not available due to conflicts, for which we did not wait
0     Number of deadlocks
0     Lock timeout value
0     Number of locks that have timed out
1000000     Transaction timeout value
0     Number of transactions that have timed out
1MB 592KB     The size of the lock region
0     The number of partition locks that required waiting (0%)
0     The maximum number of times any partition lock was waited for (0%)
0     The number of object queue operations that required waiting (0%)
0     The number of locker allocations that required waiting (0%)
165     The number of region locks that required waiting (0%)
2     Maximum hash bucket length
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock REGINFO information:
Lock     Region type
5     Region ID
__db.005     Region name
0xb781b000     Original region address
0xb781b000     Region address
0xb781b0c0     Region primary address
0     Region maximum allocation
0     Region allocated
Region allocations: 11006 allocations, 0 failures, 0 frees, 1 longest
Allocations by power-of-two sizes:
1KB     11003
2KB     0
4KB     0
8KB     0
16KB     1
32KB     0
64KB     1
128KB     1
256KB     0
512KB     0
1024KB     0
REGION_JOIN_OK     Region flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock region parameters:
104     Lock region region mutex [165/7025066 0% 5979/3075574672]
8191     locker table size
1031     object table size
600     obj_off
62608     locker_off
0     need_dd
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock conflict matrix:
0     0     0     0     0     0     0     0     0     
0     0     1     0     1     0     1     0     1     
0     1     1     1     1     1     1     1     1     
0     0     0     0     0     0     0     0     0     
0     1     1     0     0     0     0     1     1     
0     0     1     0     0     0     0     0     1     
0     1     1     0     0     0     0     1     1     
0     0     1     0     1     0     1     0     0     
0     1     1     0     1     1     1     0     1     
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by lockers:
Locker Mode Count Status ----------------- Object ---------------
4 dd= 9 locks held 1 write locks 0 pid/thread 5979/3080828608
4 READ 1 HELD ./db.db handle 0
5 dd= 8 locks held 0 write locks 0 pid/thread 5979/3072416656
6 dd= 7 locks held 0 write locks 0 pid/thread 5979/3073469328
7 dd= 6 locks held 0 write locks 0 pid/thread 5979/3073469328
8 dd= 5 locks held 0 write locks 0 pid/thread 5979/3073469328
9 dd= 4 locks held 0 write locks 0 pid/thread 5979/3072416656
a dd= 3 locks held 1 write locks 0 pid/thread 5979/3073469328
a READ 1 WAIT ./db.db page 21
a READ 2 HELD ./db.db page 20
b dd= 2 locks held 1 write locks 0 pid/thread 5979/3075574672
b READ 1 WAIT ./db.db page 21
b READ 2 HELD ./db.db page 20
c dd= 1 locks held 0 write locks 0 pid/thread 5979/3075574672
800054d9 dd= 0 locks held 1 write locks 1 pid/thread 5979/3074522000expires 05-05-12:41:36.048050000
800054d9 WRITE 2 HELD ./db.db page 21
800054da dd= 0 locks held 0 write locks 0 pid/thread 5979/3072416656
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by object:
Locker Mode Count Status ----------------- Object ---------------
800054d9 WRITE 2 HELD ./db.db page 21
a READ 1 WAIT ./db.db page 21
b READ 1 WAIT ./db.db page 21
a READ 2 HELD ./db.db page 20
b READ 2 HELD ./db.db page 20
4 READ 1 HELD ./db.db handle 0
The number of locks and lockers is somewhat low at 5,000 each, this was intentional since the same error occurs with larger numbers of locks/lockers it simply takes longer to occur. I have reviewed this output and nothing out of the ordinary seems to be occurring as far as I can tell.
Any additional insight would be appreciated.

Answer 1

Thanks to both of you for your suggestions.
I have still noticing the same issue with very similar output etc so I wont repost that. After running it some more I have started seeing another error message, it looks like its very similar and hopefully with make the issue more clear.
Here is the output of the error file:
TAS unlock failed: lock already unlocked
PANIC: Permission denied
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__os_stack+0x2b) [0xb7ee585f]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__os_abort+0x1d) [0xb7ee1bb1]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__env_panic+0xa8) [0xb7e75b8e]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__db_tas_mutex_unlock+0xa5) [0xb7d72245]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so [0xb7e27d04]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__lock_detect+0x129) [0xb7e268bf]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__lock_get_internal+0x1434) [0xb7e23504]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__lock_vec+0x167) [0xb7e20d3c]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__db_lget+0x45c) [0xb7e80fe2]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so [0xb7d7a0a5]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so [0xb7d766ce]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__dbc_get+0x598) [0xb7e6ba29]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(__dbc_get_pp+0x11f) [0xb7e7c0ec]
/usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so(_ZN3Dbc3getEP3DbtS1_j+0x3f) [0xb7d66feb]
...(user code stack layers)...
/lib/tls/i686/cmov/libpthread.so.0 [0xb79a950f]
/lib/tls/i686/cmov/libc.so.6(clone+0x5e) [0xb7a9ea0e]
Here is the output of the waits for table:
Waiter:     Waiting on:
80002249/50:     957608
6/50:     80002249 957208
Waitsfor array
Waiter:     Waiting on:
8000224a/28:     957288
a/28:     8000224a 957608
Waitsfor array
Waiter:     Waiting on:
8000224c/22:     957608
6/22:     8000224c 957368
Waitsfor array
Waiter:     Waiting on:
8000224c/22:     957608
6/22:     8000224c 957368
4/22:     8000224c 957048
Waitsfor array
Waiter:     Waiting on:
8000224d/25:     957528
7/25:     8000224d 957608
Waitsfor array
Waiter:     Waiting on:
8000224d/25:     957528
7/25:     8000224d 957608
6/25:     8000224d 957208
Waitsfor array
Waiter:     Waiting on:
8000224d/25:     957528
7/25:     8000224d 957608
6/25:     8000224d 957208
4/25:     8000224d 957288
Waitsfor array
Waiter:     Waiting on:
80002251/10:     957048
a/10:     80002251 957128
Waitsfor array
Waiter:     Waiting on:
80002257/39:     80002254 957368
80002254/39:     957288
Waitsfor array
Waiter:     Waiting on:
80002258/14:     957048
4/14:     80002258 957608
Waitsfor array
Waiter:     Waiting on:
80002258/14:     957048
8/14:     80002258 957208
4/14:     80002258 957608
Waitsfor array
Waiter:     Waiting on:
80002259/39:     80002257 957448
80002258/14:     957048
80002257/39:     957368
8/14:     80002258 957208
4/14:     80002258 957608
Waitsfor array
Waiter:     Waiting on:
80002259/39:     957448
8/39:     80002259 957288
7/39:     80002259 957048
6/39:     80002259 957528
Waitsfor array
Waiter:     Waiting on:
80002259/0:     0
80002257/0:     0
Waitsfor array
Waiter:     Waiting on:
8000225a/40:     956968
7/40:     8000225a 956888
Waitsfor array
Waiter:     Waiting on:
8000225a/40:     956968
7/40:     8000225a 956888
4/40:     8000225a 957608
It looks like there is a similar pattern here at the end with the two entries waiting on zero ("80002259/0:     0 80002257/0:     0"), which is why I suspect its the same error just manifesting itself in a slightly different manner.
I am using lock detection, I currently have it set to use DB_LOCK_MAXLOCKS, I have tried others as well with not change in behavior.
Most of my transaction code takes the following form, so if the get is successful (the id exists) then the data is updated and written back to the database. If an exception is thrown then it is caught and the transaction is aborted. As far as I can tell this should not leave and transaction handles left open or anything along those lines as the error message suggests.
void do_txn()
DbTxn *txn = NULL;
try
db->txn_begin(NULL, &txn, 0);
Data d;
//does a db->get using the txn pointer and a DB_RMW lock
//returns do_get returns false if the id does not exist
bool success = do_get(id, &d, txn);
if( !success )
txn->abort();
return;
//update data as necessary
//does a db->put using the txn pointer
do_put(id, &d, txn);
txn->commit(0);
} catch (DbDeadlockException &de) {
cout << de.what() << endl;
if( txn != NULL )
txn->abort();
}