Application hangs during accessing transactional database.

Question

Hi! Here is minimazed test which creates a database with secondary index, fills it with some records and then starts two threads. One thread adds records, another thread deletes records. The problem is that threads hangs with the following backtraces:
(dbx) thread t@2
Current function is __db_pthread_mutex_lock
204 RET_SET((pthread_cond_wait(
t@2 (l@2) stopped in ___lwp_cond_wait at 0xfffffd7ffee6e8ea
0xfffffd7ffee6e8ea: ___lwp_cond_wait+0x000a: jae ___lwp_cond_wait+0x18 [ 0xfffffd7ffee6e8f8, .+0xe ]
(dbx) where
current thread: t@2
[1] ___lwp_cond_wait(0xfffffd7ffeb88620, 0xfffffd7ffeb88608, 0x0, 0x0, 0xfffffd7ffeb88608, 0xfffffd7ffeb9dfb8), at 0xfffffd7ffee6e8ea
[2] __lwp_cond_wait(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7ffee574ec
=>[3] __db_pthread_mutex_lock(dbenv = 0x41c7f0, mutex = 118U), line 204 in "mut_pthread.c"
[4] __lock_get_internal(lt = 0x420f90, locker = 2147484656U, flags = 0, obj = 0x424eb0, lock_mode = DB_LOCK_WRITE, timeout = 0, lock = 0xfffffd7ffe9fd760), line 808 in "lock.c"
[5] __lock_vec(dbenv = 0x41c7f0, locker = 2147484656U, flags = 0, list = 0xfffffd7ffe9fd748, nlist = 2, elistp = 0xfffffd7ffe9fd740), line 116 in "lock.c"
[6] __db_lget(dbc = 0x424df0, action = 2, pgno = 5U, mode = DB_LOCK_WRITE, lkflags = 0, lockp = 0xfffffd7ffe9fd8d8), line 1006 in "db_meta.c"
[7] __bam_search(dbc = 0x424df0, root_pgno = 1U, key = 0xfffffd7ffe9fdc68, flags = 1410U, slevel = 1, recnop = (nil), exactp = 0xfffffd7ffe9fda6c), line 489 in "bt_search.c"
[8] __bam_c_search(dbc = 0x424df0, root_pgno = 0, key = 0xfffffd7ffe9fdc68, flags = 25U, exactp = 0xfffffd7ffe9fda6c), line 2479 in "bt_cursor.c"
[9] __bam_c_get(dbc = 0x424df0, key = 0xfffffd7ffe9fdc68, data = 0xfffffd7ffe9fdce0, flags = 25U, pgnop = 0xfffffd7ffe9fdb34), line 953 in "bt_cursor.c"
[10] __db_c_get(dbc_arg = 0x42f6f0, key = 0xfffffd7ffe9fdc68, data = 0xfffffd7ffe9fdce0, flags = 25U), line 689 in "db_cam.c"
[11] __db_c_put(dbc_arg = 0x424460, key = 0xfffffd7ffe9fdf28, data = 0xfffffd7ffe9fdf00, flags = 20U), line 1327 in "db_cam.c"
[12] __db_put(dbp = 0x41d070, txn = 0x423f30, key = 0xfffffd7ffe9fdf28, data = 0xfffffd7ffe9fdf00, flags = 20U), line 380 in "db_am.c"
[13] __db_put_pp(dbp = 0x41d070, txn = 0x423f30, key = 0xfffffd7ffe9fdf28, data = 0xfffffd7ffe9fdf00, flags = 20U), line 1500 in "db_iface.c"
[14] Db::put(this = 0x41c708, txnid = (nil), key = 0xfffffd7ffe9fdf28, value = 0xfffffd7ffe9fdf00, flags = 20U), line 340 in "cxx_db.cpp"
[15] Storage::put(this = 0x41c690, key = 2003, data = 2003), line 114 in "bdb_put_del_hang.cpp"
[16] reading_thread_func(arg = (nil)), line 161 in "bdb_put_del_hang.cpp"
[17] thrsetup(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7ffee6b40b
[18] lwpstart(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7ffee6b640
(dbx) thread t@3
Current function is __db_pthread_mutex_lock
204 RET_SET((pthread_cond_wait(
t@3 (l@3) stopped in ___lwp_cond_wait at 0xfffffd7ffee6e8ea
0xfffffd7ffee6e8ea: ___lwp_cond_wait+0x000a: jae ___lwp_cond_wait+0x18 [ 0xfffffd7ffee6e8f8, .+0xe ]
(dbx) where
current thread: t@3
[1] ___lwp_cond_wait(0xfffffd7ffeb88668, 0xfffffd7ffeb88650, 0x0, 0x0, 0xfffffd7ffeb88650, 0xfffffd7ffeb9dfb8), at 0xfffffd7ffee6e8ea
[2] __lwp_cond_wait(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7ffee574ec
=>[3] __db_pthread_mutex_lock(dbenv = 0x41c7f0, mutex = 119U), line 204 in "mut_pthread.c"
[4] __lock_get_internal(lt = 0x420f90, locker = 2147484655U, flags = 0, obj = 0x424b80, lock_mode = DB_LOCK_WRITE, timeout = 0, lock = 0xfffffd7ffe7fd7d0), line 808 in "lock.c"
[5] __lock_vec(dbenv = 0x41c7f0, locker = 2147484655U, flags = 0, list = 0xfffffd7ffe7fd7b8, nlist = 2, elistp = 0xfffffd7ffe7fd7b0), line 116 in "lock.c"
[6] __db_lget(dbc = 0x424ac0, action = 2, pgno = 5U, mode = DB_LOCK_WRITE, lkflags = 0, lockp = 0xfffffd7ffe7fd948), line 1006 in "db_meta.c"
[7] __bam_search(dbc = 0x424ac0, root_pgno = 1U, key = 0xfffffd7ffe7fdc48, flags = 1410U, slevel = 1, recnop = (nil), exactp = 0xfffffd7ffe7fdadc), line 489 in "bt_search.c"
[8] __bam_c_search(dbc = 0x424ac0, root_pgno = 0, key = 0xfffffd7ffe7fdc48, flags = 8U, exactp = 0xfffffd7ffe7fdadc), line 2479 in "bt_cursor.c"
[9] __bam_c_get(dbc = 0x424ac0, key = 0xfffffd7ffe7fdc48, data = 0xfffffd7ffe7fdc70, flags = 8U, pgnop = 0xfffffd7ffe7fdba4), line 871 in "bt_cursor.c"
[10] __db_c_get(dbc_arg = 0x424790, key = 0xfffffd7ffe7fdc48, data = 0xfffffd7ffe7fdc70, flags = 8U), line 689 in "db_cam.c"
[11] __db_c_del_primary(dbc = 0x425120), line 2249 in "db_cam.c"
[12] __db_c_del(dbc = 0x425120, flags = 0), line 285 in "db_cam.c"
[13] __db_del(dbp = 0x41d070, txn = 0x42f5d0, key = 0xfffffd7ffe7fdf30, flags = 0), line 492 in "db_am.c"
[14] __db_del_pp(dbp = 0x41d070, txn = 0x42f5d0, key = 0xfffffd7ffe7fdf30, flags = 0), line 485 in "db_iface.c"
[15] Db::del(this = 0x41c708, txnid = (nil), key = 0xfffffd7ffe7fdf30, flags = 0), line 226 in "cxx_db.cpp"
[16] Storage::del(this = 0x41c690, key = 501), line 131 in "bdb_put_del_hang.cpp"
[17] writing_thread_func(arg = (nil)), line 173 in "bdb_put_del_hang.cpp"
[18] thrsetup(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7ffee6b40b
[19] lwpstart(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7ffee6b640
Looks like some problems with locking, because if I set_lk_detect(DB_LOCK_DEFAULT) some methods fails with DB_LOCK_DEADLOCK, but i do not understand where is raise condition, because both threads calls just a single operations. Can anybody explain the problem?
Here is soruce code:
#include <set>
#include "pthread.h"
#include "bdb/db_cxx.h"
static const char *__THIS_FILE__ = __FILE__;
class Storage
private:
DbEnv m_bdbEnv;
Db m_bdbLongKeys, m_bdbLongKeysSec;
static void bdb_error_call( const DbEnv dbEnv, const char errpfx, const char *msg )
printf( "%s\n", msg );
static int bdb_sec_callback( Db secondary, const Dbt pkey, const Dbt pdata, Dbt skey )
skey->set_data( pdata->get_data() );
skey->set_size( pdata->get_size() );
return 0;
public:
Storage( void )
: m_bdbEnv( DB_CXX_NO_EXCEPTIONS ),
m_bdbLongKeys( &m_bdbEnv, DB_CXX_NO_EXCEPTIONS ),
m_bdbLongKeysSec( &m_bdbEnv, DB_CXX_NO_EXCEPTIONS )
int open( const char *storageURL )
m_bdbEnv.set_errcall( bdb_error_call );
static const u_int32_t envFlags =
DB_CREATE | /* create if not exists */
DB_INIT_MPOOL | /* memory pool */
DB_INIT_LOCK | /* locking */
DB_INIT_LOG | /* recovery */
DB_INIT_TXN | /* transactions */
DB_THREAD;
int error = m_bdbEnv.open( storageURL, envFlags, 0 );
if (error != 0)
m_bdbEnv.err( error, "DbEnv::open() failed." );
return -1;
static const u_int32_t dbFlags =
DB_CREATE | /* create if not exists */
DB_AUTO_COMMIT | /* auto commit */
DB_THREAD;
m_bdbLongKeys.set_errcall( bdb_error_call );
error = m_bdbLongKeys.open( NULL, "longKeys.db", NULL, DB_BTREE, dbFlags, 0 );
if (error != 0)
m_bdbLongKeys.err( error, "Db::open() failed." );
return -1;
m_bdbLongKeysSec.set_errcall( bdb_error_call );
m_bdbLongKeysSec.set_flags( DB_DUP | DB_DUPSORT );
if (error != 0)
m_bdbLongKeysSec.err( error, "Db::set_flags() failed (%s:%d).",
__THIS_FILE__, __LINE__ );
return -1;
error = m_bdbLongKeysSec.open( NULL, "longKeys-sec.db", NULL, DB_BTREE, dbFlags, 0 );
if (error != 0)
m_bdbLongKeysSec.err( error, "Db::open() failed (%s:%d).",
__THIS_FILE__, __LINE__ );
return -1;
error = m_bdbLongKeys.associate( NULL, &m_bdbLongKeysSec, bdb_sec_callback, DB_AUTO_COMMIT );
if (error != 0)
m_bdbLongKeys.err( error, "Db::associate() failed (%s:%d).",
__THIS_FILE__, __LINE__ );
return -1;
return 0;
int close( void )
m_bdbLongKeysSec.close(0);
m_bdbLongKeys.close(0);
m_bdbEnv.close(0);
return 0;
int put( long key, long data )
Dbt dbtKey, dbtData;
dbtKey.set_data( (void*) &key );
dbtKey.set_size( sizeof(key) );
dbtData.set_data( (void*) &data );
dbtData.set_size( sizeof(data) );
int error = m_bdbLongKeys.put( NULL, &dbtKey, &dbtData, DB_NOOVERWRITE );
if (error != 0)
m_bdbLongKeys.err( error, "Db::put() failed (%s:%d)", __THIS_FILE__, __LINE__ );
return -1;
return 0;
int del( long key )
Dbt dbtKey;
dbtKey.set_data( (void*) &key );
dbtKey.set_size( sizeof(key) );
int error = m_bdbLongKeys.del( NULL, &dbtKey, 0 );
if (error != 0)
m_bdbLongKeys.err( error, "Db::del() failed (%s:%d)", __THIS_FILE__, __LINE__ );
return -1; }
return 0;
Storage *g_storage = NULL;
static long g_numberTotal = 1000;
static long g_numberToRead = g_numberTotal/2;
static std::set<long> g_toRead;
static std::set<long> g_toWrite;
extern "C" void reading_thread_func( void arg );
extern "C" void writing_thread_func( void arg );
void reading_thread_func( void arg )
std::set<long>::const_iterator it = g_toRead.begin();
std::set<long>::const_iterator it_end = g_toRead.end();
for (int j = 2*g_numberTotal+1; it != it_end; ++it) {
++j;
if (g_storage->put(j, j) != 0)
printf( "put() failed (%s:%d).\n", __THIS_FILE__, __LINE__ );
return NULL;
void writing_thread_func( void arg )
std::set<long>::const_iterator it = g_toWrite.begin();
std::set<long>::const_iterator it_end = g_toWrite.end();
for (; it != it_end; it++) {
if (g_storage->del(*it) == 0)
printf( "Deleted key %d\n", *it );
else
printf( "Failed to delete key %d\n", *it );
return NULL;
int main( int argc, char *argv[] )
pthread_t read_tid, write_tid;
if (g_numberToRead > g_numberTotal)
return 1;
g_storage = new Storage;
if (g_storage->open("test_storage") != 0)
return 3;
for (int i = 0; i < g_numberTotal; ++i) {
if (g_storage->put(i, i) != 0)
std::cout << "Failed to add key '" << i << "'" << std::endl;
if (i <= g_numberToRead)
g_toRead.insert(i);
else
g_toWrite.insert(i);
pthread_create( &read_tid, NULL, reading_thread_func, NULL );
pthread_create( &write_tid, NULL, writing_thread_func, NULL );
pthread_join( read_tid, NULL );
pthread_join( write_tid, NULL );
if (g_storage != NULL) {
g_storage->close();
delete g_storage;
return 0;
}

Answer 1

Hi,
Thank you for you answer, but I still not happywith
it.
I attentively read the documents you providedbefore
posted the message on the forum. Ok. Let me explain again.
I could enclose these calls in each owntransaction
(and I tried it, but had the same result) but itdoes
not make a sence if DB_AUTO_COMMIT flag wasspecified
for the Db::open()Using transaction does not necessarily eliminate the
chance of deadlock. Practically any application that
uses locking may deadlock. Can you post here the
example in which you are trying it transactionally?
And maybe I will try to correct it.The example is in the my first message. It would be nice if you could suggest a good solution to workaround the deadlock.
There is only ONE OPERATION from the BerkeleyDBAPI
point of view in each thread under the transaction
and I would expect that the one single Db::put()or
Db::del() can not cause the deadlock. Why it
happens?Because "With the exception of the Queue access
method, the Berkeley DB access methods do page-level
locking." (Locking granularity:
http://www.oracle.com/technology/documentation/berkele
y-db/db/ref/lock/page.html)
The only solution I see now is use
DbEnv::set_lk_detect() and handle the
DB_LOCK_DEADLOCK error code on each DB operationand
retry the operation on the deadlock, but it is the
last thing I would like to use.
The conclusion:
- why single API call of Db::put() or Db::del()under
transaction made in the same time cause adeadlock?
Because both of them will require write locks on the
same page.
"The first component of the infrastructure, deadlock
detection, is not so much a requirement specific to
transaction-protected applications, but instead is
necessary for almost all applications in which more
than a single thread of control will be accessing the
database at one time. Even when Berkeley DB
automatically handles database locking, it is
normally possible for deadlock to occur. Because the
underlying database access methods may update
multiple pages during a single Berkeley DB API call,
deadlock is possible even when threads of control are
making only single update calls into the database."
(Deadlock detection:
http://www.oracle.com/technology/documentation/berkele
y-db/db/ref/transapp/deadlock.html)
Please let me know if this time I was clear enough.
Bogdan Coman