DB_DUP causes transaction out of memory

Question

HI I have been using berkeley DB for storig high volume real-time tick data (25k-50k ticks/sec). Its been working like a charm for some time. Today I tried to allow duplicates (entries with the same key) with the DB_DUP option. all of a sudden i started getting the folllowing error...
Error No: 12, Db::put: Cannot allocate memory
Unable to allocate memory for transaction detail
my DB_CONFIG file has the following entries...
set_cachesize 0 536870912 0
set_lg_regionmax 1000000
set_tx_max 2000
I googled and found that increasing tx_max may solve this, I went upto 10000 but still the same (it did take a longer time to start getting the errors)
any help is appreciated.
db_ = new Db(env_, 0);
db_->open(NULL, sfile.c_str(), NULL, DB_BTREE, DB_CREATE | DB_AUTO_COMMIT | DB_THREAD | DB_DUP, 0);
void store_in_db(Db db_, Dbt k_, Dbt *v_) {
int i_tries=0;
while (i_tries < 10) {
try {
int ret = db_->put(NULL, k_, v_, 0);
if (ret != 0) {
cout << "Could not write to DB(Err Code: " << ret << ")\n\tKey: " << (char *)(k_->get_data()) << endl;
return;
} catch (DbDeadlockException &e) {
i_tries++;
cout << "Deadlock detected: Try #" << i_tries << endl;
} catch(DbException &e) {
cout << "Could not write to DB, Key: " << (char *)(k_->get_data()) << endl << "Error No: " << e.get_errno() << ", " << e.what() << endl;
return;
}

Answer 1

Hi Mike, thanks for your response. I ran the code again and did db_stat -t twice when it was running and once when it started throwing the "Error No: 12, Db::put: Cannot allocate memory..Unable to allocate memory for transaction detail" errors. As you can see below, ' Snapshot transactions' and 'Maximum snapshot transactions' are same and keep increasing. the application fails when they go above 2000 (which is the limit I have set in DB_CONFIG). I open the environment with the following code...
     u_int32_t envFlags      = DB_CREATE | DB_RECOVER | DB_INIT_MPOOL | DB_INIT_TXN | DB_INIT_LOCK | DB_INIT_LOG | DB_THREAD;
     env_ = new DbEnv(0);
     try {
          env_->set_lk_detect(DB_LOCK_MINWRITE);
          env_->set_lg_max(1000*1024*1024);
          env_->log_set_config(DB_LOG_AUTO_REMOVE, 1);
          env_->set_flags(DB_TXN_WRITE_NOSYNC, 1);     //1 is to set the flag, 0 is to clear the flag
          env_->open("/RSIGrid/TickStore", envFlags, 0);
     } catch(DbException &e) {
          cout << "DbException while opening environment: " << e.get_errno() << ", " << e.what() << endl;
          return 1;
[nmittal@development TickStore]$ db_stat -t
1/6583654 File/offset for last checkpoint LSN
Wed Jul 1 08:15:19 2009 Checkpoint timestamp
0x80015b7c Last transaction ID allocated
2000 Maximum number of active transactions configured
0 Active transactions
5 Maximum active transactions
88956 Number of transactions begun
0 Number of transactions aborted
88956 Number of transactions committed
612 Snapshot transactions
612 Maximum snapshot transactions
0 Number of transactions restored
768KB Transaction region size
2412 The number of region locks that required waiting (0%)
Active transactions:
[nmittal@development TickStore]$
[nmittal@development TickStore]$ db_stat -t
1/28447874 File/offset for last checkpoint LSN
Wed Jul 1 08:16:49 2009 Checkpoint timestamp
0x8002e37d Last transaction ID allocated
2000 Maximum number of active transactions configured
0 Active transactions
5 Maximum active transactions
189309 Number of transactions begun
0 Number of transactions aborted
189309 Number of transactions committed
1340 Snapshot transactions
1340 Maximum snapshot transactions
0 Number of transactions restored
768KB Transaction region size
5105 The number of region locks that required waiting (0%)
Active transactions:
[nmittal@development TickStore]$
[nmittal@development TickStore]$ db_stat -t
1/45426368 File/offset for last checkpoint LSN
Wed Jul 1 08:18:20 2009 Checkpoint timestamp
0x800481da Last transaction ID allocated
2000 Maximum number of active transactions configured
0 Active transactions
5 Maximum active transactions
295386 Number of transactions begun
0 Number of transactions aborted
295386 Number of transactions committed
2134 Snapshot transactions
2134 Maximum snapshot transactions
0 Number of transactions restored
768KB Transaction region size
8210 The number of region locks that required waiting (0%)
Active transactions:
[nmittal@development TickStore]$