Inconsistent behavior for DB- put

Question

Hi,
I am writing real-time tick data to a Berkeley DB database. my keys are symbolid and field name, for ex.
399432107170_last_trade
399432506589_last_trade
399435809008_last_trade
there are about 3000 symbols (keys) like that. Problem is that while most keys (&values) get written to the db, some dont. Also, all put statement return zero value, so I am assuming all puts were a success. in my 3 example keys above, the first one gets written to DB, but the 2nd and 3rd dont.
my write code is...
envFlags      = DB_CREATE | DB_INIT_MPOOL | DB_INIT_TXN | DB_INIT_LOCK | DB_THREAD;
dbFlags      = DB_CREATE | DB_THREAD;
void clBPOD::add_in_db(Db db_, Dbt k_, Dbt *v_, bool print) {
     try {
          int ret = db_->put(NULL, k_, v_, 0);
          if (ret != 0) {
               cout << "Could not write to DB(Err Code: " << ret << ")\n\tKey: " << (char *)(k_->get_data()) << endl;
     catch(DbException &e) {
          cout << "Could not write: " << e.what() << endl;
I know that those keys are not being written as when I try to read through the database using a cursor, those keys are not returned. here is my read code (Perl)...
     my $cur = $dbh_->db_cursor();
     while ($cur->c_get($key, $val, DB_NEXT) != DB_NOTFOUND) {
          $key = unpack("Z*", $key);
          $val = unpack("d", $val);
          if (defined($val)) {
               print "$key = $val\n";     
     $cur->c_close();
I am stumped and dont know how to fix this. Please help.
thanks
Nishant

Answer 1

yes Ben, this is very troubling.. specially given that I am loosing production data.
also, I am thinking that this is not concurrency related as its always the same keys that dont get written. This is weird as market data events happen at random. Why would only a specific set of queries always not be written.
As for transactions, I had initially set the DB_AUTO_COMMIT flag while opening DBs but that was draining my CPU and hard drive. I currently get about 15K ticks/sec (average, max of 50K ticks/sec). So I decided to remove that flag.
Also, I read that you need transactions when there are multiple processes writing to the DB concurrently. In my application, there is one multithreaded process that writes to the DB. All other processes only read.
any insight/help is greatly appreciated.
thanks
Nishant