The Bug about 'DB_SECONDARY_BAD' still exists in BerkeleyDB4.8!

Question

The Bug about 'DB_SECONDARY_BAD' still exists in BerkeleyDB4.8?
I'm sorry for my poor English, But I just cannot find anywhere else for help.
Thanks for your patience first!
I'm using BDB4.8 C++ API on Ubuntu 10.04, Linux Kernel 2.6.32-24-generic
$uname -a
$Linux wonpc 2.6.32-24-generic #43-Ubuntu SMP Thu Sep 16 14:17:33 UTC 2010 i686 GNU/Linux
When I update(overwrite) a record in database, I may get a DB_SECONDARY_BAD exception,
What's worse, This case doesn't always occures, it's random. So I think it probably a bug
of BDB, I have seen many issues about DB_SECONDARY_BAD with BDB4.5,4.6...
To reproduce the issue, I make a simplified test program from my real program.
The data to be stroed into database is a class called 'EntryData', It's defined in db_access.h,
where also defines some HighLevel API functions that hide the BDB calls, such as
store_entry_data(), which use EntryData as its argument. The EntryData have a string-type
member-data 'name' and a vector<string>-type mem-data 'labels', So store_entry_data() will
put the real data of EntryData to a contiguous memory block. The get_entry_data() returns
an EntryData builed up from the contiguous memory block fetched from database.
The comlete test program is post following this line:
/////////db_access.h////////////
#ifndef __DB_ACCESS_H__
#define __DB_ACCESS_H__
#include <string>
#include <vector>
#include <db_cxx.h>
class EntryData;
//extern Path DataDir; // default value, can be changed
extern int database_setup();
extern int database_close();
extern int store_entry_data(const EntryData&, u_int32_t = DB_NOOVERWRITE);
extern int get_entry_data(const std::string&, EntryData*, u_int32_t = 0);
extern int rm_entry_data(const std::string&);
class DBSetup
// 构造时调用database_setup, 超出作用域自动调用database_close .
// 该类没有数据成员.
public:
DBSetup() {
database_setup();
~DBSetup() {
database_close();
class EntryData
public:
typedef std::vector<std::string> LabelContainerType;
EntryData() {}
EntryData(const std::string& s) : name(s) {}
EntryData(const std::string& s, LabelContainerType& v)
: name(s), labels(v) {}
EntryData(const std::string&, const char*[]);
class DataBlock;
// 直接从内存块中构建, mem指针将会从数据库中获取,
// 它就是EntryData转化成的DataBlock中buf_ptr->buf的内容.
EntryData(const void* mem_blk, const int len);
~EntryData() {};
const std::string& get_name () const { return name; }
const LabelContainerType& get_labels() const { return labels; }
void set_name (const std::string& s) { name = s; }
void add_label(const std::string&);
void rem_label(const std::string&);
void show() const;
//　get contiguous memory for all:
DataBlock get_block() const { return DataBlock(*this); }
class DataBlock
// contiguous memory for all.
public:
DataBlock(const EntryData& data);
// 引进一块内存作为 buf_ptr->buf 的内容.
// 例如从数据库中获取结果
DataBlock(void* mem, int len);
// 复制构造函数:
DataBlock(const DataBlock& orig) :
data_size(orig.data_size),
capacity(orig.capacity),
buf_ptr(orig.buf_ptr) { ++buf_ptr->use; }
// 赋值操作符:
DataBlock& operator=(const DataBlock& oth)
data_size = oth.data_size;
capacity = oth.capacity;
if(--buf_ptr->use == 0)
delete buf_ptr;
buf_ptr = oth.buf_ptr;
return *this;
~DataBlock() {
if(--buf_ptr->use == 0) { delete buf_ptr; }
// data()函数因 Dbt 构造函数不支持const char*而被迫返回 char*
// data() 返回的指针是应该被修改的.
const char* data() const { return buf_ptr->buf; }
int size() const { return data_size; }
private:
void pack_str(const std::string& s);
static const int init_capacity = 100;
int data_size; // 记录数据块的长度.
int capacity; // 已经分配到 buf 的内存大小.
class SmartPtr; // 前向声明.
SmartPtr* buf_ptr;
class SmartPtr
friend class DataBlock;
char* buf;
int use;
SmartPtr(char* p) : buf(p), use(1) {}
~SmartPtr() { delete [] buf; }
private:
std::string name; // entry name
LabelContainerType labels; // entry labels list
}; // class EntryData
#endif
//////db_access.cc/////////////
#include <iostream>
#include <cstring>
#include <cstdlib>
#include <vector>
#include <algorithm>
#include "directory.h"
#include "db_access.h"
using namespace std;
static Path DataDir("~/mydict_data"); // default value, can be changed
const Path& get_datadir() { return DataDir; }
static DbEnv myEnv(0);
static Db db_bynam(&myEnv, 0); // using name as key
static Db db_bylab(&myEnv, 0); // using label as key
static int generate_keys_for_db_bylab
(Db* sdbp, const Dbt* pkey, const Dbt* pdata, Dbt* skey)
EntryData entry_data(pdata->get_data(), pdata->get_size());
int lab_num = entry_data.get_labels().size();
Dbt* tmpdbt = (Dbt*) malloc( sizeof(Dbt) * lab_num );
memset(tmpdbt, 0, sizeof(Dbt) * lab_num);
EntryData::LabelContainerType::const_iterator
lab_it = entry_data.get_labels().begin(), lab_end = entry_data.get_labels().end();
for(int i = 0; lab_it != lab_end; ++lab_it, ++i) {
tmpdbt[ i ].set_data( (void*)lab_it->c_str() );
tmpdbt[ i ].set_size( lab_it->size() );
skey->set_flags(DB_DBT_MULTIPLE | DB_DBT_APPMALLOC);
skey->set_data(tmpdbt);
skey->set_size(lab_num);
return 0;
//@Return Value: return non-zero at error
extern int database_setup()
const string DBEnvHome (DataDir + "DBEnv");
const string dbfile_bynam("dbfile_bynam");
const string dbfile_bylab("dbfile_bylab");
db_bylab.set_flags(DB_DUPSORT);
const u_int32_t env_flags = DB_CREATE | DB_INIT_MPOOL;
const u_int32_t db_flags = DB_CREATE;
rmkdir(DBEnvHome);
try
myEnv.open(DBEnvHome.c_str(), env_flags, 0);
db_bynam.open(NULL, dbfile_bynam.c_str(), NULL, DB_BTREE, db_flags, 0);
db_bylab.open(NULL, dbfile_bylab.c_str(), NULL, DB_BTREE, db_flags, 0);
db_bynam.associate(NULL, &db_bylab, generate_keys_for_db_bylab, 0);
} catch(DbException &e) {
cerr << "Err when open DBEnv or Db: " << e.what() << endl;
return -1;
} catch(std::exception& e) {
cerr << "Err when open DBEnv or Db: " << e.what() << endl;
return -1;
return 0;
int database_close()
try {
db_bylab.close(0);
db_bynam.close(0);
myEnv.close(0);
} catch(DbException &e) {
cerr << e.what();
return -1;
} catch(std::exception &e) {
cerr << e.what();
return -1;
return 0;
// 返回Dbt::put()的返回值
int store_entry_data(const EntryData& e, u_int32_t flags)
int res = 0;
try {
EntryData::DataBlock blk(e);
// data()返回的buf中存放的第一个字符串便是 e.get_name().
Dbt key ( (void*)blk.data(), strlen(blk.data()) + 1 );
Dbt data( (void*)blk.data(), blk.size() );
res = db_bynam.put(NULL, &key, &data, flags);
} catch (DbException& e) {
cerr << e.what() << endl;
throw; // 重新抛出.
return res;
// 返回 Db::get()的返回值, 调用成功时 EntryData* e的值才有意义
int get_entry_data
(const std::string& entry_name, EntryData* e, u_int32_t flags)
Dbt key( (void*)entry_name.c_str(), entry_name.size() + 1 );
Dbt data;
data.set_flags(DB_DBT_MALLOC);
int res = db_bynam.get(NULL, &key, &data, flags);
if(res == 0)
new (e) EntryData( data.get_data(), data.get_size() );
free( data.get_data() );
return res;
int rm_entry_data(const std::string& name)
Dbt key( (void*)name.c_str(), name.size() + 1 );
cout << "to remove: \'" << name << "\'" << endl;
int res = db_bynam.del(NULL, &key, 0);
return res;
EntryData::EntryData(const std::string& s, const char* labels_arr[]) : name(s)
{   // labels_arr 需要以 NULL 结尾.
for(const char** i = labels_arr; *i != NULL; i++)
labels.push_back(*i);
EntryData::EntryData(const void* mem_blk, const int len)
const char* buf = (const char*)mem_blk;
int consumed = 0; // 已经消耗的mem_blk的大小.
name = buf; // 第一串为 name
consumed += name.size() + 1;
for (string label = buf + consumed;
consumed < len;
consumed += label.size() + 1)
label = buf + consumed;
labels.push_back(label);
void EntryData::add_label(const string& new_label)
if(find(labels.begin(), labels.end(), new_label)
== labels.end())
labels.push_back(new_label);
void EntryData::rem_label(const string& to_rem)
LabelContainerType::iterator iter = find(labels.begin(), labels.end(), to_rem);
if(iter != labels.end())
labels.erase(iter);
void EntryData::show() const {
cout << "name: " << name << "; labels: ";
LabelContainerType::const_iterator it, end = labels.end();
for(it = labels.begin(); it != end; ++it)
cout << *it << " ";
cout << endl;
EntryData::DataBlock::DataBlock(const EntryData& data) :
data_size(0),
capacity(init_capacity),
buf_ptr(new SmartPtr(new char[init_capacity]))
pack_str(data.name);
for(EntryData::LabelContainerType::const_iterator \
i = data.labels.begin();
i != data.labels.end();
++i) { pack_str(*i); }
void EntryData::DataBlock::pack_str(const std::string& s)
int string_size = s.size() + 1; // to put sting in buf separately.
if(capacity >= data_size + string_size) {
memcpy(buf_ptr->buf + data_size, s.c_str(), string_size);
else {
capacity = (data_size + string_size)*2; // 分配尽可能大的空间.
buf_ptr->buf = (char*)realloc(buf_ptr->buf, capacity);
memcpy(buf_ptr->buf + data_size, s.c_str(), string_size);
data_size += string_size;
//////////// test_put.cc ///////////
#include <iostream>
#include <string>
#include <cstdlib>
#include "db_access.h"
using namespace std;
int main(int argc, char** argv)
if(argc < 2) { exit(EXIT_FAILURE); }
DBSetup setupup_mydb;
int res = 0;
EntryData ed(argv[1], (const char**)argv + 2);
res = store_entry_data(ed);
if(res != 0) {
     cerr << db_strerror(res) << endl;
     return res;
return 0;
// To Compile:
// $ g++ -ldb_cxx -lboost_regex -o test_put test_put.cc db_access.cc directory.cc
//////////// test_update.cc ///////////
#include <iostream>
#include <cstdlib>
#include <string>
#include <boost/program_options.hpp>
#include "db_access.h"
using namespace std;
namespace po = boost::program_options;
int main(int argc, char** argv)
if(argc < 2) { exit(EXIT_SUCCESS); }
DBSetup setupup_mydb;
int res = 0;
po::options_description cmdopts("Allowed options");
po::positional_options_description pos_opts;
cmdopts.add_options()
("entry", "Specify the entry that will be edited")
("addlabel,a", po::value< vector<string> >(),
"add a label for specified entry")
("removelabel,r", po::value< vector<string> >(),
"remove the label of specified entry")
pos_opts.add("entry", 1);
po::variables_map vm;
store( po::command_line_parser(argc, argv).
options(cmdopts).positional(pos_opts).run(), vm );
notify(vm);
EntryData entry_data;
if(vm.count("entry")) {
const string& entry_to_edit = vm["entry"].as<string>();
res = get_entry_data( entry_to_edit, &entry_data );
switch (res)
case 0:
break;
case DB_NOTFOUND:
cerr << "No entry named: \'"
<< entry_to_edit << "\'\n";
return res;
break;
default:
cerr << db_strerror(res) << endl;
return res;
} else {
cerr << "No entry specified\n";
exit(EXIT_FAILURE);
EntryData new_entry_data(entry_data);
typedef vector<string>::const_iterator VS_CI;
if(vm.count("addlabel")) {
const vector<string>& to_adds = vm["addlabel"].as< vector<string> >();
VS_CI end = to_adds.end();
for(VS_CI i = to_adds.begin(); i != end; ++i) {
new_entry_data.add_label(*i);
if(vm.count("removelabel")) {
const vector<string>& to_rems = vm["removelabel"].as< vector<string> >();
VS_CI end = to_rems.end();
for(VS_CI i = to_rems.begin(); i != end; ++i) {
new_entry_data.rem_label(*i);
cout << "Old data| ";
entry_data.show();
cout << "New data| ";
new_entry_data.show();
res = store_entry_data(new_entry_data, 0); // set flags to zero permitting Over Write
if(res != 0) {
cerr << db_strerror(res) << endl;
return res;
return 0;
// To Compile:
// $ g++ -ldb_cxx -lboost_regex -lboost_program_options -o test_update test_update.cc db_access.cc directory.cc

Answer 1

////////directory.h//////
#ifndef __DIRECTORY_H__
#define __DIRECTORY_H__
#include <string>
#include <string>
#include <sys/types.h>
using std::string;
class Path
public:
Path() {}
Path(const std::string&);
Path(const char* raw) { new (this) Path(string(raw)); }
Path upper() const;
void operator+= (const std::string&);
// convert to string (char*):
//operator std::string() const {return spath;}
operator const char*() const {return spath.c_str();}
const std::string& str() const {return spath;}
private:
std::string spath; // the real path
inline Path operator+(const Path& L, const string& R)
Path p(L);
p += R;
return p;
int rmkdir(const string& path, const mode_t mode = 0744, const int depth = -1);
#endif
///////directory.cc///////
#ifndef __DIRECTORY_H__
#define __DIRECTORY_H__
#include <string>
#include <string>
#include <sys/types.h>
using std::string;
class Path
public:
Path() {}
Path(const std::string&);
Path(const char* raw) { new (this) Path(string(raw)); }
Path upper() const;
void operator+= (const std::string&);
// convert to string (char*):
//operator std::string() const {return spath;}
operator const char*() const {return spath.c_str();}
const std::string& str() const {return spath;}
private:
std::string spath; // the real path
inline Path operator+(const Path& L, const string& R)
Path p(L);
p += R;
return p;
int rmkdir(const string& path, const mode_t mode = 0744, const int depth = -1);
#endif
//////////////////// All the code is above ////////////////////////////////
Use the under command
$ g++ -ldb_cxx -lboost_regex -o test_put test_put.cc db_access.cc directory.cc
to get a test program that can insert a record to database.
To insert a record, use the under command:
$ ./test_put ubuntu linux os
It will store an EntryData named 'ubuntu' and two labels('linux', 'os') to database.
Use the under command
$ g++ -ldb_cxx -lboost_regex -lboost_program_options -o test_update test_update.cc db_access.cc directory.cc
to get a test program that can update the existing records.
To update the record, use the under command:
$ ./test_update ubuntu -r linux -a canonical
It will update the with the key 'ubuntu', with the label 'linux' removed and a new
label 'canonical'.
Great thanks to you if you've read and understood my code!
I've said that the DB_SECONDARY_BAD exception is random. The same operation may cause
exception in one time and may goes well in another time.
As I've test below:
## Lines not started with '$' is the stdout or stderr.
$ ./test_put linux os linus
$ ./test_update linux -r os
Old data| name: linux; labels: os linus
New data| name: linux; labels: linus
$ ./test_update linux -r linus
Old data| name: linux; labels: linus
New data| name: linux; labels:
dbfile_bynam: DB_SECONDARY_BAD: Secondary index inconsistent with primary
Db::put: DB_SECONDARY_BAD: Secondary index inconsistent with primary
terminate called after throwing an instance of 'DbException'
what(): Db::put: DB_SECONDARY_BAD: Secondary index inconsistent with primary
已放弃
Look! I've received a DB_SECONDARY_BAD exception. But thus exception does not always
happen even under the same operation.
For the exception is random, you may have not the "luck" to get it during your test.
So let's insert a record by:
$ ./test_put t
and then give it a great number of labels:
$ for((i = 0; i != 100; ++i)); do ./test_update t -a "label_$i"; done
and then:
$ for((i = 0; i != 100; ++i)); do ./test_update t -r "label_$i"; done
Thus, the DB_SECONDARY_BAD exception is almost certain to happen.
I've been confused by the problem for times. I would appreciate if someone can solve
my problem.
Many thanks!
Wonder