Segmentation Fault in db- put() fun with BDB ver 4.5.20/4.6.19 in linux

Question

I am working on porting an application that runs on sun solaris sparc (OS ver 5.8) with Berkeley DB ver 4.2.50 into suse linux [ Linux version 2.6.5-7.244-smp (gcc version 3.3.3 (SuSE Linux)) ]. I have compiled the application in linux and i am getting segmentation fault error while running the application. The error occurs while the objects are loaded from the database into berkeley cache. The application fails in db->put function while loading the objects. This segmentation fault error is not consistently occuring every time on the same object. Sometimes it throws the error while loading the first object itself or sometimes it throws after loading couple of objects. But the stack trace shows the same function whenever it crashes.
Given below is the stack trace of the application when it throws SIGSEGV.
[INFO] [-1756589376] 14:23:24.406 BerkeleyCache : restoring database table metadata for table [BSCOffice]...
[INFO] [-1756589376] 14:23:24.406 BerkeleyCache : restoring database index metadata for table [BSCOffice]...
[INFO] [-1756589376] 14:23:24.408 Obj [BSCOffice] Col [BSCOfficeId] : Typ[TEXT] MaxLen[10] Null[Y]
[INFO] [-1756589376] 14:23:24.408 Obj [BSCOffice] Col [City] : Typ[TEXT] MaxLen[100] Null[Y]
[INFO] [-1756589376] 14:23:24.408 BerkeleyCache : creating table [BSCOffice.tbl]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 182927004352 (LWP 14638)]
0x0000002a9643203a in __db_check_txn () from /opt/home/pesprm/local/db-4.6.19/db-4.6.19/build_unix/.libs/libdb-4.6.so
(gdb) where
#0 0x0000002a9643203a in __db_check_txn () from /opt/home/pesprm/local/db-4.6.19/db-4.6.19/build_unix/.libs/libdb-4.6.so
#1 0x0000002a9643475b in __db_put_pp () from /opt/home/pesprm/local/db-4.6.19/db-4.6.19/build_unix/.libs/libdb-4.6.so
#2 0x000000000043586b in esp::BerkeleyCache::insert_i (this=0x7fbfffe380, transactionId=866640768, pObj=0x7fbfffd3e0) at BerkeleyCache.cpp:838
#3 0x00000000004119c5 in RefServer::loadObject (this=0x7fbfffdbe0, objInfo=<value optimized out>, strPrimaryObjName=@0x7fbfffd6f0,
procParams=<value optimized out>) at BerkeleyCache.h:569
#4 0x0000000000419166 in RefServer::loadObjects (this=0x7fbfffdbe0) at RefServer.cpp:579
#5 0x0000000000419748 in main (argc=<value optimized out>, argv=<value optimized out>) at RefServer.cpp:296
The code that calls the db->put() function is given below:
<pre>
bool BerkeleyCache::insert_i(size_t transactionId, const CachePersistable* pObj)
     bdbcache::Table* pTable = findTable_i(pObj->getPersistInfo().getObjectName());
     if (pTable == NULL)
          return false;
     DB_TXN* txnp = (DB_TXN*)transactionId;
     if (txnp == NULL)
          return false;
     bdbcache::Index* pPrimaryKey = pTable->getPrimaryKey();
     if (pPrimaryKey == NULL)
          return false;
     if (pTable->getDataOffset() == -1)
          pTable->setDataOffset( (int)(pObj->getDataBufferStartPos() - pObj->getDataStartPos()) );
     DB* pdb = pTable->getDB();
     int rc = 0;
     Synchronize sync(pPrimaryKey->getKeyBufferCriticalSection());
     pPrimaryKey->getKeyValues(pObj, pPrimaryKey->getKeyBuffer());
     DBT key, data;
     memset(&key, 0, sizeof(DBT));
     key.flags = DB_DBT_USERMEM;
     key.data = (void*)pPrimaryKey->getKeyBuffer().getBuffer();
     key.ulen = key.size = pPrimaryKey->getKeyBuffer().getBufferLength();
     memset(&data, 0, sizeof(DBT));
     data.flags = DB_DBT_USERMEM;
     if (pTable->isPrimaryDb())
          data.data = (void*)pObj->getDataBufferStartPos();
          data.ulen = data.size = pObj->getDataBufferSize();
          rc = pdb->put(pdb, (DB_TXN*)transactionId, &key, &data, DB_NOOVERWRITE);
     else
          const PersistInfo::Property* prop = pTable->getJoinProperty();
          Variant var = pObj->getValue( prop );
          int n = 0;
          switch (prop->_eType)
               case PersistInfo::CHAR_ARRAY:
               case PersistInfo::STRING:
               case PersistInfo::VAR_STRING:
                    const char* pch = (const char*)var;
                    data.data = (void*)pch;
                    data.ulen = data.size = (int)strlen(pch) + 1;
                    break;
               case PersistInfo::INT:
                    n = (int)var;
                    data.data = (void*)&n;
                    data.ulen = data.size = sizeof(int);
                    break;
          rc = pdb->put(pdb, (DB_TXN*)transactionId, &key, &data, 0);
     if (rc!= DB_SUCCESS)
          //_app.logError("BerkeleyCache : DB->put() failed [%s]", db_strerror(rc));
          _app.logInfo("ERROR:BerkeleyCache : DB->put() failed [%s]", db_strerror(rc));
          return false;
     return true;
</pre>
At the end of this function, the return value of db->put is checked for DB_SUCCESS. The application never executed this line when it throws SIGSEGV.
I have tried using the BDB versions 4.5.20/4.6.18/4.6.19 with the application and it throws segmentation fault error with all versions of BDB. Following are the cache related configurable parameters that we use in our application.
cache size = 16 MB
page size = 16 KB
max locks = 3000000
dirty read = N
We are using the Sybase Adaptive DB server running in Solaris/Linux servers. The same application that runs in Solaris connecting to Sybase DB (in solaris) is working perfectly fine. And the application that i am working in linux connects to the Sybase DB running in linux server.
Please let me know what could be the issue that is causing our application to segmentation fault.
Thanks
Senthil

Answer 1

I am working on porting an application that runs on
sun solaris sparc (OS ver 5.8) with Berkeley DB ver
4.2.50 into suse linux [ Linux version
2.6.5-7.244-smp (gcc version 3.3.3 (SuSE Linux)) ].
I have compiled the application in linux and i am
getting segmentation fault error while running the
application. The error occurs while the objects areWhich indicates in almost every typical case, for a well used library like berkeleydb, an application issue.
loaded from the database into berkeley cache. The
application fails in db->put function while loading
the objects. This segmentation fault error is not
consistently occuring every time on the same object.Also indicative of typical misuse of heap, pointer errors, or otherwise undefined behavior on the application's part.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 182927004352 (LWP 14638)]
0x0000002a9643203a in __db_check_txn () from
/opt/home/pesprm/local/db-4.6.19/db-4.6.19/build_unix/
.libs/libdb-4.6.so
(gdb) where
#0 0x0000002a9643203a in __db_check_txn () from
/opt/home/pesprm/local/db-4.6.19/db-4.6.19/build_unix/
.libs/libdb-4.6.so
#1 0x0000002a9643475b in __db_put_pp () from
/opt/home/pesprm/local/db-4.6.19/db-4.6.19/build_unix/
.libs/libdb-4.6.so
#2 0x000000000043586b in
esp::BerkeleyCache::insert_i (this=0x7fbfffe380,
transactionId=866640768, pObj=0x7fbfffd3e0) at
BerkeleyCache.cpp:838
#3 0x00000000004119c5 in RefServer::loadObject
(this=0x7fbfffdbe0, objInfo=<value optimized out>,
strPrimaryObjName=@0x7fbfffd6f0,
procParams=<value optimized out>) at
BerkeleyCache.h:569
4 0x0000000000419166 in RefServer::loadObjects
(this=0x7fbfffdbe0) at RefServer.cpp:579
#5 0x0000000000419748 in main (argc=<value optimized
out>, argv=<value optimized out>) at
RefServer.cpp:296Did you build the libraries stripped or otherwise remove debugging information?
run "file /opt/home/pesprm/local/db-4.6.19/db-4.6.19/build_unix/.libs/libdb-4.6.so"
Also, if I'm not mistaken this is Linux you're porting to, right? Might want to drop the "/opt" Solaris-ism as you continue to port more to Linux.
The code that calls the db->put() function is given
below:
     if (pTable->getDataOffset() == -1)Does this return an unsigned or signed value?
pTable->setDataOffset(
t( (int)(pObj->getDataBufferStartPos() -
pObj->getDataStartPos()) );You cast to int here, what's the background reasoning for it?
     DB* pdb = pTable->getDB();
     int rc = 0;You don't check pdb is valid.
Synchronize
e sync(pPrimaryKey->getKeyBufferCriticalSection());
pPrimaryKey->getKeyValues(pObj,
, pPrimaryKey->getKeyBuffer());
     DBT key, data;
     memset(&key, 0, sizeof(DBT));
     key.flags = DB_DBT_USERMEM;
key.data =
= (void*)pPrimaryKey->getKeyBuffer().getBuffer();Do you verify this is even valid? Have you also checked the code within a debugger and verified the logic is correct?
key.ulen = key.size =
= pPrimaryKey->getKeyBuffer().getBufferLength();Same thing.
     memset(&data, 0, sizeof(DBT));
     data.flags = DB_DBT_USERMEM;
     if (pTable->isPrimaryDb())
          data.data = (void*)pObj->getDataBufferStartPos();
          data.ulen = data.size = pObj->getDataBufferSize();How do we know these are even valid? You're coming from the perspective that your code makes no mistakes when I have a hunch it does.
rc = pdb->put(pdb, (DB_TXN*)transactionId, &key,
y, &data, DB_NOOVERWRITE);Right, and this will most definitely result in a segmentation violation if you don't pass it valid data.
You're on Linux, I highly suggest you download and build Valgrind, run your application w/ valgrind --tool=memcheck, and get back to us. Also, don't strip debugging information from the berkeley db libraries during this phase of troubleshooting - especially when you're still porting something over.
I have tried using the BDB versions
4.5.20/4.6.18/4.6.19 with the application and it
throws segmentation fault error with all versions of
BDB.Because the issue isn't within BDB.
We are using the Sybase Adaptive DB server running in
Solaris/Linux servers. The same application that
runs in Solaris connecting to Sybase DB (in solaris)
is working perfectly fine.More correctly, it was working perfectly fine under Solaris. That doesn't even mean that it's not taking part in undefined behavior - it just means it "worked."