Core dump(abort) due to double free at STL string == operator,with mtmalloc

Question

We have a probrem; our program crashes due to SIGABRT.
Our program is multithreaded (about 40 thread total),and uses libmtmalloc for more performance.
By stack trace , when handling of '==' operator of STL string ,
free() and abort() is called like this:
(dbx) lwp l@25
Current function is WorkSessionCollection::findBySessionId
207 if (session->getSessionId() == sessionId)
t@null (l@25) stopped in (unknown) at 0xfedbd7fc
0xfedbd7fc: PROCEDURELINKAGE_TABLE_+0x0820 [PLT]: bcc,a,pt %icc,_PROCEDURE_LINKAGE_TABLE_+0x830 [PLT] ! 0xfedbd80c
(dbx) where
[1] 0xfedbd7fc(0x6, 0x0, 0xfeda1d04, 0x42568, 0xfede4280, 0x6), at 0xfedbd7fb
[2] getopt(0x0, 0x1, 0xff39403c, 0xa652c, 0xfede7298, 0x0), at 0xfed3de58
[3] free(0x3bc6e00, 0x3bc7020, 0x0, 0x10, 0x0, 0x10000000), at 0xff380b00
[4] libC_errors::get_msg(0x3bc7020, 0xfffb, 0x0, 0xffffffff, 0xa, 0xfc00), at 0xfeef62e0
[5] std::basic_string<char,std::char_traits<char>,std::allocator<char> >::~basic_string(0x294, 0x0, 0x3bc7020, 0x3bc7048, 0x3bc7048, 0xf82fa638), at 0xfdb4cccc
=>[6] WorkSessionCollection::findBySessionId(this = 0xbe4750, msgId = CLASS), line 207 in "WorkSessionCollection.cpp"
[7] BusinessProxy::process(this = 0xc18ff8, message = CLASS), line 159 in "BusinessProxy.cpp"
[8] TcpProtocolControl::handleMessage(this = 0x3eaff28, message = 0x3f16270), line 221 in "ProtocolControl.cpp"
[9] SocketHandler::handle_input(this = 0x3871860, _ARG2 = 200), line 229 in "SocketHandler.cpp"
[10] 0xff2e6c90(0x2aa44b8, 0xf82fbc28, 0x0, 0x0, 0x1, 0x0), at 0xff2e6c8f
[11] 0xff2e6640(0x2aa44b8, 0xf82fbcb0, 0xf82fbd2c, 0x0, 0x0, 0x0), at 0xff2e663f
[12] 0xff2e61ac(0x2aa44b8, 0x0, 0xf82fbd2c, 0x0, 0x0, 0x0), at 0xff2e61ab
[13] 0xff2e5e10(0x2aa44b8, 0x0, 0xfede8bc0, 0xfda95800, 0x1e99410, 0x0), at 0xff2e5e0f
[14] 0xff2af7c0(0x281fda0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xff2af7bf
[15] event_loop(arg = 0x281fda0), line 40 in "FrontEnd.cpp"
[16] 0xff2d4610(0x1ea82b8, 0x8f0798, 0x0, 0x0, 0x0, 0x0), at 0xff2d460f
[17] 0xff2d44ec(0x1ea82b8, 0x0, 0x0, 0x0, 0x0, 0x1), at 0xff2d44eb
[18] 0xff21c590(0x1ea82b8, 0xf82fc000, 0x0, 0x0, 0x7e3f9c, 0x1), at 0xff21c58f
(dbx) list
207 if (session->getSessionId() == sessionId)
208 break; // for loop
209 session = NULL;
210 }
211
(dbx) print sessionId
dbx: cannot access address 0xfdf68f10
(dbx) print *session
dbx: cannot access address 0xfdf68f10
and here is the compiler environment:
versionMachine hardware: sun4u
OS version: 5.9
Processor type: sparc
Hardware: SUNW,Sun-Blade-1500
The following components are installed on your system:
Sun Studio 11
Sun Studio 11 C Compiler
Sun Studio 11 C++ Compiler
Sun Studio 11 Tools.h++ 7.1
Sun Studio 11 C++ Standard 64-bit Class Library
Sun Studio 11 Garbage Collector
Sun Studio 11 Fortran 95
Sun Studio 11 Debugging Tools (including dbx)
Sun Studio 11 IDE
Sun Studio 11 Debugger GUI
Sun Studio 11 Performance Analyzer (including collect, ...)
Sun Studio 11 X-Designer
Sun Studio 11 VIM editor
Sun Studio 11 XEmacs editor
Sun Studio 11 Native Connector Tool
Sun Studio 11 LockLint
Sun Studio 11 Building Software (including dmake)
Sun Studio 11 Documentation Set
version of "/apps/studio_11/SUNWspro/bin/./../prod/bin/../../bin/cc": Sun C 5.8 Patch 121015-04 2007/01/10
version of "/apps/studio_11/SUNWspro/bin/./../prod/bin/../../bin/CC": Sun C++ 5.8 Patch 121017-10 2007/02/21
version of "/apps/studio_11/SUNWspro/bin/./../prod/bin/../../bin/f90": Sun Fortran 95 8.2 2005/10/13
version of "/apps/studio_11/SUNWspro/bin/./../prod/bin/../../bin/dbx": Sun Dbx Debugger 7.5 2005/10/13
version of "/apps/studio_11/SUNWspro/bin/./../prod/bin/../../bin/analyzer": Sun Performance Analyzer 7.5 2005/10/13
version of "/apps/studio_11/SUNWspro/bin/./../prod/bin/../../bin/dmake": Sun Distributed Make 7.7 2005/10/13
could you please help us?
Thanks in advance.

Answer 1

This kind of crash can be due to referencing a string object after its lifetime has ended, or due to heap corruption.
The most common sources of heap corruption are
- using an invalid pointer
- uninitialized
- dangling (points to object that no longer exists)
- incremented or decremented outside object bounds
- points other than to an object start address
- deleting an object more than once
- reading or writing outside object bounds
- failing to guard a shared object in multithreaded code
- race condition in multithreaded code
The program failure usually occurs far away in space and time from the actual error condition.
You might also have run into a bug in the compiler or in system libraries. (You do not have the current Studio 11 patches.)
First, try running under the dbx with Run-Time Checking enabled:
% dbx myprogram
(dbx) check -all
(dbx) run
RTC will report many of the error conditions listed above.
Sun Studio 12 has a thread analyzer, which can help find race conditions in multithreaded code. You could download Studio 12 and try it out. If you do not need to run on Solaris 8, you can upgrade to Studio 12.
Whether you stay with Studio 11 or upgrade to Studio 12, you can get all the current patches here:
http://developers.sun.com/sunstudio/downloads/patches/
At minumum, get the current C++ compiler patch, the Common compiler patch, and the C++ Runtime LIbrary patch.
By following the links to the patches, you can see a list of bugs fixed since the patch levels you currently have.