GNU compiled app crashes with signal 11: Segmentation Fault

Question

Hi Community,
I know this is not a GNU oriented forum, but maybe this is a common issue.
We have a C++ multi-threaded application running properly in Solaris 9 Sparc. Due to some issues mostly related to the NICs we needed to have the application running in Solaris 10 x86. The problem is that the compilation is ok (always with GNU) but during run time, we have the application crashing in different instructions because of a signal 11.
It always crashes in a malloc call within the libstdc++.so.6
It was compiled with the option -mt and -lthread. Is this a known problem? Do you recommend any direction to start seeking for a solution?
Pls, find attached the gdb outcomes after the crash.
Thanks in advance, Pablo
quiterio{root}# gdb cord /usr/nguser/core
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-pc-solaris2.10"...
Reading symbols from /lib/libsocket.so.1...done.
Loaded symbols for /lib/libsocket.so.1
Reading symbols from /export/home/mysql/mysql-5.0.51/lib/mysql/libmysqlclient.so.15...done.
Loaded symbols for /opt/mysql/mysql/lib/mysql/libmysqlclient.so.15
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libthread.so.1...
warning: Lowest section in /lib/libthread.so.1 is .dynamic at 00000074
done.
Loaded symbols for /lib/libthread.so.1
Reading symbols from /usr/local/lib/libmysqlpp.so.2...done.
Loaded symbols for /usr/local/lib/libmysqlpp.so.2
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/local/lib/libstdc++.so.6...done.
Loaded symbols for /usr/local/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.2...done.
Loaded symbols for /lib/libm.so.2
Reading symbols from /usr/local/lib/libgcc_s.so.1...done.
Loaded symbols for /usr/local/lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.1...done.
Loaded symbols for /lib/libc.so.1
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libaio.so.1...done.
Loaded symbols for /lib/libaio.so.1
Reading symbols from /lib/libmd.so.1...done.
Loaded symbols for /lib/libmd.so.1
Reading symbols from /export/home/mysql/mysql-5.0.51/lib/mysql/libmysqlclient_r.so.15...done.
Loaded symbols for /opt/mysql/mysql/lib/mysql/libmysqlclient_r.so.15
Reading symbols from /lib/libpthread.so.1...
warning: Lowest section in /lib/libpthread.so.1 is .dynamic at 00000074
done.
Loaded symbols for /lib/libpthread.so.1
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
Core was generated by `/export/home/egasco/cord/cord'.
Program terminated with signal 11, Segmentation fault.
#0  0xfebd4ad2 in t_splay () from /lib/libc.so.1
(gdb) bt
#0  0xfebd4ad2 in t_splay () from /lib/libc.so.1
#1  0xfebd49b0 in t_delete () from /lib/libc.so.1
#2  0xfebd46ea in realfree () from /lib/libc.so.1
#3  0xfebd42ee in _malloc_unlocked () from /lib/libc.so.1
#4  0xfebd4138 in malloc () from /lib/libc.so.1
#5  0x080bdf64 in PMData::addData (this=0x818ae70, header=
          {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<const std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<const std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<std::basic_string<char, std::char_traits<char>, std::allocator<char> >,std::basic_string<char, std::char_traits<char>, std::allocator<char> >,bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x81ade48, _M_left = 0x81c2310, _M_right = 0x81ae900}, _M_node_count = 8}}},
    data=0x81c1a20 "v=0\r\no=50 2890844526 2890842807 IN IP4 192.168.1.245\r\ns=SDP seminar\r\nc=IN IP4 192.168.1.245\r\nt= 0 0\r\nm=audio 9092 RTP/AVP 8 18\r\n", dest=ONEP_SIP) at PMData.cpp:23
#6  0x080b3a52 in UserCall::process_initialization (this=0x81c2630, packet=0x81ad128) at UserCall.cpp:505
#7  0x080b056f in UserCall::process (this=0x81c2630, packet=0x81ad128) at UserCall.cpp:132
#8  0x080791d7 in ONEPPacketManager::processPacket () at ActionNotAllowException.h:15
#9  0x080aa159 in onep_processing_thread (arg=0x8127e88) at CordApplication.cpp:1317
#10 0xfec34672 in _thr_setup () from /lib/libc.so.1
#11 0xfec34960 in L3_doit () from /lib/libc.so.1
#12 0xfe7e2400 in ?? ()
#13 0x00000000 in ?? ()

Answer 1

A crash in malloc in multi-threaded code could be a bug in the system malloc or a failure to use a thread-safe malloc. The default Solaris malloc in /usr/lib/libc.so.* is thread-safe.
A crash in malloc can also be caused by a heap corruption. Probably the most common causes of heap corruption are
- writing beyond the bounds of a buffer or variable (off by one, for example)
- deleting the same object more than once
- using an invalid pointer:
--- uninitialized
--- pointing to a deleted object
--- pointing to an out-of-scope object
- failure to guard a critical region
- failing to declare shared objects as volatile