Compiling own app with -X02 makes external dll core dump.

Question

Hello,
I have a C++ app that calls a function w/in an external shared library. The external dll does not have debug symbols and is probably optimized. When my app is compiled with -xO2 or higher, I get a core dump (no mapping at the fault address) deep within the external dll. When I am compiling my app with -xO1 or with debug info, I do not get the core dump within the external dll.
My questions are:
1. Does this mean that there is a bug w/in my code or within the external dll's code?
2. If not error in either, is there any way to overcome this core dump without lowering the level of optimization (and thus slowing down the app)
Here is a stack trace I got by attaching dbx to my app:
t@1 (l@1) signal SEGV (no mapping at the fault address) in num2qword at 0xfd320248
[1] num2qword(0x345cfd8, 0xffffff85, 0x63ad0, 0xfd323ba0, 0xa800, 0xa8d8), at 0xfd320248
[2] evalexpr(0xffbf7f58, 0x345cee8, 0x1, 0x0, 0xfd27e558, 0xfd3230d0), at 0xfd323ba0
[3] eval_expr_full(0x328e388, 0x345d04c, 0x345cfd8, 0xfd27e558, 0x1, 0x0), at 0xfd3249e8
[4] cm_eval_expr(0x328e388, 0x345d04c, 0x5800, 0x5944, 0xb400, 0xb400), at 0xfd1eea68
[5] cm_optredeem_can_redeem(0x328e388, 0x345d1e0, 0x16ddec, 0x5c00, 0x0, 0x345d1e4), at 0xfd21620c
[6] cm_optredeem_condition(0x328e388, 0x345d1e0, 0x0, 0x0, 0x0, 0x135eb29), at 0xfd2162d8
[7] cm_maybe_execute_optredeem(0x328e388, 0x345d1e0, 0x0, 0x0, 0x0, 0x1), at 0xfd216568
[8] cm_pay_1_period(0x328e388, 0xfd3ba364, 0x345d1e0, 0x345d52c, 0xfd383c98, 0x0), at 0xfd21ab5c
[9] cm_payloop(0x328e388, 0x1, 0x0, 0x324df00, 0x0, 0x3), at 0xfd21ae08
[10] icmo_cashflows(0x0, 0x0, 0x168, 0x328b7b0, 0x0, 0x328b7b0), at 0xfd2d1700
[11] CMOInstrument::getICMOCashFlow(0x1627b28, 0xfd48bdb8, 0x570, 0x2b1c, 0x2800, 0x400), at 0xfd442410
[12] NE_CMOAnalytics::value(0x39b71e0, 0x98, 0xfd48bdb8, 0xe, 0xffbfc268, 0x2e93b08), at 0xfd42d560
[13] NumericExpression::value(0x1643800, 0x1667b48, 0x1667b48, 0x2eabde0, 0x2eabde0, 0x0), at 0xc13410
[14] NE_CMOPriceTimesBalance::value(0x2b02078, 0x2ead230, 0x1667b48, 0x2eabea0, 0x2eac06c, 0xfd48ec8c), at 0xfd42da9c
[15] NumericExpression::value(0x1643800, 0x1667b48, 0x1667b48, 0x2eabea0, 0x2eabea0, 0x0), at 0xc13410
[16] ExchangeableModuleBase::evaluateExpression(0x31259c4, 0x15e4400, 0x0, 0x2eabea0, 0x1667b48, 0x2e93b08), at 0x746e54
[17] ExchangeableModuleBase::evaluateExpressions(0x31259c4, 0x2f3d7c8, 0x746ca0, 0x1, 0x164c800, 0x15c8414), at 0x746c68
[18] ExchangeableModuleBase::recalc(0x31259c4, 0x0, 0xffffffc8, 0x164c800, 0x0, 0x746c40), at 0x746c10
[19] ExchangeableModule::recalc(0x31258d4, 0x15e5730, 0x15c81ac, 0x164c800, 0x7fe8c0, 0x0), at 0x4a8ff4
[20] BatchInstrument::commandDispatch(0x15fd850, 0x1ac, 0x1, 0x619b2c, 0xc, 0x2ebbbc0), at 0x619ce0
[21] Callback(0x2eee48c, 0x3, 0x5, 0x1, 0xffbfccf0, 0xffbfcd6c), at 0xe652f0
[22] CallExtRtn(0xffbfe1d0, 0x2, 0x1, 0xffffdffe, 0x1, 0xfcd0fc86), at 0xe727d4
[23] PcodeExecute(0x2eee8ac, 0xef5c18, 0x1, 0x15b9750, 0xef5c18, 0x4), at 0xeab848
[24] ExpStartThread(0x2eee48c, 0x0, 0x0, 0x1, 0x1, 0x0), at 0xe6e1e0
[25] BasicScriptCodeObject::executeThread(0x2ebb070, 0x2eee48c, 0x0, 0x2ebb074, 0x2eeed68, 0x0), at 0xa0fcb0
[26] BasicScriptCodeObject::run(0x2ebb070, 0x0, 0x0, 0x0, 0x0, 0x2ebb070), at 0xa0fe24
[27] BasicScriptManager::loadAndExecInitScriptFile(0x2d7eb08, 0x1, 0xa, 0x0, 0x1, 0x2eef5f0), at 0x587f80
[28] main(0x32db9b8, 0xffff3400, 0x308b998, 0x0, 0x1c1fd20, 0x1c0db58), at 0x438c6c
[10] icmo_cashflows(...) is the call to the external dll. From 9 and up to 1 the source code is not visible to me..
OS/CC Versions:
CC: Sun C++ 5.5 Patch 113817-05 2004/01/13
SunOS: 5.9 Generic_117171-05 sun4u sparc SUNW,Sun-Fire-880
compilation options:
-dalign -xarch=v8plusa -xlibmopt -D__solaris_sparc__ -errtags=yes -errwarn=wnoretvalue,wnoelseretvalue +w2 -erroff=doubunder,reftotemp,notemsource,hidef,hidevf,wbadlinkage_w,noexthrow,noextry,notused -DLIC_MGR -DUSE_SMART_HEAP -DUSE_SH_ALLOC -template=wholeclass -instances=explicit -features=no%except -features=rtti -Drindex=rindex -Dindex=index   -DADD_DYNAMIC_SYMBOLS     -KPIC   -DNDEBUG   -mt   -xO1 ( -xO2 .. -xO5 or -fast give above noted core dump)
Thanks in advanice,
Alex

Answer 1

From the symptoms you describe, it is possible that the shared library contains a bug, but it is more likely that the problem is in your code, passing bad data to the shared library.
The problem might be a compiler bug, but I think an error in your code is more likely.
For example, your code as written might have undefined behavior that works by accident when compiling without optimizing, but fails when the compiler improves the runtime code.
Some examples:
1. A classic MT programming error is failing to declare a variable volatile when it is shared by more than one thread. In each thread, the compiler assumes it can remember the value of the variable and not reload it between references.
2. "x = ++x + b;" Modifying a variable twice in an expression withtout sequence points has undefined behavior -- compiler optimization can affect the result.
3." foo(x, x+=3);" the order of evaluation of function arguments is unspecified, so the actual values passed to foo could depend on optimization.
With C++ 5.5, you get some warnings from system headers that you can't do anything about, which I suspect is why you have disabled some warnings. Unfortunately, you might be disabling warnings that you need to see. That is, some of the disabled warnings could point to invalid code that results in the program crash.
I suggest using w instead of w2 (to avoid uninteresting warnings), and fix your code to eliminate the warnings you get.
In the current compiler, C++ 5.7 in Sun Studio 10, we have made some adjustments to warnings to make w and w2 more useful. You should consider upgrading.You can get a 60-day free trial version here:
http://www.sun.com/software/products/studio/index.xml
Also get the current patches for it here:
http://developers.sun.com/prodtech/cc/downloads/patches/index.html