How reliable is stack trace?

Question

Hi there,
I am debugging a core on solaris 10. the shared libraries are compiled and linked without -g0 and majority of the symbols are removed using a linker map. Now when I loaded dbx and do a "where", the stack trace only shows
[1] 0xfcb01510(0x0, 0xfebe09d8, 0xfe30b500, 0x433a70, 0x23c428, 0x1), at 0xfcb01510
[2] 0xfcb014a8(0x3937450, 0x0, 0x0, 0x18e7, 0xfccbc4a8, 0x185004a4), at 0xfcb014a8
[3] 0xfdd7b1a4(0x433a70, 0x3ec720, 0xfe30b500, 0x433a70, 0xfe31f1e8, 0xfdd7adc4), at 0xfdd7b1a4
[4] 0xfdd66890(0x433a70, 0x3ec720, 0x8, 0xff1436f8, 0xfc06edb4, 0x23c428), at 0xfdd66890
[5] 0xfc1375fc(0x3ec708, 0x3, 0x3ec720, 0xfc3fddd8, 0x837568, 0x2c0468), at 0xfc1375fc
[6] 0xfc12f3c0(0x3ec708, 0x3ec860, 0x3ec860, 0x5df89a8, 0x0, 0xa770408), at 0xfc12f3c0
[7] 0xfc1d9cfc(0x3ec714, 0x3ec860, 0x8c, 0xb8, 0xfc40002c, 0xfc174638), at 0xfc1d9cfc
[8] 0xfc1d8680(0x0, 0x17, 0x3ec860, 0x2c0558, 0x5, 0x3ec714), at 0xfc1d8680
[9] 0xfc1d8998(0x3ec8a8, 0x17, 0x3ec860, 0x2c0558, 0x3, 0x5), at 0xfc1d8998
[10] 0xfc1339c4(0x3ec708, 0x17, 0x3ec860, 0xff141b0c, 0x5, 0x2c0020), at 0xfc1339c4
[11] 0xfc1341bc(0x3ec708, 0x17, 0x3ec860, 0xa770408, 0xffffff, 0xfc3fd610), at 0xfc1341bc
[12] 0xfc133878(0x3ec708, 0x12, 0x3ec860, 0xfc3f5c60, 0x0, 0xfea56158), at 0xfc133878
[13] 0xfc134b08(0xfa87bd9c, 0xfffad938, 0x0, 0x3ec708, 0xfc40017c, 0xfc3f5c60), at 0xfc134b08
[14] 0xfea7b168(0x23c428, 0x2692c0, 0x23c4c8, 0x3ec998, 0xfeb9bc61, 0xfea56158), at 0xfea7b168
[15] 0xfea80c30(0x2692c0, 0x2692c0, 0xfefecbc0, 0xfd6e2800, 0x4eaa0, 0x0), at 0xfea80c30
[16] os_thread::runThreadFunction(0x21d550, 0xfa87c000, 0x0, 0x0, 0xff141b0c, 0x1), at 0xff11309c
I am not surprised because .so does not have too many symbols to begin with. I used the manual technique to map each frame to a shared library:
1. take address of the function in each frame,
2. compare against base address of a shared library from output of "proc -map" (it shows where each so is loaded into the process).
3. see 1. falls into the range of which library displayed in 2.
4. substract base address from 1. (used for later)
After that, I went ahead to locate function, what I did was pull the exact source code and compile them without applying the linker map (still without -g0). I hope this newly compiled so is the same as the production ones in terms of function offsets:
1. nm the_faulty_so_recompiled.so
2. sorted functions in nm output by its offset,
3. compare 4. in previous step against these offsets.
but I find the result is sometimes not reliable because there is no way some of the function is called by previous frame, it is just impossible if you look at the source code.
Can someone tell me if I am doing something terribly wrong here?
BTW, parameters for each functions show in the "where" stack frames are not the same as I expected, occasionally there is something like 0x1 or 0x17, what are those?
Thanks.

Answer 1

but the core is not reproducible, -g0 version is not particularly helpful in that case. What I am hoping is the following:
1. the release version is compiled without -g0 and with a linker map to eliminate 99% of symbols.
2. recompiled version is with exact same source code and without -g0 and without linker map.
same compiler and everything else. I would expect the function offset is the same in those 2 cases. correct? If not, what have I done wrong?
I am ok with just knowing which function is at fault so without -g0 is ok for me. But why I am seeing stack trace still does not make sense? What is wrong with my calculation of function offset?
Is there some kind of remapping the loaded address of library going on here so I have to manually calculate some offset to get the real function offset?
Thanks.