Application stuck in call of slow_lock

Question

During running of our application under Solaris 9 using SunStudio 8,
the application get stuck, and the process stack shows the following thread stack:
----------------- lwp# 6 / thread# 6 --------------------
fcd95994 lwp_park (0, 0, 0)
fcd9166c mutex_lock_queue (fcda8b44, 0, fcfc0590, fcda8000, 0, 1) + 104
fcd9206c slow_lock (fcfc0590, fc760a00, 2bf3e8, fcfbc000, 2bfa20, 7ea48ec) + 58
fcf46cf8 malloc (1, c8474c, 546872, 7efefeff, 81010100, ff0000) + 18
fd398f98 __1cQCORBA_string_dup6Fpkc_pc_ (1, 0, 21ba0, fcfbc000, 0, c84748) + 28
fd399158 __1cWCORBA_it_string_dup_eh6Fpkc_pc_ (c84748, fc760a00, 4a4974, fd399158, 365878, fd83dab8) + 2c
002a1268 __1cXCORBA_IT_Cxx_String_mgr2t5B6M_v_ (53cc35c, 0, 21ba0, fcfbc000, 40f8f48, 4f775e0) + 10
0029e480 __1cKglobaldefsUNameAndStringValue_T2t6M_v_ (53cc358, d53e40, 4a4974, fcd92634, 48, fd83dab8) + 20
Several threads were in the same state and the application seems to be in a deadlock.
Is any body knows the meaning of the system call slow_lock ?
Is it a known behavior ?
Thanks for any tips.
Yaakov Berkovitch
[email protected]

Answer 1

What you see here is the result of calls to the memory allocator being serialized through a single lock. This is described in the following SunSolve document:
SRDB 71583 - Threaded applications making heavy use of the memory allocator may scale poorly
However, this will not cause a deadlock, rather just poor scalability. The thread stack you supplied, illustrates a thread blocked in a call to malloc() waiting to aquire this lock. You should find another thread in malloc, free, memalign, calloc, realloc, or valloc which is not blocked on this lock. It will be holding the lock, and blocking the other threads...perhaps.
Would you be able to supply a full threadlist, such as an output from pstack?
Anyway, if you want to use a more scalable memory allocator use libumem as follows:
$ LD_PRELOAD=libumem.so ./my_program
or if you are running an older version of Solaris you won't have libumem. In this case use mtmalloc, which for your purposes will achieve the same goal.....a proper multi-thread scalably memory allocator:
LD_PRELOAD=libmtmalloc.so ./my_program
See manpages for mtmalloc(3MALLOC) and libumem(3LIB) for more information.