Solaris 10 on t1000 - write() to /dev/poll hangs on asm instrn

Question

hi,
i've got a streaming server which uses /dev/poll and once in a while it hangs. upon debugging with dbx i found that sometimes it gets stuck inside the write() call to remove the fd from polled fd structure at the assembly instruction "ta 8" which is for a syscall, correct? and, sometimes it gets stuck inside sosendmsg() at "ta 8" too.
this bug seems to reproduce intermittently only when the server is configured with 33 (the cpu has 32 coolthreads) worker pthreads each of which can add/remove fds from the poll device. each thread opens the /dev/poll seperately, so they have different fds to write to /dev/poll.
the problem happens after all of the threads have started up, so it does not appear to be an issue with thread spawning. if the server is configured for lesser threads this behavior doesn't happen. i haven't increased the stress level on the server to say for certain that that is the case, but under the same load it doesn't happen.
(1) the _write() call hang happens in the single thread that is responsible for handling all the incoming connections.
(2) the sosendmsg() hang happens in any one of the 32 threads which handle all out-going data (streaming).
hardware: t1000 with 8 cores (8G ram)
compiler:s SUN C/C++ 5.8 2005/10/13
application: helix server
questions:
(0) there is no need to lock the poll device even though mulltiple threads are writing to /dev/poll thru different fds, right?
(1) is this a known bug?
(2) how can i use dtrace (or any other tool) to probe beyond the syscall trap when the server hangs?
here is the stack of the last frame in dbx for _write:
0xff2412a4: _write       :      st       %o0, [%sp + 68]
0xff2412a8: _write+0x0004:      mov      4, %g1
=> 0xff2412ac: _write+0x0008:      ta       8
0xff2412b0: write+0x000c:      bcc,pt   %icc,write+0x20 ! 0xff2412c4
0xff2412b4: _write+0x0010:      cmp      %o0, 91
0xff2412b8: write+0x0014:      be,a,pn  %icc,write+0x4 ! 0xff2412a8
0xff2412bc: _write+0x0018:      ld       [%sp + 68], %o0
0xff2412c0: write+0x001c:      ba,a     cerror ! 0xff1a2380
0xff2412c4: _write+0x0020:      retl    
0xff2412c8: _write+0x0024:      nop     
for sosendmsg:
0xff240458: sosendmsg : st %o0, [%sp + 68]
0xff24045c: sosendmsg+0x0004: mov 241, %g1
0xff240460: sosendmsg+0x0008: ta 8
=> 0xff240464: sosendmsg+0x000c: bcc,pt %icc,_so_sendmsg+0x20 ! 0xff240478
0xff240468: sosendmsg+0x0010: cmp %o0, 91
0xff24046c: sosendmsg+0x0014: be,a,pn %icc,_so_sendmsg+0x4 ! 0xff24045c
0xff240470: sosendmsg+0x0018: ld [%sp + 68], %o0
0xff240474: sosendmsg+0x001c: ba,a _cerror        ! 0xff1a2380
0xff240478: sosendmsg+0x0020: retl
0xff24047c: sosendmsg+0x0024: nop
any help is appreciated.
sincerely,
-atin

Answer 1

hi,
To look and see what the applications threads are doing once the application hangs
as root run
# mdb -k
you should get a ">" prompt.
then you could run..
$<threadlist and look for your application process name amongst all the stack traces.
or ::ps -t and find your process, that will list a load of kernel threads for your app.
then you can do "threadpointer"::findstack for each of your threads eg
here is ::ps output for vold
R 342 1 342 342 0 0x4a014000 000006000987dbc0 vold
T 0x300019c4cc0 <TS_SLEEP>
T 0x300018ea380 <TS_SLEEP>
T 0x300019dd960 <TS_SLEEP>
T 0x30001152380 <TS_SLEEP>
it has the 4 sleeping threads. lets see where they are stopped.
0x300019c4cc0::findstackstack pointer for thread 300019c4cc0: 2a10116d061
[ 000002a10116d061 cv_wait_sig_swap_core+0x11c() ]
000002a10116d111 poll_common+0x4e8()
000002a10116d201 pollsys+0xd8()
000002a10116d2e1 syscall_trap32+0xcc()
and so on for all four threads.
that should give us an idea as to where in the devpoll code your thread is stopped
but.. lets look at devpoll.c especially at where could dpwrite() block
ah there is a comment saying that we wait for any other writers via dpwrite or any
readers in dpioctl(), it keeps a reference count per fd that is opened into /dev/poll
Is it possible that you have a thread in a longterm poll on the same fd as the blocking write, it looks like the write will block till the poll completes, pstack should give you the
fd for each write/ioctl as the first argument.
tim