NIO bug? selector doesn't return

Question

Hello,
I have a strange problem with my server. When a number of concurrent connections became more than 1000 the server hangs up.
I use NIO API to manage connections. The server logic is IRC-style, it broadcasts a large number of small messages. In one moment it may disconnect all the users and doesn't accept anymore incoming connections in spite of the java runtime continue working. I wrote a daemon that periodically makes thread dump to log file. When the server works normally the selector thread looks like this:
IOManager:
    sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
    sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
    sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
    sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
    sun.nio.ch.SelectorImpl.select(Unknown Source)
    ioserver.IOManager.run(IOManager.java:143)But when it doesn't respond appear two new "hepler" threads and the dump looks like:
IOManager:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Unknown Source)
    sun.nio.ch.WindowsSelectorImpl$FinishLock.waitForHelperThreads(Unknown Source)
    sun.nio.ch.WindowsSelectorImpl$FinishLock.access$600(Unknown Source)
    sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
    sun.nio.ch.SelectorImpl.select(Unknown Source)
    ioserver.IOManager.run(IOManager.java:143)
Thread-600:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Unknown Source)
    sun.nio.ch.WindowsSelectorImpl$StartLock.waitForStart(Unknown Source)
    sun.nio.ch.WindowsSelectorImpl$StartLock.access$2400(Unknown Source)
    sun.nio.ch.WindowsSelectorImpl$SelectThread.run(Unknown Source)
Thread-599:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Unknown Source)
    sun.nio.ch.WindowsSelectorImpl$StartLock.waitForStart(Unknown Source)
    sun.nio.ch.WindowsSelectorImpl$StartLock.access$2400(Unknown Source)
    sun.nio.ch.WindowsSelectorImpl$SelectThread.run(Unknown Source)The selector thread never returns from select operation. I don't know what is the cause of this behavior but I think that a problem is not in my java code. What can I do to solve the problem?
P.S.
The hardware isn't so powerful (2 CPUs Pentium III 500) thus it always has 50%-80% of load. Operating system is Windows 2003 Server for Small Business Server (without SPs). JRE: 1.5.0_09 Server VM. Socket receive buffer size: 1024, send buffer size: 4096.
Ant�n

Answer 1

Now having looked at the code and where the helper threads are and the main selector thread it seems that they are in a deadlock. From the code it seems very unlikely, since the timing has to be just right.
As an example,
Assumptions:
1) we do a select and because of the number of keys we select on, we need to create 3 helper threads. (so we have the main thread and 3 helpers = 4 total)
2) The system is quite busy and we are doing 100's of selects / minute and we also wakup the selector for adding and removing keys (cancelling keys etc.)
Event sequence:
1) We call select which calls the WindowsSelectorImpl
2) It creates 3 helper thread (also sets threadsToFinish = 3)
3) First thread gets created and waits on startLock
4) Other 2 helper threads get pre-empted out before they can wait on startLock
5) main thread wakes up everyone waiting on startLock (so one gets started, the other 2 don't)
6) Main thread enters poll and exits pol
7) 1st helperThread that was started goes into poll and finishes, calls wakeup on other threads (but they are not waiting yet)
8) 2 remaining helper threads now get to wait on startLock (not sure if this part can take that long)
9) main thread now checks threadsToFinish and sees 1 thread is finished and doesn't call wakeup, goes to code where it waits for the helper threads, but they are waiting and missed all the wakeup calls because they took "long" to start.
So there is a couple of things that need to fall in place for this to happen:
1) lots of calls into doSelect
2) relatively busy system (so some of the helpers take long to start)
3) a key in the main thread and a key in the 1st started helper triggers almost immediately
Seems that there is a need in the main thread to check if there are any helpers that might have missed the notification to start on startLock, if so, it might need to call it again ??
Reinier
PS: The funny thing though is that at the time I only had 1023 keys in the selector, yet there are 2 helper threads that suggest 1024 (main) + 1024 (helper 1) + 1024 (helper 2) = 3072 channels / keys ?
Edited by: rbezuide on Apr 2, 2008 11:09 AM