NIO Socket implementation - delay between select and get data from socket

Question

Hi all,
I have implemented a internal CallAPI for RPC over a socket connection. It works fine but if there are more than five clients and some load I have the phenomena that the READ selector returns a SelectorKey but I did not get any data from the socket.
My implementation is based on NIO has following components:
+ Accept-Thread
Thread handles new clients.
+ Read-Thread
Thread handles the data from the socket for the registered client. Each request is handled in an own Process-Thread. A Thread-Pool implementation is used for processing.
+ Process-Thread
The Process-Thread reads the data from the socket and starts the processing of the logical request.
In my tests I get the notification of data at the socket. The Process-Thread want to read the data for the socket, but no data are available. In some situations if have to read about 20 times and more to get the data. Between each read attempt I have inserted a sleep in the Process-Thread if no data was available. This have improved the problem, but it already exists. I tested the problem with several systems and jvm's but it seams that it is independent from the system.
What can I to do improve the situation?
I already include the read implementation from the grizzly-Framework. But it doesn't improve the situation.
Socket - Init
     protected void openSocket( String host, int port ) throws IOException
          serverChannel = ServerSocketChannel.open();
          serverChannel.configureBlocking( false );
          serverSocket = serverChannel.socket();
          serverSocket.setReuseAddress( true );
          this.serverhost = host;
          this.serverport = port;
          this.srvAcceptSelector = Selector.open();
          this.srvReadSelector = Selector.open();
          InetSocketAddress isa = null;
          if ( serverhost != null )
               isa = new InetSocketAddress( this.serverhost, this.serverport );
          else
               isa = new InetSocketAddress( this.serverport );
          serverSocket.bind( isa, 50 );
          serverChannel.register( this.srvAcceptSelector, SelectionKey.OP_ACCEPT );
     }New Client � Init
     // New Client
     if ( key.isAcceptable())
          keyCountConnect++;
          ServerSocketChannel actChannel =
               (ServerSocketChannel) key.channel();
          // Socket akteptieren
          SocketChannel actSocket = actChannel.accept();
          if ( actSocket != null )
               actSocket.finishConnect();
               actSocket.configureBlocking( false );
               actSocket.socket().setTcpNoDelay( true );
               this.registerSocketList.add( actSocket );
               this.srvReadSelector.wakeup();
     }Read Data from Socket
    protected int readDatafromSocket( ByteArrayOutputStream socketdata )
         throws IOException
         int readedChars = 0;
        int count = -1;
        Selector readSelector = null;
        SelectionKey tmpKey = null;
        if ( sc.isOpen())
              ByteBuffer inputbuffer = null;
             try
                  inputbuffer = bufferpool.getBuffer();
                  while (( count = sc.read( inputbuffer )) > 0 )
                       readedChars += count;
                      inputbuffer.flip();
                       byte[] tmparray=new byte[inputbuffer.remaining()];
                       inputbuffer.get( tmparray );
                       socketdata.write( tmparray );
                      inputbuffer.clear();
                  if ( count < 0 )
                       this.closeSocket();
                       if( readedChars == 0 )
                            readedChars = -1;
                       if ( log.isDebug())
                              log.debug( "Socket is closed! " );
                  else if ( readedChars == 0 )
                       if ( log.isDebug())
                              log.debug( "Reread with TmpSelector" );
                       // Glassfish/Grizzly-Implementation
                     readSelector = SelectorFactory.getSelector();
                     if ( readSelector == null )
                          return 0;
                      count = 1;
                      tmpKey = this.sc.register( readSelector, SelectionKey.OP_READ );
                     tmpKey.interestOps(
                          tmpKey.interestOps() | SelectionKey.OP_READ );
                     int code = readSelector.select( 500 );
                     tmpKey.interestOps(
                         tmpKey.interestOps() & ( ~SelectionKey.OP_READ ));
                     if ( code == 0 )
                         return 0;
                         // Return on the main Selector and try again.
                       while (( count = sc.read( inputbuffer )) > 0 )
                            readedChars += count;
                           inputbuffer.flip();
                            byte[] tmparray=new byte[inputbuffer.remaining()];
                            inputbuffer.get( tmparray );
                            socketdata.write( tmparray );
                           inputbuffer.clear();
                       if ( count < 0 )
                            this.closeSocket();
                            if( readedChars == 0 )
                                 readedChars =-1;
                       else if ( count == 0 )
                              // No data
             finally
                  if ( inputbuffer != null )
                       bufferpool.releaseBuffer( inputbuffer );
                       inputbuffer = null;
                  // Glassfish-Implementierung
                if ( tmpKey != null )
                    tmpKey.cancel();
                if ( readSelector != null)
                    // Bug 6403933
                     try
                        readSelector.selectNow();
                     catch (IOException ex)
                    SelectorFactory.returnSelector( readSelector );
        return readedChars;
    }Thanks for your time.

Answer 1

I've commented on that blog before. It is rubbish:
- what does 'overloading the main Selector' actually mean? if anything?
- 'Although this not clearly stated inside the NIO API documentation': The API documentation doesn't say anything about which Selector you should register channels with. Why would it? Register it with any Selector you like ...
- 'the cost of maintaining multiple Selectors can reduce scalability instead of improving it' Exactly. So what is the point again?
- 'wrapping a ByteBuffer inside a ByteBufferInputStream:' Code is rubbish and redundant. java.nio.channels.Channels has methods for this.
There is no a priori advantage to using multiple Selectors and threads unless you have multiple CPUs. And even then not much, as non-blocking reads and writes don't consume significant amounts of CPU. It's the processing of the data once you've got it that takes the CPU, and that should be done in a separate thread.
So I would re-evaluate your strategy. I suspect you're getting the channel registered with more than one Selector at a time. Implement it the simple way first then see if you really have a problem with 'overloading the main Selector' ...