FileChannel.transferTo() using non-blocking SocketChannel

I'm looking to use FileChannel.transfer(From/To) for performing file transfers over a network. On the uploading side I find that even though I set the SocketChannel to non-blocking mode, the loop manages to send files as large as 30MB in only a single transferTo() invocation. What I'm hoping for is to have a series of partial writes so that I might generate progress notifications, which does occur on the downloading side with transferFrom(). I don't think a file that large should be transferred in one pass so I'm thinking that this is caused by either some internal buffer allocation issue or a goof-up on my part.
Thanks in advance for any input and here's the uploading code:
public void upload( String fileName ) {
FileInputStream fileStrm = null;
FileChannel fileChan = null;
FileLock fileLock = null;
SocketChannel sockChan = null;
Selector selector = null;
try {
// Lock the source file.
file = new File( fileName );
fileStrm = new FileInputStream( file );
fileChan = fileStrm.getChannel();
fileLock = fileChan.lock( 0L, Long.MAX_VALUE, true );
// Open a server socket bound to an arbitrary port.
servChan = ServerSocketChannel.open();
servChan.socket().bind( new InetSocketAddress( 0 ) );
// Wait for a single connection and close the server socket.
sockChan = servChan.accept();
servChan.close();
sockChan.configureBlocking( false );
selector = Selector.open();
SelectionKey selectorKey = sockChan.register( selector, SelectionKey.OP_WRITE );
// Loop until transfer has completed.
int fileSize = ( int ) file.length();
int sizeLeft = fileSize;
while ( sizeLeft > 0 ) {
// Wait for data to read, then transfer it.
if ( selector.select() == 1 && selectorKey.isWritable() ) {
sizeLeft -= ( int ) fileChan.transferTo( fileSize - sizeLeft, sizeLeft, sockChan );
// Generate a progress notification here such as:
// monitor.bytesTransferred( fileSize - sizeLeft );
catch ( IOException ex ) {
ex.printStackTrace();
finally {
try {
// Cleanup.
if ( selector != null )
selector.close();
if ( sockChan != null )
sockChan.close();
if ( fileLock != null )
fileLock.release();
if ( fileChan != null )
fileChan.close();
if ( fileStrm != null )
fileStrm.close();
catch ( IOException ex ) {
ex.printStackTrace();
-Edwin

Actually, the sending process appears way ahead of the receiving one, where the send seems to finish in a blink while the receive takes several seconds to complete. In other words, the transferTo() completes in one loop, while the transferFrom() performs many with delays in between. I'd guess all that data is sitting in some large outbound buffer at the sender, which would explain why it seems to finish too quickly. If I split the send into smaller chunks, the two sides run quite nearly at the same pace as expected.
The receiver already sleeps by way of a select() against a non-blocking SocketChannel, so I'll try reducing the buffer sizes as you suggest.

Similar Messages

Are the experts wrong about non-blocking SocketChannels?

Everyone says to use the new SocketChannel and Selector for "highly scalable" server applications. So I ran a couple of tests using non-blocking (via Selector) and thread-per-connection SocketChannels.
The Selector version consumes 8x more cpu than the thread-per-connecton version!
Using JDK 1.4.1 FCS on Win2K, with 1000 socket connections each sending 1k bytes per second, the Selector version was consuming 40% cpu with 10 threads; the thread-per-connection version (using blocking SocketChannels) was only consuming about 5% cpu with 1009 threads.
So, are the experts wrong? Is there a performance problem when number of SocketChannels exceed a certain threshold?
For anyone interested, here's the source code:
Non-Blocking Server using Selector
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;
import java.nio.charset.CharsetEncoder;
import java.nio.channels.FileChannel;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
import java.util.Collections;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import java.util.Set;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Server4 implements Runnable
private static int port = 80;
public static void main(String args[]) throws Exception
Server4 server = new Server4();
Thread thread = new Thread(server);
thread.setDaemon(true);
thread.start();
thread.join();
public Server4() throws IOException
ServerSocketChannel server = ServerSocketChannel.open();
InetSocketAddress isa = new InetSocketAddress(port);
server.configureBlocking(false);
server.socket().bind(isa);
m_selector = Selector.open();
server.register(m_selector, SelectionKey.OP_ACCEPT);
Charset utf8 = Charset.forName("UTF-8");
m_decoder = utf8.newDecoder();
m_encoder = utf8.newEncoder();
public void run()
int count = 0;
try
ByteBuffer buffer = ByteBuffer.allocateDirect(2048);
//FileOutputStream fos = new FileOutputStream("server4.dat");
//FileChannel fc = fos.getChannel();
while (m_selector.select() > 0)
Set keys = m_selector.selectedKeys();
for (Iterator itr = keys.iterator(); itr.hasNext(); )
SelectionKey key = (SelectionKey) itr.next();
itr.remove();
if (key.isAcceptable())
System.out.println("accept: " + (++count));
ServerSocketChannel server
= (ServerSocketChannel) key.channel();
SocketChannel channel = server.accept();
channel.configureBlocking(false);
channel.register(m_selector, SelectionKey.OP_READ);
else
SocketChannel channel = null;
try
if (key.isReadable())
channel = (SocketChannel) key.channel();
int bytes = channel.read(buffer);
if (bytes <= 0) // Linux does not throw IOException
channel.close(); // will also cancel key
System.out.println("connection closed " + count);
else
buffer.flip();
//fc.write(buffer);
buffer.clear();
catch (IOException ioe) // connection closed by client
System.out.println("readable: " + ioe.getMessage());
sm_logger.log(Level.INFO, ioe.getMessage(), ioe);
Throwable cause = ioe.getCause();
if (cause != null)
System.out.println("cause: "
+ cause.getClass().getName()
+ ": " + cause.getMessage());
channel.close(); // will also cancel key
--count;
catch (IOException e)
System.out.println("run: " + e.getMessage());
sm_logger.log(Level.SEVERE, e.getMessage(), e);
catch (Exception e)
System.out.println("run: " + e.getMessage());
sm_logger.log(Level.SEVERE, e.getMessage(), e);
private Selector m_selector;
private CharsetDecoder m_decoder;
private CharsetEncoder m_encoder;
private static Logger sm_logger = Logger.getLogger("Server");
Thread-Per-Connection Server
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;
import java.nio.charset.CharsetEncoder;
import java.nio.channels.FileChannel;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
import java.util.Collections;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import java.util.Set;
import java.util.logging.Level;
import java.util.logging.Logger;
public class MultiThreadServer implements Runnable
private static int port = 80;
public static void main(String[] args) throws Exception
ServerSocketChannel server = ServerSocketChannel.open();
InetSocketAddress isa = new InetSocketAddress(port);
server.socket().bind(isa);
int count = 0;
while (true)
SocketChannel channel = server.accept();
System.out.println("accept: " + (++count));
MultiThreadServer worker = new MultiThreadServer(channel);
Thread thread = new Thread(worker);
thread.setDaemon(true);
thread.start();
public MultiThreadServer(SocketChannel channel) throws IOException
m_channel = channel;
public void run()
ByteBuffer buffer = ByteBuffer.allocateDirect(2048);
int bytes = 0;
try
while ((bytes = m_channel.read(buffer)) > 0)
buffer.flip();
// process buffer
buffer.clear();
System.out.println("connection closed");
m_channel.close();
catch (IOException e)
System.out.println("run: " + e.getMessage());
sm_logger.log(Level.SEVERE, e.getMessage(), e);
catch (Exception e)
System.out.println("run: " + e.getMessage());
sm_logger.log(Level.SEVERE, e.getMessage(), e);
private SocketChannel m_channel;
private static Logger sm_logger = Logger.getLogger("MultiThreadServer");
Client
import java.io.*;
import java.net.*;
import java.nio.*;
import java.nio.channels.*;
import java.nio.charset.*;
import java.util.Arrays;
import java.util.Iterator;
import java.util.Set;
import java.util.logging.Level;
import java.util.logging.Logger;
public class MultiClient implements Runnable
public static void main(String[] args) throws Exception
if (args.length < 1)
System.out.println("usage: java MultiClient number [host]");
System.exit(1);
int number = Integer.parseInt(args[0]);
String host = (args.length == 2) ? args[1] : "localhost" ;
Thread[] threads = new Thread [number];
InetSocketAddress address = new InetSocketAddress(host, 80);
for (int i = 0; i < number; i++)
MultiClient client = new MultiClient(address, Integer.toString(i));
threads[i] = new Thread(client);
threads.setDaemon(true);
for (int i = 0; i < number; i++)
threads[i].start();
for (int i = 0; i < number; i++)
threads[i].join();
public MultiClient(InetSocketAddress address, String id)
throws InterruptedException, IOException
m_id = id;
Charset charset = Charset.forName("UTF-8");
m_decoder = charset.newDecoder();
m_encoder = charset.newEncoder();
m_channel = SocketChannel.open();
m_channel.connect(address);
if (id.equals("0"))
Socket socket = m_channel.socket();
System.out.println("SO_SNDBUF=" + socket.getSendBufferSize()
+ ",SO_TIMEOUT=" + socket.getSoTimeout()
+ ",SO_KEEPALIVE=" + socket.getKeepAlive());
byte[] buf = new byte [1024]; // bufsize = 1K
Arrays.fill(buf, (byte) m_id.charAt(0));
m_buffer = ByteBuffer.allocateDirect(1024);
m_buffer.put(buf);
m_buffer.flip();
Thread.currentThread().sleep(50L);
public void run()
System.out.print(m_id);
try
while (true)
m_channel.write(m_buffer);
m_buffer.rewind();
Thread.currentThread().sleep(1000L);
catch (IOException ioe)
ioe.printStackTrace();
catch (InterruptedException ie)
System.err.println(ie.toString());
private String m_id;
private CharsetEncoder m_encoder;
private CharsetDecoder m_decoder;
private SocketChannel m_channel;
private ByteBuffer m_buffer;

{This is a crosspost. I posted this earlier today at http://forum.java.sun.com/thread.jsp?forum=4&thread=319822 before I stumbled on a search phrase that located this older thread.
All follow-ups should be on haam's thread instead of mine. The important point below is that NIO select() behavior (vs. threading IO) is [b]worse under Windows but better under Solaris. This seems fundamentally broken. }
My company sells a scalable multi-user server platform built on Java 2.
It runs under Java 1.3.1 (and 1.4.0 windows) using multiple threads for communications, and 1.4.x (unix) using NIO. We were happy to see that 1.4.1 (windows) fixed the problem drastically limiting the number of selectable ports. :-)
The bad news is that whatever the VM is doing "under the sheets" to fix the problem seems to perform very poorly in terms of CPU:
I compared connecting 500 simulated users to a Solaris 8 and a Win2K box. These users were in 25 chat rooms, each sending a chat message every 30 seconds. (There was plenty of memory on each machine. There was no swapping in either case. Clock/CPU type doesn't matter as this isn't about comparing a machine to a machine, but different load characteristics -within- a machine environment.)
                Threaded IO      NIO/Select
Solaris 1.4.1     20-30%           15- 20%
Windows 1.4.1     40-50%           95-100%Numbers are % of CPU as reported by 'top' and the Win2K task manager.
Both platforms showed the expected significant improvement in memory usage when moving from standard threaded IO to NIO.
Strangely, the Windows implementation of the VM showed a significant (and unexpected) degradation of NIO performance vs. the threaded model, whereas the Solaris VM behaved as expected: NIO outperformed threaded IO.
Our best guess is that the Selector fix in 1.4.1 is implemented in some cpu-intensive way; perhaps polling. As a result, we still can't use NIO for Wintel boxes running our server. :-( To us, Selector
is still broken.
Has anyone else seen results like this? Have we missed some configuration parameter that will fix this?
I thought the big upside of using select() was supposed to be performance. :-P
F. Randall Farmer
State Software, Inc.
http://www.statesoftware.com

Non-blocking SocketChannels

I'm trying to learn how to use non-blocking socket channles, but I haven't found much info (nor luck) so far.
To learn, I'm building a server and a client. The server accepts input from the clients and process it in only one thread. The clients should send Objects to the server, and the server process them and return the result also as an Object. For this, I'm trying to use ObjectOutputStream and ObjectInputStream.
The problem I don't know how to solve is that the SocketChannel is in non-bolcking mode, so I can't use their input/output streams (I get a IllegalBlockingModeException). In the server process loop I can reconfigure the SocketChannel to blocking mode to be able to read the Object, but I can't configure it to non-blocking mode again because I get a CancelledKeyException.
Does anyone know how to work with InputStreams and non-blocking channels? Or where to find more info about it?
Here are the relevant part of the server code:
Set ready = selector.selectedKeys();
Iterator i = ready.iterator();
while (i.hasNext()) {
   try {
      SelectionKey sk = i.next();
      i.remove();
      if (sk.isAcceptable()) {
         ServerSocketChannel ssc = (ServerSocketChannel)sk.channel();
         SocketChannel sc = ssc.accept();
         sc.configureBlocking(false);
         sc.register(selector, SelectionKey.OP_READ);
      } else if (sk.isReadable()) {
         SocketChannel sc = (SocketChannel)sk.channel();
         // PROBLEM: If the channel is in non-blocking mode
         // I cannot use InputStreams
         sk.cancel();
         sc.configureBlocking(true);
         // Read the object sent by the client
         ObjectInputStream in = new ObjectInputStream(Channels.newInputStream(sc));
         Object o = in.readObject();
         // PROBLEM: Can't put the channel back to non-blocking mode
         sc.configureBlocking(false);
         sc.register(selector, SelectionKey.OP_READ); // CancelledKeyException
   } catch (...){
}

In my client, this is working fine:
ObjectOutputStream oos = null;
ObjectInputStream ois = null;
for (int i = 0; i < 30000; i++) {
   oos = new ObjectOutputStream(sc.socket().getOutputStream());
   oos.writeObject(object);
   oos.flush();
   ois = new ObjectInputStream(sc.socket().getInputStream());
   Object o = ois.readObject();
}But trying to do it like this throws a StreamCorruptedException at the server side.
ObjectOutputStream oos = new ObjectOutputStream(sc.socket().getOutputStream());
ObjectInputStream ois = new ObjectInputStream(sc.socket().getInputStream());
for (int i = 0; i < 30000; i++) {
   oos.writeObject(object);
   oos.flush();
   Object o = ois.readObject();
}Do you know why?

Non-Blocking SocketChannel read give Connection timed out

Hi,
My program is using Non-Block NIO SocketChannel for both Server and Client sides. However, if the connection between server and client has been idle for a while, when the client try to communicate with server, it gets blocked for a while (15 minutes) and then receives the following exception:
java.io.IOException: Connection timed out
     at sun.nio.ch.FileDispatcher.read0(Native Method)
     at sun.nio.ch.SocketDispatcher.read(Unknown Source)
     at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
     at sun.nio.ch.IOUtil.read(Unknown Source)
     at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
How can this be since the read is in Non-Blocking mode? Also, is there anyway to determine the timeout without being blocked?

This would mean that you are trying to read from the socket without having properly completed the connection, which timed out. I would say you are connecting in non-blocking mode but not calling finishConnect() when you get OP_CONNECT.

Java.io.IOException: Connection timed out while using non-blocking NIO

Hi,
I am using non-blocking NIO on jdk 1.6 to transfer files between two processes. The problem is that the sender thread times out on the write call even though it's in non-blocking mode. It doesn't make sense. The receiver on the other end is running fine, and even tries to read from the socket after the sender drops the connection, and throws a Connection reset by peer exception. I should mention that the receiver is sometimes busy doing other stuff, and may be a second or two late in issuing the read call. But if I understand correctly, default TCP timeout is about 2 minutes, so there's no reason why it should timeout in a second or two. I'd appreciate it if anyone has any insight.
Thanks!

No I have definitely completed the connection since I it happens a while after the transfer has actually started and part of the file has already been transferred. Oh btw, I should mention that it doesn't always happen. It's intermittent. And it's not a network issue since iperf tests between the two nodes show nothing unusual.

NIO: Strange problem when using ByteBuffer with non-blocking SocketChannel

Hi,
I have a server that uses multiplexed, non-blocking I/O with java.nio. When a client connects, the server waits for the message: <system cmd="knock"/>, returns a message and disconnects the client. The clients are shortly serviced in less than a second.
But the server newer receive anything from about 20% of the clients - even though it is sent. Or with other words: it is received and the data is contained in the ByteBuffer - SocketChannel.read(ByteBuffer) - but a call to ByteBuffer.remaing() returns 0 !!
ByteBuffer receiveBuf = ByteBuffer.allocate(65536);
receiveBuf.clear(); // the code is elsewhere used for longer living clients
int readBytes = channel.read(receiveBuf);
receiveBuf.flip();
StringBuffer sb = new StringBuffer();
System.out.println(" * Remaining: "+receiveBuf.remaining()); // writes: ' * Remaining: 0'
System.out.println(" * Received: "+new String(receiveBuf.array())); // writes: ' * Received: <system cmd="knock"/>'
while(receiveBuf.remaining() >= 2) {
byte b = receiveBuf.get();
sb.append((char)b);
System.out.println(" * sb content: "+sb.toString()); // writes: ' * sb content: 'The ByteBuffer clearly receives the correct data, but the ByteBuffer.remaining() returns 0 and therefore the StringBuffer will never have any content.
The problem seems to occur randomly and for about 20% of the clients (simulated from the same computer and therefore has the same bandwidth and so on).
Anyone knows what is going on, and how to solve the problem !?

It's always possible in any read that the number of bytes read is less than the number of bytes requested. You need to keep reading until you have got everything you expected, and cope with every possible error condition on each iteration.
EJP

Using non blocking connect() call for SCTP sockets in Solaris10

Hi,
I have a problem with non blocking connect call on SCTP socket.
I am using the sctp stack support in Solaris10.
When the connect is successful, I can get the pollout event on the socket.
But there is no event observed when the peer does not exist. In other words, I could not get the pollout event on connection failure. This logic works fine with TCP sockets on both Solaris and Suse10.
I am working with SCTP one-to-one style sockets.
Is there any way to handle this issue?
Do I need to load any patch to resolve this issue?
It will be great if I get a solution in this regard.
Thanks in advance.
Best Regards,
Bipin.

There are at least two problems here.
A. In the receiver you should test for -1 from the read() immediately, rather than continue with the loop and try to write -1 bytes to the file.
B. In the sender you are ignoring the return value of client.write(), which can be anything from 0 to the buffer length. If you get 0 you should wait for another OP_WRITE to trigger; if you get a 'short write' you need to retry it until you've got nothing left to write from the current buffer, before you read any more data. This is where the data is vanishing.

Non-blocking SocketChannel and close - huh?

It looks like closing a socketchannel that is in non-blocking mode can result in a dead drop of the connection, with some bytes that have already been sent and accepted (as in, 'consumed' from the buffer) being completely dropped.
In fact, I'm generally confused: Actual C non-blocking code has a similar setup for 'close' as you can see in SocketChannel's OP_CONNECT behaviour - just because you want to connect the socket doesn't mean it magically happends without blocking. Wait for a ready flag, then try again.
It should work the same way with close (you try to close, but it may not be possible without blocking).
Is this a huge gaping bug that no one quite figured out just yet, or did I miss something? I loathe to turn on linger, as that's just a crapshoot (you never know if it actually gets through. You could run into funny surprises once the server gets a bit busy. I'd rather not) and tends to cause massive leaks on linux, at least according to some google searches.
There seems to be slightly better performance (in that I have always received all data sofar) if I close the socket instead of the channel (socketChannel.socket().close() instead of socketChannel.close()) - but this has been a random attempt at 'making it work', and I can't find any documentation that backs up that this will DEFINITELY not lose any information without letting me know somehow. That still sounds impossible with this approach.

Actual C non-blocking code has a similar setup for
'close' as you can see in SocketChannel's
OP_CONNECT behaviour ...No it doesn't. I don't know what you mean by this.
Closing a socket is asynchronous, but it shouldn't lose any data - if the data can be delivered it will be, followed by the FIN. You don't know when it is delivered, and you don't get to hear about any errors such as an RST coming from the other end, say if it decided to close before reading all the data, or if some intermediate router hung you up.
I'm wondering if you are really dealing with all the short write and zero length write possibilities, i.e. whether the data isn't really still in your output buffer. Don't wish to teach you to suck eggs but there are some subtleties here.
Setting a positive linger timeout doesn't really help because Java doesn't tell you if the timeout expired. (I only asked for this about four years ago). You get to wait while any pending data is delivered, but you still have to guess about whether it all went or the timeout expired, and the behaviour after the timeout expires is platform-dependent: some (Windows) issue an RST, others (Unix) keep trying.
Blocking or non-blocking mode shouldn't make any difference to this (except that Linux will block on a positive linger timeout even in non-blocking mode, which is wrong), and whether you close the channel or the socket is immaterial as one calls the other anyway.
The reason why OP_CONNECT is different in blocking/non-blocking modes is that in blocking mode it won't return until the SYN-ACK is received, while in non-blocking mode it just sends the SYN and relies on you calling connect() again (at the C level) to collect the SYN-ACK, or not - this is what finishConnect() tells you.
tends to cause massive leaks on linux, at least
according to some google searchesIt can't, unless you are referring to the Helix client thing, which appears to be just a badly designed C++ class library with, for some reason, its own linger implementation outside the kernel.

HTTP Web Proxy using Non Blocking IO

Resolved Thanks................

client.register(selector, SelectionKey.OP_READ
| SelectionKey.OP_WRITE);OP_READ only at this stage. OP_WRITE is always 'ready' unless the socket send buffer is full, so your selector will just spin. Don't register OP_WRITE until you have something to write.
ByteBuffer bferClient = ByteBuffer.allocate(4096);You're doing this every time around the loop, and you don't even know whether the key is readable yet. Do it when you accept a connection, and save it as the key attachment for that channel.
client.configureBlocking(false);It's already in non-blocking mode. Why are you doing this again?
try {
if ((bytesRead = client.read(bferClient)) != -1) {
bferClient.flip();
String msgFromClient = new String(bferClient.array());new String(bferClient.array(), 0, bferClient.limit()-1), but you're assuming you got a complete command here. You may not have. That's another reason why you need a ByteBuffer per channel.
channel.connect(socketAddress);You should do that in non-blocking mode, register the channel for OP_CONNECT, and let the select loop take care of finishing the connection.
channel.write(bferClient);The incoming command was CONNECT. You don't write that upstream.
if (!"".equals(msgFromClient.trim())) {
int bytesReadServer;
byte[] reply = new byte[4096];
try {
ByteBuffer bufferNIO = ByteBuffer.allocateDirect(1024);
while ((channel.read(bufferNIO)) != -1) {
bufferNIO.flip();
client.write(bufferNIO);
bufferNIO.clear();
}And you don't do any of this here. You register the new channel for OP_CONNECT, then when you get that do finishConnect() and then register it for OP_READ, and proceed around your select loop. You also need to associate this channel with the channel you read the CONNECT from. So you really need a key attachment object for the original channel that contains its byte buffer and the upstream channel; similarly the upstream channel needs a key attachment that contains its buffer and its downstream channel. The effect of doing all this stuff here instead of around the select loop is that you are never getting back to the select loop: you are handling this one client here, in blocking mode, until the connection is dropped. Completely wrong.
} catch (IOException e) {Hang on. If bytesRead was -1 you must close the channel.
e.printStackTrace();And if you get any IOException here you must also close the channel.
if (channel != null) {
try {
channel.close();This is a really strange place to close this channel.
When you get all that fixed up, post your code and I'll tell you how to handle OP_WRITE.

File transfer using non-blocking sockets - data mysteriously vanish

Hello,
I need to pass some big amount of data using sockets. I my appliaction I have noticed that sometimes I didn't get all bytes. To check it out I wrote simple client + server to figure out what is happening. So:
- I have a sender and receiver application
- I'm trying to transfer 5MB text file.
- On receiver side, output file is never larget than 3MB
- If I put some timeout on sender side (1ms timeout between write operations) everything works fine.
Could someone tell me what I do wrong? Why data disappears and when? The same file transfered using old sockets goes always fine...
Thanks in advance!
Here is complete source for receiver and sender:
RECEIVER:
import java.io.FileOutputStream;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
import java.util.Iterator;
public class ReceiverA {
      * @param args
     public static void main(String[] args) {
          String outputFile = "c:\\outputA.txt", host = "127.0.0.1";
          int port = 8001, bufferSize = 10240;
          try {
               ByteBuffer buffer = ByteBuffer.allocate(bufferSize);
               Selector selector = Selector.open();
               ServerSocketChannel server = ServerSocketChannel.open();
               server.configureBlocking(false);
               server.socket().bind(new InetSocketAddress(host, port));
               server.register(selector, SelectionKey.OP_ACCEPT);
               System.out.println("Server started");
               while(true)
                    selector.select();
                    Iterator<SelectionKey> iterator = selector.selectedKeys().iterator();
                    while (iterator.hasNext()) {
                         SelectionKey key = (SelectionKey) iterator.next();
                         iterator.remove();
                         if (key.isAcceptable()) {
                              SocketChannel client = server.accept();
                              client.configureBlocking(false);
                              client.register(selector, SelectionKey.OP_READ);
                              continue;
                         SocketChannel channel = (SocketChannel) key.channel();
                         int counter = 1;
                         if ( key.isReadable() ) {
                              FileOutputStream os = new FileOutputStream(outputFile);
                              int res;
                              do
                                   buffer.clear();
                                   res = channel.read(buffer);
                                   counter += res;
                                   System.out.println(res);
                                   buffer.flip();
                                   os.write(buffer.array(), 0, buffer.limit());
                              while( res >= 0 );
                              channel.close();
                              os.close();
                              System.out.println("Receiver: " + counter);
                              return;
          } catch (Exception e) {
               e.printStackTrace();
}SENDER:
import java.io.FileInputStream;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.SocketChannel;
import java.util.Iterator;
public class SenderA {
     public static void main(String[] args) {
          String inputFile = "c:\\inputA.txt" , host = "127.0.0.1";
          int port = 8001, bufferSize = 10240;
          try {
               ByteBuffer buffer = ByteBuffer.allocate(bufferSize);
               byte[] byteArr = new byte[buffer.capacity()];
               Selector selector = Selector.open();
               SocketChannel connectionClient = SocketChannel.open();
               connectionClient.configureBlocking(false);
               connectionClient.connect(new InetSocketAddress(host, port));
               connectionClient.register(selector, SelectionKey.OP_CONNECT);
               while(true) {
                    selector.select();
                    Iterator<SelectionKey> iterator = selector.selectedKeys()
                              .iterator();
                    while (iterator.hasNext()) {
                         SelectionKey key = (SelectionKey) iterator.next();
                         iterator.remove();
                         SocketChannel client = (SocketChannel) key.channel();
                         if (key.isConnectable()) {
                              if (client.isConnectionPending()) {
                                   System.out.println("Trying to finish connection");
                                   try {
                                        client.finishConnect();
                                   } catch (IOException e) {
                                        e.printStackTrace();
                              client.register(selector, SelectionKey.OP_WRITE);
                              continue;
                         if(key.isWritable()) {
                              FileInputStream is = new FileInputStream(inputFile);
                              int res;
                              int counter = 0;
                              do
                                   buffer.clear();
                                   res = is.read(byteArr, 0, byteArr.length);
                                   System.out.println(res);
                                   if ( res == -1 ) break;
                                   counter += res;
                                   buffer.put(byteArr, 0, Math.min(res, buffer.limit()));
                                   buffer.flip();
                                   client.write(buffer);
                                    * When I remove comment file transfer goes OK!
                                   //Thread.sleep(1);
                              while( res != -1 );
                              client.close();
                              is.close();
                              System.out.println("Receiver: " + counter);
                              return;
          catch(Exception e) {
               e.printStackTrace();
}

There are at least two problems here.
A. In the receiver you should test for -1 from the read() immediately, rather than continue with the loop and try to write -1 bytes to the file.
B. In the sender you are ignoring the return value of client.write(), which can be anything from 0 to the buffer length. If you get 0 you should wait for another OP_WRITE to trigger; if you get a 'short write' you need to retry it until you've got nothing left to write from the current buffer, before you read any more data. This is where the data is vanishing.

Using non-blocking NIO, need Main Thread wait?

I'm trying to utilize the New IO libraries to create a simple url check application. The application just tests return codes to make sure the list of urls I have are still valid.
I'm using the NIO libraries because I had problems with the URLConnection object hanging when I ran into a url where the server never replied. It just sat there waiting for a reply.
Now I have a new problem, the connection to the web server is now in a new thread and I need to wait for the connection to finish before using it. I can't figure out how to make the main app wait for the connection. All of the documentation I find for the Wait() method refers to threads and I can't find how to refer to the main application thread.
Any suggestions?
Josh

Ok, this is were I have some gaps in my Java knowledge. How do I wait for OP_CONNECT to fire. Is it like setting up a ActionListener but for a non-visual object?
Also what do you mean when you refer to deregistering it?
You don't need to explain everything. I would be happy if I was just pointed at the appropriate pages and I'll do the leg work.
Josh
If you're using NIO the connection can be handled in
the same thread as the I/O. Just wait for OP_CONNECT
to fire, deregister it, and away you go.

Non-blocking Vectored read from SocketChannel fails

Hi.
Please look at the code example:
ByteBuffer tmp1 = ByteBuffer.allocate(500);
ByteBuffer tmp2 = ByteBuffer.allocate(500);
ByteBuffer[] tmp = {tmp1, tmp2};
while ((count = socketChannel.read(tmp)) > 0) {
+ "bytes.");
When I run this code, (using non-blocking socket channel), the first buffer is read ok, but on the second round of the while loop, I get an IOException, saying: "a non blocking socket operation could not be completed immediately".
If I put 'tmp1' instead of 'tmp' in the read(), (i.e - use a normal buffer), the read successfully finishes.
Why?
I found nothing in the documentation, and in the 'Java NIO' book (Ron Hitchens).
Help?
Thanks.

Please ignore the + "bytes."); inside the while.

SO timeout relevance in non-blocking NIO ServerSocketChannel

I converted a threaded blocking server from using the standard blocking IO with a thread for each connection to using non-blocking NIO channels. Works great but I'm trying to understand the SO timeout relevance in the context of non-blocking NIO channels. Is there any relevance and should I set an SO timeout on the ServerSocket associated with a ServerSocketChannel and if so how should I handle it?

No. Socket timeouts are for blocking mode. If you need timeouts in non-blocking mode you have to do them yourself, taking advantage of the Selector.select(long timeout) method and keeping track of activity times per channel yourself. You should use that select() method anyway, rather than just blocking indefinitely, as it gives you a chance to catch up on housekeeping, dead channels, etc. For example as a very naive approach if nothing happens in say a select timeout of 10 minutes you might want to close all accepted SocketChannels.

Please help!!!!! on non blocking io

How can I use non blocking IO on client side? is it possible with BufferedReader(new InputStreamReader) on read????
pl. hjelp me, it's urgetn.. :(
Thanx in advance

FoxyLady No, it's Client/Server environment

Parsing Non-Blocking Data

I'm presently using non blocking sockets to receive data from a server. The problem is that I receive several chunks of data all at once. Is there a way to parse the data using a terminating byte instead of the usual -1 EOF?

I receive data continuously so given: channel.read(buffer) I just run out of bufferspace before I am given the chance to parse it. Bottomline I want to replace this blocking code:
DataInputStream dis = new DataInputStream(socket.getInputStream());
while((currentByte = dis.read()) != ETX) //ETX is the terminating byte
buffer.put((byte)currentByte);
with its nonblocking counterpart.

FileChannel.transferTo() using non-blocking SocketChannel

Similar Messages

Maybe you are looking for