Streaming large amounts of data of socket causes corruption?

Question

I'm wrinting an app to transfer large amounts of data via a simple client/server architecture between two machines.
Problem: If I send the data too 'fast', the data arrives corrupted:
- Calls to read() returns wrong data (wrong 'crc')
- Subsequent calls to read() do not return -1 but allow me to read e.g. another 60 or 80 KBytes.
- available() returns always '0'; but I'll get rid of that method anyway (as recommended in other forum entries).
The behaviour is somewhat difficult to repeat, but it fails for me reliably when transferring the data between two separate machines and when setting the number of packets (Sender.TM) to 1000 or larger.
Workaround: Reduce number of packages send to e.g. 1; or intruduce the 'sleep' on the Sender side. Another workaround: Changing alone to java.nio.* did not help, but when I got rid of the Streams and used solely ByteBuffers, the problem disappeared. Unfortunately the Streams are required by other parts of my application.
I'm running the code on two dual-CPU machines connected via
Below are the code of the Sender and the Listener. Please excuse the style as this is only to demonstrate the problem.
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.nio.channels.Channels;
import java.nio.channels.SocketChannel;
import java.util.Arrays;
public class SenderBugStreams {
    public static void main(String[] args) throws IOException {
        InetSocketAddress targetAdr = new InetSocketAddress(args[0], Listener.DEFAULT_PORT);
        System.out.println("connecting to: "+targetAdr);
        SocketChannel socket = SocketChannel.open(targetAdr);
        sendData(socket);
        socket.close();
        System.out.println("Finished.");
    static final int TM = 10000;
    static final int TM_SIZE = 1000;
    static final int CRC = 2;
    static int k = 5;
    private static void sendData(SocketChannel socket) throws IOException {
        OutputStream out = Channels.newOutputStream(socket);
        byte[] ba = new byte[TM_SIZE];
        Arrays.fill(ba, (byte)(k++ % 127));
        System.out.println("Sending..."+k);
        for (int i = 0; i < TM; i++) {
            out.write(ba);
//            try {
//                Thread.sleep(10);
//            } catch (InterruptedException e) {
//                // TODO Auto-generated catch block
//                e.printStackTrace();
//                throw new RuntimeException(e);
        out.write(CRC);
        out.flush();
        out.close();
import java.io.IOException;
import java.io.InputStream;
import java.net.InetSocketAddress;
import java.nio.channels.Channels;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
public class ListenerBugStreams {
    static int DEFAULT_PORT = 44521;
     * @param args
     * @throws IOException
    public static void main(String[] args) throws IOException {
        ServerSocketChannel serverChannel = ServerSocketChannel.open();
        serverChannel.socket().bind(new InetSocketAddress(DEFAULT_PORT));
        System.out.print("Waiting...");
        SocketChannel clientSocket = serverChannel.accept();
        System.out.println(" starting, IP=" + clientSocket.socket().getInetAddress() +
            ", Port="+clientSocket.socket().getLocalPort());
        //read data from socket
        readData(clientSocket);
        clientSocket.close();
        serverChannel.close();
        System.out.println("Closed.");
    private static void readData(SocketChannel clientSocket) throws IOException {
        InputStream in = Channels.newInputStream(clientSocket);
        //read and ingest objects
        byte[] ba = null;
        for (int i = 0; i < SenderBugStreams.TM; i++) {
            ba = new byte[SenderBugStreams.TM_SIZE];
            in.read(ba);
            System.out.print("*");
        //verify checksum
        int crcIn = in.read();
        if (SenderBugStreams.CRC != crcIn) {
            System.out.println("ERROR: Invalid checksum: "+SenderBugStreams.CRC+"/"+crcIn);
        System.out.println(ba[0]);
        int x = in.read();
        int remaining = 0;
        while (x != -1) {
            remaining++;
            x = in.read();
        System.out.println("Remaining:"+in.available()+"/"+remaining);
        System.out.println(" "+SenderBug.TM+" objects ingested.");
        in.close();
}

Answer 1

Here is your trouble:
in.read(ba);read(byte[]) does not read N bytes, it reads up to N bytes. If one byte has arrived then it reads and returns that one byte. You always need to check the return value of read(byte[]) to see how much you got (also check for EOF). TCP chops up the written data to whatever packets it feels like and that makes read(byte[]) pretty random.
You can use DataInputStream which has a readFully() method; it loops calling read() until it gets the full buffer's worth. Or you can write a little static utility readFully() like so:
    // Returns false if hits EOF immediately. Otherwise reads the full buffer's
    // worth. If encounters EOF in mid-packet throws an IOException.
    public static boolean readFully(InputStream in, byte buf[])
        throws IOException
        return readFully(in, buf, 0, buf.length);
    public static boolean readFully(InputStream in, byte buf[], int pos, int len)
        throws IOException
        int got_total = 0;
        while (got_total < len) {
            int got = in.read(buf, pos + got_total, len - got_total);
            if (got == -1) {
                if (got_total == 0)
                    return false;
                throw new EOFException("readFully: end of file; expected " +
                                       len + " bytes, got only " + got_total);
            got_total += got;
        return true;
    }