Socket latency

Question

I am experiancing long round-trip latency when chaining many Solaris processs together with INET blocking sockets. On NT, I see little latency.
Is there any way of tuning the solaris internals to make the socket respoonse times falster ? Is there an underlying poll timeout I can change ?
my setup is 4 processes:
server: has server socket.
proxy 2: has server socket and client socket to client.
proxy 1: has server socket and client socket to proxy 2.
client: has connection to proxy 1
The client APP sends 40 100byte messages. these are proxied up to the server, which then responds with a 100byte response message for each request. All sockets are blocking with dedicated reader and writer threads. All apps are threaded C++ apps that have been Purify'ed extensively.
If I run the processes on seperate SOLARIS boxes I see the time difference between the first message sent by the client to the last response recieved at 0.5 of a second ! I soon deduced that this was due to the nagle algorithm my sending an extra few placebo messages after the last response to effectively flush the nagle algorithm. I then got it down to ~300ms.
I have now tried running with all processes on the same SunFire-880 machine and the timing is still ~300ms. If I do the same on my Pentium IV Win2000 machine I get round-trip times of ~100 ms or less.
Does anyone know of any issues that affect the response times of sockets ?
I don't believe this is my code as I can sustain transaction rates of ~6000 requests and responses per second ! This is what makes these small-burst issues so fustrating.
Neil

Answer 1

This is what I get, what do you get:
java -server Loopback1000000 messages in 3004 ms
1000000 messages in 2752 ms
1000000 messages in 2706 ms
1000000 messages in 2728 ms
import java.io.*;
import java.net.*;
public class Loopback
    static final int PORT = 6666;
    public static void main(String args[])
     throws Exception
     new Listener().start();
     Socket socket = new Socket("localhost", PORT);
     DataOutputStream out = new DataOutputStream(new BufferedOutputStream(socket.getOutputStream()));
     int count = 0;
     long start_time = System.currentTimeMillis();
     while (true) {
         out.writeBytes("hello world");
         out.flush();
         if (++count % 1000000 == 0) {
          long now = System.currentTimeMillis();
          System.out.println("1000000 messages in " + (now - start_time) + " ms");
          start_time = now;
    static class Listener
     extends Thread
     ServerSocket server_socket;
     Listener()
         throws IOException
         server_socket = new ServerSocket(PORT);
     public void run()
         try {
          while (true) {
              Socket socket = server_socket.accept();
              InputStream in = socket.getInputStream();
              byte buf[] = new byte[8192];
              while (true) {
               if (in.read(buf) == -1)
                   break;
              in.close();
              socket.close();
         } catch (IOException e) {
          System.out.println("error in listener: " + e);
}