SocketChannel.write() blocking application

Greets,
I'm developping a huge application and got a latency/block problem using the write(ByteBuffer) method on a Socket from a socketchannel connection.
Running java 1.5 (diablo) on Freebsd 6 servers, 4Gb ram (2.2 allocated to jvm), with dual xeon dual-core (total 4 cores)
Here is the application schema :
- A thread accepting connexion on the socketchannel
- A thread selecting keys with data to process, enqueuing it after some basic checks on a command FIFO
- A thread getting commands from the FIFO and processing 'em, generating answers on 4 answer FIFOs
- 4 threads (1 per FIFO) to get answers and send 'em back to the socket.
The application usually runs with 4500-5000 simultaneous clients.
My problem is that the only write() method sometimes takes over 20ms to write a message, with a length smaller than 50 bytes.
As I got about 25000 answers to process each second, when some of 'em decide to be slow, the 4 threads runs slowly, and all the connected clients are suffering of that latency, for the few minutes needed to empty the FIFOs.
Every client socket get about 5 answers per second.
On about 1 hour running, there are about 3 'peaks' of slowness, that I cannot explain yet. That's why I'm in need of advices !
I monitored the application when such case happens. TOP indicates 40% cpu idle, JVM memory got >500Mb free, network runs @ about 1.2Mbps, where maximal transfer rate is >20Mbps. netstat -m told me no erros, and a large amount of free buffers available.
As the only slow process is the write() method that usually runs faster than 1ms each call, but in those case I got delays over 20ms.
freebsd tcp default sendbuffer size is 64k, receive buffer is 32k
Commands average received size is below 1k, Answers average sending size below 8k.
This application is running live, and as I cannot emulate 5000+ connections with a similar beahviour to test withour being sure that won't crash all.
What points could be responsible of such slow write() calls ? Seems it's not CPU, not RAM, not network itself...
I suppose it's the network buffers that are causing problems. But I don't really know if I have to fit 'em to a lower size, fitting my requirements, or to a larger size, to be sure there won't be full buffers blocking all ?
I need advices. Thanks for your ideas !
Bill

Hmm. So you're happy to lose data?
A few comments:
(a) SocketChannels are thread-safe. I don't think you need the synchronization at all, unless maybe multiple writing threads are possible. I would eliminate that possibility and the sync myself.
(b) If you're getting write delays of 30ms occasionally, the sync must also take 30ms at the same points if it is doing anything at all, i.e. if the possibility of multiple writing threads does exist. So maybe it doesn't?
(c) I would have a good look at this:
http://forum.java.sun.com/thread.jspa?threadID=459338
and specifically the part on how to manage a channel that presents write blocks, using OP_WRITE when it happens and turning it off when it doesn't.
(d) You seem to be using one output buffer for all channels. You might be better off using a small one per channel. Then that way you don't clear, you just do put/flip/write/compact, and if the write returned 0 just post OP_WRITE for next time around the select loop. Then you won't lose any data at all, except to a client who really isn't reading: you can detect that situation by keeping track of the last successful write time to a channel, and when there is pending data and the last write is too long ago have a think about what the block means in terms of the application. Maybe you should just disconnect the client?
(e) It would be interesting to know how many times the write loop looped when you get these large delays, and also what the data size was, and also to know that for the other cases to see if there is a difference.
(f) Generally from a fairness point of view I prefer not to have write loops, just one attempt and if it returns even a short read I post OP_WRITE as above. Otherwise you're spending too long servicing one channel.
You can contact me offline via http://www.telekinesis.com.au if you like.

Similar Messages

Does SocketChannel.write() block

I have a server communicating with it's clients using NIO and therefore SocketChannel's. When data are send to clients, the method SocketChannel.write(ByteBuffer) are used. The question is then, does the execution time depend of SocketChannel.write() depend on the speed of the client to receive data? Does the the method block in any way, or does it just send what is possible and then returns?

If you have the channel in blocking mode, it also depends on how fast the client is reading. If the client isn't reading at all, ultimately its receive buffer will fill up, then the sender's sending buffer will fill, and then the write will block waiting for space in the send buffer; once the client starts reading again, space will become available and the write can complete. That's not to say that every write waits for the completion of every prior read, it's a matter of buffering and windowing, and the effect is rather decoupled because of the presence of two intermediate buffers. However it certainly can occur.

SocketChannel.write()

I've written a non-blocking server and am trying to test it with a client that uses the traditional (blocking) I/O. The problem I'm running into is when the server tries to write data to the client, it seems to take an inordinate amount of time.
Often times the SocketChannel.write() method will return zero and write nothing to the socket when it has a perfectly good ByteBuffer available to be written. The Javadoc for the SocketChannel.write() states this is normal:
Some types of channels, depending upon their state, may write only some of the bytes or possibly none at all. A socket channel in non-blocking mode, for example, cannot write any more bytes than are free in the socket's output buffer.
So I put the following statement in the code:
while ((retVal = socketChannel.write(outputBuffer)) == 0);However, this is proving to be quite slow, sometimes taking over half a second to complete a write of around 1000 bytes.
While this solution delivers all of the data, it is slow. Has anybody run into this, and if so, found a better solution?
Thanks,
Ken

I feel compelled to reply to myself to keep this alive, so to add:
It seems like the delays being experience here would not happen with blocking sockets and that this is a function of NIO. What I don't understand is why socket channel takes so long to know when it can write again. Waiting over half a second for one write of a 1000 byte message is not acceptable.
Is it because I am not using a Selector? Would that tell me the channel is ready any faster? I wouldn't see why. Or is it because the client application is reading the data too slowly?
Any help would be appreciated.
Ken

Socketchannel write(bytebuffer)

I use nio socketchannel write(bytebuffer [])method.
In my program,as following...
buff.clear();
sc.read(buff);
buff.flip();
sc.write();
sc.close();//Ok,when this line is not been marked.
In another program,
msg(String s)
buff.get(s.getBytes());
buff.flip();
sc.write(buff);//No close(),and problems come,in design,
sc can not close()
How to solve the problem?

You didn't show any client-side code, did you? Are you using a selector? If so, it could be that your select logic is wrong, causing you to miss when the socket first becomes readable (happened to me, for instance).
When you ctrl-c the server, a TCP FIN packet gets sent to the client - if you were blocked in a select on the client side, it'd break out - and since the data in the TCP receive queue would still be there to read, your application would be able to read it out before getting an EOF on the socket.
--bruce

Exceptions not thrown on interrupted SocketChannel.write()

I just noticed a behaviour of java.nio.channels.SocketChannel.write() that makes me wonder. If write() blocks(), and the channel is closed during this blocking by another thread, AsynchronousCloseException should be thrown.
However, in most cases this does not happen in my little test app. If any part of the data passed to write() has already been written to TCP before the socket was closed, write() returns without exception.
Similar behaviour is observed with intterupting. If the thread blocked in write() is interrupted by another thread, it returns immediately and has the interrupted Flag set, but in most cases no Exception is thrown.
ClosedByInterruptException is only thrown if not any part of the data passed to write() has been passed to TCP.
Is this a bug or a feature ?

Yes, i'm pretty sure that it blocks. In my test, the server just accepts the connection and then goes to sleep for a looooong time. The client just connects, and sends 10Meg of fata in one write() call. If i do not interrupt the client, it blocks as long as the server sleeps. In this case, this is the client's stack while blocking:
Thread [main] (Suspended)
     FileDispatcher.write0(FileDescriptor, long, int) line: not available [native method]
     SocketDispatcher.write(FileDescriptor, long, int) line: 29
     IOUtil.writeFromNativeBuffer(FileDescriptor, ByteBuffer, long, NativeDispatcher, Object) line: 104
     IOUtil.write(FileDescriptor, ByteBuffer, long, NativeDispatcher, Object) line: 75
     SocketChannelImpl.write(ByteBuffer) line: 334
     Channels.write(WritableByteChannel, ByteBuffer) line: 60
     Channels.access$000(WritableByteChannel, ByteBuffer) line: 47
     Channels$1.write(byte[], int, int) line: 134
     Channels$1(OutputStream).write(byte[]) line: 58
     SocketClient.main(String[]) line: 75     If i start another thread before calling write(), that closes the socket after 3 seconds, the following happens: The call of IOUtil.write() returns ( with a value n that i cannot see in the debugger ), This value n is tested inside SocketChannelImpl.write() via (n > 0 || (n == IOStatus.UNAVAILABLE), what return true. For that reason, AbstractInterruptibleChannel.end(boolean), does not throw an exception.
Once the server wakes up later, it is able to read about 200K from the socket...
I tried this on a linux system (kernel 2.6.17, glibc 2.4) with jdk 1.6.0_03. I'm now gonna try it under windows, hold on..
Of course, if anybody is interested, i'll post the test proggy...

SocketChannel.write() on J2SDK1.4.2_04/Windows 2000

Hello,
I have a question regarding SocketChannel.write() (as I mentioned before, I am using J2SDK 1.4.2_04 on Win 2000 pro).
I prepared a ByteBuffer with some binary data. When I invoke SocketChannel.write(), the other application receives the data only until the first null (byte = 0) character is found??? What am I doing wrong??
Thanks.

On the other side, I get the message
Read: ZYKT
(only the first 4 bytes of the message)
I experimented adding more non-null characters and the
result was
Read: ZYKT08@AYes, but (again) what does the CODE look like? Maybe it's written incorrectly to stop when it gets a null byte. Or maybe it stuffs it into a C-style char array (which is null-terminated by convention) and just happens to print up to the null (as C-style char array functions are designed to do). Or, who knows? You're not showing what that client process looks like.

Write blocked waiting on read

Hi all,
I have been experiencing difficulties with simultaneous read/write with AsyncIO. For my scenario that the client/server application is being developed, a client may send requests or status information at any given time. Hence continual monitoring on the incoming stream is required.
The server shall respond depending upon the status message requiring a write to the client (so it won't respond to all messages received). Likewise, the client in some instances will only send status messages depending on the last message it received from the server.
I've been experiencing difficulties with writes blocking because of reading. Essentially I would like to continually poll reading whilst allowing writes to be flushed immediately.
Using traditional blocking read/writes things hang indefinitely whilst attempting to write a message as reading has blocked waiting for input.
Using the IBM AsyncIO package (which is purported to be faster implementation of NIO), writing blocks for some time until reading (i assume) relinquishes the socket to allow writing to occur before resuming reading again. The lag time in this situation is significant.
Is someone able to provide an example using non-blocking R/W in which a server can sit reading (on one thread) whilst the writing is attempted on another thread that doesn't cause any lag?
Below is a basic overview of what is happening in my software:
public class MessageQueue {
   private LinkedList<Message> queue;
   /** Creates a new instance of MessageQueue */
   public MessageQueue() {
      queue = new LinkedList<Message>();
   public synchronized void put(Message message) {
      queue.add( message );
      notifyAll();
   public synchronized boolean isEmpty() {
      return queue.isEmpty();
   public synchronized Message get() {
      while( queue.isEmpty() ) {
         try {
            wait();
         } catch( InterruptedException ie ) {
      Message message = ( Message )queue.removeFirst();
      return message;
   public synchronized void close() {
      queue.clear();
      queue = null;
public class InputReader implements Runnable {
private MessageQueue messages;
private AsyncSocketChannel async_channel;
public InputReader(MessageQueue messages, AsyncSocketChannel async_channel) {
    this.messages = messages;
    this.async_channel = async_channel;
public long read(ByteBuffer b) {
     long bytes_read = 0;
     helper = new AsyncSocketChannelHelper( this.async_channel );
     future = channel.read(b);
     bytes_read = future.getByteCount( );
     return bytes_read;
public void run() {
     ByteBuffer b = ByteBuffer.allocateDirect(Message.SIZE);
     boolean running = true;
     while(running) {
        if (read(b) == 0)
          running = false;
        else
          messages.put(new Message(b));
public class OutputWriter implements Runnable {
private MessageQueue messages;
private AsyncSocketChannel async_channel;
public OutputWriter(MessageQueue messages, AsyncSocketChannel async_channel) {
    this.messages = messages;
    this.async_channel = async_channel;
public long write(ByteBuffer b) {
      long bytes_written = 0;
      try {
         AsyncSocketChannelHelper helper = new AsyncSocketChannelHelper( this.async_channel );
         IAsyncFuture future = helper.write(b, 20000); // write or timeout
         // wait for completion of write, or for the timeout to happen
         bytes_written = future.getByteCount( );
         // THIS IS WHERE THE PROBLEM LIES. The write does not happen straight away because of the read operation. With traditional blocking IO this locks completely.
      } catch ( AsyncTimeoutException ate) {
              System.err.println("Timed out after 20 seconds");
      return bytes_written;
   public void run() {
     boolean running = true;
     while(running) {
        Message m = this.messages.get();
        if (write(m.getByteBuffer()) == 0)
          running = false;
        else
          messages.put(new Message(b));
public class Controller {
   public Controller(AsyncSocketChannel async_channel) {
        MessageQueue in = new MessageQueue();
        MessageQueue out = new MessageQueue();
        InputReader ir = new InputReader(out, async_channel);
        OutputWriter ow = new OutputWriter(out, async_channel);
        new Thread(ow).start();
        new Thread(ir).start();
        boolean running = true;
        Message m;
        while(running) {
           m = in.get();
           if (m.getStatus() == "REQUIRES_RESPONSE"))
             out.put(m); // dummy example to demonstrate that once the right condition is met, a new message must be written
}

That makes me wonder what the problem is I am experiencing then.
I initially had stock-standard java.net IO for socket reading and writing. The approach I took was to set up an input reader on its own thread and an output writer on its own thread.
When it came to writing data however, things locked up. Stepping through the code in debug mode allowed me to see that the write method was not completing because the read method was waiting for input to come in.
I tested it using traditional buffered output which made a call to flush() afterwards, but it was getting stuck on the write() call.
I came to the conclusion that the read must be blocking the write from completing because of the response to this thread: http://forum.java.sun.com/thread.jspa?forumID=536&threadID=750707
On further debugging, write() never locked up when I didn't allow the input reader to perform a simultaneous read() on the socket.
Hence my belief that the java.net socket does block when one operation is being performed.
After dealing with IBM's AsyncIO package I'd be willing to wager that their ibmaio package is 50x more complex to use than standard Java sockets (barely any documentation/examples) so 10x complexity seems positively lightweight ;-)
So ejp, to clarify, would NIO help solve this blocking problem or do you think something else is the culprit? It is hard to see what as to test things out I made two bare bones testing programs (one client and the other a server) so I don't feel it could be anything else in the code.
Thoughts?

SelectionKey.OP_WRITE versus SocketChannel.write(ByteBuffer)

I'm writing a small Socket server using Non blocking approach based on the ractor design pattern.
Many samples use the OP_WRITE on the selection keys to know when the channel is ready to accept writes.
I use the SocketChannel.write(ByteBuffer) whenever I need to write instead.
Is there a reason why SelectionKey.OP_WRITE is prefered in the samples??????
Thanks
Stéphane

Rainman4500 wrote:
I'm writing a small Socket server using Non blocking approach based on the ractor design pattern.
Many samples use the OP_WRITE on the selection keys to know when the channel is ready to accept writes.
I use the SocketChannel.write(ByteBuffer) whenever I need to write instead.
Is there a reason why SelectionKey.OP_WRITE is prefered in the samples??????
So that you don't attempt a write unless the channel can accept some data from you. In your example, suppose you actually transfer 0 bytes because the output buffer is still full. What do you do then?

Precision on "Write Intensive" applications

Hi,
We dont have experience with Oracle so far.
From Oracle faq, I saw the information "...RAID 5 is good for read intensive, but not write intensive applications."
What is write intensive application? We are planning to have 200-300 mb of transactions per day in a database.
Thanks.
Claude

this is a topic that generates significant debate. based on my experiences, the oracle documentation i have read, and input from large disk manufacturers, i generally recommend raid-1 for many of the reasons mentioned in the previous response.
databases typically use random, low-block-size (16k and under) i/o that is better suited for raid-1. raid-5 excels in highly sequential, large block size operations. even though raid-1 "costs" more in acquisition cost, i believe the relatively low cost of disk storage is worth the extra price in terms of the performance benefit you will receive (in both normal and in degraded modes) for larger, higher volume databases.
no matter what raid mode you use, most raid controllers allow you to enable write caching, which can speed the performance of write operations. make sure you have battery backup on your hardware if you enable this option.
another very important factor you should address is the distribution of your raid arrays over multiple host channels. this means that you should not hang all of your oracle data off of a single scsi or fiber disk bus. distributing your data over multiple buses also opens up the ability to store frequently-accessed data on faster disk. so you could use a combination of raid-1 and raid-5 if you wanted to compromise. but you always have to be monitoring the i/o performance of your hardware and database, so you can balance the i/o appropriately.

How to write a application using WDJ with the adobe form ?

Hi, experts,
I don't know how to write a application using !!webdynpro for java!! with the adobe form so that I can fill data to the adobe form and get data from the adobe form.
Note: I have configed the ADS(adobe document services),and I can create a application with a interactiveform in webdynpro for abap and run it successfully, so that I can fill data to the adobe form and get data from the adobe form.
Do you give me some hint?
Thanks a lot.
Best regards,
tao
Edited by: wang tao on Sep 9, 2008 8:59 AM

Hi,
Refers the following links.
https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/4fd2d690-0201-0010-de83-b4fa0c93e1a9
https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/5f27e290-0201-0010-ff82-c21557572da1
https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/70c9a954-aa20-2a10-64bb-ea0835204e03
Thanks
Abhilasha.

Access Restrictions - Blocked Applications does not save

Under the Access Restrictions section it is possible to Block Applications.
I enter a name for the application, the port range (from and to) and whether it is TCP, UDP or both. I then hit ADD which adds the application to the list of Blocked Applications in the box above. I then scroll down thru that box to find the App and [>>] add it to the Blocked List. I then hit SAVE SETTINGS
Upon returning to the policy and the Blocked Apps list the record I just created is nowhere to be found. Not in the Blocked List or under Applications.
What gives ???

Wireless-N Gigabit Router
Model: WRT310N
Firmware Version: v1.0.06
All PCs run VISTA
I don't see how a HARD RESET and POWER RECYCLE creates a fix for this. All that does is reboot the existing firmware and causes me to have to reenter all the router and security settings. The documentation for the lastest firmware (version Ver.1.0.9 build 4 12/11/2009) and the previous upgrade (1.0.07 build 14) does not include references to a fix for this problem. Has this been identified before and what firmware version is the fix included in ?
I have held off performing the 1.0.9 firmware upgrade as I have no confidence that this will fix this problem.

DIO Port Config & DIO Port Write Block Diagram Errors (Call Library Function Node:library not found or failed to load)

Hi Guys, need help on this.
I have this LabVIEW program that used to work on the old computer.
The old computer crashes most of the time, so I upgraded the computer
and used its Hard Drive as slave to the new computer.
I have no idea where are its installers since the guy that made the program
is not in my department anymore.
I downloaded all the drivers needed from NI: NIDAQ9.0, NIVISA,NI488.2,
and drivers of some instruments needed in the setup. I'm using LabVIEW8.2.
Everything's fine until I open the LabVIEW program for our testing.
Here goes the error:
   DIO Port Config
   DIO Port Write
Block Diagram Errors
   Call Library Function Node: library not found or failed to load
Attachments:
ErrorList.JPG ‏200 KB

Honestly, I'm a newbie on Labview. I just want this old program to run on the new computer.
The guys that installed the drivers on the old computer are no longer here in my department.
And I have no idea where the drivers are. So I just downloaded the drivers needed for my hardware and instruments.
Here's my hardware: (cards: PCI-DIO-96, PCI-GPIB), (instruments: SCB100,E4407B, HP83623, HP3458, HP8657)
OS: Windows XP Pro
By the way, I have unzipped the TraditionalDAQ drivers. First I tried the 7.4.1, but installation error appeared.
I thought maybe the installer is corrupted, so I downloaded the 7.4.4 and unzipped it.
But, still same installation error appears. I don't understand, both TraditionalDAQ drivers have same installation error.
Now I have tried the DAQmx8.7.2 driver, bu still the DIO Port Config and DIO Port Write have errors.

Error on Mac: couldn't write the application to the hard drive

A Mac user can load the Connect meeting interface but when he
tries to share screen he gets a pop-up that says:
"Couldn't write the application to the hard drive. Please
verify the hard disk is available" with options to retry or cancel.
On retry same error.
Any idea of what is going on and how to fix it?
thanks

Hi rpembroke,
I don't know if this will help you or not. I found the fix for me, for this problem and several others I was having.
I created, or I should say, since I am still working on this, a new user account. Once I created a new account and tried to install no more can't install to harddrive. Many of the other problems I was having are now gone also.
I am in the process of moving over just what I need as far as apps, docs, book marks etc... So far everything is working fine. So you can try creating a new account and see.
Hope this works for you.
Z

How do I write javacard applications and run them in Eclipse 4.2

Greetings-
I have seen some content online about javacard on older versions of Eclipse. I like to ask if it is possible to write javacard applications in Eclipse 4.2? I have an Android development environment, which is basically the bundle that Google provides. I want to know if it is possible for me to use this instance of Eclipse. If not, which version of Eclipse is the one to go with please?

I will recommend you this plugin (JCDE for eclipse)
http://sourceforge.net/projects/eclipse-jcde/
Note that it will only work with JavaCard framework 2.2.2, I am developing applets on Eclipse Kepler (4.2 probably).
If you dont want to use this plugin, I suggest you try to find or write ANT script which would convert your class files into cap. You would be implementing the applet using standard Java project.

WRT600N - Blocked Applications

Can anyone elucidate the method by which Blocked Applications works?
If you set up the "Deny" internet access policy (between days and hours etc.) and list the PC's (Mac addresses) - internet access is indeed restricted within the time noted. However, if you set up the Blocked applications - i.e. filter out the port ranges - it does not work. i.e. if you set up a Test policy that spans the port range of 1-65535 on both UDP and TCP - all access is still granted. There are no restrictions whatsoever. What makes the setup especially confusing is that the top half of the setup page allows a blanket allow or deny of PC's listed access to the internet within a given time range, whereas the bottom half of the page allows you to filter ports, but with no indication as to whether a) this filtering operates within the time restrictions you placed in the policy (this would be moot because if you had set the policy to deny internet anyway and thus this would be redundant), and b) the converse would allow you to have too few choices as the list only allows three blocked applications.
If anyone knows a solution to be able to restrict access to certain ports while leaving the rest open that would be great. PS - my Cisco router at work does not have this problem.

Davet55 wrote:
DD-WRT ???
Still some issues with DD-WRT, not everything is fully functional, yet.
A+ Certified, over 15 years of hands on computer and home networking experience...and I still get lost!

SocketChannel.write() blocking application

Similar Messages

Maybe you are looking for