Uploading large files using nio in http client

Question

Hi,
I'm developing a multithreaded Swing client which needs to be capable of uploading and downloading large ( 100mb ) files to servlets via http.
Downloading has presented no problem, but writing to the OutputStream of a URLConnection, whilst fine for small files, gave me 'out of memory' errors on large ones. ( As I understand it, the whole stream is buffered before sending and it just runs out of room.)
So I replaced the URLConnection with a SocketChannel, and after writing the http header, used a loop to write chunks of data from a FileChannel.
I needed to monitor any incoming data to truncate the upload if the servlet gives a premature response indicating an error condition, so I set the SocketChannel to non-blocking and put a read into the loop.
Immediately I got a bunch of Swing exceptions and the frame was unable to render itself, I guess due to thread starvation problems.
I currently have this clunky code that seems to work ok:
int percent=0;
long total = 0L;
long chunk = 1024*8;
while(total<length ){
chunk = ((length-total)<chunk)?length-total:chunk;
long ii= fc.transferTo(total,chunk,socketChannel);
total+=ii;
yield();
socketChannel.configureBlocking(false);
if(socketChannel.read(buffer)>0 )break;
socketChannel.configureBlocking(true);
What I can't figure out is why Swing objects if I permanently set the SocketChannel to non-blocking rather than just whilst I read it, and why I get the following exception from a polling thread that runs concurrently:
'No buffer space available (maximum connections reached?)'
Anyone got any ideas or a better way to do it?
Chris

Answer 1

I assume that the OutputStream obtained fromURLConnection is internally buffered and URLConnection
waits until the whole stream has been input before
inserting the initial Content-length header. Only when
it knows the complete length is the content sent. Hmm, this is quite unfortunate. Especially since it is not necessary: HTTP allows the data stream close to serve as the end-of-data marker (Content-Length: is an optional header, in its absence the receiver simply reads all of the data until the sender closes the stream).
Has anyone else had experience of sending large
(100mb) files through Java http clients on small
pc's?I've sent large-ish (~1-10Mb) payloads via URLConnection but not as large as yours. Interesting issue to figure out... I am out of ideas for the moment.