URLConnection with large amount of data causes OutOfMemoryError: Java heap

Question

I am setting up a system where my customers can sent me their data files to my server. If I use a simple socket with a server permanently running on a chosen port, my customers are able to transfer files of any size without problem. However, If I adopt a different architecture, using a web server and a CGI program to receive their submissions, the client program that they run will crash if the file they are sending is any larger than about 30 Mbytes, with the exception OutOfMemoryError: Java heap.
The code in the two architectures is almost identical:
Socket con = new Socket( host, portno);
//URL url = new URL("http://"+host+":"+portno+path);
//URLConnection con = url.openConnection();
//con.setDoOutput(true);
File source_file = new File(file_name);
FileInputStream source = new FileInputStream(source_file);
out = new BufferedOutputStream(con.getOutputStream());
// First, Send a submission header.
data_out = submit_info.getBytes();
len = data_out.length + 1;
data_len[0] = (byte)len;
out.write(data_len, 0, 1);
out.write(data_out, 0, data_out.length);
// Then send the file content.
buff = new byte[(int)buffSize];
content_length = source_file.length();
long tot = 0;
while ( tot < content_length )
// Read data from the file.
readSize = (int) Math.min( content_length-tot, buffSize );
nRead = source.read(buff, 0, readSize);
tot += nRead;
if ( nRead == 0 ) break;
// Send data.
out.write(buff, 0, nRead);
"buffSize" is 4096. This code works fine, but if the first line is commented out and the next three are uncommented, the OutOfMemory exception is thrown within the loop when "tot" is around 30 million.
I have tried calling the garbage collector within the loop but it makes no difference. I am unable to anticipate the size of files that my customers will submit, so I cannot set the heap size in advance to cope with what they will sent. Fortunately, using a simple Socket avoids the problem, so there seems to be something wrong with how URLConnection works.

Answer 1

Set the URLConnection to use chunked mode. This saves it from having to buffer the entire contents before sending to ascertain its length.