Uploading Very Large Files via HTTP

Question

I am developing some classes that must upload files to a web server via HTTP and multipart/form-data. I am using Apache's Tomcat FileUpload library contained within the commons-fileupload-1.0.jar file on the server side. My code fails on large files or large quantities of small files because of the memory restriction of the VM. For example when uploading a 429 MB file I get this exception:
java.lang.OutOfMemoryError
Exception in thread "main"I have never been successful in uploading, regardless of the server-side component, more than ~30 MB.
In a production environment I cannot alter the clients VM memory setting, so I must code my client classes to handle such cases.
How can this be done in Java? This is the method that reads in a selected file and immediately writes it upon the output stream to the web resource as referenced by bufferedOutputStream:
private void write(File file) throws IOException {
  byte[] buffer = new byte[bufferSize];
  BufferedInputStream fileInputStream = new BufferedInputStream(new FileInputStream(file));
  // read in the file
  if (file.isFile()) {
    System.out.print("----- " + file.getName() + " -----");
    while (fileInputStream.available() > 0) {
      if (fileInputStream.available() >= 0 &&
          fileInputStream.available() < bufferSize) {
        buffer = new byte[fileInputStream.available()];
      fileInputStream.read(buffer, 0, buffer.length);
      bufferedOutputStream.write(buffer);
      bufferedOutputStream.flush();
    // close the files input stream
    try {
      fileInputStream.close();
    } catch (IOException ignored) {
      fileInputStream = null;
  else {
    // do nothing for now
}The problem is, the entire file, and any subsequent files being read in, are all being packed onto the output stream and don't begin actually moving until close() is called. Eventually the VM gives way.
I require my client code to behave no different than the typcial web browser when uploading or downloading a file via HTTP. I know of several commercial applets that can do this, why can't I? Can someone please educate me or at least point me to a useful resource?
Thank you,
Henryiv

Answer 1

Are you guys suggesting that the failures I'm
experiencing in my client code is a direct result of
the web resource's (servlet) caching of my request
(files)? Because the exception that I am catching is
on the client machine and is not generated by the web
server.
trumpetinc, your last statement intrigues me. It
sounds as if you are suggesting having the client code
and the servlet code open sockets and talk directly
with one another. I don't think out customers would
like that too much.Answering your first question:
Your original post made it sound like the server is running out of memory. Is the out of memory error happening in your client code???
If so, then the code you provided is a bit confusing - you don't tell us where you are getting the bufferedOutputStream - I guess I'll just assume that it is a properly configured member variable.
OK - so now, on to what is actually causing your problem:
You are sending the stream in a very odd way. I highly suspect that your call to
buffer = new byte[fileInputStream.available()];is resulting in a massive buffer (fileInputStream.available() probably just returns the size of the file).
This is what is causing your out of memory problem.
The proper way to send a stream is as follows:
     static public void sendStream(InputStream is, OutputStream os, int bufsize)
                 throws IOException {
          byte[] buf = new byte[bufsize];
          int n;
          while ((n = is.read(buf)) > 0) {
               os.write(buf, 0, n);
     static public void sendStream(InputStream is, OutputStream os)
                 throws IOException {
          sendStream(is, os, 2048);
     }the simple implementation with the hard coded 2048 buffer size is fine for almost any situation.
Note that in your code, you are allocating a new buffer every time through your loop. The purpose of a buffer is to have a block of memory allocated that you then move data into and out of.
Answering your second question:
No - actually, I'm suggesting that you use an HTTPUrlConnection to connect to your servlet directly - no need for special sockets or ports, or even custom protocols.
Just emulate what your browser does, but do it in the applet instead.
There's nothing that says that you can't send a large payload to an http servlet without multi-part mime encoding it. It's just that is what browsers do when uploading a file using a standard HTML form tag.
I can't see that a customer would have anything to say on the matter at all - you are using standard ports and standard communication protocols... Unless you are not in control of the server side implementation, and they've already dictated that you will mime-encode the upload. If that is the case, and they are really supporting uploads of huge files like this, then their architect should be encouraged to think of a more efficient upload mechanism (like the one I describe) that does NOT mime encode the file contents.
- K