Download big files

Question

I am trying to download a big file from the internet. Currently I am doing the following:
            URLConnection conn = downloadURL.openConnection();
            InputStream in = conn.getInputStream();
            OutputStream out = new FileOutputStream(dst);
            // Transfer bytes from in to out
            byte[] buf = new byte[SIZE];
            int len;
            while ((len = in.read(buf)) > 0 && !cancelled) {
                out.write(buf, 0, len);
            }That code works for me most of the time, but sometimes the file is not downloaded correctly and I do not know how to test if the download was completed or not, and I do not know how to guarantee that the file is downloaded completely.
Is there some way to make this?
Greetings,
Magus

Answer 1

That statement makes no sense. I'm programming in Java, and Java has a fine mechanism for throwing exceptions.No, that statement makes no sense. Your Java code is talking TCP/IP to a server, which doesn't have such a mechanism. It can close or reset the connection; that's it. If it had reset the connection, your read() would block forever, or timeout. If it had closed the connection, your read() would have returned -1, as it did, so that is what happened.
1. Can you expound upon what makes you think that is the reason that -1 was returned, as opposed to the connection timing out, the connection dropping, etc.?Because no exception was thrown. Or else one was thrown and you have swallowed it, but the code you posted doesn't indicate that.
2. Can you indicate why Java would return a -1 instead of throwing a some I/O exception?Because the server, or possibly an intermediate firewall, closed the connection, rather than Java incurring some exception at the client end such as a timeout. The documentation you quoted at me before bears that out. That's what the -1 means. You quoted that at me yourself.
The whole point here is that, according to the API documentation, -1 apparently indicates that the end of the stream was reached normally, and that error conditions are indicated by exception.Exactly so. So the server closed the connection normally, or an intermediate firewall did, but before it had sent all the data. Why is another question. Have a look at the server logs, or investigate the firewall configuration.
I guess I'm asking if anyone has seen this behavior, and has any insight on why it doesn't seem to follow the API.It does.
you seem to think that this behavior is in line with the API documentation; I disagree.Well you've quoted enough of it yourself: have another look, or another think. Premature (but normal) closing of the connection by the server or a firewall is the only possible explanation for what you're seeing. If it wasn't closed you wouldn't be getting the -1; if there was a timeout you would get a SocketTimeoutException; if there was any other exception you would catch that.
I've seen plenty of short downloads in my life, but the server doesn't have a way of indicating that via TCP/IP. You have to count.
NB some firewalls do close long-lived connections on their own account. Is there a client-side firewall that might be doing that.
Or else the transmitted content-length is wrong; for example, it overflows an integer.