Problem using HttpURLConnection

Question

Hi
I am using HttpURLConnection,to download the contents from the URL,in my application.
When only a limitted no of threads are started then it is working fine and the stuffs are getting downloaded into a file as per my requirement However if if sponning a large number of thread or if the http server is slow then
instead of downloading the contents slowly...the inputsteam is showing eof and the downloading is completed without actual completino.Hence i am failing to dowload all the contents{color:#008000}.(The application requires to start around 100 threads and download the contents of all the urls into a single file){color}
Following code has been used :
{color:#3366ff}+url = new URL(httpUrl);+
conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setAllowUserInteraction(false);
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.connect();
appendToTempFile("");
appendToTempFile("");
appendToTempFile("Downloading " httpUrl);+
in = conn.getInputStream();
BufferedReader reader =BufferedReader(new InputStreamReader(in));
String text;
+while ((text = reader.readLine()) != null) {+
if (!text.startsWith(" "))
appendToTempFile(text);
+}+
+}+
conn.disconnect();
*{color:#000000}Please suggest me some solution.{color}*
{color}

Answer 1

1.
Are you writing to the file with all threads at the same time? Then this could be a problem. My approach would be a consumer/producer pattern, where the download threads are the producers. They download the data and store it at a central place. Another thread, the consumer, fetches the data from this central place and writes it the file. This way you only have one thread, which is working with the file and you don't get problems like one thread is overwriting the data of another thread.
2.
If you start many download threads, there yould be limitations, which could result in timeouts or something like that. Some servers only allow a defined number of connections from one IP. Same goes for your browser. Firefox for example can be configured to open only a specified number of connections to one server. There are some os settings, which could be limitating, too. So you should try to not use too many threads. You could use a thread pool, for example.