Transfer monitor with concurrency issues

Question

I'm currently working on a project where the user may choose a domain name from a source list to fetch a bunch of subpages. To the user this looks like a single, large download. Since this may or may not take a substantial amount of time each download is attached to a TransferMonitor instance that reports the currently downloaded datasize and transfer rate back to the GUI. This is working fine when there is only one active download, but if there are several simultaneous downloads the monitor fails. After digging into it a bit I can see how it fails, but I think my limited experience with threads are keeping me from seeing why and what I can do to correct it. To illustrate the problem I've created a very simple fetcher:
public class Fetcher implements Runnable {
    private int bytes = 0;
    private int millis = 0;
    public void run() {
        for (int i = 0; i < 50; i++) {
            HttpURLConnection connection = null;
            InputStreamReader reader = null;
            try {
                // Setup
                connection = (HttpURLConnection)new URL(
                                    URI).openConnection();
                StringWriter writer = new StringWriter();
                reader = new InputStreamReader(
                    connection.getInputStream());
                // Process
                char[] buffer = new char[4096];
                int count = 0, n = 0;
                long start = System.currentTimeMillis();
                while ((n = reader.read(buffer)) != -1) {
                    writer.write(buffer, 0, n);
                    count += n;
                // Report
                millis += System.currentTimeMillis() - start;
                bytes += count;
            } catch (MalformedURLException e) {
            } catch (IOException e) {
            } finally {
                // Finalize
                IOUtils.closeQuietly(reader);
                connection.disconnect();
        System.out.println(Thread.currentThread().getName() + "\n" +
                        "Transfer rate: " + new DecimalFormat("0.00",
                        new DecimalFormatSymbols(Locale.UK)).format(
                        ((long)bytes * 1024) / (millis * 1000)) +
                        " kb/s\nAvg millis/cycle: " + millis / 50 + "\n");
}This class fetches the same webpage over and over (50 times/cycles). My real one of course does not. Finally it prints the average transfer rate and the number of milliseconds used per cycle. The following code is used to run this fetcher:
public static void main(String[] args) throws InterruptedException {
    long start = System.currentTimeMillis();
    Thread thread1 = new Thread(new Fetcher(), "Thread 1");
    thread1.start();
    thread1.join();
    long end = System.currentTimeMillis();
    System.out.println("Total millis: " + (end - start));
}Gives me the following print out:
Thread 1
Transfer rate: 908.00 kb/s
Avg millis/cycle: 58
Total millis: 11343Considering my bandwidth is 12 mbit (~1500 kb/s) it seems a bit low. How come?
Either way I've learned to live with this so I'll try using the fetcher with several threads at the same time:
public static void main(String[] args) throws InterruptedException {
    long start = System.currentTimeMillis();
    Thread thread1 = new Thread(new Fetcher(), "Thread 1");
    thread1.start();
    Thread thread2 = new Thread(new Fetcher(), "Thread 2");
    thread2.start();
    Thread thread3 = new Thread(new Fetcher(), "Thread 3");
    thread3.start();
    Thread thread4 = new Thread(new Fetcher(), "Thread 4");
    thread4.start();
    Thread thread5 = new Thread(new Fetcher(), "Thread 5");
    thread5.start();
    thread1.join();
    thread2.join();
    thread3.join();
    thread4.join();
    thread5.join();
    long end = System.currentTimeMillis();
    System.out.println("Total millis: " + (end - start));
}Gives me the following print out:
Thread 5
Transfer rate: 728.00 kb/s
Avg millis/cycle: 72
Thread 4
Transfer rate: 704.00 kb/s
Avg millis/cycle: 75
Thread 3
Transfer rate: 678.00 kb/s
Avg millis/cycle: 78
Thread 1
Transfer rate: 716.00 kb/s
Avg millis/cycle: 74
Thread 2
Transfer rate: 681.00 kb/s
Avg millis/cycle: 77
Total millis: 13109The number of milliseconds per cycle per thread has only increased from 58 to ~75 making the five threads combined only 15% slower to fetch five times the content - with a combined transfer rate of ~3500 kb/s, which of course is not right. If I throttle the bandwidth at e.g 30 kb/s using NetLimiter it actually reports the correct rates (~6 kb/s per thread). As already mentioned I probably do not yet have a good enough understanding of threads (one may fetch while another is busy doing something else etc.).
The question is; How can I set this up so that it reports the correct transfer rate independantly of the number of threads fetching? Perhaps there even exists an API that can do this for me?

Answer 1

HTTP networks have caching proxies, so repeated requests to the same URL don't have to travel neyond the cache, until the page expires. Your test is at best fetching the page from the target server once and then getting it from the nearest cache 49 times, or 499 times. That's OK as long as you're aware that that is what you're measuring, and as long as that situation has some correspondence with reality in your actual application.Certainly a good point and something I did not considerate. The Fetcher below is simplified, but works fairly similar to the real thing. It extracts data from a Calendar object to compute the URL. At the end of each cycle one hour is added to the Calendar:
public class Fetcher implements Runnable {
    private Calendar c;
    private int bytes = 0;
    private int millis = 0;
    public Fetcher(Calendar c) {
        this.c = c;
    private HttpURLConnection getConnection() throws IOException {
        // Return a connection from a URL computed using c
    public void run() {
        for (int i = 0; i < 500; i++) {
            HttpURLConnection connection = null;
            InputStreamReader reader = null;
            try {
                // Setup
                connection = getConnection();
                StringWriter writer = new StringWriter();
                reader = new InputStreamReader(
                        connection.getInputStream());
                // Read-Write
                char[] buffer = new char[4096];
                int count = 0, n = 0;
                long start = System.currentTimeMillis();
                while ((n = reader.read(buffer)) != -1) {
                    writer.write(buffer, 0, n);
                    count += n;
                // Report
                millis += System.currentTimeMillis() - start;
                bytes += count;
                } catch (MalformedURLException e) {
                } catch (IOException e) {
                } finally {
                    // Finalize
                    IOUtils.closeQuietly(reader);
                    connection.disconnect();
                    c.add(Calendar.HOUR_OF_DAY, 1);
            System.out.println(Thread.currentThread().getName() + "\n" +
                                    "Transfer rate: " + new DecimalFormat("0.00",
                                    new DecimalFormatSymbols(Locale.UK)).format(
                                    ((long)bytes * 1024) / (millis * 1000)) +
                                    " kb/s\nAvg millis/cycle: " + millis / 500 + "\n");
}However, this modification did not change the result.
Your figures also suggest that each transfer is 200k. Is that realistic?Based on the figure your calculations are correct, but I knew for a fact that was not the case. So I ran the code again (single thread) and the value of 'millis' turns out to be ~28 sec giving me a ~933 kb/s rate. (933 kb/s * 28 sec) / 500 cycles = ~52kb per transfer/webpage which is the right size. So to correct myself; When you asked me two posts above if the size of all transferred data was 10MB it was infact about 2,5MB, but you could not have known that.
This got me really puzzled. The total time spent is 113 sec (measured in the main method), but only 28 of them (given by variable millis) were spent actually fetching - so what is the actual transfer rate? And where has the remaining 85 seconds gone? Is the other stuff really that expensive? Or are my measurements wrong? I think I need some concrete suggestions instead of just guessing on my own here.
In your parallel thread case, you are transferring 5 times the data in the multi-threaded case, so the basis for comparison is rather invalid. It might be more interesting to time 5 x thread.run() as against 5 x thread.start() followed by 5 x thread.join().You'll have to spoon feed me this one. Anyway, at this point I don't even know if my rate is correct for a single thread let alone multi-threaded.