Optimization problems in downloading data of greater size from any URL.

Question

Hi everyone !
I'm trying to download some resource from the internet by providing valid URL. My code takes almost 50-55 minutes to download. The same resource is downloaded within 3-5 minutes by using Internet Explorer and 3-4 minutes using Fire Fox at the same network and Kbps. The sample code is as under:
//I'm using apache's HTTP APIs
public static String url = "http://www.sk.ee/crls/esteid/esteid.crl"; //file size is approx: 10 MB
//httpMethod is instance of HttpMethodBase and I have initialized it with GET method
obj_httpMethod = new GetMethod( obj_httpUrl.getEscapedURI()
InputStream obj_is = obj_httpMethod.getResponseBodyAsStream();
ByteArrayOutputStream obj_data = new ByteArrayOutputStream();
//above code is working fine and giving proper input stream got from response.
//problem start from here....although this code is working ok but it is not optimized
//Is there any way to optimize the code
MyHttpTransporter.copyStream(obj_is, obj_data, 524288); //Buffer Size: 512K [512*1024]
//This method is working fine with files of small sizes but it is not doing well with greater file sizes e.g. 10MB
public static void copyStream(InputStream a_objIs, OutputStream a_objOs,
                                int a_iblockSize) throws Exception{
    System.out.println( "Buffer Size..: " +  a_iblockSize);
    byte [] byte_Buffer = new byte[a_iblockSize];
    int i_byteRead = -1;
    while ( (i_byteRead = a_objIs.read(byte_Buffer, 0, a_iblockSize)) > -1) {
      System.out.println( "reading stream...." );
      a_objOs.write(byte_Buffer, 0, i_byteRead);
      System.out.println( "writing stream...." );
      System.out.println( i_byteRead );
System.out.println( i_byteRead );
The one thing found in the above code is that, it is ignoring my provided buffer size and always fetching max 2047 bytes from the resource.
Please help me in this regard as I'm unable to progress further in it.
br,
KS

Answer 1

ASC_KS wrote:
Thanx for the help.
It makes some difference.
the sample code is:
//iblockSize = 524288 [512K]
//1
BufferedInputStream bis = new BufferedInputStream(a_objIs, a_iblockSize);
BufferedOutputStream bos = new BufferedOutputStream(a_objOs, a_iblockSize);
//2
BufferedInputStream bis = (BufferedInputStream)a_objIs;
BufferedOutputStream bos = (BufferedOutputStream)a_objOs;
//what do you suggest 1 or 2 ? BTW I'm using 1
If (2) works then behind the scenes the Apache code is already using Buffered streams (otherwise you would get a class cast exception) so there is going to be little advantage to using further buffering. Also, casting to buffered streams does not offer any performance advantage since any reading/writing you do will already be using the methods on the Buffered streams.
Do I need to concentrate on the block size [reduce or increase what is good practice]. Because again it is fetching maximum 5387B and on average it is fetching the 2047B. But it is improved a little bit.Though the buffer size of the Buffered streams can be set when constructed they cannot be changed later so if the Apache code is not setting the buffer size when it constructs the buffer then you are stuffed. Check the Apache documentation to see if you can change the buffering.
P.S. Of course, since it is open source, you can always modify the Apache code to increase the buffer size.
P.P.S. I always use HttpURLConnection for this and it seems pretty quick though I have never downloaded anything beyond a few hundred KBytes.
Edited by: sabre150 on Nov 21, 2007 11:18 AM