Can't fully download long web-pages using HttpURLConnection

Question

I'm using these two methods:
/**keeps document encoding or sets Cp1251 as default
      * @param uri is uri to document
      * @return {@link InputStreamReader} with encoding
     public static InputStreamReader getInputStreamReader(String uri) throws IOException{
          URL url = new URL(uri);
          HttpURLConnection urlConn = (HttpURLConnection)url.openConnection();
          urlConn.setConnectTimeout(timeout);
          urlConn.setDefaultUseCaches( false );
        urlConn.setUseCaches( false );
        urlConn.setRequestProperty( "Cache-Control", "max-age=0,no-cache" );
        urlConn.setRequestProperty( "Pragma", "no-cache" );
        String encoding = urlConn.getContentEncoding();
        if(encoding == null || encoding.equals("") ){
             String Content_Type = urlConn.getHeaderField("Content-Type");
             if(Content_Type.contains("charset=")){
                 String[] t = Content_Type.split("charset=");
                 if(t != null && t.length>0){
                      encoding = t[t.length-1];
        if(encoding == null || encoding.equals("")){
             encoding = "windows-1251";
        InputStreamReader isr = new InputStreamReader(urlConn.getInputStream(), encoding);
        return isr;
     /**Returns content in default charset
      * @param uri is uri to document you want to get
      * @return String with encoding equal to {@link Charset.defaultCharset()}
     public static String fetchEncodedURI(String uri) throws IOException{
          InputStreamReader isr = getInputStreamReader(uri);
          ByteArrayOutputStream baos = new ByteArrayOutputStream();
          OutputStreamWriter osw = new OutputStreamWriter( baos, Charset.defaultCharset());
          char[] buffer = new char[256];
          int n = -1;
          while ( (n = isr.read(buffer))!= -1 ){
               osw.write(buffer, 0, n);
        String outputString = new String(baos.toByteArray());
        osw.flush();
        osw.close();
        baos.flush();
        baos.close();
        return outputString;
     }The problem is that some pages are cut! I make output of result string (outputString) ans see for example just 2/3 of the real page content.
Is it possible to overcome this difficulty?
Edited by: Holod on 01.10.2008 11:31

Answer 1

ejp wrote:
Personally, I'd suggest reading the whole page as bytes into baos and covert to string with one call to boas.toString(encoding)Or use a StringWriter. The way you're doing it is the worst of both worlds.Can you help me please with a sample link?
Or please tell, is it right solution?
         /**Returns content in default charset
      * @param uri is uri to document you want to get
      * @return String with encoding equal to {@link Charset.defaultCharset()}
     public static String fetchEncodedURI(String uri) throws IOException{
          InputStreamReader isr = getInputStreamReader(uri);
          ByteArrayOutputStream baos = new ByteArrayOutputStream();
          //OutputStreamWriter osw = new OutputStreamWriter( baos, Charset.defaultCharset());
          StringWriter sw = new StringWriter();
          char[] buffer = new char[256];
          int n = -1;
          int previousPosititon = 0;
          while ( (n = isr.read(buffer))!= -1 ){
               sw.write(buffer,previousPosititon,n);
               previousPosititon+=n;
               //osw.write(buffer, 0, n);
        //osw.flush();
     //String outputString = new String(baos.toByteArray());
        String outputString = sw.toString();
        //osw.close();
        baos.flush();
        baos.close();
        return outputString;
     }Edited by: Holod on 03.10.2008 5:41