HttpURLConnection and HEAD/GET methods

Question

I am attempting to validate whether an HTML page exists or not.
I have found that, for about 7% of the pages checked, HEAD and GET methods return different response codes.
I have structured my code such:
1) make initial check using HEAD method
2) for non valid (200) response codes, recheck the page using the GET method.
In this case about 75% of the pages that failed using the HEAD method will work when using the GET method.
So, I guess my questions are:
1) Does anybody know why HEAD/GET return different response codes?
2) Does anybody know a better way to check if a page exists?
Here is the sample program I am using with a few URLs that exhibit this behaviour:
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.InetAddress;
import java.net.URL;
import java.net.UnknownHostException;
public class Internet
     private final static String DEFAULT_LOCAL_HOST = "127.0.0.1";
     private URL url;
     private HttpURLConnection http;
     private int responseCode;
     private String responseMessage;
     public Internet(URL url)
          this.url = url;
     public boolean isValid()
          try
               //  Make first attempt using a HEAD request
               http = (HttpURLConnection)url.openConnection();
               http.setRequestMethod( "HEAD" );
               http.connect();
               System.out.println( "head: " + http.getResponseCode()
               + " : " + http.getResponseMessage() );
               //  GET seems to do a better job, try again
               if ( http.getResponseCode() != HttpURLConnection.HTTP_OK)
                    http = (HttpURLConnection)url.openConnection();
                    http.setRequestMethod( "GET" );
                    http.connect();
                    System.out.println( "get:  " + http.getResponseCode() );
               responseCode = http.getResponseCode();
               responseMessage = http.getResponseMessage();
               if (http.getResponseCode() == HttpURLConnection.HTTP_OK)
                    return true;
               else
                    System.out.println( http.getResponseMessage() );
                    return false;
          catch (IOException e)
               responseCode = -1;
               responseMessage = e.getMessage();
               System.out.println( e );
               return false;
     public static void main(String[] args)
          throws Exception
          URL url = new URL( "http://www.trca.on.ca" );
          Internet internet = new Internet( url );
          System.out.println( internet.isValid() );
          url = new URL( "http://school.discovery.com/sciencefaircentral" );
          internet = new Internet( url );
          System.out.println( internet.isValid() );
          url = new URL( "http://www.amazon.com" );
          internet = new Internet( url );
          System.out.println( internet.isValid() );
}

Answer 1

Using my sample program:
1) about 3K of data is transferred
2) it runs in about 8 seconds
Using InputStream in = http.getInputStream():
1) about 73K of data is transferred
2) it runs in about 15 seconds
Using the getInputStream() method causes the entire file to be transmitted (even though I don't use any read() methods). I don't care about the data.
I want the check to be as fast as possible which is why I use the HEAD method. It doesn't transfer the contents of the file. But why in some cases does it show a response code other than 200?