Gathering HTML from Google News

Question

Hi there,
Using the following code i'm trying to simply print the source code of http://news.google.co.uk/?ned=uk&topic=n
but i get an IOException, i was wondering if anybody knew why, and if there is a way around it.
public static String newsPage = "http://news.google.co.uk/?ned=uk&topic=n";
try {
            URL url = new URI(newsPage).toURL();
            BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
            String inputLine;
            while((inputLine = in.readLine())!=null) {
                newsHTML = inputLine;
                System.out.println(newsHTML);
        } catch (MalformedURLException e) {System.out.println("Malformed URL Exception");
        } catch (URISyntaxException e) {System.out.println("URI Syntax Exception");
        } catch (IOException e) {System.out.println("IO Exception");
        }Thanks in advance :)

Answer 1

Ok, that returned the following error
Server returned HTTP response code: 403 for URL:
http://news.google.co.uk/?ned=uk&topic=n
java.io.IOException: Server returned HTTP response
code: 403 for URL:
http://news.google.co.uk/?ned=uk&topic=n
at
at
at
at
sun.net.www.protocol.http.HttpURLConnection.getInputSt
ream(HttpURLConnection.java:1149)
at java.net.URL.openStream(URL.java:1007)
at grabber.Main.getHeadlines(Main.java:161)
at grabber.Main.main(Main.java:69)403 is forbidden. I suspect that google is preventing programs such as yours from doing what it is are trying to do.
To circumvent this you will likely have to do something weasely with the HTTP headers. Like fool google into thinking you are IE or Firefox or something.
But I am not going to give more advice than that because the fact it is 403 in the first place strongly suggests that you shouldn't be doing what you are trying to do for legal reasons.