Converting html forms to java

Question

having viewed an earlier post in this forum i managed to create the following code but am having a few problems.
here are the active parts of the html form on googles search, engine. i am trying to convert this into java form and have so far succeeeded in generating the code shown below, but keep getting the following error
"IOException; Server returned HTTP response code: 501 for URL: http://www.google.co.uk/search"
the form:
<form action=/search name=f>
<input type=hidden name=hl value=en>
<input maxlength=2048 size=55 name=q value="" title = "Google Search">
<input type=submit value="Google Search" name=btnG>
<input type=submit value="I'm Feeling Lucky" name=btnI>
</form>
in the following code baseURL is the domain of the server... in this case http://www.google.co.uk/, method is the method as defined in the form method attirbute, if the attribute is not there, as in this form the default method is called which is GET
            URL url;
            HttpURLConnection urlConn;
            DataOutputStream printout;
            BufferedReader input;
            if(action.startsWith("http")) {
                url = new URL(action);
            } else if(action.startsWith("/")) {
                url = new URL(baseURL+action.substring(1,action.length()));
            } else {
                url = new URL(baseURL+action);
            urlConn = (HttpURLConnection) url.openConnection();
            urlConn.setDoInput(true);
            urlConn.setDoOutput(true);
            urlConn.setUseCaches(false);
            //      set request method
            urlConn.setRequestMethod(method);
            //      set request type
            urlConn.setRequestProperty("Content-Type",
                    "application/x-www-form-urlencoded");
            //       data-value pairs are separated by &
            String content ="hl=en&q="
            // imitate someone searching for "hello" in i feel lucky
            content+="hello&btnI=I'm Feeling Lucky";
            urlConn.setRequestProperty("Content-Length", content.length() + "");
            //       Send POST output.
            printout = new DataOutputStream(urlConn.getOutputStream());
            printout.writeBytes(content);
            printout.flush();
            printout.close();
            //       Get response data.
            input = new BufferedReader(new InputStreamReader(urlConn.getInputStream()));
            String str;
            while (null != ((str = input.readLine())))
                System.out.println(str);
            input.close();
        } catch (MalformedURLException me) {
            System.err.println("MalformedURLException; " + me);
        } catch (IOException ioe) {
            System.err.println("IOException; " + ioe.getMessage());
            ioe.printStackTrace();
        }i also tried self adding in the question mark that appears after the search text bu again get the same error. any ideas what im doing wrong when connecting to the server? thanks in advance for any help

Answer 1

Just saw this topic on the forum,
For each web crawlers the web sites maintain a file called Robots.txt which can be assessed by : http://www.google.co.uk/robots.txt
You would see following on that file :
User-agent: *
Allow: /searchhistory/
Disallow: /search
Disallow: /groups
which means that crawlers are not allowed to scrap this site on search, due to which our code wasnt working
Thanks