Parsing Content From A Site That Requires Authentication

Question

Ive been scanning websites using the Java HTML parser from htmlparser.sourceforge.net to gather useful data into a more easily used format, in this case it is stored in a mySQL database.
The problem that has stumped me for the past few days is how to get around the login authenication page required to access content from the website www.racingpost.com . I am a registered user but am having difficulty logging in via Java and managing the cookies to bypass the login page HTML I keep getting instead of the race data.
I am unsure when and where I should be passing cookies around. Also logging in using POST is confusing me.
Thanks for any help you can offer.

Answer 1

this should help you out with posting to a URL. It is taken from:
http://javaalmanac.com/egs/java.net/Post.html
// e135. Sending a POST Request Using a URL
    try {
        // Construct data
        String data = URLEncoder.encode("key1", "UTF-8") + "=" + URLEncoder.encode("value1", "UTF-8");
        data += "&" + URLEncoder.encode("key2", "UTF-8") + "=" + URLEncoder.encode("value2", "UTF-8");
        // Send data
        URL url = new URL("http://hostname:80/cgi");
        URLConnection conn = url.openConnection();
        conn.setDoOutput(true);
        OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
        wr.write(data);
        wr.flush();
        // Get the response
        BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
        String line;
        while ((line = rd.readLine()) != null) {
            // Process line...
        wr.close();
        rd.close();
    } catch (Exception e) {
    }