Retrieving cookies when logging into java forums through a java application

Question

Hello --
I am currently working on a spider application that will have the ability to login to sites of interest and then begin crawling.
I thought an easy place to start would be to login to java forums. However, I have run into a bit of a snag with all of the cookies....
The following is the information that I am posting:
action = "login"
url = "http://forum.java.sun.com/index.jsp"
UserId     // my username
Password // my password
Submit = "Log in"
AutoLogin = "1"
I am doing this using HttpURLConnection that was created using http://developer.java.sun.com/servlet/SessionServlet as the URL. Then I create an output stream etc:
OutputStream os = huc.getOutputStream();
os.write(sb.toString().getBytes("US-ASCII")); // where sb is a StringBuffer which contains the above information separated with &'s
os.close();
This is when I run into my problem...Next, I try to obtain the session cookies (using the same HttpURLConnection that I posted with) but I am not getting all of the cookies.
As a test, I set my browser to prompt for cookies and I found that when you submit your login information three cookies are set: JDC, JSessionId, and SessionCredentials. Then two more cookies are set JSessionId again and jiveUserLastVisited -- I think these cookies are actually set while getting the http://forum.java.sun.com/index.jsp page. Anyway the only cookies that I get through my program are JSessionId and jiveUserLastVisited.
So to remedy this I tried setting followRedirects to false on my HttpURLConnection (before I perform my post) and then I tried to get the Set-Cookie header immediately following my post. What I found was after calling getFieldHeader() my connection has changed -- when I do a getConnection on my HttpURLConnection the connection has changed to the second page (http://forum.java.sun.com/index.jsp) even with redirects set to false.
I guess my inexperience with web applications has caught up with me. Does anybody have any ideas?
Thanks,
Lisa

Answer 1

Lisa
Did you get any reply on this ?? I am also writing a similar web crawler and i also ran into similar problems.
let me know if you found the solution. thanks.
-P