How do you keep the same URL connection connected continuously?

Question

I am trying to extract information from a web page, but it puts a cust_id in the middle of the URL. It generates the cust_id randomly everytime you use the site. I connect to the initial URL to find the cust_id, I substring that to the new URL and then make a new connection to extract the information, but the web page gives me an error saying that it doesn't recognise the cust_id. As if it was a new connection with the wrong cust_id.
How can I keep the first connection open and with the same cust_id information and then jump to another web page but using the initial connection. Using IE6 I can just cut and paste the second URL and get to the page, but with my java application it doesn?t work. Please have a look at my code and let me know if I am doing anything wrong.
URL inputURL;
     String inputLine;
     URLConnection connection;
     String page = "";
     int x;
     String cust_id1 = "";
     String cust_id2 = "";
     String site;
     File outputFile = new File("web.txt");
     FileWriter out = new FileWriter(outputFile);
   try
     inputURL = new URL("http://www-ets.woolworths.com.au/ets/owa/login?topage=search");
     connection = inputURL.openConnection();
     BufferedReader inputStream = new BufferedReader(new InputStreamReader(connection.getInputStream()));
     while ((inputLine = inputStream.readLine()) != null)
          page = page + inputLine;
          System.out.println(inputLine);
          x = inputLine.indexOf("cust_id=");
          if (x != -1)
          cust_id1 = inputLine.substring(x, x + 14);
          PrintWriter file = new PrintWriter(new FileOutputStream("web.txt"));
          out.write(page);
          out.close();
          System.out.println("1st time successful");
          System.out.println(cust_id1);
          //inputStream.close();
          //extracted the customer id, now able to search for the product
          //test for milk
     site = "http://www-ets.woolworths.com.au/ets/owa/items?search_pattern=milk&";
     site = site + cust_id1;
     site = site + "&imode=search&search_option=";
     System.out.println(site);     
     inputURL = new URL(site);
     BufferedReader inagain = new BufferedReader(new InputStreamReader(inputURL.openStream()));
     //tried the connection way
     //connection = inputURL.openConnection();          
     //BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
     while ((inputLine = inagain.readLine()) != null)
          //System.out.println(inputLine);          
          x = inputLine.indexOf("cust_id=");
          if (x != -1)
          cust_id2 = inputLine.substring(x, x + 14);
          System.out.println(cust_id1);
          System.out.println(cust_id2);
          System.out.println("2nd time completed");
          inputStream.close();
   catch(ArrayIndexOutOfBoundsException obe)
            System.out.println("Usage: FetchURL url");
   catch(IOException ioe)
            System.out.println("IO Exception");

Answer 1

Please have a quick look at my code and see if I am doing anything wrong. I havent been able to set the correct cookies and the output html page comes up with an error saying that it could not retrieve my request. This time there is no mention of invalid cust_id.
          try
               URL inputURL = new URL("http://www-ets.woolworths.com.au/ets/owa/login?topage=search");
               connection = inputURL.openConnection();
               System.out.println("filename = " + inputURL.getFile());
               BufferedReader inputStream = new BufferedReader(new InputStreamReader(connection.getInputStream()));
               while ((inputLine = inputStream.readLine()) != null)
                    //System.out.println(inputLine);
                    page = page + inputLine;     
                    x = inputLine.indexOf("cust_id=");
                    if (x != -1)
                         cust_id = inputLine.substring(x + 8, x + 14);
               out.write(page);
               out.close();
               System.out.println("extracted cust_id: " + cust_id);
               for (int i = 1; i <= 30; ++i)
                    System.out.println("    Header " + i + ":  " + connection.getHeaderFieldKey(i) + ": " + connection.getHeaderField(i));
                    header = header + connection.getHeaderFieldKey(i);
                    header = header + ": ";
                    header = header + connection.getHeaderField(i);
               outcookie.write(header);
               outcookie.close();           
               inputStream.close();          
               //extracted the customer id, now able to search for the product
               //test for milk
               String url2 = "http://www-ets.woolworths.com.au/ets/owa/items";
               //String url2 = "http://www-ets.woolworths.com.au/ets/owa/login?topage=search";
               inputURL = new URL(url2);     
               connection = inputURL.openConnection();     
               connection.setDoOutput(true);
               //connection.setUseCaches(false);
               System.out.println("filename = " + inputURL.getFile());
               String date = "; expires=Mon, 28 Jan 2002 02:53:17GMT; path/;";     
               String date1 = "expires=Sun, 31 Dec 1989 14:00:00 GMT; path/;";
               connection.setRequestProperty("cookie", "qwc=" + cust_id + date);
               connection.setRequestProperty("cookie", "cust_id=" + cust_id + "; path=/;");
               connection.setRequestProperty("cookie", "cd=2%40142%40%40%40%40N%40; path=/;");
               connection.setRequestProperty("cookie", "state=NSW; path=/;");
               connection.setRequestProperty("cookie", "dsd=1; path=/;");
               connection.setRequestProperty("cookie", "sb1=0%2A%2D645746%2D0%2A; path=/;");
               connection.setRequestProperty("cookie", "sb2=0; " + date1);
               connection.setRequestProperty("cookie", "sb3=0; " + date1);
               connection.setRequestProperty("cookie", "sb4=0; " + date1);
               connection.setRequestProperty("cookie", "sb5=0; " + date1);
               connection.setRequestProperty("cookie", "sb6=0; " + date1);
               connection.setRequestProperty("cookie", "sb7=0; " + date1);
               connection.setRequestProperty("cookie", "sb8=0; " + date1);
               connection.setRequestProperty("cookie", "sb9=0; " + date1);
               connection.setRequestProperty("cookie", "sb10=0; " + date1);
               connection.setRequestProperty("cookie", "sbn=; path=/");
               connection.setRequestProperty("cookie", "search=milk; " + date1);               
               //connection.setRequestProperty("Content-type", "application/octet-stram");
               //connection.setRequestProperty("Content-type", "application/x-www-form-urlencoded");
               //String parms = "?search_pattern=" + URLEncoder.encode("milk") + "&cust_id=";
               //parms = parms + URLEncoder.encode(cust_id);
               //parms = parms + "&imode=" + URLEncoder.encode("search") + "&search_option=";
               //System.out.println("parms : " + parms);
               String parms = "?search_pattern=milk&cust_id=";
               parms = parms + cust_id;
               parms = parms + "&imode=search&search_option=";
               System.out.println("parms : " + parms);
               PrintWriter outStream = new PrintWriter(connection.getOutputStream());
               outStream.println("parms");
               outStream.close();               
               BufferedReader inputStream1 = new BufferedReader(new InputStreamReader(connection.getInputStream()));     
               //BufferedReader inagain = new BufferedReader(new InputStreamReader(connection.getInputStream()));
               while ((inputLine = inputStream1.readLine()) != null)
                    //System.out.println(inputLine);
                    page2 = page2 + inputLine;
                    x = inputLine.indexOf("cust_id=");
                    if (x != -1)
                         cust_id2 = inputLine.substring(x + 8, x + 14);
               out2.write(page2);
               out2.close();
               System.out.println("2nd attempt cust_id" + cust_id2);
               for (int i = 1; i <= 30; ++i)
                    System.out.println("    Header " + i + ":  " + connection.getHeaderFieldKey(i) + ": " + connection.getHeaderField(i));
                    header2 = header2 + connection.getHeaderFieldKey(i);
                    header2 = header2 + ": ";
                    header2 = header2 + connection.getHeaderField(i);
               outcookie2.write(header2);
               outcookie2.close();      
               inputStream.close();
          }The original web page to get the cookie is
http://www-ets.woolworths.com.au/ets/owa/login?topage=search
Enter a search product eg milk and that is the second page I am trying to extract.
You will notice in the middle of the url is cust_id.
Am I going about this 2 pages completely wrong?