HttpURLConnection and HtmlUnit

Hello,
I am using these commands to get a webpage from my in-house webserver:
String urlStr = "http://localhost/welcome.jsp";
URL url = new URL(urlStr);
HttpURLConnection ucon = (HttpURLConnection)url.openConnection();
ucon.connect();
InputStreamReader isr = new InputStreamReader(ucon.getInputStream());
BufferedReader in = new BufferedReader(isr);
I then read the file line by line, searching the text for this input tag:
<input type=hidden name=pagecode value=123>
Is there any easier way to get the value of this hidden variable?
Does the HttpURLConnection have some way of extracting hidden values directly?
Thank you for helping.

I put the htmlunit-2.11.jar file in the web-inf/lib folder.
I added this line to my standard java file:
import com.gargoylesoftware.htmlunit.WebClient;
I tried to rebuild the java file and got this error:
cannot access com.gargoylesoftware.htmlunit.WebClient;
bad class file web-inf/lib/htmlunit.jar
class file has wrong version 50.0, should 49.0
please remove or make sure it appears in the correct subdirectory of the classpath
Any suggestions are greatly appreciated.

Similar Messages

Problem with HttpURLConnection and HttpSession.

Hi,
Problem with HttpURLConnection and HttpSession.
I created a HttpSession object in my main class and then called a URL via HttpURLConnection.
I tried to access the same session object from the servlet called by the URL . but the main session
is not returned a new seperate session is created.let me know how can we continue the same session in
the called URL also which is created in main class.
Thanks
Prasad

You are not supported to create a HttpSession by java client. Only J2EE web container can create it.

HttpURLConnection and URLConnection

hi,
what's the difference between HttpURLConnection and URLConnection ?
When should I use the first and when the second one ?
thanks

The Javadoc shows the relationship. Cast a URLConnection to an HttpURLConnection when the URL protocol is HTTP.

HttpURLConnection and HEAD/GET methods

I am attempting to validate whether an HTML page exists or not.
I have found that, for about 7% of the pages checked, HEAD and GET methods return different response codes.
I have structured my code such:
1) make initial check using HEAD method
2) for non valid (200) response codes, recheck the page using the GET method.
In this case about 75% of the pages that failed using the HEAD method will work when using the GET method.
So, I guess my questions are:
1) Does anybody know why HEAD/GET return different response codes?
2) Does anybody know a better way to check if a page exists?
Here is the sample program I am using with a few URLs that exhibit this behaviour:
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.InetAddress;
import java.net.URL;
import java.net.UnknownHostException;
public class Internet
     private final static String DEFAULT_LOCAL_HOST = "127.0.0.1";
     private URL url;
     private HttpURLConnection http;
     private int responseCode;
     private String responseMessage;
     public Internet(URL url)
          this.url = url;
     public boolean isValid()
          try
               // Make first attempt using a HEAD request
               http = (HttpURLConnection)url.openConnection();
               http.setRequestMethod( "HEAD" );
               http.connect();
               System.out.println( "head: " + http.getResponseCode()
               + " : " + http.getResponseMessage() );
               // GET seems to do a better job, try again
               if ( http.getResponseCode() != HttpURLConnection.HTTP_OK)
                    http = (HttpURLConnection)url.openConnection();
                    http.setRequestMethod( "GET" );
                    http.connect();
                    System.out.println( "get: " + http.getResponseCode() );
               responseCode = http.getResponseCode();
               responseMessage = http.getResponseMessage();
               if (http.getResponseCode() == HttpURLConnection.HTTP_OK)
                    return true;
               else
                    System.out.println( http.getResponseMessage() );
                    return false;
          catch (IOException e)
               responseCode = -1;
               responseMessage = e.getMessage();
               System.out.println( e );
               return false;
     public static void main(String[] args)
          throws Exception
          URL url = new URL( "http://www.trca.on.ca" );
          Internet internet = new Internet( url );
          System.out.println( internet.isValid() );
          url = new URL( "http://school.discovery.com/sciencefaircentral" );
          internet = new Internet( url );
          System.out.println( internet.isValid() );
          url = new URL( "http://www.amazon.com" );
          internet = new Internet( url );
          System.out.println( internet.isValid() );
}

Using my sample program:
1) about 3K of data is transferred
2) it runs in about 8 seconds
Using InputStream in = http.getInputStream():
1) about 73K of data is transferred
2) it runs in about 15 seconds
Using the getInputStream() method causes the entire file to be transmitted (even though I don't use any read() methods). I don't care about the data.
I want the check to be as fast as possible which is why I use the HEAD method. It doesn't transfer the contents of the file. But why in some cases does it show a response code other than 200?

HttpURLConnection and the killer Pound (#)

I'm trying to create a part of my application that posts data to an ASP page (on an affiliate's web site).
Originally, I was using the following code to post the data:
URL url = new URL(sPostURL);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
// include fields to post
conn.setRequestProperty("address", sAddress);
conn.setRequestProperty("ste/apt#", sApt);
DataOutputStream dos = new DataOutputStream(conn.getOutputStream() );
dos.writeBytes("");
dos.close();
// Retrieve response
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String sNewLine = in.readLine();
String sResp = "";
while (sNewLine != null) {
sResp += sNewLine;
sNewLine = in.readLine();
in.close();Which worked at first, but then when I went back to the code, I must have made a change because it no longer seems to post the data (sAddress and sApt) to the page. So after researching the topic in the forums, I switched to using a tokenized string to store my post data:
URL url = new URL(sPostURL);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
DataOutputStream dos = new DataOutputStream(conn.getOutputStream() );
// include fields to post
String sRequest = "&address=" + sAddress; // BUILD DATA STRING (NEW)
sRequest += "&ste/apt#=" + sApt; // BUILD DATA STRING (NEW)
dos.writeBytes("");
dos.close();
// Retrieve response
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String sNewLine = in.readLine();
String sResp = "";
while (sNewLine != null) {
sResp += sNewLine;
sNewLine = in.readLine();
in.close();which posts the data fine, but the pound sign (#) causes the server to return a response code 500. I can't find an easy way to escape the pound sign. I've even tried building the string using the URL encoded value (%23) but it still causes problems. Does anyone have any suggestions? Was I correct in abandoning the first method? if anyone has any guidance I would be very grateful.

I think the first method is more clean, but if you want to use the second one, you should put your request string through a URLEncoder to encode the string first.
--lichu

HttpURLConnection and bandwidth

I have written a program which downloads files from net using the HttpURLConnection class.
I want to specify the percentage of bandwidth that should be used for downloading the files, so that it does not slow down by other net surfing work.
Can anybody suggest me a way to do this.
Thank u

That's not something you can control via Java.
The only way you could control that is thru some network hardware, or, if the server is yours, you could set a header in the request and the server could take that as a clue to throttle the data it's going to send out. But otherwise, URLConnection class itself will not be able to do anything.

HttpUrlConnection and codification

Hello, I am trying to open a connection to a webpage that is writen in greek (using utf8), but when I try to see the page in my JSP, I don't see well the greek characters. Is my code wrong? maybe is the server configuration?
I have no idea. I try some encode, but nothing changes.
Iam using java 5.0 and tomcat 5.5.
My code:
String url = "http://www.google.gr";
URL u = new URL (url);
HttpURLConnection urlc = (HttpURLConnection)u.openConnection ();
response.setContentType("text/html;charset=UTF-8");
urlc.connect ();
URLConnection.getInputStream()
String line;
BufferedReader br = new BufferedReader (new InputStreamReader (urlc.getInputStream ()));
while ( (line = br.readLine ()) != null ) {
out.write(line);
br.close ();
urlc.disconnect ();
Thanks a lot.
isaac.

Well, you aren't specifying the encoding on your InputStreamReader, so it would be using some other encoding instead. There may be other problems but fix that one first.

HttpURLConnection and redirect

Hello,
I need help with this. I have on server xml page with some data, but before I get this page, I must login to server. With my code, I reach only redirect page. How I can login and get response after redirect?
Here is the function for communication:
package qpconnector;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
public class ServerConnect {
    private String defaultServer =
            "http://10.0.5.55/login/login.action?username=InstallAdmin&password=&" +
            "page=/touchscreens/mainDisplay.qsphtml";
    public void connectToServer () {
        try {
            URL url = new URL(defaultServer);
            URLConnection connection = url.openConnection();
            HttpURLConnection httpConn = (HttpURLConnection) connection;
            httpConn.setDoInput(true);
            httpConn.setDoOutput(true);
            InputStream is = httpConn.getInputStream();
            StringBuffer sb = new StringBuffer();
            int read;
            while ((read = is.read()) != -1) {
                sb.append((char) read);
            System.out.println (sb.toString());
        } catch (MalformedURLException ex1) {
            ex1.printStackTrace();
        } catch (Exception ex) {
            ex.printStackTrace();
    public static void main (String[] args) {
        ServerConnect sc = new ServerConnect();
        sc.connectToServer();
}

ProjectMoon wrote:
malcolmmc wrote:
By default java.net.URL connections manage cookies automatically.It appears that it will provide you with Cookie objects from the server when you make the connection. However, when you make a new connection, you will have to send your cookies with it. This is because of the way HTTP works. If that is incorrect, I apologize.It will return cookies received from a particular domain to that domain, which is generally all you need. Cookies always originate at the server end.
>
Edit: [Ah ha.|http://blogs.sun.com/CoreJavaTechTips/entry/cookie_handling_in_java_se] There is new stuff in Java 6 for cookies.
There is, it makes the cookie business more manageable, but the basic cookie mechanism is already there, at least in 1.5, it's just a lot more tedious if you want to do something complicated, like maintain more than one cookie jar in the JVM.

HttpURLConnection and DataInputStream

I posted this in the Conventional & Interruptable IO forum, but thought it might be better off here....
I'm having a strange issue with this piece of code. The input stream coming in is quite predictable and I know that this loop will only iterate twice. Once for less than 1024 bytes, and the next returns -1. The problem I'm having has to do with the second iteration. The in.read(bary); line takes around 75 seconds just to get that -1 so we can break out of the loop. I don't understand how the first read is so fast, and the second one is so slow. Any thoughts?
1 URL url = new URL("http://xyz");
2 HttpURLConnection con = (HttpURLConnection)url.openConnection();
3 DataInputStream in = new DataInputStream( con.getInputStream() );
4 StringBuffer httpResponse = new StringBuffer();
5 byte[] bary = new byte[1024];
6 while (true) {
7 int bytesRead = in.read(bary);
8 if (bytesRead <= 0) {
9 break;
10 }
11 httpResponse.append(new String(bary, 0, bytesRead));
10 }//end while
I also want to mention that this bit of code runs in J2EE apps that are deployed to 5 different environments, one of those being production which is heavly hit. In only one of these am I having the issue...not production :) . They're running on Solaris 8, sun's sdk 1.4.2_3, weblogic 8.1. Also, I've run the exact same code making this call and I know that the app on the other end is responding fast...we've also verified this using the webserver's access log.
Thanks for reading,
-Jon

The -1 check is in the while loop, it's just not indented very well. The loop starts on line 6, and ends on 10. I mentioned the data is predicatable. On the first call to read I know that I'm reading all the data that the stream has to offer, the second time i read returns -1. I also know this because The actual code has debug statements that tell me how many times it loops, and how long it takes to perform the read in millies. 100% of the time it loops twice, and 100% of the time the read that results in -1 takes 75 seconds.
Maybe i misunderstood what you were saying? Any thoughts would be greatly appreciated.
Message was edited by:
jTosca
Message was edited by:
jTosca

HttpURLConnection and cookies!!

hello evrybody,
i want to write a program which connect to a URL and load some data from there. but the problem is that site writes cookies you will only receive an hint to allow cookies. How can i write a program or simulate a client that accepts cookies to have finaly the required data.
the Programm i am using doesn�t work. Here is a snippet:
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setDoOutput( true );
con.setDoInput( true );
if (!con.getAllowUserInteraction()) con.setAllowUserInteraction( true );
con.setFollowRedirects( true );
BufferedInputStream br = new BufferedInputStream(con.getInputStream());
byte[] c = new byte[5000];
int read;
read = br.read(c,0,5000)

I believe it would be something like the following:
URL url = new URL(...);
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
conn.setRequestProperty("Cookie", "your cookie here");
conn.connect();

HttpURLConnection and valid cookies

Hi,
I have implemented a web server with a login page to authenticate a user.
When the browser has not send a valid cookie, the server redirects to the
login page, where the user can fill a form to post the username and password.
I am implementing a java stand-alone client to access this web server. I use
the HttpURLConnection class for this.
First, i authenticate the user sending a post request with the login and
password. The server responds with a cookie. To access other resources in
the server i send the cookie in each request.
My problem is: i am using java class HttpURLConnection. When the cookie
is not valid, the server redirects to the login page, but i have not
found a way to know about it without parsing the response html. I get the http
header fields but there is no info about this.
Thanks in advance

You should extraxt the sessionID from the cookie, when logging on to the system.
From then on, pass the session ID to all further connections.
You can obtain the session-ID like this:
public void retrieveSessionID() {
               String key = "";
               String id = "";
               try {
                 URL url = new URL ( "http://localhost/yourapplication/logon.jsp?username=Homer&password=donut);
                 URLConnection urlConnection = url.openConnection();
                 if (urlConnection != null)
                      for (int i = 1;(key = urlConnection.getHeaderFieldKey(i)) != null; i++)
                           if (key.equalsIgnoreCase("set-cookie"))
                                id = urlConnection.getHeaderField(key);
                                id = id.substring(0, id.indexOf(";"));
                           }     //if
                      }     //for
                 }     //if
               } catch (IOException e) {
                                                                      e.printStackTrace();
               }     //catch
                 this.jSessionId = id;
     }     //getSessionIDAnd then make every connection like this:
URL url = new URL("http:......");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
                connection.setRequestProperty("Cookie", this.jSessionId);
                connection.connect();                Hope I could help!
Greetings
Stef

HttpURLConnection and java.io.InterruptedIOException

Hello,
I am experiencing the following problem:
I created a standalone java program to open an HttpURLConnection to
an image on the internet at an https address. I am timing how long
it takes to read image through the stream so I can get an idea of my
connection speed.
It works ok, so I made it a servlet that resides in WebSphere 3.5 App server, and when I invoke it, I get:
java.io.InterruptedIOException: Operation timed out: no further information
I check the field java.io.InterruptedIOException.bytesTransferred and it is 0.
Here is a code snippet:
URL dhServlet = new URL("https://www.some-server.com/some-image.bmp");
HttpURLConnection conn = (HttpURLConnection) dhServlet.openConnection();
conn.setDoOutput(false);
conn.setDoInput(true);
conn.setRequestProperty("Accept-Language","en");
conn.setRequestProperty("Accept", "image/bmp, */*");
if (conn.getResponseCode() == 200) // success
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
int chr = in.read();
while( chr != -1 )
chr = in.read();
in.close();
Again, I am confused why it would work in void main but not as a servlet... Thanks for any help you can offer!
- Tom

No they were on different machines -
so I put the stand alone on on the server with the servlet, and
got the error message:
Exception in thread "main" java.net.NoRouteToHostException: Operation timed out:
no further information
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at TimeConnection.main(TimeConnection.java:43)
And Line 43 says
if (conn.getResponseCode() == 200) // success
Maybe the server I connect to restricts by IP? Thanks very
much for your feedback!
- Tom

HttpURLConnection and headers.

I want to send an XML document to a HttpServlet and get a new one back. I plan to send it using the POST method however, in order to enclose the document in the request I need to set the Entity-Headers.. I can't really se how I would do this with HttpURLConnection. Can I modify the headers that HttpURLConnection produce at all?
Of course, I could just use a regular URLConnection and pass all headers manually but it would be nice to avoid that.

Well, thats not the problem...
I need to send the whole document in the Request in order to get a response object with the other document. But if I do something like this:
HttpURLConnection conn = (HttpURLConnection)url.openConnection("foo");
conn.setContentType("text/plain");
conn.setDoOutput(true);
PrintWriter out = new PrintWriter( dataConn.getOutputStream() );
out.write("blah");
out.close();
then I can't call setDoInput() and open an inputstream and read the response. I get and exception saying the connection is already open. This leads me to the conclusion that when I call the SetDoInput it opens a new Connection which is not possible since it is already open.
When Is the request actually sent and how do I modify indivdual headers in the request?

HttpURLConnection and caching of authentication credentials

I am using HttpURLConnection with digest authentication. I do not want the authorization credentials remembered between requests. Is there a way to turn off this behavior or reset the AuthorizationInfo cache?

A brute-force way I have discovered is to clear out the entire AuthCacheValue map:
AuthCacheValue.setAuthCache(new AuthCacheImpl());
However, both AuthCacheValue and AuthCacheImpl are part of the sun.net.www.* classes, so this method would be very fragile and raises issues of incompatibility.

Httpurlconnection and Proxy settings

hi all,
I wrote a servlet that uses an HTTPUrlConnection to get a page from a remote server and the code looks like this :
url = new URL(urlstr);
HttpURLConnection uc = null;
System.getProperties().put( "proxySet", "true" );
System.getProperties().put( "proxyHost", "[myproxyhost]");
System.getProperties().put( "proxyPort", "[myproxyport]);
uc = (HttpURLConnection) url.openConnection();
The code works fine for the particular connection.
If I need to get another page though another connection that needs no Proxy settings, it doesn't work.
How can I deactivate the proxy settings between the two connections?
And, finally, how can I leave the System settings in the same state as before the code execution?
Thank you.
Nicola

You could trySystem.getProperties().put( "proxySet", "false" );to temporarily turn off the proxy settings. Or you could trySystem.getProperties().remove( "proxySet");
// and the other two similarlyto remove those system properties.

HttpURLConnection and HtmlUnit

Similar Messages

Maybe you are looking for