Can any body help me in reading from HTTPS URL

Question

I need to read an HTTPS URL and store the response within a table .
How will I manage to do it from within a servlet using URLConnection and openStream as it does'nt work .
How will JSSE help in this regard .
Since I also need to give the userid and password to get into the file and read the file
https://anyhost.com/readthisfile.html
somnath
Web Developer

Answer 1

Hi,
The Java Secure Socket Extension (JSSE) library from Sun Microsystems lets you access a secure Web server from behind a firewall via
proxy tunneling. To do this, the JSSE application needs to set the https.ProxyHost and https.ProxyPort system properties. The
tunneling code in JSSE checks for "HTTP 1.0" in the proxy's response. If your proxy, like many, returns "HTTP 1.1", you will get an
IOException. In this case, you need to implement your own HTTPS tunneling protocol.
In this article, I will show you how to create a secure socket that tunnels through the firewall, and pass it to the HTTPS stream handler to
open HTTPS URLs using the URLConnection class.
Open the http tunnel socket to the proxy
The first step to creating your secure socket is to open the tunneling socket to the proxy port. The code needed to do this proxy
handshaking can be found in the sample code SSLClientSocketWithTunneling.java that comes with the JSSE distribution. First, a normal socket is created that connects to
the proxy port on the proxy host (line 65). After the socket is created, it is passed to the doTunnelHandshake() method where the proxy's tunneling protocol is called:
54 SSLSocketFactory factory =
55 (SSLSocketFactory)SSLSocketFactory.getDefault();
56
57 /*
58 * Set up a socket to do tunneling through the proxy.
59 * Start it off as a regular socket, then layer SSL
60 * over the top of it.
61 */
62 tunnelHost = System.getProperty("https.proxyHost");
63 tunnelPort = Integer.getInteger("https.proxyPort").intValue();
64
65 Socket tunnel = new Socket(tunnelHost, tunnelPort);
66 doTunnelHandshake(tunnel, host, port);
In doTunnelHandshake(), an http "CONNECT" command is sent to the proxy, with the secure site's hostname and port number as the parameters (line 161). In the original
tunneling code on line 206 in JSSE, it then checks for "HTTP/1.0 200" in the proxy's reply. If your organization's proxy replies with "HTTP 1.1", an IOException will be
thrown. To get around this, the code here checks for the reply "200 Connection Established", which indicates that tunneling is successful (line 207). You can modify the
code to check for the expected corresponding response from your proxy:
139 private void doTunnelHandshake(Socket tunnel, String host, int port)
140 throws IOException
141 {
142 OutputStream out = tunnel.getOutputStream();
143 String msg = "CONNECT " + host + ":" + port + " HTTP/1.0\n"
144 + "User-Agent: "
145 + sun.net.www.protocol.http.HttpURLConnection.userAgent
146 + "\r\n\r\n";
147 byte b[];
148 try {
149 /*
150 * We really do want ASCII7 -- the http protocol doesn't change
151 * with locale.
152 */
153 b = msg.getBytes("ASCII7");
154 } catch (UnsupportedEncodingException ignored) {
155 /*
156 * If ASCII7 isn't there, something serious is wrong, but
157 * Paranoia Is Good (tm)
158 */
159 b = msg.getBytes();
160 }
161 out.write(b);
162 out.flush();
163
164 /*
165 * We need to store the reply so we can create a detailed
166 * error message to the user.
167 */
168 byte reply[] = new byte[200];
169 int replyLen = 0;
170 int newlinesSeen = 0;
171 boolean headerDone = false; /* Done on first newline */
172
173 InputStream in = tunnel.getInputStream();
174 boolean error = false;
175
176 while (newlinesSeen < 2) {
177 int i = in.read();
178 if (i < 0) {
179 throw new IOException("Unexpected EOF from proxy");
180 }
181 if (i == '\n') {
182 headerDone = true;
183 ++newlinesSeen;
184 } else if (i != '\r') {
185 newlinesSeen = 0;
186 if (!headerDone && replyLen < reply.length) {
187 reply[replyLen++] = (byte) i;
188 }
189 }
190 }
191
192 /*
193 * Converting the byte array to a string is slightly wasteful
194 * in the case where the connection was successful, but it's
195 * insignificant compared to the network overhead.
196 */
197 String replyStr;
198 try {
199 replyStr = new String(reply, 0, replyLen, "ASCII7");
200 } catch (UnsupportedEncodingException ignored) {
201 replyStr = new String(reply, 0, replyLen);
202 }
203
204 /* We check for Connection Established because our proxy returns
205 * HTTP/1.1 instead of 1.0 */
206 //if (!replyStr.startsWith("HTTP/1.0 200")) {
207 if(replyStr.toLowerCase().indexOf(
208 "200 connection established") == -1){
209 throw new IOException("Unable to tunnel through "
210 + tunnelHost + ":" + tunnelPort
211 + ". Proxy returns \"" + replyStr + "\"");
212 }
213
214 /* tunneling Handshake was successful! */
215 }
Overlay http tunnel socket with SSL socket
After you have successfully created the tunneling socket, you overlay it with the SSL socket. Again, this is not difficult to do:
54 SSLSocketFactory factory =
55 (SSLSocketFactory)SSLSocketFactory.getDefault();
56
57 /*
58 * Set up a socket to do tunneling through the proxy.
59 * Start it off as a regular socket, then layer SSL
60 * over the top of it.
61 */
62 tunnelHost = System.getProperty("https.proxyHost");
63 tunnelPort = Integer.getInteger("https.proxyPort").intValue();
64
65 Socket tunnel = new Socket(tunnelHost, tunnelPort);
66 doTunnelHandshake(tunnel, host, port);
67
68 /*
69 * Ok, let's overlay the tunnel socket with SSL.
70 */
71 SSLSocket socket =
72 (SSLSocket)factory.createSocket(tunnel, host, port, true);
73
74 /*
75 * register a callback for handshaking completion event
76 */
77 socket.addHandshakeCompletedListener(
78 new HandshakeCompletedListener() {
79 public void handshakeCompleted(
80 HandshakeCompletedEvent event) {
81 System.out.println("Handshake finished!");
82 System.out.println(
83 "\t CipherSuite:" + event.getCipherSuite());
84 System.out.println(
85 "\t SessionId " + event.getSession());
86 System.out.println(
87 "\t PeerHost " + event.getSession().getPeerHost());
88 }
89 }
90 );
The code had called the SSLSocketFactory's getDefault() method earlier to get an instance of the SSLSocketFactory (line 54, repeated above). Next, it passes the
tunneling socket that was created in the previous step to the createSocket() method of the SSLSocketFactory. The createSocket() method returns an SSLSocket that is
connected to the destination host and port via the proxy tunnel. You can optionally add a HandshakeCompletedListener to the socket if you wish to be informed when the
SSL handshaking is completed.
The SSLSocket created is basically ready for use to transfer secure contents. The startHandshake() method is called to start the SSL handshaking (line 98). After which, you
can issue the http "GET" command to retrieve the secure pages (line 105):
91
92 /*
93 * send http request
94 *
95 * See SSLSocketClient.java for more information about why
96 * there is a forced handshake here when using PrintWriters.
97 */
98 socket.startHandshake();
99
100 PrintWriter out = new PrintWriter(
101 new BufferedWriter(
102 new OutputStreamWriter(
103 socket.getOutputStream())));
104
105 out.println("GET http://www.verisign.com/index.html HTTP/1.0");
106 out.println();
107 out.flush();
However, issuing http commands to the tunneling SSL socket to access Webpages is not ideal because it would mean having to rewrite the whole http protocol handler from
scratch. Instead, you should use the HTTPS URL APIs that the JSSE already includes for that purpose. To do this, you have to pass the tunneling SSL socket to the HTTPS URL
stream handler.
Pass SSL socket to HTTPS URL stream handler
The JSSE library has an HttpsURLConnection class that is in the com.sun.net.ssl package, which extends the java.net.URLConnection class. An HttpsURLConnection object
is returned by the URL object's openConnection() method when "HTTPS" is specified as the protocol. The HttpsURLConnection class has a method, setSSLSocketFactory(),
that lets you set an SSLSocketFactory of your choice. To pass the tunneling SSL socket to the HTTPS URL stream handler, you would set the setSSLSocketFactory()
method's parameter with a socket factory that returns the tunneling SSL socket that you created previously.
To do this, you would wrap the code discussed previously in an SSLTunnelSocketFactory class that extends from the SSLSocketFactory class. The SSLSocketFactory is an
abstract class. To extend it, you must implement the createSocket() method to return the tunneling SSL socket that you created earlier:
12 public SSLTunnelSocketFactory(String proxyhost, String proxyport){
13 tunnelHost = proxyhost;
14 tunnelPort = Integer.parseInt(proxyport);
15 dfactory = (SSLSocketFactory)SSLSocketFactory.getDefault();
16 }
44 public Socket createSocket(Socket s, String host, int port,
45 boolean autoClose)
46 throws IOException,UnknownHostException
47 {
48
49 Socket tunnel = new Socket(tunnelHost,tunnelPort);
50
51 doTunnelHandshake(tunnel,host,port);
52
53 SSLSocket result = (SSLSocket)dfactory.createSocket(
54 tunnel,host,port,autoClose);
55
56 result.addHandshakeCompletedListener(
57 new HandshakeCompletedListener() {
58 public void handshakeCompleted(HandshakeCompletedEvent event) {
59 System.out.println("Handshake finished!");
60 System.out.println(
61 "\t CipherSuite:" + event.getCipherSuite());
62 System.out.println(
63 "\t SessionId " + event.getSession());
64 System.out.println(
65 "\t PeerHost " + event.getSession().getPeerHost());
66 }
67 }
68 );
69
70 result.startHandshake();
71
72 return result;
73 }
Notice that the SSLTunnelSocketFactory contains a default SSLSocketFactory object. The default SSLSocketFactory object can be instantiated from a call to the static
method getDefault() (line 15). You need this SSLSocketFactory object to overlay the tunnel socket with the SSL socket, as discussed earlier. You also call the default
object's getDefaultCipherSuites() and getSupportedCipherSuites() methods when implementing the corresponding abstract methods of the SSLSocketFactory super
class. For implementation details, please refer to the complete source code for the SSLTunnelSocketFactory in Resources.
Tunnel through the proxy via URLConnection
To tunnel through the proxy via URLConnection in your JSSE application, after you call the openConnection() method, check if the returned object is that of the
HttpsURLConnection. If so, you instantiate your SSLTunnelSocketFactory object and set it in the setSSLSocketFactory() method (lines 22 through 25):
10 public class URLTunnelReader {
11 private final static String proxyHost = "proxy.sg.ibm.com";
12 private final static String proxyPort = "80";
13
14 public static void main(String[] args) throws Exception {
15 System.setProperty("java.protocol.handler.pkgs",
16 "com.sun.net.ssl.internal.www.protocol");
17 //System.setProperty("https.proxyHost",proxyHost);
18 //System.setProperty("https.proxyPort",proxyPort);
19
20 URL verisign = new URL("https://www.verisign.com");
21 URLConnection urlc = verisign.openConnection(); //from secure site
22 if(urlc instanceof com.sun.net.ssl.HttpsURLConnection){
23 ((com.sun.net.ssl.HttpsURLConnection)urlc).setSSLSocketFactory
24 (new SSLTunnelSocketFactory(proxyHost,proxyPort));
25 }
26
27 BufferedReader in = new BufferedReader(
28 new InputStreamReader(
29 urlc.getInputStream()));
30
31 String inputLine;
32
33 while ((inputLine = in.readLine()) != null)
34 System.out.println(inputLine);
35
36 in.close();
37 }
38 }
You can then access the HTTPS URLs using the APIs provided by the URLConnection class. You don't need to worry about the format of the http GET and POST commands,
which you would if you used the SSL Socket APIs.
The complete source code for the SSLTunnelSocketFactory and the application code that connects to a secure URL using proxy tunneling is included in Resources. To
compile and run the application, you would need to download and install Sun's JSSE from its Website, also listed in Resources.
Conclusion
If your JSSE application could not tunnel through your organization's firewall, you need to implement your own tunneling socket. The sample code included with the JSSE
distribution shows you how to open an SSL socket tunnel. This article goes one step further to show you how to pass the tunneling socket to the HTTPS URL stream handler,
and saves you the trouble of rewriting a http handler
I hope this will help you.
Thanks
Bakrudeen