Download a web page, how to ?

Question

Can any one help me with code for downloading a web page given the url address ? I can download the page, but the problem is it doesn't download the associated images, javascripts etc. neither does it create an associated folder as one might expect when saving a page using browser.
Below is the code snippet -
                    URL url = new URL(address);
                    out = new BufferedOutputStream(
                         new FileOutputStream(localFileName));
                    conn = url.openConnection();
                    in = conn.getInputStream();
                    byte[] buffer = new byte[1024];
                    int numRead;
                    long numWritten = 0;
                    while ((numRead = in.read(buffer)) != -1) {
                         out.write(buffer, 0, numRead);
                         numWritten += numRead;
                    System.out.println(localFileName + "\t" + numWritten);

Answer 1

javaflex wrote:
I don't think web crawler would work
webcrawler simply takes every link or url on the given address and digs into it ..
Would it work for javascripts ? Given a url like xyz.com/a.html,
1. the code above would downlod the plain html.
2. parse html to find javascripts, images (anything else I need to look at ?)
3. download those
4. put everything in one folder (but the question is then do I need to rename the pointers in the dwnlded html to point at the other contents on the disk ? )
This is naive approach - anything better ?
thanks.More advanced web-crawlers parse the JavaScipt source files (or embedded JS sources inside HTML files) and (try) to execute the script in order to find new links. So the answer is: yes, some crawlers do. I know for a fact that Heritrix can do this quite well, but it is a rather "large" crawler and can take a while to get to work with. But it really is one of the best (if bot the best) open source Java web-crawlers around.