Regex for web page parsing

Question

I am trying to parse the html output of a web page. I need to grab all the URLs which are in hrefs, actions etc inside.
I am using the regex pattern
Pattern p = Pattern.compile("http.*?//\\S+[^\"]");
(basically grab anything that starts with http till you hit a double quote and white space)
When I do a find using matcher it gets me URLs which are like
href="http://www.abcd.com/" or
href="http://www.abcd.com/mypath" or
href="http://www.abcd.com/mypath?p1=v1&p2=v2"
But it doesn't work for something like
href="http://www.abcd.com"><img src="/abcd/my.gif"
Is there a good pattern to catch all of these varaints of URLs                    
Thanks

Answer 1

First off, your regex is wrong. After matching the double slash, it gobbles up as many non-whitespace characters as it can, then looks for a character that is not a double-quote. When the attribute value is followed by a space, the \S+ will stop at that point, then the [^"] will match the space. This at least give you a usable result; all you have to do is strip off the quote and the space. But if the value is immediately followed by a closing angle-bracket, the \S+ just keeps on gobbling. In your last example, it won't stop until it hits the space after the "img"--and now all you have is garbage.
Here's a better way to extract URL's:
  Pattern p = Pattern.compile(
    "(href|src|background)  # ...and whatever other attr's  \n" +
    "\\s*=\\s*                                              \n" +
    "(?:                                                    \n" +
    "   \"([^\"]+)\"  # anything in double quotes           \n" +
    " |               # or                                  \n" +
    "   \'([^\']+)\'  # anything in single quotes           \n" +
    " |               # or                                  \n" +
    "   ([^\\s>]+)    # any non-WS except closing bracket   \n" +
    ")", Pattern.COMMENTS);Instead of looking for the URL's themselves, it looks for the names of attributes that should have URL's as values. The attribute name will be captured in group(1) and the URL will be in group(2), group(3) or group(4), depending on whether it was double-quoted, single-quoted or not quoted. If need be, you can verify that the value is indeed a URL using the java.net.URL class.