Not well formed sites crash parser

Question

Hello,
I am trying to make a very simple proxy using a server. Right now I am having trouble parsing the html on google.com. First the unescaped ampersands was making tomcat crash. I put the html in a string and escaped all the & but found out that DocumentBuilder.parse() does not take the data as a string ( it does take a string, its supposed to be a uri, not the actual data itself). I got around this by saving the escaped html to a file and loading that into the parse method.
Now my problem is this. The html is malformed on google's site. It has an opening <meta> but no closing </meta>. Is there a method or technique to force well formedness on the html?
Here is the relavent code:
     protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException
          response.setContentType("text/plain");
          PrintWriter pw = response.getWriter();
          String param;
          param = request.getParameter("url"); //get the url to proxy for example localhost:8080/test/proxy?url=http://www.google.com
          URL url = new URL(param);
          HttpURLConnection u = (HttpURLConnection)url.openConnection();
          u.connect();
          StringBuffer result = new StringBuffer();
          BufferedReader in = new BufferedReader(new InputStreamReader(u.getInputStream()));
          String temp;
          while((temp = in.readLine())!= null)
               result.append(temp);
          String somename = result.toString();
          somename = somename.replace("&", "&"); //escape the &'s
          BufferedWriter bw = new BufferedWriter(new FileWriter("xml"));
          bw.write(somename);
          bw.flush();
          bw.close();
          DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
          DocumentBuilder db;
          Document page = null;
          try
               db = dbf.newDocumentBuilder();
               page = db.parse("xml"); //line it fails at
          catch(ParserConfigurationException pe)
               System.out.println("Parser Configuration Error");
               pe.printStackTrace();
               System.exit(0);
          catch(SAXException sax)
               System.out.println("SaxException");
               sax.printStackTrace();
}Here is the corresponding error in catalina.out
[Fatal Error] xml:1:1973: The element type "meta" must be terminated by the matching end-tag "</meta>".
SaxException
org.xml.sax.SAXParseException: The element type "meta" must be terminated by the matching end-tag "</meta>".
     at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
     at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
     at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)
     at proxy.doGet(proxy.java:62)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
     at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
     at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
     at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
     at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
     at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
     at java.lang.Thread.run(Thread.java:619)Thanks for any help

Answer 1

bob0123 wrote:
Now my problem is this. The html is malformed on google's site. It has an opening <meta> but no closing </meta>. Is there a method or technique to force well formedness on the html?Call up Sergey and Larry and tell them their site is causing you a problem? No, you were on the right track in the first place. By the way there are other HTML parsers available. TagSoup is one. (I don't have any experience with it, I just know it exists.)