Trying to parse a huge XML file

Question

I'm trying to parse a large (~200 Meg) XML file. I get out of memory errors with DOM, and so far using SAX has been a huge disaster. My program has to run on IE and Netscape as an applet. I've run into a HUGE amount of errors. While I have been doing Java programming for about 2 years, this is the first time I've ever worked with XML. The error I am currently getting is an abstract method error associated with the following code :
SAXParserFactory spf = new
                         com.sun.xml.parser.SAXParserFactoryImpl();
               spf.setValidating(false);
               try {
               out.write("Gets past SAXParserFactory creation");
               out.flush();
                    catch (Exception e) {}
               xmlReader = null;
               try {
                         // Create a JAXP SAXParser
                         saxParser = spf.newSAXParser();
                         // Get the encapsulated SAX XMLReader
                         out.write("gets to assigning an XMLReader");
***THIS LINE OF CODE ***                         xmlReader = saxParser.getXMLReader();
                         out.write("gets past assigning an XMLReader");
                         out.flush();
I've run out of ideas and would greatly appreciate any help anyone could offer on this most frustrating problem.

Answer 1

With XML, we don't have to check each line of the file, only the data in those tags of each element that we are searching on.That's true, once you have read the entire XML and parsed it. Reading the entire XML would of course take much longer than reading the text file it was generated from, so you would want to pre-parse it into a DOM and store that somehow. But then to do a search you'd have to load that DOM, which is in turn much larger than the XML it was generated from...
What you describe sounds to me a lot like a database application. Of course I don't know anything about the specifics of what you are doing with the text, but storing it in a database with indexes that support the searches seems to me like a better approach. Of course you then have the challenge of distributing the database and its supporting software in a form that doesn't require installation. But I'd respectfully suggest that banning any installation at all is a bit extreme. After all, where I live you can buy an encyclopedia on a CD, from a gas station, for $10. And when you load the CD, it runs through an installation procedure. No big deal these days.