SAX (xerces) problem

Question

I have a big problem with Apache Xerces2 Java.
I have to parse and get data from very large xml files (100 MB to 20 GB). Because the files are very large I have to use SAX parser.
If I use internal xerces in any update of jdk/jre 1.6 then whole document gets into memory. I have found a bug report related at http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6536111 . I am not sure that fix will solve my problem and fix has not delivered yet. According to the bug report it is going to be delivered with jdk6 update 14 in the mid May 2009.
I thougt maybe the problem is with the internal SAX parser. So I started to use source of xerces. (I use the last version - 2.9.1). At this point I have discovered that parse takes more time and need 24 byte for each node. Sometimes xml files have 80.000.000 nodes. It will take 1,5 - 2 GB of RAM which I don't have. Even if I have RAM that size I can not use it at windows 32 platform. (OS limits)
Has anyone got idea, solution?
Thanks..

Answer 1

Thank you both Toll and DrClap for your help. I'll take a look at Saxon, but I'm still intrigued why nobody is complaining about a tool (SAX) that's almost a standard for stream parsing... and yet not working for XSLT transformations! Maybe you were right after all when you said stream processing might not be possible for my XSLT file but I doubt it because the XML is representing "sort of" a table and therefore it's made up of thousands of structurally identical <row> elements which can be individually transformed...I can't think of anything more suitable for streaming transformation.
Thanks again for your time.
Edited: at the end I've decided to parse the document using SAX (which in my tests uses almost no memory at all and performs lightning-fast) and then applying a XSL transformation for each parsed node (I can do it in my case). But transforming a document will still be a huge problem -in terms of memory usage- for those who don't have a repeating pattern on their XML's, although I guess 99% of the times there'll probably be one for big/huge documents.
I think this will be very useful for other programmers facing the same problem I had. This code divides a xml file into several different files according to a repeating pattern (in this case InsurancePolicyData/Record) using SAX and then processes each chunk of xml separately, optimizing the use of memory:
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.Writer;
import org.dom4j.io.SAXReader;
public class SingleThreadSplitXSLT
  public static void main(String[] args)
    throws Exception
    if (args.length != 3)
      System.err.println(
        "Error: Please provide 3 inputs:” +
        “ inputXML XSLT outputXML");
      System.exit(-1);
    long startTimeMs = System.currentTimeMillis();
    File xmlFile = new File(args[0]);
    File xsltFile = new File(args[1]);
    BufferedWriter outputWriter = new
      BufferedWriter(new FileWriter(args[2]));
    styleDocument(xmlFile, xsltFile, outputWriter);
    outputWriter.close();
    long executionTime =
      System.currentTimeMillis() - startTimeMs;
    System.err.println("Successful transformation took "
      + executionTime);
  public static void styleDocument(File xmlFile,
    File xsltFile, Writer outputWriter)
    throws Exception
    // start the output file
    outputWriter.write(
      "<?xml version=\"1.0\" encoding=\"UTF-8\"?>");
    outputWriter.write("<InsurancePolicyData>");
    // read the input file incrementally
    SAXReader reader = new SAXReader();
    reader.addHandler( "/InsurancePolicyData/Record",
      new SplitFileElementHandler(
        xsltFile, outputWriter));
    reader.read(xmlFile);
    // finish output file
    outputWriter.write("</InsurancePolicyData>");
}(I found it at http://www.devx.com/xml/Article/34677/1954)
That's exactly what I was looking for, hope it helps others as well :)
Edited by: Isana on Jun 4, 2009 7:56 AM