Read/update large xml files

Hi,
My requirement is to store and update data in xml files (no other option) which can be of size > 100MB.
This is what I am doing at the moment..
1. In order to be able to modify, delete nodes easily I am using DOM (Dom4j)
2. As the file size is blowing up memory I have decided to distribute data in multiple files as one file per day. In a loop I load each file as Document, modify or delete nodes and write back to disk.
The problem is its still taking a lot of time to complete one operation and intuitively I am not comfortable with this design, I guess there is better options available..
Can anybody suggest a better alternatives for this using less memory and faster update times. I did a bit of search on google but none like JAXP or JAXB solve my problem.

I had a similar problem.
I had to read through an XML file and then output it in a certain format of text file. The normal size of the XML files I was working with would be between 200KB and 3MB.
I wrote the code for doing this using DOM, and it worked okay, and reasonably fast.
However when I tested the file with huge files (~150MB) it completely ate up the systems memory, and eventually crashed, even with a hugely extended Java heap.
So to solve my problem I re-wrote the program using SAX. This concerned me at first, for I'm sure the same reasons you're thinking now. At first it took me a while to get my head around how to use it, but now that I understand how it works, I find SAX to be a superior way to work with XML in Java.
In addition, the performance improvements are huge. The program now works over twice as fast, and uses a comparatively minimal amount of memory.
So my suggestion is to do a bit of reading about SAX and port your application to use it. If you have any questions regarding this, I'd be happy to help, especially as I've recently done the same thing.

Similar Messages

Reading large XML file using a file event generator and a JPD process

I am using a FileEventGenerator and a JPD Subscription process to read a large XML file. The large XML file basically contains repeated XML elements. My understanding is that the file subscription method reads the whole file in memory which causes lots of problem for huge file size like 1MB. Is there a way to read the file size-wise or is there a way to read chunks of data from a large size file..or any other alternative. I would like to process the file in a loop iteration by iteration.

Hitejain,
Here are a couple of pointers you could try. One is that the file event generator has a pass by reference (filename) functionality which you could use so that you could do the following inside of your process.
1) Read file name from the reference
2) Move the file to a processed directory (so it doesn't get picked up again. Note: I don't know how the embedded archive methods of the file event generator plays with pass by reference.
3) Open a stream to the file.
4) Use a SAX or SAX - DOM combined approach to parse your XML while managing the memory usage inside of your process
There is another possibility which might fit your needs and it is related to the RawData object that BEA provides. If I understand it correctly provides wrapping functionality around a stream object, but depending on your parsing methods might just postpone the problem.
Hope this helps
Chris Falling
Stormforge Software

Does the parser work with large XML files?

Is there a restriction on the XML file size that can be loaded into the parser?
I am getting a out of memory exception reading in large XML file(10MB) using the commands
DOMParser parser = new DOMParser();
URL url = createURL(argv[0]);
parser.setErrorStream(System.err);
parser.setValidationMode(true);
parser.showWarnings(true);
parser.parse(url);
Win NT 4.0 Server
Sun JDK 1.2.2
===================================
Error output
===================================
Exception in thread "main" java.lang.OutOfMemoryError
at oracle.xml.parser.v2.ElementDecl.getAttrDecls(ElementDecl.java, Compi
led Code)
at java.util.Hashtable.<init>(Unknown Source)
at oracle.xml.parser.v2.DTDDecl.<init>(DTDDecl.java, Compiled Code)
at oracle.xml.parser.v2.ElementDecl.getAttrDecls(ElementDecl.java, Compi
led Code)
at oracle.xml.parser.v2.ValidatingParser.checkDefaultAttributes(Validati
ngParser.java, Compiled Code)
at oracle.xml.parser.v2.NonValidatingParser.parseAttributes(NonValidatin
gParser.java, Compiled Code)
at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingPa
rser.java, Compiled Code)
at oracle.xml.parser.v2.ValidatingParser.parseRootElement(ValidatingPars
er.java:97)
at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingP
arser.java:199)
at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:146)
at TestLF.main(TestLF.java:40)
null

We have a number of test files that are that size and it works without a problem. However using the DOMParser does require significantly more memory than your doc size.
What is the memory configuration of the JVM that you are running with? Have you tried increasing it? Are you using our latest version 2.0.2.6?
Oracle XML Team

Problems with Large XML files

I have tried increasing the memory pool using the -mx and -ms options. It doesnt work. I am using your latest XML parser for Java v2. Please let me know if there are some specific options I should be using.
Thanx,
-Sameer
We have a number of test files that are that size and it works without a problem. However using the DOMParser does require significantly more memory than your doc size.
What is the memory configuration of the JVM that you are running with? Have you tried increasing it? Are you using our latest version 2.0.2.6?
Oracle XML Team
Is there a restriction on the XML file size that can be loaded into the parser?
I am getting a out of memory exception reading in large XML file(10MB) using the commands
DOMParser parser = new DOMParser();
URL url = createURL(argv[0]);
parser.setErrorStream(System.err);
parser.setValidationMode(true);
parser.showWarnings(true);
parser.parse(url);
Win NT 4.0 Server
Sun JDK 1.2.2
===================================
Error output
===================================
Exception in thread "main" java.lang.OutOfMemoryError
at oracle.xml.parser.v2.ElementDecl.getAttrDecls(ElementDecl.java, Compi
led Code)
at java.util.Hashtable.<init>(Unknown Source)
at oracle.xml.parser.v2.DTDDecl.<init>(DTDDecl.java, Compiled Code)
at oracle.xml.parser.v2.ElementDecl.getAttrDecls(ElementDecl.java, Compi
led Code)
at oracle.xml.parser.v2.ValidatingParser.checkDefaultAttributes(Validati
ngParser.java, Compiled Code)
at oracle.xml.parser.v2.NonValidatingParser.parseAttributes(NonValidatin
gParser.java, Compiled Code)
at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingPa
rser.java, Compiled Code)
at oracle.xml.parser.v2.ValidatingParser.parseRootElement(ValidatingPars
er.java:97)
at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingP
arser.java:199)
at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:146)
at TestLF.main(TestLF.java:40)
null

You might try using a different JDK/JRE - either a 1.1.6+ or 1.3 version as 1.2 in our experience has the largest footprint. If this doesn't work can you give us some details about your system configuration. Finally you might try the SAX interface as it does not need to load the entire DOM tree into memory.
Oracle XML Team

Parsing Large XML File

Is there a restriction on the XML file size that can be loaded into the parser?
I am getting a out of memory exception reading in large XML file(10MB) using the commands
DOMParser parser = new DOMParser();
URL url = createURL(argv[0]);
parser.setErrorStream(System.err);
parser.setValidationMode(true);
parser.showWarnings(true);
parser.parse(url);
Win NT 4.0 Server
Sun JDK 1.2.2
===================================
Error output
===================================
Exception in thread "main" java.lang.OutOfMemoryError
at oracle.xml.parser.v2.ElementDecl.getAttrDecls(ElementDecl.java, Compi
led Code)
at java.util.Hashtable.<init>(Unknown Source)
at oracle.xml.parser.v2.DTDDecl.<init>(DTDDecl.java, Compiled Code)
at oracle.xml.parser.v2.ElementDecl.getAttrDecls(ElementDecl.java, Compi
led Code)
at oracle.xml.parser.v2.ValidatingParser.checkDefaultAttributes(Validati
ngParser.java, Compiled Code)
at oracle.xml.parser.v2.NonValidatingParser.parseAttributes(NonValidatin
gParser.java, Compiled Code)
at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingPa
rser.java, Compiled Code)
at oracle.xml.parser.v2.ValidatingParser.parseRootElement(ValidatingPars
er.java:97)
at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingP
arser.java:199)
at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:146)
at TestLF.main(TestLF.java:40)
null

hi
i think you can use STAX a java lib.
since i m not having much more knowledge about this.
you can find this on Internaet.
Thanks

Parsing large xml file and display using swing

Hi all,
I want to read a large xml file and display graphically in swing as a tree structure.
I implemented it and works fine for files of 5MB size after increasing the jvm heap size (-Xmx). If the file size is larger than 5MB it throws out of memory error. I'm creating a custom datastructure from the xml and I'm using sax parsing.
After displaying the datastructure, the user could do some operation on this, like search etc.
Can any of you suggest a method, to support larger files ? What I'm looking for is create the datastructure in file system, rather than in memory.
Any other tips for memory management would be greatly appreciated
Thanks in Advance.
Nisha

Use a memory-mapped file?
http://javaalmanac.com/egs/java.nio/CreateMemMap.html

Large XML files containing time based data

Hi,
i'm having an extremely large XML files, gegas, it holds some info for displaying graphical representations for users, one of which is the updated view every time instance. i'm stuck with handling this large XML file, obviously i tried using a DOM parser, i got an out of memory exception, i though of using SAX as its event based, but then i'll be stuck going backward through the representation, i need users to go backward and forward representing the graphical representation at different time instances.

It is gigabytes?
And this file gets modified by some external application(s), rather than being created one time?
And you want to keep reprocessing it at the same time other application(s) are modifying it?
If yes to all of those, then I doubt there is any real solution except to copy it. If you attempt to sync it with the file system then you are either going to be locking it, preventing updates, or you are going to end up with corrupted data and thus ill formed xml.
If the time data is historical in that a 'user' does something and a new entry is made and you want to report in various ways on that data then parsing it into a database would seem like reasonable solution.

Changing /updating an xml file using JAXP(DOM)

Hello,
i am fairly new to xml and am using it in my degree project.I am able to retrieve and read data from a fairly large xml file using JAXP(DOM) and/or XMLBeans.I am having difficulties in updating the xml document. Any updation i believe is ito be saved into a new xml document,but dont know how to proceed with it. Any help would be appreciated.
Following is a snippet of my code using JAXP. Here i am able to retrieve data from the source file.
File document=new File("C:\\tester.xml");
try {
DocumentBuilderFactory factory
= DocumentBuilderFactory.newInstance();
DocumentBuilder parserr = factory.newDocumentBuilder();
Document doc=parserr.parse(document);
System.out.println(document + " is well-formed.");
NodeList n2=doc.getElementsByTagName("Top");
NodeList n3=doc.getElementsByTagName("Base");
int x=n2.getLength();
System.out.println("There are " x "players");
for(int g=0;g<=x;g++)
System.out.println("Top is" + n2.item(g).getFirstChild().getNodeValue()+" Base is" +n3.item(g).getFirstChild().getNodeValue());
--------------------------------------------------------------------------------

Following is my updation code to the dom tree:
NodeList list=doc.getElementsByTagName("Information");
for(int i=0; i<list.getLength();i++){
Node thissampnode=list.item(i);
Node thisNameNode=thissampnode.getFirstChild();
if(thisNameNode==null) continue;
if(thisNameNode.getFirstChild()==null)continue;
// if(thisNameNode.getFirstChild() !(instanceof org.w3c.dom.Text) continue;
String data=thisNameNode.getFirstChild().getNodeValue();
if (! data.equals("0.59")) continue;
Node newsampNode = doc.createElement("Samp");
Node newsampTopNode = doc.createElement("Top");
Text tnNode = doc.createTextNode("0.50");
newsampTopNode.appendChild(tnNode);
Element newsampRef = doc.createElement("Ref");
Text tsr = doc.createTextNode("0");
newsampRef.appendChild(tsr);
Element newsampType = doc.createElement("Type");
Text tt = doc.createTextNode("z");
newsampType.appendChild(tt);
Element newsampbase = doc.createElement("Base");
Text sb = doc.createTextNode("0.55");
newsampbase.appendChild(sb);
newsampNode.appendChild(newsampTopNode);
newsampNode.appendChild(newsampRef);
newsampNode.appendChild(newsampType);
newsampNode.appendChild(newsampbase);
rootNode.insertBefore(newsampNode, thissampnode);
Here i dont see any changes to the original xml source file.

I want to load large raw XML file in firefox and parse by DOM. But, for large XML file the firefox very slow some time crashed . Is there any option to increase DOM handling memory in Firefox

Actually i am using an off-line form to load very large XML file and using firefox to load that form. But, its taking more time to load and some time the browser crashed. through DOM parsing this XML file to my form. Is there any option to increase DOM handler size in firefox

Thank you for your suggestion. I have a question,
though. If I use a relational database and try to
access it for EACH and EVERY click the user makes,
wouldn't that take much time to populate the page with
data?
Isn't XML store more efficient here? Please reply me.You have the choice of reading a small number of records (10 children per element?) from a database, or parsing multiple megabytes. Reading 10 records from a database should take maybe 100 milliseconds (1/10 of a second). I have written a web application that reads several hundred records and returns them with acceptable response time, and I am no expert. To parse an XML file of many megabytes... you have already tried this, so you know how long it takes, right? If you haven't tried it then you should. It's possible to waste a lot of time considering alternatives -- the term is "analysis paralysis". Speculating on how fast something might be doesn't get you very far.

Query in a large xml file

Hello,
I'm trying to work with very large xml files which are created from csv files. These files may be very large - up to 1 GB ! Untill now I have managed to do several validations on these big xml files, and the only thing that works for me is SAX parser, DOM is out of the question because it fills up memory.
My next task is to do queries on these files, smth like:
select field1,field2 from file.xml
where field3 = 'A'
and (fileld4>'B' or field1='C')
order by field2.
I searched the net about finding out how to make queries on xml files (since I have never done queries on xml before), but I couldn't find which "query language" is best for large files. If I use XPath (XSLT) will that not cause me memory problems because XSLT represents the file as a memory object?
My idea is to parse the file with SAX and check every row if it fits the where condition and then write it immediately to a result xml file. But validating the where statement can be very complicated without using some tool. Also the order by statement is another problematic issue.
Does anyone have some more intelegent ideas about how I can do this? Please help! :(
The xml file looks like this:
<doc>
<row id ="1">
<column id="1" name="column1">value</column>
<column id="N" name="columnN">value</column>
</row>
<row id ="M">
<column id="1" name="column1">value</column>
<column id="N" name="columnN">value</column>
</row>
</doc>

Hi all,
Thank you very much for your replies.
First, saxon didn't work because it uses an in-memory parser, and that is what I was trying to avoid.
Different database is also out of the question, because the customer insist on XML, and also there are some files that can never be converted to a database table, because eventually with some transformations thay are changed and are not completely like the standard csv format.
I think that maybe http://exist.sourceforge.net is the rigth solution for me, but I will probably try it in the next version of my project.
For now I have managed to make the project with only SAXParser and a lot of back - end programming and it works ok, althoug it was very hard to make it, and will be harded to maintain, so I will try to look at the eXist project.
Thanks everyone for the help.

Large XML file Loading

I have a large XML file that I am converting to an
ArrayCollection to use as a dataprovider for a datagrid. It takes
sometime to fully load. Is there any way to load partial list while
the rest of the list is loading?? or does anyone know a way speed
up this process??
Thanks

I'd try to modify the autoComplete component.
You could break this processing up into smaller chunks. For
it to work, you need some outside counter or indexer that keeps
track of where you are. Have the conversion function process say
nodes 0-500, then end. Then using callLater, call that function
again, to process ne next batch of nodes.
This process will allow the UI to update between iteration
batches. If you need more responsiveness, you could try monitoring
mouse move, and stopping the conversion, until the mouse is
inactive again. That is just brainstorming. I have not tried it
(the mouse move part. I know the iterator method works to allow the
UI to update.)
Tracy

Best technology to navigate through a very large XML file in a web page

Hi!
I have a very large XML file that needs to be displayed in my web page, may be as a tree structure. Visitors should be able to go to any level depth nodes and access the children elements or text element of those nodes.
I thought about using DOM parser with Java but dropped that idea as DOM would be stored in memory and hence its space consuming. Neither SAX works for me as every time there is a click on any of the nodes, my SAX parser parses the whole document for the node and its time consuming.
Could anyone please tell me the best technology and best parser to be used for very large XML files?

Thank you for your suggestion. I have a question,
though. If I use a relational database and try to
access it for EACH and EVERY click the user makes,
wouldn't that take much time to populate the page with
data?
Isn't XML store more efficient here? Please reply me.You have the choice of reading a small number of records (10 children per element?) from a database, or parsing multiple megabytes. Reading 10 records from a database should take maybe 100 milliseconds (1/10 of a second). I have written a web application that reads several hundred records and returns them with acceptable response time, and I am no expert. To parse an XML file of many megabytes... you have already tried this, so you know how long it takes, right? If you haven't tried it then you should. It's possible to waste a lot of time considering alternatives -- the term is "analysis paralysis". Speculating on how fast something might be doesn't get you very far.

Efficient searching in a large XML file for specific elements

Hi
How can I search in a large XML file for a specific element efficiently (fast and memory savvy?) I have a large (approximately 32MB with about 140,000 main elements) XML file and I have to search through it for specific elements. What stable and production-ready open source tools are available for such tasks? I think PDOM is a solution but I can't find any well-known and stable implementations on the web.
Thanks in advance,
Behrang Saeedzadeh.

The problem with DOM parsers is that the whole document needs to be parsed!
So with large documents this uses up a lot of memory.
I suggest you look at sometthing like a pull parser (Piccolo or MPX1) which is a fast parser that is program driven and not event driven like SAX. This has the advantage of not needing to remember your state between events.
I have used Piccolo to extract events from large xml based log files.
Carl.

Procedure for read/write an XML-File using UCCX9

Hi Team!
Is there someone who can explain me the procedure of reading/writing an XML-File using Cisco UCCX Rel. 9.0?
In the last couple of days I did a lot of research on the partner community as well as reading the CRS-Editor-PDFs.
But I can’t find a solution.
There are some templates available but they all didn’t seem to work.
I have two xml-files, both uploaded to “Document Management\en_us\”.
The script itself is attached to this posting.
I build that my script according to a sample script I found in the community.
From my point of view I have configured my script exactly like the sample script.
But it still doesn’t work.
I am especially interested in where to store the the appropriate files.
I would be very happy if there is someone who can explain me, what’s wrong there.
Many thanks in advance.
Kind regards
Michael

I'm experiencing the same issue [reading XML causes errors that say things from "no such file" to some odd Java exception.] I'm able to use a template XML file and after authenticating a user, transform the template into a proper XML file and then successfully save it to the en_US Doc Repository with updated values.
However, no matter what I've tried or who I've asked, the READ on the XML file so that I can actually script something that does some routing based on the value of the XML file created in my first script? Ultra fail.
I've even worked with three separate TAC IPCC engineers who were able to reproduce the issue in their test environment, but they were unable to assist. Eventually, whe pressed, I received this response from the last IPCC engineer.:
"Hello Brad,
Yes I know what I provided you with is what you already have achieved and this missing step is precisely what you need: how to setup the script to check the status and value of this XML document and then decide routing according to the state. But unfortunately this is what I found internally. The other resources are the ones you already know: support forums, scripting guides. The other resource with suggest to customer looking for script assistance is the Cisco Account Manager of your company, they know where to get the resources for such issues. Then again as I mentioned before on our phone conversation TAC does not support scripting but certainly if I find something else on my side I will let you know for sure."
I would do ponderous cartwheels of joy if someone who has this working in version 9.x [9.0.2 specifically] could share the deep magic.

How to update an XML file?

hi,
I am having an xml file like this.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE map [ <!ELEMENT map (entry*) >
<!ELEMENT entry EMPTY >
<!ATTLIST entry key ID #REQUIRED
value CDATA "mydefault"> ]>
<map>
<entry key="key1" value="xxx"/>
<entry key="key2" value="yyy"/>
</map>
I have written a DOMparsing program to parse this XML file. And the program is this,
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.FactoryConfigurationError;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.Transformer;
import javax.xml.transform.stream.StreamSource;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import javax.xml.transform.TransformerException;
import org.xml.sax.SAXException;
import org.xml.sax.SAXParseException;
import java.io.File;
import java.io.IOException;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.NamedNodeMap;
import org.w3c.dom.DOMException;
import org.w3c.dom.Attr;
import org.w3c.dom.NodeList;
public class DomExample
public static void main(String args[])
// Obtain an XML document; this method is implemented in
Document doc = parseXmlFile("sample1.xml", true);
//Getting elements by Tag name
NodeList nodeList= doc.getElementsByTagName("entry");
// Obtain an element
Element element = (Element)(nodeList.item(0));
element.removeAttribute("value");
element.setAttribute("value", "Manivannan");
try
TransformerFactory tFactory =TransformerFactory.newInstance();
StreamSource stylesource = new StreamSource();
Transformer transformer = tFactory.newTransformer(stylesource);
DOMSource source = new DOMSource(doc);
//File f1 = new File("Sample3.xml");
StreamResult result = new StreamResult(System.out);
transformer.transform(source, result);
catch(TransformerException te)
te.printStackTrace();
public static Document parseXmlFile(String filename, boolean validating) {
try {
// Create a builder factory
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setValidating(validating);
DocumentBuilder builder = factory.newDocumentBuilder();
// Create the builder and parse the file
Document doc = builder.parse(new File(filename));
return doc;
} catch (SAXException e) {
// A parsing error occurred; the xml input is not valid
} catch (ParserConfigurationException e) {
} catch (IOException e) {
return null;
Now what i am trying to do is replace value xxx in my xml with "Manivannan" and yyy with "gopalan"..But what i am getting during running this program is i am getting MalformedURL exception..Also will this program be able to update my xml with new values.If not possible pls let me know the method to update my xml file.
Thanks for your reply in advance.

Replace
StreamSource stylesource=new StreamSource();
Transformer transformer=tfactory.newTransformer(stylesource);with
Transformer transformer=tfactory.newTransformer();

Read/update large xml files

Similar Messages

Maybe you are looking for