How to unmarshal large XML file

I have a large (170 MB) XML document. The size is due to a complex tree structure that is an array. It's taking approximately 45 minutes to unmarshal the document into the OTD. Not sure if there's a way to process this XML document in pieces (logical section)?? Suggestions are appreciated.

I tried your suggestion, but get an error when I try to export the XSD node. It's expecting a TargetNamespace. Here is the beginning of the XSD that I imported:
<?xml version="1.0" encoding="utf-8"?>
<xs:schema elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="MeterReadingDocument" nillable="true" type="MeterReadingDocument" />
<xs:complexType name="MeterReadingDocument">
<xs:complexContent mixed="false">
<xs:extension base="XmlDocumentBase">
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="1" name="Remarks" type="ArrayOfString" />
<xs:element minOccurs="0" maxOccurs="1" name="Header" type="ExportHeader" />
<xs:element minOccurs="0" maxOccurs="1" name="ImportExportParameters" type="ImportExportParameters" />
<xs:element minOccurs="0" maxOccurs="1" name="Channels" type="ArrayOfChannel" />
</xs:sequence>
</xs:extension>
</xs:complexContent>
</xs:complexType>
If I add a targetNamespace, then I get an exception when I attempt to unmarshal the file. I get the message:
There was an error during unmarshal.
Expect: '{http://www.w3.org/2001/XMLSchema}MeterReadingDocument', found: 'MeterReadingDocument'
Not sure what to do.
Thanks.

Similar Messages

  • How to insert large xml file to XMLType column?

    Hi,
    I have a table with one column as XMLType(Binary XML storage option and Free Text Indexing). When i try to insert a large XML as long as 8kb, i'm getting an error ORA-01704:string literal too long.
    Insert into TEST values(XMLTYPE('xml HERE'));
    How to insert large XML values to XMLType column?
    Regards,
    Sprightee

    For a large XML file, you basically have two options - you can load the string directly as an XMLType, or you can load the string as a CLOB and cast it on the database side to an XMLType.
    If you decide to load the XML as XmlType client-side, then you may be interested to know that versions of Oracle after 11.2.0.2 support the JDBC 4.0 SQLXML standard. See the JDBC driver release documentation here:
    http://docs.oracle.com/cd/E18283_01/java.112/e16548/jdbcvers.htm#BABGHBCC
    If you want to load as a CLOB, then you'll need to use PreparedStatement's setClob() method, or allocate an oracle.sql.clob object.
    For versions before 11.2.0.2, you can create an XMLType with a constructor that includes an InputStream or byte[] array.
    HTH
    Edited by: 938186 on Jun 23, 2012 11:43 AM

  • How to parse large xml file

    I need to parse large xml file which contains following tag. The size of the file is upto 10MB-50MB or more.
    <departments>
    <department>
    <a_depart id="124">
    <b_depart id="Bss_253">
    <bss_depart id="253">
    <attributes>
    <name_one>abc</name_one>
    </attributes>
    </bss_depart id="253">
    </b_depart id="Bss_253">
    </a_depart id="124">
    </department>
    <department>
    <a_depart id="124">
    <b_depart id="Bss_254">
    <mss_depart id="253">
              <attributes>
              <name_one>abc</name_one>
              <name_two>xyz</name_one>
              </attributes>
         </mss_depart>
         </b_depart>
    </a_depart>
    </department>
    <department>
    <a_depart id="124">
    <b_depart id="Bss_254">
    <mss_depart id="255">
              <attributes>
              <name_one>abc</name_one>
              <name_two>xyz</name_one>
              </attributes>
         </mss_depart>
         </b_depart>
    </a_depart>
    </department>
    <department>
    <a_depart id="125">
    <b_depart id="Bss_254">
    <mss_depart id="253">
              <attributes>
              <name_one>abc</name_one>
              <name_two>xyz</name_one>
              </attributes>
         </mss_depart>
         </b_depart>
    </a_depart>
    </department>
    I want to get the infomation for that xml file. like mss_depart id=233, building xpath dyanmically for every id and loading
    that using dom4j. which is very very slow.
    Is there any other solution for that to read the data using sax parser only.
    I want to execute the xpath or data for the following way.
    //a_depart/@id ------> all the ids of a_depart tags if it returns 3 values say 123,124,125
    after that i want to execute
    //a_depart[@id='123']/b_depart/@id like this ...to retrive the values of all the levels ...
         I am executing following xpath for every unique ids at all levels.
         List l = doc.selectNodes(xPathForID);
         List l1 = doc.selectNodes(xPathForAttributes+attributes.get(j)+"/text()");
    But it is very slow and taking lot of time.
    Is there any other way to solve this problem. If any please mail me it is urgent.
    I am using jdk1.4 and jdk1.5
    Is there any support for sax parser to execute xpath in jdk1.5 direclty, with out using dom4j
    Thanks in advance....

    I doubt you will find a preexisting solution to your problem.
    SAX is usually recommended for processing big files (where "big" is undefined"). It works on big files by avoiding the messy problem of storing the data -- that is left as an exercise to you.
    DOM (and its variants) works by building a Document object as the head of the tree of objects for the entire contents. With DOM, you can then use XPath, because there is something to search that is already in memory. To use XPath, you seem to have two choices, build a DOM-ish tree, or if you can find an XPath processor (I'm not sure if one exists) that can process the XML file directly, but it will be slow, since you are looking for "all" occurences of an attribute, and this means you have to read the entire file each time.
    It might be worth exploring a hybrid approach -- use SAX to get some information, and build your own objects to store the data. Maybe a HashMap as the main index. But, that will keep you from using XPath, since you do not have the data structures it expects.
    A third alternative would be to look at JAXB. It builds Java code from a Schema of your data and then when you import the data, it creates the necessary objects and fills in values. But, I don't think XPath woll work there either.
    Dave Patterson

  • How to insert large xml file ino xmltype column

    Hi all,
    iam using oracle10g. I have a table example1(key_column varchar2(10),xml_column xmltype) like this. i want to insert xml file from server to xmltype column. Can you please anybody suggest regarding this.wht is the size of the xmltype column. is it same as clob like 2GB or sth different. and one more thing is iam creating one procedure in that procedure , in that we are fetching xmltype value into clob varable. is it correct or not...
    Thanks

    Manipulating a single XMLType of 2GB is probably not sensible period. I'm guessing that the content of a file that big is probably a set of smaller XML documents which someone has wrapped a begin and end tag around. Since databases are good at handling collections of documents (or collections of anything else for that matter), you should consider de-compising the large document back into a set of smaller documents before storing it in the database

  • How to set SAXParser at command-line interface to create a large XML file

    Hi,
    I am trying to create a large XML file (more than 50 MB) by selecting from Oracle database but failed because of "out of memory" error. According to "Oracle XML Developer Guide", we should use SAXParser to parsing a large XML file. But there is no example to show how to set SAXParser at command-line
    Following is what I use to get xml files. It works only when the file is small.
    java OracleXML getXML -DateFormat -withDTD -rowsetTag PO_HDR -conn
    "jdbc:oracle:oci8:@server_name" -user "ID/password" "select * from table_name"
    When I set SAXParser at the way below,
    java oracle.xml.parser.v2.SAXParser OracleXML getXML -DateFormat -withDTD -rowsetTag PO_HDR -conn
    "jdbc:oracle:oci8:@server_name" -user "ID/password" "select * from table_name"
    it failed with the error message: "In class oracle.xml.parser.v2.SAXParser: void main(String argv[]) is not defined"
    Does anyone know how to solve the problem? I'll be appreciated very much for your help.
    Yi

    here are my ideas.
    register the xml schema.
    using xmldom, generate the desired xml output and return as xmltype.
    then you can use something like this to check.
    declare
    xmldoc xmltype ;
    begin
       -- populate xmldoc from you xmldom function
       -- validate against XML schema
       xmldoc.isSchemaValid(schema_url, root_element);
       if xmldoc.isSchemaValid = 1 then
            --valid schema
       else
            --invalid
       end if;
    end

  • Problem with parsing large XML files chunked over HTTP

    I'm trying to isolate a bug that was introduced when upgrading the JRE in use from Java 7u51 to 7u71 without changing any code. The problem appears to be very similar to: Bug ID: JDK-8027359 XML parser returns incorrect parsing results.
    Further investigation showed that it was also introduced in the same versions (7u71) where that fix was applied. Unlike that bug though, my XML is marked as version 1.0. It also appears to be with only large XML files, on the order of 10MB or so.
    The closest I've been able to narrow it down to is the code is using JAXB to unmarshall a stream that the debugger tells me is a org.apache.http.com.EofSensorInputStream / org.apache.http.impl.io.ChunkedInputStream. The exception I get is not consistent, but typically appears to be from chunks being overwritten or shuffled, resulting in letters appearing in attributes that are actually numbers, or like the following where an attribute "testAttribute" gets partially overwritten by the end of a timestamp that was in a different section of the XML.
    javax.xml.bind.UnmarshalException
    - with linked exception:
    [javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,98748]
    Message: Attribute name "testAttribu00Z" associated with an element type "testElement" must be followed by the ' = ' character.]
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.handleStreamException(UnmarshallerImpl.java:421)
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:357)
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:334)
    Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,98748]
    Message: Attribute name "testAttribu00Z" associated with an element type "testElement" must be followed by the ' = ' character.
      at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:598)
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:181)
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:355)
      ... 6 more
    Here's some code that seems to reproduce it if you can connect to an XML server that returns a large chunked XML file:
      SchemeRegistry registry = new SchemeRegistry();
      registry.register(
                    new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
      HttpClient client = new DefaultHttpClient(new BasicClientConnectionManager(registry));
      String url = "http://someUrlReturningAlargeChunkedXML";
      HttpGet method = new HttpGet(url);
      HttpResponse response = client.execute(method);
      InputStream inputStream = response.getEntity().getContent();
      XMLStreamReader responseReader = factory.createXMLStreamReader(inputStream);
      JAXBElement<JaxBObjectOfResponse> wot = unmarshaller.unmarshal(responseReader, JaxBObjectOfResponse.class);
    If you connect using URL.openStream() to the same service there is no error. If I read bytes directly and write to a file, there is no error. The error only happens when I try to unmarshal it, and it's large, and I'm using Java 7u71 (or later). It can be consistently repeated with the jsp webapp that I'm using, but didn't show the error when I used the same code with a Wikipedia dump XML file.
    How can I unmarshal in a different way to avoid this problem? Or, how can I better isolate the bug so it can be posted to the appropriate bug system?

    Apparently, adding the Woodstox XML libraries avoids the bug. Is there anyone who can reproduce this on another system? Was there any changes to the Stax implementation between u67 and u71 that may have introduced a bug like this?
    Edit: When setting the logging level to DEBUG, I once saw the overwritten buffer being logged as if that was what was received (as in the testAttribu00Z example above). I can't repeat that anymore though, and very rarely it does parses with no exception (though it may have still been corrupted). Now the error seems to be consistently on one of the buffer boundaries, as in:
    17:08:09,705 DEBUG wire:63 - << "2000[\r][\n]"
    17:08:09,705 DEBUG wire:77 - << "trend>....OTHER XML...<trend hours=""
    17:08:09,705 DEBUG wire:77 - << "634.0972777777778" datetime="2013-05-21T00:43:48.350Z" t"
    17:08:09,705 DEBUG wire:63 - << "[\r][\n]"
    17:08:09,705 DEBUG wire:63 - << "2000[\r][\n]"
    17:08:09,705 DEBUG wire:77 - << "rend-mode="0">
    Exception in thread "main" java.lang.NumberFormatException: t34.0972777777778
      at com.sun.xml.internal.bind.DatatypeConverterImpl._parseDouble(DatatypeConverterImpl.java:213)
      at mypackage.Trend_JaxbXducedAccessor_hours.parse(TransducedAccessor_field_Double.java:48)
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.StructureLoader.startElement(StructureLoader.java:194)
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallingContext._startElement(UnmarshallingContext.java:486)
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallingContext.startElement(UnmarshallingContext.java:465)
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.InterningXmlVisitor.startElement(InterningXmlVisitor.java:60)
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleStartElement(StAXStreamConnector.java:231)
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:165)
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:355)
      at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:334)
    Or:
    17:19:12,563 DEBUG wire:63 - << "2000[\r][\n]"
    17:19:12,563 DEBUG wire:77 - << ...OTHER XML...<trend index="5"
    17:19:12,563 DEBUG wire:77 - << "" label="N"
    17:19:12,563 DEBUG wire:63 - << "[\r][\n]"
    Exception in thread "main" java.lang.NumberFormatException: Not a number: N
      at com.sun.xml.internal.bind.DatatypeConverterImpl._parseInt(DatatypeConverterImpl.java:106)
      at com.sun.xml.internal.bind.DatatypeConverterImpl._parseShort(DatatypeConverterImpl.java:118)

  • Is there a way to import large XML files into HANA efficiently are their any data services provided to do this?

    1. Is there a way to import large XML files into HANA efficiently?
    2. Will it process it node by node or the entire file at a time?
    3. Are there any data services provided to do this?
    This for a project use case i also have an requirement to process bulk XML files, suggest me to accomplish this task

    Hi Patrick,
         I am addressing the similar issue. "Getting data from huge XMLs into Hana."
    Using Odata services can we handle huge data (i.e create schema/load into Hana) On-the-fly ?
    In my scenario,
    I get a folder of different complex XML files which are to be loaded into Hana database.
    Then I gotta transform & cleanse the data.
    Can I use oData services to transform and cleanse the data ?
    If so, how can I create oData services dynamically ?
    Any help is highly appreciated.
    Thank you.
    Regards,
    Alekhya

  • How to edit an XML file in ios5.1 ?

    Hi,
    I have an large xml file . I need to split it into 3 small ones .For reading the XML i am using NSXMLParser . How to write xml file ? If there is need of edit some tag , how to do it ?Please guide me ....
    Thanks
    Amiya

    I don't know what your model is but if I assume a straighforward tree-structured model PresetInfo > Modality > Body, this code is the design pattern I usually use.  If you don't actually need to load a model, but instead just extract a specific data element, look at the XPath query in PresetInfo.parseXmlElement, and how I get attribute and element values in the other parseXmlElements.
    The model interface.  I used arrays for the lists, if you need keyed access to list elements, you could use dictionaries.  The top-level class PresetInfo contains methods to read the XML from a file.
    @interface ModelBase : NSObject
    - (id)initWithXmlElement:(GDataXMLElement *)element;
    - (void)parseXmlElement:(GDataXMLElement *)element;
    @end
    @interface PresetInfo : ModelBase
    @property NSMutableArray *modalities;
    - (id)initWithPath:(NSString *)path;
    - (id)initWithXmlString:(NSString *)xmlString;
    @end
    @interface Modality : ModelBase
    @property NSString *type;
    @property NSMutableArray *bodies;
    @end
    @interface Body : ModelBase
    @property NSString *part;
    @property NSInteger ww;
    @property NSInteger wl;
    @end
    The model implementation.
    @implementation ModelBase
    - (id)initWithXmlElement:(GDataXMLElement *)element
        if (self = [self init]) {
            [self parseXmlElement:element];
        return self;
    - (void)parseXmlElement:(GDataXMLElement *)element {
        [NSException raise:NSInternalInconsistencyException format:@"You must override %@ in a subclass", NSStringFromSelector(_cmd)];  // Objective-C has no abstract methods
    @end
    @implementation PresetInfo
    - (id)init
        if (self = [super init]) {
            self.modalities = [[NSMutableArray alloc] init];
        return self;
    - (id)initWithPath:(NSString *)path
        NSError *error = nil;
        NSStringEncoding *usedEncoding = nil;
        NSString *xmlString = [NSString stringWithContentsOfFile:path usedEncoding:usedEncoding error:&error];
        if (!xmlString)
            NSLog(@"Error reading XML file: %@", [error localizedDescription]);
        return [self initWithXmlString:xmlString];
    - (id)initWithXmlString:(NSString *)xmlString
        NSError *error = nil;
        GDataXMLDocument *xmlDoc = [[GDataXMLDocument alloc] initWithXMLString:xmlString  options:0 error:&error];
        if (!xmlDoc)
            NSLog(@"Error creating XML document: %@", [error localizedDescription]);
        return [self initWithXmlElement:xmlDoc.rootElement];
    - (void)parseXmlElement:(GDataXMLElement *)element
        NSError *error = nil;
        NSArray *nodes = [element nodesForXPath:@"presetinfo/modality" error:&error];
        if (!nodes) {
            NSLog(@"Response not found: %@", [error localizedDescription]);
            return;
        for (GDataXMLElement *element in nodes)
            [self.modalities addObject:[[Modality alloc] initWithXmlElement:element]];
    @end
    @implementation Modality
    - (id)init
        if (self = [super init]) {
            self.bodies = [[NSMutableArray alloc] init];
        return self;
    - (void)parseXmlElement:(GDataXMLElement *)element {
        GDataXMLNode *node = [element attributeForName:@"type"];
        if (node) self.type = node.stringValue;
        NSArray *nodes = [element elementsForName:@"body"];
        for (GDataXMLElement *element in nodes)
            [self.bodies addObject:[[Body alloc] initWithXmlElement:element]];
    @end
    @implementation Body
    - (void)parseXmlElement:(GDataXMLElement *)element {
        GDataXMLNode *node = [element attributeForName:@"PART"];
        if (node) self.part = node.stringValue;
        node = [element attributeForName:@"WW"];
        if (node) self.ww = [node.stringValue integerValue];
        node = [element attributeForName:@"WL"];
        if (node) self.wl = [node.stringValue integerValue];
    @end
    Example of loading your XML into the model and showing the data.  This assumes the containing class (for example a view controller) has a property PresetInfo *presetInfo and the XML you posted is in a file presetinfo.xml.
    // load the XML into the model
    NSString *path = [[NSBundle mainBundle] pathForResource:@"presetinfo" ofType:@"xml"];
    self.presetInfo = [[PresetInfo alloc] initWithPath:path];
    // verify the model loaded correctly
    for (Modality *modality in self.presetInfo.modalities) {
        for (Body *body in modality.bodies) {
            NSLog(@"Modality Type = %@; Body Part = %@; WW = %d; WL = %d", modality.type, body.part, body.ww, body.wl);

  • I want to load large raw XML file in firefox and parse by DOM. But, for large XML file the firefox very slow some time crashed . Is there any option to increase DOM handling memory in Firefox

    Actually i am using an off-line form to load very large XML file and using firefox to load that form. But, its taking more time to load and some time the browser crashed. through DOM parsing this XML file to my form. Is there any option to increase DOM handler size in firefox

    Thank you for your suggestion. I have a question,
    though. If I use a relational database and try to
    access it for EACH and EVERY click the user makes,
    wouldn't that take much time to populate the page with
    data?
    Isn't XML store more efficient here? Please reply me.You have the choice of reading a small number of records (10 children per element?) from a database, or parsing multiple megabytes. Reading 10 records from a database should take maybe 100 milliseconds (1/10 of a second). I have written a web application that reads several hundred records and returns them with acceptable response time, and I am no expert. To parse an XML file of many megabytes... you have already tried this, so you know how long it takes, right? If you haven't tried it then you should. It's possible to waste a lot of time considering alternatives -- the term is "analysis paralysis". Speculating on how fast something might be doesn't get you very far.

  • Query in a large xml file

    Hello,
    I'm trying to work with very large xml files which are created from csv files. These files may be very large - up to 1 GB ! Untill now I have managed to do several validations on these big xml files, and the only thing that works for me is SAX parser, DOM is out of the question because it fills up memory.
    My next task is to do queries on these files, smth like:
    select field1,field2 from file.xml
    where field3 = 'A'
    and (fileld4>'B' or field1='C')
    order by field2.
    I searched the net about finding out how to make queries on xml files (since I have never done queries on xml before), but I couldn't find which "query language" is best for large files. If I use XPath (XSLT) will that not cause me memory problems because XSLT represents the file as a memory object?
    My idea is to parse the file with SAX and check every row if it fits the where condition and then write it immediately to a result xml file. But validating the where statement can be very complicated without using some tool. Also the order by statement is another problematic issue.
    Does anyone have some more intelegent ideas about how I can do this? Please help! :(
    The xml file looks like this:
    <doc>
    <row id ="1">
    <column id="1" name="column1">value</column>
    <column id="N" name="columnN">value</column>
    </row>
    <row id ="M">
    <column id="1" name="column1">value</column>
    <column id="N" name="columnN">value</column>
    </row>
    </doc>

    Hi all,
    Thank you very much for your replies.
    First, saxon didn't work because it uses an in-memory parser, and that is what I was trying to avoid.
    Different database is also out of the question, because the customer insist on XML, and also there are some files that can never be converted to a database table, because eventually with some transformations thay are changed and are not completely like the standard csv format.
    I think that maybe http://exist.sourceforge.net is the rigth solution for me, but I will probably try it in the next version of my project.
    For now I have managed to make the project with only SAXParser and a lot of back - end programming and it works ok, althoug it was very hard to make it, and will be harded to maintain, so I will try to look at the eXist project.
    Thanks everyone for the help.

  • Best technology to navigate through a very large XML file in a web page

    Hi!
    I have a very large XML file that needs to be displayed in my web page, may be as a tree structure. Visitors should be able to go to any level depth nodes and access the children elements or text element of those nodes.
    I thought about using DOM parser with Java but dropped that idea as DOM would be stored in memory and hence its space consuming. Neither SAX works for me as every time there is a click on any of the nodes, my SAX parser parses the whole document for the node and its time consuming.
    Could anyone please tell me the best technology and best parser to be used for very large XML files?

    Thank you for your suggestion. I have a question,
    though. If I use a relational database and try to
    access it for EACH and EVERY click the user makes,
    wouldn't that take much time to populate the page with
    data?
    Isn't XML store more efficient here? Please reply me.You have the choice of reading a small number of records (10 children per element?) from a database, or parsing multiple megabytes. Reading 10 records from a database should take maybe 100 milliseconds (1/10 of a second). I have written a web application that reads several hundred records and returns them with acceptable response time, and I am no expert. To parse an XML file of many megabytes... you have already tried this, so you know how long it takes, right? If you haven't tried it then you should. It's possible to waste a lot of time considering alternatives -- the term is "analysis paralysis". Speculating on how fast something might be doesn't get you very far.

  • Transform Large XML files with XSL

    HELP, LARGE XML FILES
    I have got 30 - 50 MB large xml file, and i would like to transform it
    with xslt, i tried but i have got OutOfMemory Exception.
    I tried to find out the solution on JAVA site, but i didn't find it.
    I can not displit my xml file. I hope for some help.
    I tried really everything.
    Thanks a lot

    What is your machine configuration ?
    The above 2 suggestions would help, but it does depend on how your software is written.
    Please post more info about your environment and object design.
    Chintan

  • Efficient searching in a large XML file for specific elements

    Hi
    How can I search in a large XML file for a specific element efficiently (fast and memory savvy?) I have a large (approximately 32MB with about 140,000 main elements) XML file and I have to search through it for specific elements. What stable and production-ready open source tools are available for such tasks? I think PDOM is a solution but I can't find any well-known and stable implementations on the web.
    Thanks in advance,
    Behrang Saeedzadeh.

    The problem with DOM parsers is that the whole document needs to be parsed!
    So with large documents this uses up a lot of memory.
    I suggest you look at sometthing like a pull parser (Piccolo or MPX1) which is a fast parser that is program driven and not event driven like SAX. This has the advantage of not needing to remember your state between events.
    I have used Piccolo to extract events from large xml based log files.
    Carl.

  • How to insert folloiwng XML file in Oracle -

    Hello
    This is on Database 11g.
    I have a very large XML file with following elements. I am new to PL SQL XML programming. How can I make Oracle read the OS file location where the XML file is located, open the XML file and insert XML data residing in their tag elements and non-tag elements ?
    <BillOfMaterialItem revisionIdentifier="--" billOfMaterialItemIdentifier="645-173-1" billOfMaterialItemUniqueIdentifier="IVI123019387" itemQuantity="6" globalProductQuantityTypeCode="PerAssembly" globalProductQuantityTypeCodeOther="" notes="" description="3/8" WIDE" proprietarySequenceIdentifier="115510509">
    - <AdditionalAttributes groupLabel="BOM Information">
      <AdditionalAttribute name="Line Number" value="20" dataType="Float" />
      </AdditionalAttributes>
      </BillOfMaterialItem>
    {code}
    Thanks,                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

    forum has many example
    look at Re: Load xml data in Oracle table
    steps:
    - create oracle directory
    - create table(s) for your data, may be with column xmltype
    - parse xml files from oracle directory and insert into table(s)
    for example
    SQL> select * from v$version where rownum=1;
    BANNER
    Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Prod
    SQL>
    SQL> select *
      2      from (xmltable('*'
      3                     passing (xmltype(bfilename('GET_XML','sample.xml'),nls_charset_id('AL32UTF8')))
      4                     columns billofmaterialitem_revisionid varchar2(10) path '/BillOfMaterialItem/@revisionIdentifier'
      5                            , billofmaterialitem_itemid varchar2(20) path '/BillOfMaterialItem/@billOfMaterialItemIdentifier'
      6                            , BillOfMaterialItem_ItUId varchar2(20) PATH '/BillOfMaterialItem/@billOfMaterialItemUniqueIdentifier'
      7                            , billofmaterialitem_itemq varchar2(20) path '/BillOfMaterialItem/@itemQuantity'
      8                            , billofmaterialitem_gqtcode varchar2(20) path '/BillOfMaterialItem/@globalProductQuantityTypeCode'
      9                            , BillOfMaterialItem_notes varchar2(20) PATH '/BillOfMaterialItem/@notes'
    10                            , BillOfMaterialItem_desc varchar2(20) PATH '/BillOfMaterialItem/@description'
    11                            , billofmaterialitem_prseqid varchar2(20) path '/BillOfMaterialItem/@proprietarySequenceIdentifier'
    12                            , additionalattributes_grlbl varchar2(20) path '/BillOfMaterialItem/AdditionalAttributes/@groupLabel'
    13                            , additionalattribute_name varchar2(20) path '/BillOfMaterialItem/AdditionalAttributes//AdditionalAttribute/@name'
    14                            , additionalattribute_value varchar2(20) path '/BillOfMaterialItem/AdditionalAttributes//AdditionalAttribute/@value'
    15                            , AdditionalAttribute_dataType varchar2(20) PATH '/BillOfMaterialItem/AdditionalAttributes//AdditionalAttribute/@dataType'
    16                    )
    17           ) x
    18  /
    BILLOFMATERIALITEM_REVISIONID BILLOFMATERIALITEM_ITEMID BILLOFMATERIALITEM_ITUID BILLOFMATERIALITEM_ITEMQ BILLOFMATERIALITEM_GQTCODE BILLOFMATERIALITEM_NOTES BILLOFMATERIALITEM_DESC BILLOFMATERIALITEM_PRSEQID ADDITIONALATTRIBUTES_GRLBL ADDITIONALATTRIBUTE_NAME ADDITIONALATTRIBUTE_VALUE ADDITIONALATTRIBUTE_DATATYPE
    --                            645-173-1                 IVI123019387             6                        PerAssembly                                         3/8                     115510509                  BOM Information            Line Number              20                        Float
    SQL>

  • Reading large XML file using a file event generator and a JPD process

    I am using a FileEventGenerator and a JPD Subscription process to read a large XML file. The large XML file basically contains repeated XML elements. My understanding is that the file subscription method reads the whole file in memory which causes lots of problem for huge file size like 1MB. Is there a way to read the file size-wise or is there a way to read chunks of data from a large size file..or any other alternative. I would like to process the file in a loop iteration by iteration.

    Hitejain,
    Here are a couple of pointers you could try. One is that the file event generator has a pass by reference (filename) functionality which you could use so that you could do the following inside of your process.
    1) Read file name from the reference
    2) Move the file to a processed directory (so it doesn't get picked up again. Note: I don't know how the embedded archive methods of the file event generator plays with pass by reference.
    3) Open a stream to the file.
    4) Use a SAX or SAX - DOM combined approach to parse your XML while managing the memory usage inside of your process
    There is another possibility which might fit your needs and it is related to the RawData object that BEA provides. If I understand it correctly provides wrapping functionality around a stream object, but depending on your parsing methods might just postpone the problem.
    Hope this helps
    Chris Falling
    Stormforge Software

Maybe you are looking for

  • How do i deduct dates using formatted search with a query

    Hi Experts, I would like to know how to get the date difference using query and formatted search. Below are the details: 1. I have created 3 Header UDF's composed of 2  Date type fields and 1 Quantity field. 2. The first one is for the Original Date,

  • Calling a PL/SQL from Forte code

    Hi, We are using the Sun UDS 5.1.3 I need to call directly from the code a PL/SQL stored procedure. It is true that this would defeat the Forte layer purpose but I have a very specific case. Is there an easy way to do so?

  • How to include psp in jsp

    can you tell me the information through which we can include psp in our jsp pages with the <jsp:include/> tag ,since it is mentioned by oracle documentation we can include psp pages in our jsp. So can somebody give me the solution since in my project

  • Anyone else have a recurring kernel panic on the 13inch i7 upon waking up from a long sleep?

    The laptop comes back without sound after restarting it.  It generally takes another restart or a shutdown/boot up to bring the sound back.  When there's no sound, I'm unable to adjust the volume as well, as you can see in the screenshot above. === O

  • Exeption  - trying to write to file, using CSVWriter.

    I have got the following exception: Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException at application.utils.GestureGuideLogger.logExcel(GestureGuideLogger.java :238) the row code is:      238:     writerExcel.writeNext(new String[]