Processing large xml file (500mb)? break into small part? load into jtree

hi,
i'm doing an assignment to processing large xml file (500mb) and
load into jree using JAVA.
can someone advice me on the algorithm to do this?
how can i load a 500mb xml in a jtree without system hang?
how to i break my file and do the loading?

1 Is the file schema based binary XML.
2. The limits are dependant on storage model and chacater set.
3. For all NON-XML content the current limit is 4GBytes (Where that is bytes not characters). So for Character content in an AL32UTF8 database the limit is 2GB
4. For XML Content stored as CLOB the limit is the same as for character data (2GB/4GB) dependant on database character set.
5. For SB Based XML content stored in Object Relatioanl storage the limit is determined by the complexity and structures defiend in the XML Schema

Similar Messages

Process Large XML files

Hi,
I am working on a File-File scenarion and I have around 300 MB XML files files coming in.
My interface cud successfully process smaller files but it is not able to handle larger files.
Group pls direct me.
Thanks,
Nandini

Nandini,
Check out this thread...
File Adpater: Size of your processed messages
Other than parameter and hardware setting, use JAVA mapping rather than Message / XSLT mapping...Java mapping is much faster compare to any other sort of mapping...
Nilesh

How to load a base64-stream (always WORD-document) in large XML files ( 1 MB) into ORACLE 10 table

On our Oracle Server there are multipe XML files that I have to read and put the data into an ORACLE 10 table.
I have to threath these XML files one by one.
In the XML there is also een base64-stream (alwasys a WORD-document) . This base64-stream exists of 1.000.000 characters.
How can I read this base64-stream from the XML-file into a BLOB-column of my ORACLE 10 table
ORACLE 10
1 XML = +/- 1 MB
PL/SQL
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<slo xmlns="http://www.myschema.com/gasdedectie">
<LekKey>1999_036371_509627</LekKey>
<HuiNum>46</HuiNum>
<Res></Res>
<InfLig>TEST STRUI AFGESTORVEN - PLAANSTRAAT 46</InfLig>
<XWGS>3.637028</XWGS>
<YWGS>50.962667</YWGS>
<Pei>KESTENS</Pei>
<DatPei>1999-11-30T10:17:36.000+01:00</DatPei>
<Kan> </Kan>
<Doc>UEsDBBQABgAIAAAAIQB5gHbnswEAAHcGAAATAAgCW0NvbnRlbnRfVHlwZXNdLnhtbCCiBAIooAACAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC0VUtP20AQviP1P1h7rewNPVRVFYcDj2OL1FTluqzH
ycK+tDMB8u8764AVwMSBiIsle/d7zLc74+nJg7PFHSQ0wdfiuJqIArwOjfGLWvydX5Q/RIGkfKNs8FCL
NaA4mX05ms7XEbBgtMdaLIniTylRL8EprEIEzyttSE4Rv6aFjErfqgXIb5PJd6mDJ/BUUuYQs+kZtGpl
qTh/4M8bJ9EvRHG62ZelamFcxufvchCRwOILiIrRGq2Ia5N3vnnhq3z0VDGy24NLE/ErG39DIa8897Qt
8DbuJg7XchOhK+Y3559MA8WlSvRLOS5W3ofUyCboleOgqt3KA6WFtjUaenxmiyloQOSDdbbqV5wy/qnk
IR96hRTclbPSELjLFCIeH2ynJ818kMhAH/uQhy4LpLUFPFj6VRIb3l0RbMn/M7Q8b1vQfKvHz8RhmbHV
RmILO64GRHxQ+4g877Vy7ODxkXnUwj1c//k0F1vko0ZaHgJzdW1hj8TfGUZPPWqCeLCB7J6HX/+OZpck
d2fXaTwo0wfKfpprGV1y2+/RYr0iT7SDc4Y8xhtoBrRl99uY/QcAAP//AwBQSwMEFAAGAAgAAAAhAB6R
GrfzAAAATgIAAAsACAJfcmVscy8ucmVscyCiBAIooAACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAACMkttKA0EMhu8F32HIfTfbCiLS2d5IoXci6wOEmewBdw7MpNq+vaMgulDbXub058tP1puDm9Q7
pzwGr2FZ1aDYm2BH32t4bbeLB1BZyFuagmcNR86waW5v1i88kZShPIwxq6Lis4ZBJD4iZjOwo1yFyL5U
upAcSQlTj5HMG/WMq7q+x/RXA5qZptpZDWln70C1x1g2X9YOXTcafgpm79jLiRXIB2Fv2S5iKmxJxnKN
ain1LBpsMM8lnZFirAo24Gmi1fVE/1+LjoUsCaEJic/zfHWcA1peD3TZonnHrzsfIVksFn17+0ODsy9o
PgEAAP//AwBQSwMEFAAGAAgAAAAhAGtucxBiAQAA1AUAABwACAF3b3JkL19yZWxzL2RvY3VtZW50Lnht
bC5yZWxzIKIEASigAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAArJRNTwIxEIbvJv6HTe+2LCioYeGiJlwVI9fSnd2t
bj/SDir/3goBF2HXg7006TSd9+nbmRlPP1WdvIPz0uiMpLRHEtDC5FKXGXmeP1xck8Qj1zmvjYaMrMGT
6eT8bPwINcdwyVfS+iRk0T4jFaK9ZcyLChT31FjQ4aQwTnEMW1cyy8UbL4H1e70hc80cZHKQM5nlGXGz
POjP1zYo/53bFIUUcGfESoHGExJMqqAdEnJXAmZEQS75Njigr7Yk7DTDoIVBSeGMNwVSYRTbyn/Ljg5f
xjyua/AvEqv7ogCB/gfh6IgGw9o4Ri0cJ3z+jxf9Li/6MRm2r//tRpcFaUx5sfJo1CIYvi8JStk+yiSC
SrtohjFp2oszpVa3FudVTIYPWD4BYuj+Ro02gl1mpGF2xGtVDCOk0aqbLdusnT9yGZPBH1mxi3T5cBMT
oTAa53xZN6zYh3YQ7GAWT74AAAD//wMAUEsDBBQABgAIAAAAIQA/ebUM0KkAAO1pAwARAAAAd29yZC9k
b2N1bWVudC54bWzsfdmS4kiz5v2YzTuk5ZjNDadKC1pQ9l91TEhiBwmxc9OmFQTakIQEHDtm/zuc2xmz
uZoHmUf5n2Q8JMhMcs+qrK7qbrDuSq0eER4e7qHwL9z/8e87z71KrSh2Av/LNfEZv76yfCMwHX/x5Xo0
rH2qXF/Fieabmhv41pfrvRVf//vX//7f/pHdmIGx9Sw/uQISfnyThcaX62WShDcYFhtLy9Piz55jREEc
2MlnI/CwwLYdw8KyIDIxEifw/CiMAsOKYyhP0PxUi6+P5LzH1ILQ8qEsO4g8LYk/B9EC87RovQ0/AfVQ
SxzdcZ1kD7Rx5kQm+HK9jfybY4U+3VYIvXJTVOj45/RG9KgVT5RbvCkeOZCXiEWWC3UI/HjphHfN+FZq
0MTlqUrpS41IPff0XBYS1KPybpv8lj4QIy2Drrgj+IjcE8wwi5c8t+AD6t+7Xn1IkcBfasyxRxCJ2zq8
pQrnZZ5q4mmOf0vm21hzn7kwIr5HvutRsA1vqxM630et6a9vaaGB+Y6a4Uw+8u43LX4XgUdDd7DUQuv6
yjNumgs/iDTdhRplBHWFJPL6KygLPTD36G94ld2AsjHVL9c4XqtRFUK4Pl0SLVvbugm6w1C0SPD5m6ES
5S8Okr1rwaOp5n65HjqJa11j6EawTVzHtzqpe7qJoxsYlFW8qQfBGumJQaJFCTzjmFACKtTXPKjm7/Wg
qhnrgtjpWck3b5/MS8nrEBUE/UCJgsDOr8eHU6lkXmx242r+4nTNjj5VJVSUpcUJHzval+viUlHBI73j
+IHGhDeabyyD6Mp04mSYVxMdVW+POqCjCaqMQ/3RDeDi6TR2vNC1lAB6Em4Wuii1GpazWAJHSZpgWAKn
4JZuLR3fBNUF715fuYGxtgqGuNoeeNn0BcsFDsM9zXWDTAbj4GohupBX8Lacq11e0h79i1gR3oRB7CAF
2LgtvhYF3pdrI3C3nl+8Ds/Ith1byddPRc2ho/I3j1dPpwWhM7LjB2RDLdIWkRYuH1LGT0SeojnOaVq7
BFkuA9pQVAMYY0BTaIIgWPrYHsu2LSORikcRSzichueAnYjD+b/6XeszqEoPbGTBCzCOSnSFRI0EC1pI
Gm/rluUi03oFF4vnjF5aR21wjFoE4ogETLvJW3W80oEOio82EcTnwTB/XRsXOtAPhCXIpcXHITQJdWYu
geHNS+V/b6n3miJqiXa1jR7rvNcbEDpGso0s6GQ4uoH/j9WCo29gxzk1P1Uc6KeCNLDi2GXQvUWPwV1U
9rG74F3Er+MrGqpL0TuPuXt1eymKgmxpaSaMy4Lp51QwdHpWDd11wprjuoh76PgqurE8HY3RqGnCvMaA
6VgCeiuMHD9B3ajdxJGhQrcWx0lkJcYSXbaByvE6Bg+dbuRF3pWCKhCDqrzSs25gAmFtmwTAbe1mZ0ce
+gtG+cFg125gAL00eqC809thFCd1K/Cu0AE0AiqaU9fSToyqDI+eHkGF+QFqfN4U17/KvlxzNEnnL9y7
4zmJFV25DiiXCo5+BR8Qn0Fz5y8nmuMWx1CA60M5p4YeD+E0L/w42JCE3j+H41t1jI5vbrU0HIPEwD+g
1x+aMnQ9+TocDQZSry13B8MrnlfFOj9AFJKcDryLbBN68o3GsEywVJU6mUkFVA5+vJQTKYwS6HSoj2vZ
wGOSZEn0uO0Ayz
pQQAALNCCQAAAA==</Doc>
</slo>

If I understand your question correctly, the discussion in this thread
ORA-31167: 64k size limit for XML node
shows you do not have a lot of options.
As you are needing to extract the contents of a node from an XML document, and that node is > 64K, you are going to need to treat the entire XML as a CLOB and use CLOB functionality (aka INSTR, SUBSTR) to find and extract what you need.
Once your DB version is 11.1 or greater, then you can use the built-in that Billy mentions.
Note: mdrake is Mark Drake from Oracle.

OSB - Iterating over large XML files with content streaming

Hi @ll
I have to iterate over all item in large XML files and insert into a oracle database.
The file is about 200 MB and contains around 500'000, and I am using OSB 10gR3.
The XML structure is something like this:
<allItems>
<item>.....</item>
<item>.....</item>
<item>.....</item>
<item>.....</item>
<item>.....</item>
</allItems>
Actually I thought about using a proxy service with enabled content streaming and a "for each" action for iterating
over all items. But for this the whole XML structure has to be materialized into a variable otherwise it is not possible!
More about streaming large files can be found here:
[http://download.oracle.com/docs/cd/E13159_01/osb/docs10gr3/userguide/context.html#large_messages]
There is written "When you enable streaming for large message processing, you cannot use the ... for each...".
And for accessing single items you should should use an assign action with a xpath like "$body/allItems/item[1]";
this works fine and not the whole XML stream has to be materialized.
So my idea was to use the "for each" action and processing seqeuntially all items with a xpath like:
$body/allItems/item[$counter]
But the "for each" action just allows iterating over a sequence of xml items by defining an selection xpath
and the variable that contains all items. I would like to have a "repeat until" construct that iterates as long
$body/allItems/item[$counter] returns not null. Or can I use the "for each" action differently?
Does the OSB provides any other iterating mechanism? I know there is this spli-join construct that supports
different looping techniques, but as far I know it does not support content streaming, is this correct?
Did I miss somehting?
Thanks a lot for helping!
Cheers
Dani
Edited by: user10095731 on 29.07.2009 06:41

Hi Dani,
Yes, according to me this would be the best approach. You can use content-streaming to pass this large xml to ejb and once it passes successfully EJB should operate on this. If you want any result back (for further routing), you can get it back from EJB.
EJB gives you power of java to process this file and from java perspective 150 MB is not a very LARGE data. Ensure that you are using buffering. Check out this link for an explanation on Java IO Streams and, in particular, buffered streams -
http://java.sun.com/developer/technicalArticles/Streams/ProgIOStreams/
Try dom4J with xpp (XML Pull Parser) parser in case you have parsing requirement. We had worked with 1.2GB file using this technique.
Regards,
Anuj

Loading, processing and transforming Large XML Files

Hi all,
I realize this may have been asked before, but searching the history of the forum isn't easy, considering it's not always a safe bet which words to use on the search.
Here's the situation. We're trying to load and manipulate large XML files of up to 100MB in size.
The difference from what we have in our hands to other related issues posted is that the XML isn't big because it has a largly branched tree of data, but rather because it includes large base64-encoded files in the xml itself. The size of the 'clean' xml is relatively small (a few hundred bytes to some kilobytes).
We had to deal with transferring the xml to our application using a webservice, loading the xml to memory in order to read values from it, and now we also need to transform the xml to a different format.
We solved the webservice issue using XFire.
We solved the loading of the xml using JAXB. Nevertheless, we use string manipulations to 'cut' the xml before we load it to memory - otherwise we get OutOfMemory errors. We don't need to load the whole XML to memory, but I really hate this solution because of the 'unorthodox' manipulation of the xml (i.e. the cutting of it).
Now we need to deal with the transofmation of those XMLs, but obviously we can't cut it down this time. We have little experience writing XSL, but no experience on how to use Java to use the XSL files. We're looking for suggestions on how to do it most efficiently.
The biggest problem we encounter is the OutOfMemory errors.
So I ask several questions in one post:
1. Is there a better way to transfer the large files using a webservice?
2. Is there a better way to load and manipulate the large XML files?
3. What's the best way for us to transform those large XMLs?
4. Are we missing something in terms of memory management? Is there a better way to control it? We really are struggling there.
I assume this is an important piece of information: We currently use JDK 1.4.2, and cannot upgrade to 1.5.
Thanks for the help.

I think there may be a way to do it.
First, for low RAM needs, nothing beats SAX. as the first processor of the data. With SAX, you control the memory use since SAX only processes one "chunk" of the file at a time. You supply a class with methods named startElement, endElement, and characters. It calls the startElement method when it finds a new element. It calls the characters method when it wants to pass you some or all of the text between the start and end tags. It calls endElement to signal that passing characters is over, and to let you get ready for the next element. So, if your characters method did nothing with the base-64 data, you could see the XML go by with low memory needs.
Since we know in your case that the characters will process large chunks of data, you can expect many calls as SAX calls your code. The only workable solution is to use a StringBuffer to accumulate the data. When the endElement is called, you can decode the base-64 data and keep it somewhere. The most efficient way to do this is to have one StringBuffer for the class handling the SAX calls. Instantiate it with a big enough size to hold the largest of your binary data streams. In the startElement, you can set the length of the StringBuilder to zero and reuse it over and over.
You did not say what you wanted to do with the XML data once you have processed it. SAX is nice from a memory perspective, but it makes you do all the work of storing the data. Unless you build a structured set of classes "on the fly" nothing is kept. There is a way to pass the output of one SAX pass into a DOM processor (without the binary data, in this case) and then you would wind up with a nice tree object with the rest of your data and a group of binary data objects. I've never done the SAX/DOM combo, but it is called a SAXFilter, and you should be able to google an example.
So, the bottom line is that is is very possible to do what you want, but it will take some careful design on your part.
Dave Patterson

Is there a way to import large XML files into HANA efficiently are their any data services provided to do this?

1. Is there a way to import large XML files into HANA efficiently?
2. Will it process it node by node or the entire file at a time?
3. Are there any data services provided to do this?
This for a project use case i also have an requirement to process bulk XML files, suggest me to accomplish this task

Hi Patrick,
I am addressing the similar issue. "Getting data from huge XMLs into Hana."
Using Odata services can we handle huge data (i.e create schema/load into Hana) On-the-fly ?
In my scenario,
I get a folder of different complex XML files which are to be loaded into Hana database.
Then I gotta transform & cleanse the data.
Can I use oData services to transform and cleanse the data ?
If so, how can I create oData services dynamically ?
Any help is highly appreciated.
Thank you.
Regards,
Alekhya

Reading large XML file using a file event generator and a JPD process

I am using a FileEventGenerator and a JPD Subscription process to read a large XML file. The large XML file basically contains repeated XML elements. My understanding is that the file subscription method reads the whole file in memory which causes lots of problem for huge file size like 1MB. Is there a way to read the file size-wise or is there a way to read chunks of data from a large size file..or any other alternative. I would like to process the file in a loop iteration by iteration.

Hitejain,
Here are a couple of pointers you could try. One is that the file event generator has a pass by reference (filename) functionality which you could use so that you could do the following inside of your process.
1) Read file name from the reference
2) Move the file to a processed directory (so it doesn't get picked up again. Note: I don't know how the embedded archive methods of the file event generator plays with pass by reference.
3) Open a stream to the file.
4) Use a SAX or SAX - DOM combined approach to parse your XML while managing the memory usage inside of your process
There is another possibility which might fit your needs and it is related to the RawData object that BEA provides. If I understand it correctly provides wrapping functionality around a stream object, but depending on your parsing methods might just postpone the problem.
Hope this helps
Chris Falling
Stormforge Software

Large XML file Loading

I have a large XML file that I am converting to an
ArrayCollection to use as a dataprovider for a datagrid. It takes
sometime to fully load. Is there any way to load partial list while
the rest of the list is loading?? or does anyone know a way speed
up this process??
Thanks

I'd try to modify the autoComplete component.
You could break this processing up into smaller chunks. For
it to work, you need some outside counter or indexer that keeps
track of where you are. Have the conversion function process say
nodes 0-500, then end. Then using callLater, call that function
again, to process ne next batch of nodes.
This process will allow the UI to update between iteration
batches. If you need more responsiveness, you could try monitoring
mouse move, and stopping the conversion, until the mouse is
inactive again. That is just brainstorming. I have not tried it
(the mouse move part. I know the iterator method works to allow the
UI to update.)
Tracy

Large XML File in JMS Message

Hi Experts,
I got to send larg XML files (+20MB) to the Message listener for processing. Would you please comment if it would be a good approach or suggest some tested and authentic solution for this purpose. Prompt response will be highly appreciated.
Regards
Shahzad mahmood

You can use the following algo to send large messages
- use BytesMessage to send the large message
- break the message into chunks (chunk size can be configurable)
- in the header of the first message add a header prioperty to notify that this is the first message
- in the header add a property for the size of the each message
- in the header of the last message add a property to notify that its the last message
on the receiver side
- see the header property of the message
- if its first message, open a stream (may be file stream)
- keep writing messages to this stream till you receive the last message (identified by the header property)
- close the stream after processing the last message.
BTW some JMS products like FioranoMQ (www.fiorano.com) provide built-in support for large messages.
You might want to look into such products.
Thanks
Bhuvan

Extract data from xml file and insert into another exiting xml fil

hello,
i am searching extract data from xml file and insert into another exiting xml file by a java program. I understood it is easy to extract data from a xml file, and how ever without creating another xml file. We want to insert the extracted data into another exiting xml file. Suggestions?
Regards,
Zhuozhi

If the files are small, you can load the target file in a DOM document, insert the data from the source file and persist the DOM.
If the files are large, you probably want to use a StAX or SAX.

Problem with parsing large xml files

Hello All,
I am parsing a large xml file of 20MB and I use DocumentBuilder.parse(File). This method works for small xml files with size less than 20MB but the application hangs and doesn't through any error message when parsing 20MB xml files. Please let me know what I have to do at this point ?
Thanks & Regards,
Kumar.

Well... i can't agree.
If you have such structure:
<task>
<task/>
<task>
     <task>
        <task/>
     </task>
     <task/>
</task>
</task>
...you may always keep stack of tasks (at startElement push to top, and at endElement pop), so at every leaf of tree you will have all parents of that leaf.
for such structure:
<task id="1" parent="0"/>
<task id="2" parent="1"/>
<task id="3" parent="1"/>
<task id="4" parent="2"/>
<task id="5" parent="3"/>
...it will be much faster to go thro document by sax several times to build tree of tasks, than to load all document into memory...

Loading Large XML files using plsql

I have a process where there is a need to load large xml files (i.e. easily over 500k or more) into Oracle via an interface. Preference would be to use plsql or some plsql based utility if possible. I am looking for any suggestions on the best method to accomplish this. Currently running on 9.2.0.6. Thanks in advance.

I have a process where there is a need to load large xml files (i.e. easily over 500k or more) into Oracle via an interface. Preference would be to use plsql or some plsql based utility if possible. I am looking for any suggestions on the best method to accomplish this. Currently running on 9.2.0.6. Thanks in advance.

Import Large XML File to Table

I have a large (819MB) XML file I'm trying to import into a table in the format:
<ROW_SET>
<ROW>
<column_name>value</column_name>
</ROW>
<ROW>
<column_name>value</column_name>
</ROW>
</ROW_SET>
I've tried importing it with xmlsequence(...).extract(...) and ran into the number of nodes exceed maximum error.
I've tried importing it with XMLTable(... passing XMLTYPE(bfilename('DIR_OBJ','large_819mb_file.xml'), nls_charset_id('UTF8'))) and I gave up after it ran for 15+ hours ( COLLECTION ITERATOR PICKLER FETCH issue ).
I've tried importing it with:
insCtx := DBMS_XMLStore.newContext('schemaname.tablename');
DBMS_XMLStore.clearUpdateColumnList(insCtx);
DBMS_XMLStore.setUpdateColumn(insCtx,'column1name');
DBMS_XMLStore.setUpdateColumn(insCtx,'columnNname');
ROWS := DBMS_XMLStore.insertXML(insCtx, XMLTYPE(bfilename('DIR_OBJ','large_819mb_file.xml'), nls_charset_id('UTF8')));
and ran into ORA-04030: out of process memory when trying to allocate 1032 bytes (qmxlu subheap,qmemNextBuf:alloc).
All I need to do is read the XML file and move the data into a matching table in a reasonable time. Once I have the data in the database, I no longer need the XML file.
What would be the best way to import large XML files?
Oracle Database 11g Release 11.2.0.1.0 - 64bit Production
PL/SQL Release 11.2.0.1.0 - Production
"CORE     11.2.0.1.0     Production"
TNS for Linux: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production

This (rough) approach should work for you.
CREATE TABLE HOLDS_XML
        (xml_col XMLTYPE)
      XMLTYPE xml_col STORE AS SECUREFILE BINARY XML;
INSERT INTO HOLDS_XML
VALUES (xmltype(bfilename('DIR_OBJ','large_819mb_file.xml'), nls_charset_id('UTF8')))
-- Should be using AL32UTF8 for DB character set with XML
SELECT ...
FROM HOLD_XML HX
       XMLTable(...
          PASSING HX.xml_col ...)How it differs from your approach.
By using the HOLDS_XML table with SECUREFILE BINARY XML storage (which became the default in 11.2.0.2) we are providing a place for Oracle to store a parsed version of the XML. This allows the XML to be stored on disk instead of in memory. Oracle can then access the needed pieces of XML from disk by streaming them instead of holding the whole XML in memory and parsing it repeatedly to find the information needed. That is what COLLECTION ITERATOR PICKLER FETCH means. A lot of memory work. You can search on that term to learn more about it if needed.
The XMTable approach then simply reads this XML from disk and should be able to parse the XML with no issue. You have the option of adding indexes to the XML, but since you are simply reading it all one time and tossing it, there is no advantage to indexes (most likely)

Generating large xml files

i am generating large xml files these are generated using XML Classgen from a DTD, but the since the XML being generated is a large file in 250+ Mb range it invariables runs into memory problems.
also i need the output to be a single file, so is there anything that can be done.
Another problem i am facing is the setting the standalone attribute in stored in the <?xml ...?> tag since neither the CGDocument and its CGNode contain to methods to set this
null

If the XML is produced from a database query, I'd recommend using the XML SQL Utility's getXMLSAX() method to process XML query results of arbitrarily large size.
You implement a SAX2 ContentHandler and pass that to the getXMLSAX() method.

How to set SAXParser at command-line interface to create a large XML file

Hi,
I am trying to create a large XML file (more than 50 MB) by selecting from Oracle database but failed because of "out of memory" error. According to "Oracle XML Developer Guide", we should use SAXParser to parsing a large XML file. But there is no example to show how to set SAXParser at command-line
Following is what I use to get xml files. It works only when the file is small.
java OracleXML getXML -DateFormat -withDTD -rowsetTag PO_HDR -conn
"jdbc:oracle:oci8:@server_name" -user "ID/password" "select * from table_name"
When I set SAXParser at the way below,
java oracle.xml.parser.v2.SAXParser OracleXML getXML -DateFormat -withDTD -rowsetTag PO_HDR -conn
"jdbc:oracle:oci8:@server_name" -user "ID/password" "select * from table_name"
it failed with the error message: "In class oracle.xml.parser.v2.SAXParser: void main(String argv[]) is not defined"
Does anyone know how to solve the problem? I'll be appreciated very much for your help.
Yi

here are my ideas.
register the xml schema.
using xmldom, generate the desired xml output and return as xmltype.
then you can use something like this to check.
declare
xmldoc xmltype ;
begin
   -- populate xmldoc from you xmldom function
   -- validate against XML schema
   xmldoc.isSchemaValid(schema_url, root_element);
   if xmldoc.isSchemaValid = 1 then
        --valid schema
   else
        --invalid
   end if;
end

Processing large xml file (500mb)? break into small part? load into jtree

Similar Messages

Maybe you are looking for