Generating large xml files

i am generating large xml files these are generated using XML Classgen from a DTD, but the since the XML being generated is a large file in 250+ Mb range it invariables runs into memory problems.
also i need the output to be a single file, so is there anything that can be done.
Another problem i am facing is the setting the standalone attribute in stored in the <?xml ...?> tag since neither the CGDocument and its CGNode contain to methods to set this
null

If the XML is produced from a database query, I'd recommend using the XML SQL Utility's getXMLSAX() method to process XML query results of arbitrarily large size.
You implement a SAX2 ContentHandler and pass that to the getXMLSAX() method.

Similar Messages

Reading large XML file using a file event generator and a JPD process

I am using a FileEventGenerator and a JPD Subscription process to read a large XML file. The large XML file basically contains repeated XML elements. My understanding is that the file subscription method reads the whole file in memory which causes lots of problem for huge file size like 1MB. Is there a way to read the file size-wise or is there a way to read chunks of data from a large size file..or any other alternative. I would like to process the file in a loop iteration by iteration.

Hitejain,
Here are a couple of pointers you could try. One is that the file event generator has a pass by reference (filename) functionality which you could use so that you could do the following inside of your process.
1) Read file name from the reference
2) Move the file to a processed directory (so it doesn't get picked up again. Note: I don't know how the embedded archive methods of the file event generator plays with pass by reference.
3) Open a stream to the file.
4) Use a SAX or SAX - DOM combined approach to parse your XML while managing the memory usage inside of your process
There is another possibility which might fit your needs and it is related to the RawData object that BEA provides. If I understand it correctly provides wrapping functionality around a stream object, but depending on your parsing methods might just postpone the problem.
Hope this helps
Chris Falling
Stormforge Software

How to set SAXParser at command-line interface to create a large XML file

Hi,
I am trying to create a large XML file (more than 50 MB) by selecting from Oracle database but failed because of "out of memory" error. According to "Oracle XML Developer Guide", we should use SAXParser to parsing a large XML file. But there is no example to show how to set SAXParser at command-line
Following is what I use to get xml files. It works only when the file is small.
java OracleXML getXML -DateFormat -withDTD -rowsetTag PO_HDR -conn
"jdbc:oracle:oci8:@server_name" -user "ID/password" "select * from table_name"
When I set SAXParser at the way below,
java oracle.xml.parser.v2.SAXParser OracleXML getXML -DateFormat -withDTD -rowsetTag PO_HDR -conn
"jdbc:oracle:oci8:@server_name" -user "ID/password" "select * from table_name"
it failed with the error message: "In class oracle.xml.parser.v2.SAXParser: void main(String argv[]) is not defined"
Does anyone know how to solve the problem? I'll be appreciated very much for your help.
Yi

here are my ideas.
register the xml schema.
using xmldom, generate the desired xml output and return as xmltype.
then you can use something like this to check.
declare
xmldoc xmltype ;
begin
   -- populate xmldoc from you xmldom function
   -- validate against XML schema
   xmldoc.isSchemaValid(schema_url, root_element);
   if xmldoc.isSchemaValid = 1 then
        --valid schema
   else
        --invalid
   end if;
end

What are the best tools for opening very large XML files and examining the tree and confirming they are valid?

I am generating some very large XML files (600,000+ lines, 50MB+ characters). I finally have them all being valid XML and valid UTF-8.
But the files are so large Safari and Chrome will often not open them. FireFox will though.
Instead of these browsers, I was wondering if there are there any other recommended apps for the Mac for opening and viewing the XML, getting an error message if they are not valid for some reason and examing the XML tree?
I opened the file in the default app for XML which is Xcode, but that is just like opening it in a plain text editor. You can't expand/collapse the XML tree like you can with a browser, and it doesn't report errors.
Thanks,
Doug

Hi Tom,
I had not seen that list. I'll look it over.
I'm also in touch with the developer of BBEdit (they are quite responsive) and they are willing to look at the file in question and see why it is not reporting UTF-8 errors while Chrome is.
For now I have all the invalid characters quashed and things are working. But it would be useful in the future.
By the by, some of those editors are quite pricey!
doug

XSU: Dealing with large tables / large XML files

Hi,
I'm trying to generate a XML file from a "large" table (about 7 million lines, 512Mbytes of storage) by means of XSU. I get into "java.lang.OutOfMemoryError" even after raising the heap size up to 1 Gbyte (option -Xmx1024m of the java cmd line).
For the moment, I'm involved in an evaluation process. But in a near future, our applications are likely to deal with large amount of XML data, (typically hundreds of Mbytes of storage, which means possibly Gbytes of XML code), both in updating/inserting data and producing XML streams from existing data in relationnal DB.
Any ideas about memory issues regarding XSU? Should we consider to use XMLType instead of "classical" relational tables loaded/unloaded by means of XSU?
Any hint appreciated.
Regards,
/Hervi QUENIVET
P.S. our environment is Linux red hat 7.3 and Oracle 9.2.0.1 server

Hi,
I'm trying to generate a XML file from a "large" table (about 7 million lines, 512Mbytes of storage) by means of XSU. I get into "java.lang.OutOfMemoryError" even after raising the heap size up to 1 Gbyte (option -Xmx1024m of the java cmd line).
For the moment, I'm involved in an evaluation process. But in a near future, our applications are likely to deal with large amount of XML data, (typically hundreds of Mbytes of storage, which means possibly Gbytes of XML code), both in updating/inserting data and producing XML streams from existing data in relationnal DB.
Any ideas about memory issues regarding XSU? Should we consider to use XMLType instead of "classical" relational tables loaded/unloaded by means of XSU?
Any hint appreciated.
Regards,
/Hervi QUENIVET
P.S. our environment is Linux red hat 7.3 and Oracle 9.2.0.1 server Try to split the XML before you process it. You can take look into XMLDocumentSplitter explained in Building Oracle XML Applications Book By Steven Meunch.
The other alternative is write your own SAX handler and send the chuncks of XML for insert

How to generate a XML file from a XML schema...?

Hi guys...!
I have a XML schema and I want to write a XML file based on that schema. I have heard about this feature that eclipse has, which given a XML schema, can generate a XML file for you.
Does anyone of you know about this feature? or is there another way to do it. The schema that I have is very huge...
I will really appreciate any help from you guys...
Thank...you..
--- Spirit Away...

don't cross post
http://forum.java.sun.com/thread.jspa?threadID=649205&tstart=10

Is there a way to import large XML files into HANA efficiently are their any data services provided to do this?

1. Is there a way to import large XML files into HANA efficiently?
2. Will it process it node by node or the entire file at a time?
3. Are there any data services provided to do this?
This for a project use case i also have an requirement to process bulk XML files, suggest me to accomplish this task

Hi Patrick,
I am addressing the similar issue. "Getting data from huge XMLs into Hana."
Using Odata services can we handle huge data (i.e create schema/load into Hana) On-the-fly ?
In my scenario,
I get a folder of different complex XML files which are to be loaded into Hana database.
Then I gotta transform & cleanse the data.
Can I use oData services to transform and cleanse the data ?
If so, how can I create oData services dynamically ?
Any help is highly appreciated.
Thank you.
Regards,
Alekhya

Generating a XML file from a JSP request Page

Hi,
I am very new to JAVA with XML. My need is i want to capture some data in in a JSP page then after submitting ,it will generate a XML file. How can i do that??? Any help will be highly appriciated. If anybody can give me a good example....
Thx

<BLOCKQUOTE><font size="1" face="Verdana, Arial">quote:</font><HR>Originally posted by Steven Muench ([email protected]):
Are you using our XML SQL Utility that comes with the Oracle XDK for Java? Is the correct result produced, but you are having trouble writing it to a file?<HR></BLOCKQUOTE>
hi steven,
yes i am using xml sql utility and have solved the problem..... as i was working from a remote system i didnot give the correct path that is the c:\.....etc of the machine i was giving http:// thats why i did not write to a file... now its working

Generate an XML file from a DOM tree

Hi,
I'm trying to generate an XML file from a DOM tree that I obtained from another XML file with the Xerces library. I used the following operation :
public static void writeXmlFile(Document doc, String filename) {
try {
Source source = new DOMSource(doc);
File file = new File(filename);
Result result = new StreamResult(file);
Transformer xformer = TransformerFactory.newInstance().newTransformer();
xformer.transform(source, result);
} catch (TransformerConfigurationException e) {
System.err.println(e);
System.exit(1);
} catch (TransformerException e) {
System.err.println(e);
System.exit(1);
But I have this Windows/Linux problem. When I execute this on Windows, everything is correct. But if I try on Linux (every distribution does the same thing), I obtain an XML file where everything is written on just ONE LINE : there is neither identation nor carriage return at the end of a line ... And that is pretty annoying 'cause I need to SEE what I generate ...
Thanks for your answer,
Blueberryfin.

Actually I would think that no indents and no new-lines would be more correct, but maybe it's an option for parsers to do it either way if you don't specify it. If you want to specify it, look at this post from last month:
http://forum.java.sun.com/thread.jsp?forum=34&thread=383400

How to generate an XML file from SSIS (based on an XSD)?

I have an XSD that describes an XML format. I also have a sample XML file as well from the XSD. I am trying to understand SSIS's capability to generate the XML file in an ETL process. I would need to create the XML file based on the XSD that
I have. It is safe to assume that all the data elements are in the DB that I am incorporating in the ETL and it is also safe to assume that the data in the XML may be from multiple different tables (so massaging is probably necessary).
Would this be a job for C#/VB or can I do this through an SSIS process/task? Please help me understand my options. Thanks.

Dear All,
Indeed the post by RamJaddu did not answer the question. I am having the same issue: I have been provided an .xsd file which contains a data schema, and I have all of the necessary
data in a .mdf file in order to create the required .xml file. Having queried SQL Server and received my data in "myDataSet" I can use C# to create the .xml ignoring the schema using
string strSchemaFile = @"F:\Documents\UserCost\VictorianDataCollection\VCDCSchema_v2.xsd";
string strXmlFile = "testXmlExport.xml";
myDataSet.WriteXmlSchema(strSchemaFile);
myDataSet.WriteXml(strXmlFile, XmlWriteMode.IgnoreSchema);
but how do I export the .xml file so that is conforms with the .xsd format?
If any of you have worked this out the answer would be most appreciated.
All the best,
Nick
"Everything should be made as simple as possible, but not simpler" - Einstein

How generate a XML file & how to save data in that file?

hi,
How to generate an xml file & how to add data in that xml file? is there anyone who can help me about this? Firstly i need to open a xml file and after that i need to save user data in that file? how i will do it? What will be the solution of this? please give me solution........
with regards
Bina

With an API like Xerces you can parse an XML document into a DOM tree. You can do modifications on that tree manually and then write it back out as XML data.
http://xerces.apache.org/xerces2-j/

In SAP-XI stop generating of xml files

Hi,
In sap-xi i created a simple file to file scenario. After completion
of end to end scenario the xml files are keep on generating. I have selected delete option in CommunicationChannel. The xml file from source folder is deleted. Still the taget xml is generating in Destination folder.
How to stop the generating of xml files
Any one tell me the solution for this.
Thanks,
Ram

Hi,
Do the Following,
1) Make sure that you deleted all the Files present in Source Directory
2)Select the Processing mode as Delete.
3)Now Place the File in the Source DIR path
4)See Whether the file got deleted from the Source or not.
5) Check the Sender CC in RWB
6) Check the SXMB_MONI to get the Success Flag.
7) Check the Receiver CC in RWB.
8) Check the Receiver Directory whether received the file or not.
Please check these one-by-one
Regards
Seshagiri

How I could generate an XML file from a report in version 4.0B

Good morning,
How could I generate an XML file from a report? Please note that I am using version 4.0B
I don't have access to
Billy Vital

Hi,
In the Class CL_XML_DOCUMENT,
we have a method EXPORT_TO_FILE to download an XML file.

My report contains a BLOB (image), why it can't generate an XML file?

Hi guys,
I created a report (Purchase Order) and it has a Signature field.
which is a BLOB column of an image (.JPEG) of the signature of the Purchase Order Approver.
When i run the report in the report builder its working and it shows the image of the signature.
BUT when i try to generate xml file to use it for the XML Publisher, it generates an xml file with error. which can't be loaded into the .rtf file.
i Googled around and it seems that there are limitations in XML Publisher that can not load a BLOB or LONG RAW fields.
So Please guys i need your help! what can i do now ?
They need the Purchase Order in a custom PDF format, which i can only make it with XML Publisher.
Best Regards,

Hello,
For questions about BI Publisher (Formelly XML Publisher) :
BI Publisher
Regards

Performance Problem in parsing large XML file (15MB)

Hi,
I'm trying to parse a large XML file(15 MB) and facing a clear performance problem. A Simple XML Validation using the following code snippet:
DBMS_LOB.fileopen(targetFile, DBMS_LOB.file_readonly);
DBMS_LOB.loadClobfromFile
tempCLOB,
targetFile,
DBMS_LOB.getLength(targetFile),
dest_offset,
src_offset,
nls_charset_id(CONSTANT_CHARSET),
lang_context,
conv_warning
DBMS_LOB.fileclose(targetFile);
p_xml_document := XMLType(tempCLOB, p_schema_url, 0, 0);
p_xml_document.schemaValidate();
is taking 30 mins on a HP-UX (4GB ram, 2 CPU) machine (Oracle version : 9.2.0.4).
Please explain what could be going wrong.
Thanks In Advance,
Vineet

Thanks Mark,
I'll open a TAR and also upload the schema and instance XML.
If i'm not changing the track too much :-) one more thing in continuation:
If i skip the Schema Validation step and directly insert the instance document into a Schema linked XMLType table, what does OracleXDB do in such a case?
i'm getting a severe performance hit here too... the same file as above takes almost 40 mins to Insert.
code snippet:
DBMS_LOB.fileopen(targetFile, DBMS_LOB.file_readonly);
DBMS_LOB.loadClobfromFile
tempCLOB,
targetFile,
DBMS_LOB.getLength(targetFile),
dest_offset,
src_offset,
nls_charset_id(CONSTANT_CHARSET),
lang_context,
conv_warning
DBMS_LOB.fileclose(targetFile);
p_xml_document := XMLType(tempCLOB, p_schema_url, 0, 0);
-- p_xml_document.schemaValidate();
insert into INCOMING_XML values(p_xml_document);
Here table INCOMING_XML is :
TABLE of SYS.XMLTYPE(XMLSchema "http://INCOMING_XML.xsd" Element "MatchingResponse") STORAGE Object-
relational TYPE "XDBTYPE_MATCHING_RESPONSE"
This table and type XDBTYPE_MATCHING_RESPONSE were created using the mapping provided in the registered XML Schema.
Thanks,
Vineet

Generating large xml files

Similar Messages

Maybe you are looking for