Huge XML file

Hello,
I have a program that must store on file a lot of words and information about them; I chose as outout format XML; but there are so many words
that the file is 10MB; this under is an example of a row (there are 80000 like that); is there a way (XML) to save space? (I don't know, maybe changing something inside the XML)
<word hampost="2.7340257709269167E-6" spampost="2.255630617930008E-6" hamFreq="1" spamFreq="0"> ferreirinho </word>thanks,

mickey0 wrote:
Hello,
I have a program that must store on file a lot of words and information about them; I chose as outout format XML; but there are so many words
that the file is 10MB; this under is an example of a row (there are 80000 like that); is there a way (XML) to save space? (I don't know, maybe changing something inside the XML)
<word hampost="2.7340257709269167E-6" spampost="2.255630617930008E-6" hamFreq="1" spamFreq="0"> ferreirinho </word>thanks,The overhead of XML in this case is only about 50% or less--50% of the storage is spent on XML tags. That's not that much...it's not like it inflates the size by an order of magnitude.
You use 115 bytes in that example. About 65 of them are XML overhead, but then a further 20 or so are sunk in using a text format at all. You made the choice to use a human-readable format, but if you abandon that, you could shrink the size by at least 70%. And probably make it much faster to read.
If that matters. But when you're storing data like this, I assume you're going to want to be able to read it in an ad-hoc manner later on.

Similar Messages

Reading huge xml files in OSB11gR1(11.1.1.6.0)

Hi,
I want to read a huge xml file of size 1GB in OSB(11.1.1.6.0)?
I will be creating a (JCA)file adapter in jdeveloper and importing artifacts to OSB.
Please let me know the maximum file size that could be handled in OSB?
Thanks in advance.
Regards,
Suresh

Depends on what you intend to do after reading the file.
Do you want to parse the file contents and may be do some transformation? Or do you just have to move the file from one place to another for ex. reading from local system and moving to a remote system using FTP?
If you just have to move the file, I would suggest using JCA File/FTP adapter's Move operation.
If you have to parse and process the file contents within OSB, then it may be possible depending on the file type and what logic you need to implement. For ex. for very large CSV files you can use JCA File Adapter batching to read a few records at a time.

How to break up a huge XML file and generate serialized JSP pages

I have a huge xml file, with about 100 <item> nodes.
I want to process this xml file with pagination and generate jsp pages.
For example:
Display items from 0 to 9 on page 1 page1.jsp , 10 to 19 on page2.jsp, and so on...
Is it possible to generate JSP pages like this?
I have heard of Velocity but dont know if it will be the right technology for this kind of a job.

Thank you for your reply, I looked at the display tag library and it looks pretty neat with a lot of features, I could definitely use it in a different situation.
The xml file size is about 1.35 MB, and the size is unpredictable it could shrink or grow periodically depending on the number of items available.
I was hoping to create a documentation style (static pages) of the xml feed instead of having 1 jsp with dynamic pages
I was looking at Anakia : http://jakarta.apache.org/velocity/docs/anakia.html , may be it has features that enable me to create static pages but not very sure.
I think for now, I will transform the xml with an xsl file and pass the page numbers as input parameters to the xsl file
null

How to compare two huge xml files(50MB+) using Java Code

I want to compare two huge xml files using java code and need to find the difference of those xml files
is there any API for that

You should find third party API

Getting a iterator over a huge xml-file

Hi !
I have to persist data from some huge xml-files.
The structure of the xml is simple, like in:
<items>
<item>
</item>
<item>
</item>
</items>
The best way for me, is to get an iterator over the item-elements represented as a DOM strcture for easy access.
Because the files are so huge (80-150 MBytes) and I have no control over the creation of them, the underlying parser should be SAX oriented.
So I need a solution which will parse until one item-element (maybe spezified through a XPATH query) is completed and build a DOM-tree from it.
Then I can pass this DOM-element to my persistence layer. When this item is persisted, and the Iterator hasNext(), then I could do a simple .next() to get the next DOM representation of the item.
Is there such a solution out ? Or do I have to 'invent' such kind of parsing ?
Would by great to get some hints ... this problem is driving me crazy :)
Thank you in advance,
Gerald

Vaguely familiar, http://lists.xml.org/archives/xml-dev/200201/msg00032.html
I can't think of a DOM builder that would work off a fragment; you would probably have to create one yourself.
There's also http://drizzle.stanford.edu/~peastman/pax.html which might solve your problem.
Pete

URGENT -- Perofrmance issue while creating Huge XML file

All XML Experts Please Help....Thanks a lot in Advance
We are trying to create a XML file for a huge table.. 5 million rows and the performance is ver very bad.. Can some body help by giving me an idea what what my best approch could be... or what am I doing wrong in in the code below
CREATE OR REPLACE PROCEDURE Sales_1_Generate_Xml IS
temp_clob CLOB;
temp_buffer VARCHAR2(1);
amount BINARY_INTEGER := 1;
position INTEGER := 1;
filehandle utl_file.file_type;
error_number NUMBER;
error_message VARCHAR2(100);
length_count INTEGER;
qryctx dbms_xmlgen.ctxhandle;
BEGIN
qryctx := dbms_xmlgen.newcontext('select /* INDEX UF_SALES(UF_SALES_IX16) */
TRANSACTION_NUMBER     "Transaction_Number",
TRANSACTION_TYPE_ID     "Transaction_Type_ID",
PROCESS_FISCAL_DATE_ID     "Process_Fiscal_Date_ID",
INVOICE_FISCAL_DATE_ID     "Invoice_Fiscal_Date_ID",
ORDER_FISCAL_DATE_ID     "Order_Fiscal_Date_ID",
PROCESS_CALENDAR_DATE_ID     "Process_Calendar_Date_ID",
INVOICE_CALENDAR_DATE_ID     "Invoice_Calendar_Date_ID",
ORDER_CALENDAR_DATE_ID     "Order_Calendar_Date_ID",
CURRENT_TM_ID     "Current_TM_ID",
CUSTOMER_ID     "Customer_ID",
CUSTOMER_TYPE_ID     "Customer_Type_ID",
CUSTOMER_LEVEL_ID     "Customer_Level_ID",
ACCOUNT_TYPE_ID     "Account_Type_ID",
TRADE_CLASS_ID     "Trade_Class_ID",
DISTRIBUTOR_ID     "Distributor_ID",
PRODUCT_ID     "Product_ID",
ORDERED_PRODUCT_ID     "Ordered_Product_ID",
BRAND_TYPE_ID     "Brand_Type_ID",
LABEL_TYPE_ID     "Label_Type_ID",
BRAND_LABEL_ID     "Brand_Label_ID",
PRICED_BY_ID     "Priced_By_ID",
SALES_UOM_ID     "Sales_UOM_ID",
PURCHASING_UOM_ID     "Purchasing_UOM_ID",
PRICING_UOM_ID     "Pricing_UOM_ID",
NET_COST     "Net_Cost",
NPA_S     "NPA_S",
CMA_S     "CMA_S",
NOT_S     "NOT_S",
TOTAL_NATIONAL_ALLOWANCE_S     "Total_National_Allowance_S",
LPA_S     "LPA_S",
LMA_S     "LMA_S",
LOT_S     "LOT_S",
TOTAL_LOCAL_ALLOWANCE_S     "Total_Local_Allowance_S",
TOTAL_ALLOWANCES_S     "Total_Allowances_S",
LPC     "LPC",
LPC_EXTENDED     "LPC_Extended",
LPF     "LPF",
LPF_EXTENDED     "LPF_Extended",
TRUE_COST     "True_Cost",
CDE     "CDE",
LPP     "LPP",
SURCHARGE     "Surcharge",
COMBINED_SURCHARGE     "Combined_Surcharge",
TOTAL_SURCHARGES     "Total_Surcharges",
MARKET_COST     "Market_Cost",
INSIDE_PAD     "Inside_Pad",
SALES_REP_COST     "Sales_Rep_Cost",
SALES_REP_MARGIN     "Sales_Rep_Margin",
SALES_PRICE     "Sales_Price",
SALES_TRUE_MARGIN     "Sales_True_Margin",
NVD     "NVD",
LVD     "LVD",
NID     "NID",
LID     "LID",
TOTAL_VD     "Total_VD",
TOTAL_ID     "Total_ID",
TOTAL_DEVIATIONS     "Total_Deviations",
GP1     "GP1",
GP2     "GP2",
DEVIATED_COST     "Deviated_Cost",
ACTUAL_COST     "Actual_Cost",
SALES_TAX     "Sales_Tax",
QUANTITY_ORDERED     "Quantity_Ordered",
QUANTITY_SHIPPED     "Quantity_Shipped",
QUANTITY_DEVIATED     "Quantity_Deviated",
QUANTITY_SUBBED     "Quantity_Subbed",
UNITS_ORDERED     "Units_Ordered",
EACHES_ORDERED     "Eaches_Ordered",
EACH_CONVERSION_FACTOR     "Each_Conversion_Factor",
UNITS_SHIPPED     "Units_Shipped",
EACHES_SHIPPED     "Eaches_Shipped",
SHIP_WEIGHT     "Ship_Weight",
ACTUAL_GP_DLR     "Actual_GP_Dlr",
TRUE_GP_DLR     "True_GP_Dlr",
LANDED_GP_DLR     "Landed_GP_Dlr",
LANDED_ACTUAL_GP_DLR     "Landed_Actual_GP_Dlr",
INVOICE_GP_DLR     "Invoice_GP_Dlr",
INVOICE_ACTUAL_GP_DLR     "Invoice_Actual_GP_Dlr",
ADJUSTED_ACTUAL_GP_DLR     "Adjusted_Actual_GP_Dlr",
EB_S     "EB_S",
MB_S     "MB_S",
ACTUAL_TM_ID     "Actual_TM_ID",
ACTUAL_TM_NAME     "Actual_TM_Name",
ACTUAL_DSM_ID     "Actual_DSM_ID",
ACTUAL_DSM_NAME     "Actual_DSM_Name",
INVOICE_NUMBER      "Invoice_Number ",
CONTRACT_NUMBER     "Contract_Number",
CUSTOMER_NUMBER     "Customer_Number",
CUSTOMER     "Customer",
PRODUCT_NUMBER     "Product_Number",
MASTER_DISTRIBUTOR_ID     "Master_Distributor_ID",
ORDERED_PRODUCT_NUMBER     "Ordered_Product_Number",
NATIVE_PRODUCT_STATUS     "Native_Product_Status",
NATIVE_PRICED_BY_INDICATOR     "Native_Priced_By_Indicator",
EXTRACTION_TIME     "Extraction_Time"
from uf_sales where distributor_id in (''5139'',
''5140'',
''5145'',
''5150'',
''5160'',
''5175'',
''5180'',
''5210'',
''5220'',
''5230'')
DBMS_XMLGen.setRowTag(qryctx,'Sales_Record');
DBMS_XMLGen.setRowSetTag(qryctx,'Sales_Set');
temp_clob:=dbms_xmlgen.getxml(qryctx);
length_count := dbms_lob.getlength(temp_clob);
dbms_output.put_line('Internal LOB size is: ' || length_count);
filehandle := utl_file.fopen('DATA_EXTRACT','Sales_1.xml','Wb',32767);
WHILE length_count <> 0 LOOP
dbms_lob.read (temp_clob, amount, position, temp_buffer);
--utl_file.put (filehandle, temp_buffer);
utl_file.put_raw(filehandle, utl_raw.cast_to_raw(temp_buffer));
position := position + 1;
length_count := length_count - 1;
temp_buffer := null;
END LOOP;
dbms_output.put_line('Exit the loop');
utl_file.fclose(filehandle);
DBMS_XMLGen.closeContext(qryctx);
dbms_output.put_line('Close the file');
EXCEPTION
WHEN OTHERS THEN
BEGIN
error_number := sqlcode;
error_message := substr(sqlerrm ,1 ,100);
dbms_output.put_line('Error #: ' || error_number);
dbms_output.put_line('Error Message: ' || error_message);
utl_file.fclose_all;
END;
END;
/

OK, so you are writing the file with UTL_FILE. How long is the whole process taking. Have you timed the time taken to generate the temp_clob with the result Vs the time to write the output to a file.

Trying to parse a huge XML file

I'm trying to parse a large (~200 Meg) XML file. I get out of memory errors with DOM, and so far using SAX has been a huge disaster. My program has to run on IE and Netscape as an applet. I've run into a HUGE amount of errors. While I have been doing Java programming for about 2 years, this is the first time I've ever worked with XML. The error I am currently getting is an abstract method error associated with the following code :
SAXParserFactory spf = new
                         com.sun.xml.parser.SAXParserFactoryImpl();
               spf.setValidating(false);
               try {
               out.write("Gets past SAXParserFactory creation");
               out.flush();
                    catch (Exception e) {}
               xmlReader = null;
               try {
                         // Create a JAXP SAXParser
                         saxParser = spf.newSAXParser();
                         // Get the encapsulated SAX XMLReader
                         out.write("gets to assigning an XMLReader");
***THIS LINE OF CODE ***                         xmlReader = saxParser.getXMLReader();
                         out.write("gets past assigning an XMLReader");
                         out.flush();
I've run out of ideas and would greatly appreciate any help anyone could offer on this most frustrating problem.

With XML, we don't have to check each line of the file, only the data in those tags of each element that we are searching on.That's true, once you have read the entire XML and parsed it. Reading the entire XML would of course take much longer than reading the text file it was generated from, so you would want to pre-parse it into a DOM and store that somehow. But then to do a search you'd have to load that DOM, which is in turn much larger than the XML it was generated from...
What you describe sounds to me a lot like a database application. Of course I don't know anything about the specifics of what you are doing with the text, but storing it in a database with indexes that support the searches seems to me like a better approach. Of course you then have the challenge of distributing the database and its supporting software in a form that doesn't require installation. But I'd respectfully suggest that banning any installation at all is a bit extreme. After all, where I live you can buy an encyclopedia on a CD, from a gas station, for $10. And when you load the CD, it runs through an installation procedure. No big deal these days.

XML publisher report not generating output for huge XML files

Changed Depreciation Projections Report output type to XML.
Defined a Data Definition and a new Data Template (RTF) for this report.
Ran the Depreciation Projection Report to generate the XML output.
Ran the XML Report Publisher report to generate teh PDF/Excel output of the above report.
Output generated for smaller XML files. When XML size is big, the program is running for hours without generating the output.
Teh RTF template is basically a matrix report in which the number of columns in the report is based on the number of periods the report is run for.
The same is not working in the Desktop version also. The system is hanging when i try to view the preview pdf.
The XML file size is approximately 33 MB.
Please let me know if there is any way we can increase the memory size to see the output.
Thanks,
Ram.

for publisher use Category: E-Business Suite

Reading huge XML file in ECC 6.0

Hi ABAPexperts,
Is there any way to read/parse a huge xml in ABAP (ECC 6.0)?
I am looking at ixml and sxml, but ixml have memory issues and but is sxml available in ECC 6.0?
Any example will help me.
thank you for your help.

Have you considered using Simple Transformation?
It is should perform well on huge data due to linear processing.

Memory overflow problems when processing huge XML files

Hi All,
We are in need of processing very large XML file.(more than 100MB)..
We ran this job in background and it resulted in runtime errors.
Is there any way of processing this file as a whole?
Edited by: Thomas Zloch on Nov 17, 2010 4:16 PM - subject adjusted

Normally such memory problems can be avoided by using block processing and clearing tempory data inbetween the blocks. However all XML techniques that I know (DOM, XSLT, ST) require all data to reside in an internal table at once. I will be facing a similar problem soon, so I'm quite interested in a solution.
One way would be to upgrade the hardware and allow more memory to be allocated to the workprocess (system administration). Some background information:
http://help.sap.com/saphelp_nw70ehp1/helpdata/en/49/32eb1fe92e3504e10000000a421937/frameset.htm
I wonder if there are other workarounds, let's see if there will be additional replies.
Thomas

Want parse huge xml file for CDATA

HI all
I want to parse hude xml file for getting values of CDATA
can u plz give me sample code for that
here i am attaching the part of my xml file
<initParams>
<param description="simulation mode " name="smtpSimulationMode" passOn="false" required="false" type="bool" varSubstitute="false">
<![CDATA[false]]>
</param>
<param description="full name of smtp host" name="smtpHost" passOn="false" required="true" type="string" varSubstitute="false">
<![CDATA[222222]]>
</param>
<param description="smtpUserName for authentication" name="smtpUserName" passOn="false" required="false" type="string" varSubstitute="false"/>
<param description="smtpUserPassword for authentication" name="smtpUserPassword" passOn="false" required="false" type="string" varSubstitute="false"/>
<param description="ip address for remote server" name="serverId" passOn="false" required="true" type="string" varSubstitute="false">
<![CDATA[1111111]]>
</param>
<param description="location of file on remote server side" name="fileRemoteLocation" passOn="false" required="false" type="string" varSubstitute="false">
<![CDATA[test/]]>
</param>
<param description="user name for authentication" name="userName" passOn="false" required="true" type="string" varSubstitute="false">
<![CDATA[abc]]>
</param>
</initParams>
thanks in advance

There are several Java XML API's available. SAX, JXpath, DOM4J, Xerces, etcetera.

How to plot line chart for huge xml files?

Guys,
I would like to plot files over 2000 lines in a line chart!
but the application is getting very slow!
does anyone have any tips to improve performance?

Can we see how you implement the LineChart and bind the XML please?
It should be fast normally.

JDOM parser fails to parse Huge Size Xml Files ??? Unable to Parse ???

Hi All,
When i transformed or parsed many XML Files of huge size...I am getting java.lang.OutOfMemory error for all the huge xml files. It is working fine for files which is of small size.
I've 2GB ram in my system. I have also set heapsize for the JVM.
-Xms512M -Xmx1800M (or) -Xms512M -Xmx1500M
I like to know what are the drawbacks of JDOM parser ?
What is the maximum size of the Xml which JDOM can parse ?
I've read a long time before that parsers have certain limitations. Can anybody tell me the limitations of the parses?
Please help. i'm currently using jdom.jar to parse the xml files.
Thanks,
J.Kathir

Hi All,
When i transformed or parsed many XML Files of huge
size...I am getting java.lang.OutOfMemory error for
all the huge xml files. It is working fine for files
which is of small size.Of course it is.
>
I've 2GB ram in my system. I have also set heapsize for the JVM.
-Xms512M -Xmx1800M (or) -Xms512M -Xmx1500MYou can't always get what you want. Your JVM is competing with all the other processes on your machine, including the OS, for RAM.
I like to know what are the drawbacks of JDOM parser ?You just discovered it: all DOM parsers create an in-memory representation of the entire document. A SAX parser does not.
What is the maximum size of the Xml which JDOM can parse ?The max amount of memory that you have available to you.
I've read a long time before that parsers have
certain limitations. Can anybody tell me the
limitations of the parses?See above.
Please help. i'm currently using jdom.jar to parse the xml files.You can try a SAX parser or a streaming parser. Maybe those can help.
%

Error while loading XML file to XML_type column

I am trying to load a huge xml file in table having xml_type column. I am getting an error
The following error has occurred:
ORA-20010: Error: ORA-31011: XML parsing failed
ORA-19202: Error occurred in XML processing
LPX-00216: invalid character 140 (0x8C)
Error at line 2 at location: 10
Any suggestions why I might be getting this errors?
Is there any size restriction on the file that we can load with loadfromfile?
THis is the procedure I have written to load the file--
CREATE OR REPLACE procedure vsave_xml_file_to_table(
p_directory in varchar2,
p_filename in varchar2 )
AUTHID DEFINER
as
v_filelocator bfile;
v_cloblocator clob;
v_filelength number;
v_loc number :=10;
begin
dbms_lob.createtemporary(v_cloblocator,true);
v_filelocator := bfilename(p_directory, p_filename);
     dbms_lob.fileopen(v_filelocator, dbms_lob.file_readonly);
     dbms_lob.loadfromfile(v_cloblocator, v_filelocator,dbms_lob.getlength(v_filelocator));
     v_filelength := dbms_lob.getlength(v_filelocator);
DBMS_OUTPUT.PUT_LINE('File is loaded and its size is :'|| v_filelength);
     dbms_lob.fileclose(v_filelocator);
     --insert into vxml values(1, XMLType.createXML(v_cloblocator));
     insert into vxml values(1, XMLType(v_cloblocator,null,0,0));
     DBMS_LOB.FREETEMPORARY(v_cloblocator);
exception
when others then
-- close the cursor and file, and reraise.
if dbms_lob.fileisopen(v_filelocator) = 1 then
dbms_lob.fileclose(v_filelocator);
end if;
     rollback;
raise_application_error (-20010,'Error: '||sqlerrm|| ' at location: '||v_loc);
end vsave_xml_file_to_table;
The input file is of size 5K.
The procedure loads a file of size around 100 bytes with no problem.
I am on Oracle9i Enterprise Edition Release 9.2.0.4.0 - 64bit Production
Please help
Thanks!

As you said, i have changed the Code, But still the same error,
CREATE OR REPLACE FUNCTION fn_Ins_CLOBDOCUMENT(FILENAME IN VARCHAR2,
                                           CHARSET IN VARCHAR2 DEFAULT NULL)
RETURN CLOB DETERMINISTIC IS
FILE        BFILE := BFILENAME('FXLMBAH', FILENAME);
CHARCONTENT CLOB := ' ';
TARGETFILE BFILE;
LANG_CTX    NUMBER := DBMS_LOB.DEFAULT_LANG_CTX;
CHARSET_ID NUMBER := 0;
SRC_OFFSET NUMBER := 1;
DST_OFFSET NUMBER := 1;
WARNING     NUMBER;
BEGIN
IF CHARSET IS NOT NULL THEN
    CHARSET_ID := NLS_CHARSET_ID(CHARSET);
END IF;
TARGETFILE := FILE;
DBMS_LOB.FILEOPEN(TARGETFILE, DBMS_LOB.FILE_READONLY);
DBMS_LOB.LOADCLOBFROMFILE(CHARCONTENT,
                            TARGETFILE,
                            DBMS_LOB.GETLENGTH(TARGETFILE),
                            SRC_OFFSET,
                            DST_OFFSET,
                            CHARSET_ID,
                            LANG_CTX,
                            WARNING);
DBMS_LOB.FILECLOSE(TARGETFILE);
RETURN CHARCONTENT;
END;
INSERT INTO XMLTABLE
VALUES(XMLTYPE(fn_Ins_CLOBDOCUMENT('purchaseorder.xml')))
ORA-22288: file or LOB operation FILEOPEN failed
No such file or directory
ORA-06512: at "SYS.DBMS_LOB", line 523
ORA-06512: at "LCC.FN_INS_CLOBDOCUMENT", line 17
ORA-06512: at line 1raja
Edited by: KrChowdary on Feb 18, 2009 3:36 PM

1 xml file to multiple xml files with 200 records in each

Hello Experts,
I have below scenario.
Please suggest me which might be the best approch to work on.
1) XML file to XML file
I will have 1 pretty huge XML file which i need to break in multiple XML files with 200 records each.
1) first approch i can have bpm in which i can split the file according to my requirement.
2) Second approch i can create 2 scenarios in which 1st scenario will pick up XML file and create multiple flat files with File content conversion. Second scenario will pick up all these flat files and create XML files.
2) XML file to XML file
Or i can have multiple files with 1 record per file and i need to merge in multiple XML files with 200 records in each.
So its kind of 1:N or M:N scenarios.
Please tell me which is might be better performance and design wise.
Or if you have any idea in any other way i can do this, then please reply as soon as possbile.
Please tell me if you have OS command for the same or some script to run or anything which i can implement.
Thanks,
Hetal

what is your senario? is it File to File?
You can use multi mapping concept without BPM. You can handle the 200 records per message logic in the multimapping.
Regards,
Praveen Gujjeti.

Huge XML file

Similar Messages

Maybe you are looking for