Memory overflow problems when processing huge XML files
Hi All,
We are in need of processing very large XML file.(more than 100MB)..
We ran this job in background and it resulted in runtime errors.
Is there any way of processing this file as a whole?
Edited by: Thomas Zloch on Nov 17, 2010 4:16 PM - subject adjusted
Normally such memory problems can be avoided by using block processing and clearing tempory data inbetween the blocks. However all XML techniques that I know (DOM, XSLT, ST) require all data to reside in an internal table at once. I will be facing a similar problem soon, so I'm quite interested in a solution.
One way would be to upgrade the hardware and allow more memory to be allocated to the workprocess (system administration). Some background information:
http://help.sap.com/saphelp_nw70ehp1/helpdata/en/49/32eb1fe92e3504e10000000a421937/frameset.htm
I wonder if there are other workarounds, let's see if there will be additional replies.
Thomas
Similar Messages
-
Problem in attaching a xml file in Email Notification
Hi,
I am new to BPEL. I am using 10.1.3.4 Jdev. Facing a problem in attaching a xml file in Email Notification process. I do not get any email when ever the attachment is a xml. Whereas when the attachment is a pdf, i get the pdf in the mail. I have put the mime type as text/xml. Used ora:doc() to read the xml. Dont know what i am missing. Please help me out. Thanks.
Edited by: friendsforever on Oct 6, 2010 1:05 AMactually i want that when a user open my website register form then he must attached their resume in it
so i want to know how can i make a field of attaching.
please reply
I am waiting for your reply -
Getting a iterator over a huge xml-file
Hi !
I have to persist data from some huge xml-files.
The structure of the xml is simple, like in:
<items>
<item>
</item>
<item>
</item>
</items>
The best way for me, is to get an iterator over the item-elements represented as a DOM strcture for easy access.
Because the files are so huge (80-150 MBytes) and I have no control over the creation of them, the underlying parser should be SAX oriented.
So I need a solution which will parse until one item-element (maybe spezified through a XPATH query) is completed and build a DOM-tree from it.
Then I can pass this DOM-element to my persistence layer. When this item is persisted, and the Iterator hasNext(), then I could do a simple .next() to get the next DOM representation of the item.
Is there such a solution out ? Or do I have to 'invent' such kind of parsing ?
Would by great to get some hints ... this problem is driving me crazy :)
Thank you in advance,
GeraldVaguely familiar, http://lists.xml.org/archives/xml-dev/200201/msg00032.html
I can't think of a DOM builder that would work off a fragment; you would probably have to create one yourself.
There's also http://drizzle.stanford.edu/~peastman/pax.html which might solve your problem.
Pete -
Error when Uploading _fmb.xml files
Hi...
I'm a newbie in Apex. I'm trying to create a migration project (ie migrating Oracle Forms 6i/10g to xml via Apex). I've converted the *.fmb files to _fmb.xml using frmf2xml.bat utility. My problem is when I tried to upload the _fmb.xml, this error appears:
+1 error has occurred+
+•Select the Forms File you want to upload. For Forms Modules, upload the XML file which is generated by the Forms2XML conversion tool, containing the information of your Forms Module. (Go to error)+
the error message is - javascript:apex.jQuery('#P3_FORMS_XML_FILE').focus();void(0)
the url returns: http://localhost:8080/apex/wwv_flow.accept
I've also tried to upload any type of file but it still returns the same error message. Please, anyone can help me...
Thanks.when i tried the online workspace as in http://apex.oracle.com/pls/apex/, i successfully created the migration project. But still failed to do it locally. Obviosly my locally setup Apex is having problem uploading the _fmb.xml files. Where have i gone wrong? Could anyone please help me?
-
Reading huge xml files in OSB11gR1(11.1.1.6.0)
Hi,
I want to read a huge xml file of size 1GB in OSB(11.1.1.6.0)?
I will be creating a (JCA)file adapter in jdeveloper and importing artifacts to OSB.
Please let me know the maximum file size that could be handled in OSB?
Thanks in advance.
Regards,
SureshDepends on what you intend to do after reading the file.
Do you want to parse the file contents and may be do some transformation? Or do you just have to move the file from one place to another for ex. reading from local system and moving to a remote system using FTP?
If you just have to move the file, I would suggest using JCA File/FTP adapter's Move operation.
If you have to parse and process the file contents within OSB, then it may be possible depending on the file type and what logic you need to implement. For ex. for very large CSV files you can use JCA File Adapter batching to read a few records at a time. -
How to break up a huge XML file and generate serialized JSP pages
I have a huge xml file, with about 100 <item> nodes.
I want to process this xml file with pagination and generate jsp pages.
For example:
Display items from 0 to 9 on page 1 page1.jsp , 10 to 19 on page2.jsp, and so on...
Is it possible to generate JSP pages like this?
I have heard of Velocity but dont know if it will be the right technology for this kind of a job.Thank you for your reply, I looked at the display tag library and it looks pretty neat with a lot of features, I could definitely use it in a different situation.
The xml file size is about 1.35 MB, and the size is unpredictable it could shrink or grow periodically depending on the number of items available.
I was hoping to create a documentation style (static pages) of the xml feed instead of having 1 jsp with dynamic pages
I was looking at Anakia : http://jakarta.apache.org/velocity/docs/anakia.html , may be it has features that enable me to create static pages but not very sure.
I think for now, I will transform the xml with an xsl file and pass the page numbers as input parameters to the xsl file
null -
Initiate to process the xml file
Dear Experts,
I have a requirement to initiate to process of .xml file when .txt file is available in FTP folder,If the .xml file alone in that FTP folder no need to process or pick the xml file from sender side.
Scenerio is File(xml)->PI->RFC.
Is there any option in PI 7.1.kindly guide how to acheive this.
Best Regards,
MonikandanWe have this option for NFS protocol but i don't think FTP has got it yet though the idea is submitted in idea forum.
Q4 :http://wiki.scn.sap.com/wiki/display/XI/Sender+File+Adapter+Frequently+Asked+Questions
I think you have to count on executing OS commands feature.., -
Problem with parsing large xml files
Hello All,
I am parsing a large xml file of 20MB and I use DocumentBuilder.parse(File). This method works for small xml files with size less than 20MB but the application hangs and doesn't through any error message when parsing 20MB xml files. Please let me know what I have to do at this point ?
Thanks & Regards,
Kumar.Well... i can't agree.
If you have such structure:
<task>
<task/>
<task>
<task>
<task/>
</task>
<task/>
</task>
</task>
...you may always keep stack of tasks (at startElement push to top, and at endElement pop), so at every leaf of tree you will have all parents of that leaf.
for such structure:
<task id="1" parent="0"/>
<task id="2" parent="1"/>
<task id="3" parent="1"/>
<task id="4" parent="2"/>
<task id="5" parent="3"/>
...it will be much faster to go thro document by sax several times to build tree of tasks, than to load all document into memory... -
How to process an XML file if I don't now it's name before execution time .
Hi All,
I've a problem that some of you may be already covered.
Basically i have to process some XML files (with the same structure) without prior knowledge of their filenames.
The question is, how I've to modify the jdbc url of the XML datastore to use a variable and not an hardcoded filename ?
Example:
At the moment I'm using the following url :jdbc:snps:xml?f=c:/oditest.xml&ro=true&dod=yes
but I would like to replace the hardcoded filename (c:/oditext.xml) with a danamyc variable refreshed at run time.
Thanks a lot for your help
BenHi I've found the solution :
look at
https://s3.amazonaws.com/Ora/XML_7_Reading_a_dynamic_named_xml_file.swf
Thanks to all
Ben -
Itunes 10.6.1.7 problem: when I change the file "media type" from 'Music' to 'Podcast' the file disapears from ITUNES. I do this via (1) right click, (2) select 'Get Info', (3) select 'options' tab, and (4) change media type. What is the problem?
Hi Memalyn
Essentially, the bare issue is that you have a 500GB hard drive with only 10GB free. That is not sufficient to run the system properly. The two options you have are to move/remove files to another location, or to install a larger hard drive (eg 2TB). Drive space has nothing to do with SMC firmware, and usually large media files are to blame.
My first recommendation is this: download and run the free OmniDiskSweeper. This will identify the exact size of all your folders - you can drill down into the subfolders and figure out where your largest culprits are. For example, you might find that your Pictures folder contains both an iPhoto Library and copies that you've brought in from a camera but are outside the iPhoto Library structure. Or perhaps you have a lot of purchased video content in iTunes.
If you find files that you KNOW you do not need, you can delete them. Don't delete them just because you have a backup, since if the backup fails, you will lose all your copies.
Don't worry about "cleaners" for now - they don't save much space and can actually cause problems. Deal with the large file situation first and see how you get on.
Let us know what you find out, and if you manage to get your space back.
Matt -
Oracle reports problem in distribution with xml file
Hi ,
I am having problem while I am trying to distribute the file with specifing the destination details in the xml. If I am giving the destination details in url then it is working.
Ex.
This is Working
http://bryxh91:8889/reports/rwservlet?report=devlopersuit/reports/test.jsp&userid=anju/[email protected]&desformat=pdf&destype=file&desname=devlopersuit/reports/output_file.pdf
Not working :
http://bryxh91:8889/reports/rwservlet?report=devlopersuit/reports/test.jsp&userid=anju/[email protected]&DISTRIBUTE=YES&destination=devlopersuit/reports/dis_test.xml
xml file
<destinations>
<file id="test" name="output_file.pdf" format="pdf">
<include scr="mainSection"/>
</file>
</destinations>
Error : REP-34304: Distribution failed to complete; review the distribution lists
Its been week I tried many things , but not working.
Right now I am trying to distribute to single file but my goal is to burst and distribute on basis of account number.
This is the first stage of the project.
Thanks a lot for the help
AnjuHello
see answer in Oracle reports problem in distribution with xml file
Regards -
How to compare two huge xml files(50MB+) using Java Code
I want to compare two huge xml files using java code and need to find the difference of those xml files
is there any API for thatYou should find third party API
-
2nd generation ipod touch shows error message 3259 when trying to update with version 4 software from itunes. Looks as though downloading update but problem when processing. Any advice? thanks
Try here. If not successful, try looking at the previous posts on the right under the heading: More Like This that have your same problem Start with the one with the green checkmark.
iTunes Store reporting error - 3259 -
Processing large xml file (500mb)? break into small part? load into jtree
hi,
i'm doing an assignment to processing large xml file (500mb) and
load into jree using JAVA.
can someone advice me on the algorithm to do this?
how can i load a 500mb xml in a jtree without system hang?
how to i break my file and do the loading?1 Is the file schema based binary XML.
2. The limits are dependant on storage model and chacater set.
3. For all NON-XML content the current limit is 4GBytes (Where that is bytes not characters). So for Character content in an AL32UTF8 database the limit is 2GB
4. For XML Content stored as CLOB the limit is the same as for character data (2GB/4GB) dependant on database character set.
5. For SB Based XML content stored in Object Relatioanl storage the limit is determined by the complexity and structures defiend in the XML Schema -
Problem when load more swf files work with xml files into my movie
hi ;
I have one flash file & more of swf files which work with xml files .
when I load one swf file into my flash file and remove it and load anther one on the same movieclip in my flash file it load the old swf file.
when i load one on movieclip and remove it and load anther swf on anther movieclip the file doesn`t work and stoped.
when test my flash file to load and remove swf files without xml file it work fine but when repleaced the swf files with other work with xml files the problem hapend.
thanks;YOu should trace the names of the files that are being targeted for loading to see if they agree with what you expect. If you want help with the coding you will need to show the code that is relevant to your problem (not all of it)
Maybe you are looking for
-
HT2518 how do I move PC files from a carbonite account to a mac?
I just wiped and sold my PC. All my data is backed up on a Carbonite account. I just bought a Mac and would like to transfer ~85G from the Carbonite account to my new machine. This is basically all my photos, music, and documents. Carbonite does not
-
Landing Page Default to Contract Management
Hello Experts, Basically, Whenever a user login to the CLM system, he will be landed on Home Page(Over view page) by default. Pls refer the below snapshot. I have a requirement where in, the customer wants to make Contract Management Workbench as a d
-
Reducing size of pattern in Layers Style
Hello. I'm using Photoshop CS3 and CS6 and have one strange problem. Working with Pattern Overlay menu I set 99% (or less) size of image and unfortunately quality is reducing. This is very strange. 100% image scale (original) have great quality 101%
-
Hi friends I want to add new fields in me21n can anybuddy help me?
-
How do i copy the picture i made on photoshop and paste it on to word
how do i copy the picture i made on photoshop and paste it on to word