Memory overflow problems when processing huge XML files

Hi All,
We are in need of processing very large XML file.(more than 100MB)..
We ran this job in background and it resulted in runtime errors.
Is there any way of processing this file as a whole?
Edited by: Thomas Zloch on Nov 17, 2010 4:16 PM - subject adjusted

Normally such memory problems can be avoided by using block processing and clearing tempory data inbetween the blocks. However all XML techniques that I know (DOM, XSLT, ST) require all data to reside in an internal table at once. I will be facing a similar problem soon, so I'm quite interested in a solution.
One way would be to upgrade the hardware and allow more memory to be allocated to the workprocess (system administration). Some background information:
http://help.sap.com/saphelp_nw70ehp1/helpdata/en/49/32eb1fe92e3504e10000000a421937/frameset.htm
I wonder if there are other workarounds, let's see if there will be additional replies.
Thomas

Similar Messages

  • Problem in attaching a xml file in Email Notification

    Hi,
    I am new to BPEL. I am using 10.1.3.4 Jdev. Facing a problem in attaching a xml file in Email Notification process. I do not get any email when ever the attachment is a xml. Whereas when the attachment is a pdf, i get the pdf in the mail. I have put the mime type as text/xml. Used ora:doc() to read the xml. Dont know what i am missing. Please help me out. Thanks.
    Edited by: friendsforever on Oct 6, 2010 1:05 AM

    actually i want that when a user open my website register form then he must attached their resume in it
    so i want to know how can i make a field of attaching.
    please reply
    I am waiting for your reply

  • Getting a iterator over a huge xml-file

    Hi !
    I have to persist data from some huge xml-files.
    The structure of the xml is simple, like in:
    <items>
    <item>
    </item>
    <item>
    </item>
    </items>
    The best way for me, is to get an iterator over the item-elements represented as a DOM strcture for easy access.
    Because the files are so huge (80-150 MBytes) and I have no control over the creation of them, the underlying parser should be SAX oriented.
    So I need a solution which will parse until one item-element (maybe spezified through a XPATH query) is completed and build a DOM-tree from it.
    Then I can pass this DOM-element to my persistence layer. When this item is persisted, and the Iterator hasNext(), then I could do a simple .next() to get the next DOM representation of the item.
    Is there such a solution out ? Or do I have to 'invent' such kind of parsing ?
    Would by great to get some hints ... this problem is driving me crazy :)
    Thank you in advance,
    Gerald

    Vaguely familiar, http://lists.xml.org/archives/xml-dev/200201/msg00032.html
    I can't think of a DOM builder that would work off a fragment; you would probably have to create one yourself.
    There's also http://drizzle.stanford.edu/~peastman/pax.html which might solve your problem.
    Pete

  • Error when Uploading _fmb.xml files

    Hi...
    I'm a newbie in Apex. I'm trying to create a migration project (ie migrating Oracle Forms 6i/10g to xml via Apex). I've converted the *.fmb files to _fmb.xml using frmf2xml.bat utility. My problem is when I tried to upload the _fmb.xml, this error appears:
    +1 error has occurred+
    +•Select the Forms File you want to upload. For Forms Modules, upload the XML file which is generated by the Forms2XML conversion tool, containing the information of your Forms Module. (Go to error)+
    the error message is - javascript:apex.jQuery('#P3_FORMS_XML_FILE').focus();void(0)
    the url returns: http://localhost:8080/apex/wwv_flow.accept
    I've also tried to upload any type of file but it still returns the same error message. Please, anyone can help me...
    Thanks.

    when i tried the online workspace as in http://apex.oracle.com/pls/apex/, i successfully created the migration project. But still failed to do it locally. Obviosly my locally setup Apex is having problem uploading the _fmb.xml files. Where have i gone wrong? Could anyone please help me?                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

  • Reading  huge xml files in OSB11gR1(11.1.1.6.0)

    Hi,
    I want to read a huge xml file of size 1GB in OSB(11.1.1.6.0)?
    I will be creating a (JCA)file adapter in jdeveloper and importing artifacts to OSB.
    Please let me know the maximum file size that could be handled in OSB?
    Thanks in advance.
    Regards,
    Suresh

    Depends on what you intend to do after reading the file.
    Do you want to parse the file contents and may be do some transformation? Or do you just have to move the file from one place to another for ex. reading from local system and moving to a remote system using FTP?
    If you just have to move the file, I would suggest using JCA File/FTP adapter's Move operation.
    If you have to parse and process the file contents within OSB, then it may be possible depending on the file type and what logic you need to implement. For ex. for very large CSV files you can use JCA File Adapter batching to read a few records at a time.

  • How to break up a huge XML file and generate serialized JSP pages

    I have a huge xml file, with about 100 <item> nodes.
    I want to process this xml file with pagination and generate jsp pages.
    For example:
    Display items from 0 to 9 on page 1 page1.jsp , 10 to 19 on page2.jsp, and so on...
    Is it possible to generate JSP pages like this?
    I have heard of Velocity but dont know if it will be the right technology for this kind of a job.

    Thank you for your reply, I looked at the display tag library and it looks pretty neat with a lot of features, I could definitely use it in a different situation.
    The xml file size is about 1.35 MB, and the size is unpredictable it could shrink or grow periodically depending on the number of items available.
    I was hoping to create a documentation style (static pages) of the xml feed instead of having 1 jsp with dynamic pages
    I was looking at Anakia : http://jakarta.apache.org/velocity/docs/anakia.html , may be it has features that enable me to create static pages but not very sure.
    I think for now, I will transform the xml with an xsl file and pass the page numbers as input parameters to the xsl file
    null

  • Initiate to process the xml file

    Dear Experts,
         I have a requirement to initiate to process of .xml file when .txt file is available in FTP folder,If the .xml file alone in that FTP folder no need to process or pick the xml file from sender side.
    Scenerio is File(xml)->PI->RFC.
    Is there any option in PI 7.1.kindly guide how to acheive this.
    Best Regards,
    Monikandan

    We have this option for NFS protocol but i don't think FTP has got it yet though the idea is submitted in idea forum.
    Q4 :http://wiki.scn.sap.com/wiki/display/XI/Sender+File+Adapter+Frequently+Asked+Questions
    I think you have to count on executing OS commands feature..,

  • Problem with parsing large xml files

    Hello All,
    I am parsing a large xml file of 20MB and I use DocumentBuilder.parse(File). This method works for small xml files with size less than 20MB but the application hangs and doesn't through any error message when parsing 20MB xml files. Please let me know what I have to do at this point ?
    Thanks & Regards,
    Kumar.

    Well... i can't agree.
    If you have such structure:
    <task>
      <task/>
      <task>
         <task>
            <task/>
         </task>
         <task/>
      </task>
    </task>
    ...you may always keep stack of tasks (at startElement push to top, and at endElement pop), so at every leaf of tree you will have all parents of that leaf.
    for such structure:
    <task id="1" parent="0"/>
    <task id="2" parent="1"/>
    <task id="3" parent="1"/>
    <task id="4" parent="2"/>
    <task id="5" parent="3"/>
    ...it will be much faster to go thro document by sax several times to build tree of tasks, than to load all document into memory...

  • How to process an XML file if I don't now it's name before execution time .

    Hi All,
    I've a problem that some of you may be already covered.
    Basically i have to process some XML files (with the same structure) without prior knowledge of their filenames.
    The question is, how I've to modify the jdbc url of the XML datastore to use a variable and not an hardcoded filename ?
    Example:
    At the moment I'm using the following url :jdbc:snps:xml?f=c:/oditest.xml&ro=true&dod=yes
    but I would like to replace the hardcoded filename (c:/oditext.xml) with a danamyc variable refreshed at run time.
    Thanks a lot for your help
    Ben

    Hi I've found the solution :
    look at
    https://s3.amazonaws.com/Ora/XML_7_Reading_a_dynamic_named_xml_file.swf
    Thanks to all
    Ben

  • Itunes 10.6.1.7 problem: when I change the file "media type" from 'Music' to 'Podcast' the file disapears from ITUNES. I do this via (1) right click, (2) select 'Get Info', (3) select 'options' tab, and (4) change media type. What is the problem?

    Itunes 10.6.1.7 problem: when I change the file "media type" from 'Music' to 'Podcast' the file disapears from ITUNES. I do this via (1) right click, (2) select 'Get Info', (3) select 'options' tab, and (4) change media type. What is the problem?

    Hi Memalyn
    Essentially, the bare issue is that you have a 500GB hard drive with only 10GB free. That is not sufficient to run the system properly. The two options you have are to move/remove files to another location, or to install a larger hard drive (eg 2TB). Drive space has nothing to do with SMC firmware, and usually large media files are to blame.
    My first recommendation is this: download and run the free OmniDiskSweeper. This will identify the exact size of all your folders - you can drill down into the subfolders and figure out where your largest culprits are. For example, you might find that your Pictures folder contains both an iPhoto Library and copies that you've brought in from a camera but are outside the iPhoto Library structure. Or perhaps you have a lot of purchased video content in iTunes.
    If you find files that you KNOW you do not need, you can delete them. Don't delete them just because you have a backup, since if the backup fails, you will lose all your copies.
    Don't worry about "cleaners" for now - they don't save much space and can actually cause problems. Deal with the large file situation first and see how you get on.
    Let us know what you find out, and if you manage to get your space back.
    Matt

  • Oracle reports problem in distribution with xml file

    Hi ,
    I am having problem while I am trying to distribute the file with specifing the destination details in the xml. If I am giving the destination details in url then it is working.
    Ex.
    This is Working
    http://bryxh91:8889/reports/rwservlet?report=devlopersuit/reports/test.jsp&userid=anju/[email protected]&desformat=pdf&destype=file&desname=devlopersuit/reports/output_file.pdf
    Not working :
    http://bryxh91:8889/reports/rwservlet?report=devlopersuit/reports/test.jsp&userid=anju/[email protected]&DISTRIBUTE=YES&destination=devlopersuit/reports/dis_test.xml
    xml file
    <destinations>
    <file id="test" name="output_file.pdf" format="pdf">
    <include scr="mainSection"/>
    </file>
    </destinations>
    Error : REP-34304: Distribution failed to complete; review the distribution lists
    Its been week I tried many things , but not working.
    Right now I am trying to distribute to single file but my goal is to burst and distribute on basis of account number.
    This is the first stage of the project.
    Thanks a lot for the help
    Anju

    Hello
    see answer in Oracle reports problem in distribution with xml file
    Regards

  • How to compare two huge xml files(50MB+) using Java Code

    I want to compare two huge xml files using java code and need to find the difference of those xml files
    is there any API for that

    You should find third party API

  • My 2nd generation ipod touch almost completes update of version 4 then error code 3259. any advice to help. Seems to download but problem when processing

    2nd generation ipod touch shows error message 3259 when trying to update with version 4 software from itunes. Looks as though downloading update but problem when processing. Any advice? thanks

    Try here. If not successful, try looking at the previous posts on the right under the heading: More Like This that have your same problem  Start with the one with the green checkmark.
    iTunes Store reporting error - 3259

  • Processing large xml file (500mb)? break into small part? load into jtree

    hi,
    i'm doing an assignment to processing large xml file (500mb) and
    load into jree using JAVA.
    can someone advice me on the algorithm to do this?
    how can i load a 500mb xml in a jtree without system hang?
    how to i break my file and do the loading?

    1 Is the file schema based binary XML.
    2. The limits are dependant on storage model and chacater set.
    3. For all NON-XML content the current limit is 4GBytes (Where that is bytes not characters). So for Character content in an AL32UTF8 database the limit is 2GB
    4. For XML Content stored as CLOB the limit is the same as for character data (2GB/4GB) dependant on database character set.
    5. For SB Based XML content stored in Object Relatioanl storage the limit is determined by the complexity and structures defiend in the XML Schema

  • Problem when load more swf files work with xml files into my movie

    hi ;
    I have one flash file & more of swf files which work with xml files .
    when I load one swf file into my flash file  and remove it and load anther one on the same movieclip in my flash file it load the old swf file.
    when i load one on movieclip and remove it and load anther swf  on anther movieclip the file doesn`t work  and stoped.
    when test my flash file to load and remove swf files without xml file it work fine but when repleaced the swf files with other work with xml files the problem hapend.
    thanks;

    YOu should trace the names of the files that are being targeted for loading to see if they agree with what you expect.  If you want help with the coding you will need to show the code that is relevant to your problem (not all of it)

Maybe you are looking for