Export large XML-file

Hi!
I want to export XML-files, but when i select 50 rows of my object view i only get 43 in my utl_file.
Here is the code:
CREATE OR REPLACE PROCEDURE Export_XML_to_file_with_Schema IS
qryCtx DBMS_XMLGEN.ctxHandle;
result XMLTYPE;
result_1 Clob;
Abfrage VARCHAR2(4000);
lob_length integer;
read_amount integer;
read_offset integer;
buffer varchar2(100);
loc varchar2(100) := 'usr_dir';
f_hand utl_file.file_type;
BEGIN
-- Setting up offset and no. of chars to be read in
-- in one go from clob datatype.
read_offset := 1;
read_amount := 75;
--Opening file
f_hand := Utl_File.Fopen(location =>'d:\oracle\utl_file\',
filename =>'test.xml',
open_mode =>'w',
max_linesize => 32767);
Abfrage :='Select * from I_GESAMT_V where rownum < 50';
qryctx := dbms_xmlgen.newContext(Abfrage);
Returns a new context
PARAMETERS: queryString (IN)- the query string, the result of which needs to be converted to XML
RETURNS: Context handle.
Call this function first to obtain a handle that you can use in the getXML()
and other functions to get the XML back from the result.
dbms_xmlgen.setRowSetTag(qryCtx,NULL);
rowsetTag (IN) -
the name of the document element.
NULL indicates that you do not want
the ROW element to be present.
Call this to set the name of the document root element,
if you do not want the default "ROWSET" name in the output.
You can also set this to NULL to suppress the printing of this element.
However, this is an error if both the row and the rowset are null and there
is more than one column or row in the output.
dbms_xmlgen.setRowTag(qryCtx,'INSP_PDA');
rowTag (IN) -
the name of the ROW element.
NULL indicates that you do not want the ROW element to be present.
Call this function to set the name of the ROW element,
if you do not want the default "ROW" name to show up.
You can also set this to NULL to suppress the ROW element itself.
Its an error if both the row and the rowset are null and there
is more than one column or row in the output.
-- now get the result
result := DBMS_XMLGEN.getXMLType(qryCtx);
select xmlelement("INSOBJ",
xmlattributes('http://www.w3.org/2001/XMLSchema-instance' AS "xmlns:xsi",'http://sv6:8080/sys/schemas/SCOTT/sv6:8080/public/mydocs/inspection_pda_schema.xsd' AS "xsi:noNamespaceSchemaLocation"), result) into result from dual;
result_1 :=result.getClobval();
-- Getting the length of the data stored in Clob
lob_length := dbms_lob.getlength(result_1);
utl_file.put(f_hand,'<?xml version="1.0" encoding="windows-1252"?>');
Utl_File.New_Line (f_hand,1);
-- Reading data from clob variable and writng into file.
while (lob_length > 0) loop
dbms_lob.read(result_1,read_amount,read_offset,buffer);
utl_file.put(f_hand,buffer);
read_offset := read_offset+read_amount;
lob_length := lob_length-read_amount;
if lob_length < read_amount then
read_amount := lob_length;
end if;
end loop;
utl_file.fclose(f_hand);
END Export_XML_to_file_with_Schema ;
Is there something wrong with my code?
Thanks for help.
With best regards
Nicole

HI!
I've seen that the utl_file max_linesize is the problem, is there an alternative except 'Spool'!
Thanks for help.
With best regards
Nicole

Similar Messages

  • Export large xml file in PL/SQL

    How to export large or big xml (XMLTYPE) into file system.
    I have tried with...
    1. UTL_FILE.PUT -- it has a limitation of 32767 chars
    2. DBMS_XSLPROCESSOR.CLOB2FILE supports little more than UTL_FILE but not in MB's
    i'm looking for options to export a xml file which is more than 5mb
    any suggestions?

    user12020576 wrote:
    I have tried with...
    1. UTL_FILE.PUT -- it has a limitation of 32767 chars
    It might have but this is not a problem, if you programmatically break it up in pieces.
    Have a look at https://forums.oracle.com/message/9748097#9748097
    The method demonstrates how to dump/write a CSV in binary format to disk.
    I used it to dump/write files way bigger than 10++ MB in size...

  • Is there a way to import large XML files into HANA efficiently are their any data services provided to do this?

    1. Is there a way to import large XML files into HANA efficiently?
    2. Will it process it node by node or the entire file at a time?
    3. Are there any data services provided to do this?
    This for a project use case i also have an requirement to process bulk XML files, suggest me to accomplish this task

    Hi Patrick,
         I am addressing the similar issue. "Getting data from huge XMLs into Hana."
    Using Odata services can we handle huge data (i.e create schema/load into Hana) On-the-fly ?
    In my scenario,
    I get a folder of different complex XML files which are to be loaded into Hana database.
    Then I gotta transform & cleanse the data.
    Can I use oData services to transform and cleanse the data ?
    If so, how can I create oData services dynamically ?
    Any help is highly appreciated.
    Thank you.
    Regards,
    Alekhya

  • How to set SAXParser at command-line interface to create a large XML file

    Hi,
    I am trying to create a large XML file (more than 50 MB) by selecting from Oracle database but failed because of "out of memory" error. According to "Oracle XML Developer Guide", we should use SAXParser to parsing a large XML file. But there is no example to show how to set SAXParser at command-line
    Following is what I use to get xml files. It works only when the file is small.
    java OracleXML getXML -DateFormat -withDTD -rowsetTag PO_HDR -conn
    "jdbc:oracle:oci8:@server_name" -user "ID/password" "select * from table_name"
    When I set SAXParser at the way below,
    java oracle.xml.parser.v2.SAXParser OracleXML getXML -DateFormat -withDTD -rowsetTag PO_HDR -conn
    "jdbc:oracle:oci8:@server_name" -user "ID/password" "select * from table_name"
    it failed with the error message: "In class oracle.xml.parser.v2.SAXParser: void main(String argv[]) is not defined"
    Does anyone know how to solve the problem? I'll be appreciated very much for your help.
    Yi

    here are my ideas.
    register the xml schema.
    using xmldom, generate the desired xml output and return as xmltype.
    then you can use something like this to check.
    declare
    xmldoc xmltype ;
    begin
       -- populate xmldoc from you xmldom function
       -- validate against XML schema
       xmldoc.isSchemaValid(schema_url, root_element);
       if xmldoc.isSchemaValid = 1 then
            --valid schema
       else
            --invalid
       end if;
    end

  • Performance Problem in parsing large XML file (15MB)

    Hi,
    I'm trying to parse a large XML file(15 MB) and facing a clear performance problem. A Simple XML Validation using the following code snippet:
    DBMS_LOB.fileopen(targetFile, DBMS_LOB.file_readonly);
    DBMS_LOB.loadClobfromFile
    tempCLOB,
    targetFile,
    DBMS_LOB.getLength(targetFile),
    dest_offset,
    src_offset,
    nls_charset_id(CONSTANT_CHARSET),
    lang_context,
    conv_warning
    DBMS_LOB.fileclose(targetFile);
    p_xml_document := XMLType(tempCLOB, p_schema_url, 0, 0);
    p_xml_document.schemaValidate();
    is taking 30 mins on a HP-UX (4GB ram, 2 CPU) machine (Oracle version : 9.2.0.4).
    Please explain what could be going wrong.
    Thanks In Advance,
    Vineet

    Thanks Mark,
    I'll open a TAR and also upload the schema and instance XML.
    If i'm not changing the track too much :-) one more thing in continuation:
    If i skip the Schema Validation step and directly insert the instance document into a Schema linked XMLType table, what does OracleXDB do in such a case?
    i'm getting a severe performance hit here too... the same file as above takes almost 40 mins to Insert.
    code snippet:
    DBMS_LOB.fileopen(targetFile, DBMS_LOB.file_readonly);
    DBMS_LOB.loadClobfromFile
    tempCLOB,
    targetFile,
    DBMS_LOB.getLength(targetFile),
    dest_offset,
    src_offset,
    nls_charset_id(CONSTANT_CHARSET),
    lang_context,
    conv_warning
    DBMS_LOB.fileclose(targetFile);
    p_xml_document := XMLType(tempCLOB, p_schema_url, 0, 0);
    -- p_xml_document.schemaValidate();
    insert into INCOMING_XML values(p_xml_document);
    Here table INCOMING_XML is :
    TABLE of SYS.XMLTYPE(XMLSchema "http://INCOMING_XML.xsd" Element "MatchingResponse") STORAGE Object-
    relational TYPE "XDBTYPE_MATCHING_RESPONSE"
    This table and type XDBTYPE_MATCHING_RESPONSE were created using the mapping provided in the registered XML Schema.
    Thanks,
    Vineet

  • I want to load large raw XML file in firefox and parse by DOM. But, for large XML file the firefox very slow some time crashed . Is there any option to increase DOM handling memory in Firefox

    Actually i am using an off-line form to load very large XML file and using firefox to load that form. But, its taking more time to load and some time the browser crashed. through DOM parsing this XML file to my form. Is there any option to increase DOM handler size in firefox

    Thank you for your suggestion. I have a question,
    though. If I use a relational database and try to
    access it for EACH and EVERY click the user makes,
    wouldn't that take much time to populate the page with
    data?
    Isn't XML store more efficient here? Please reply me.You have the choice of reading a small number of records (10 children per element?) from a database, or parsing multiple megabytes. Reading 10 records from a database should take maybe 100 milliseconds (1/10 of a second). I have written a web application that reads several hundred records and returns them with acceptable response time, and I am no expert. To parse an XML file of many megabytes... you have already tried this, so you know how long it takes, right? If you haven't tried it then you should. It's possible to waste a lot of time considering alternatives -- the term is "analysis paralysis". Speculating on how fast something might be doesn't get you very far.

  • OSB - Iterating over large XML files with content streaming

    Hi @ll
    I have to iterate over all item in large XML files and insert into a oracle database.
    The file is about 200 MB and contains around 500'000, and I am using OSB 10gR3.
    The XML structure is something like this:
    <allItems>
    <item>.....</item>
    <item>.....</item>
    <item>.....</item>
    <item>.....</item>
    <item>.....</item>
    </allItems>
    Actually I thought about using a proxy service with enabled content streaming and a "for each" action for iterating
    over all items. But for this the whole XML structure has to be materialized into a variable otherwise it is not possible!
    More about streaming large files can be found here:
    [http://download.oracle.com/docs/cd/E13159_01/osb/docs10gr3/userguide/context.html#large_messages]
    There is written "When you enable streaming for large message processing, you cannot use the ... for each...".
    And for accessing single items you should should use an assign action with a xpath like "$body/allItems/item[1]";
    this works fine and not the whole XML stream has to be materialized.
    So my idea was to use the "for each" action and processing seqeuntially all items with a xpath like:
    $body/allItems/item[$counter]
    But the "for each" action just allows iterating over a sequence of xml items by defining an selection xpath
    and the variable that contains all items. I would like to have a "repeat until" construct that iterates as long
    $body/allItems/item[$counter] returns not null. Or can I use the "for each" action differently?
    Does the OSB provides any other iterating mechanism? I know there is this spli-join construct that supports
    different looping techniques, but as far I know it does not support content streaming, is this correct?
    Did I miss somehting?
    Thanks a lot for helping!
    Cheers
    Dani
    Edited by: user10095731 on 29.07.2009 06:41

    Hi Dani,
    Yes, according to me this would be the best approach. You can use content-streaming to pass this large xml to ejb and once it passes successfully EJB should operate on this. If you want any result back (for further routing), you can get it back from EJB.
    EJB gives you power of java to process this file and from java perspective 150 MB is not a very LARGE data. Ensure that you are using buffering. Check out this link for an explanation on Java IO Streams and, in particular, buffered streams -
    http://java.sun.com/developer/technicalArticles/Streams/ProgIOStreams/
    Try dom4J with xpp (XML Pull Parser) parser in case you have parsing requirement. We had worked with 1.2GB file using this technique.
    Regards,
    Anuj

  • Query in a large xml file

    Hello,
    I'm trying to work with very large xml files which are created from csv files. These files may be very large - up to 1 GB ! Untill now I have managed to do several validations on these big xml files, and the only thing that works for me is SAX parser, DOM is out of the question because it fills up memory.
    My next task is to do queries on these files, smth like:
    select field1,field2 from file.xml
    where field3 = 'A'
    and (fileld4>'B' or field1='C')
    order by field2.
    I searched the net about finding out how to make queries on xml files (since I have never done queries on xml before), but I couldn't find which "query language" is best for large files. If I use XPath (XSLT) will that not cause me memory problems because XSLT represents the file as a memory object?
    My idea is to parse the file with SAX and check every row if it fits the where condition and then write it immediately to a result xml file. But validating the where statement can be very complicated without using some tool. Also the order by statement is another problematic issue.
    Does anyone have some more intelegent ideas about how I can do this? Please help! :(
    The xml file looks like this:
    <doc>
    <row id ="1">
    <column id="1" name="column1">value</column>
    <column id="N" name="columnN">value</column>
    </row>
    <row id ="M">
    <column id="1" name="column1">value</column>
    <column id="N" name="columnN">value</column>
    </row>
    </doc>

    Hi all,
    Thank you very much for your replies.
    First, saxon didn't work because it uses an in-memory parser, and that is what I was trying to avoid.
    Different database is also out of the question, because the customer insist on XML, and also there are some files that can never be converted to a database table, because eventually with some transformations thay are changed and are not completely like the standard csv format.
    I think that maybe http://exist.sourceforge.net is the rigth solution for me, but I will probably try it in the next version of my project.
    For now I have managed to make the project with only SAXParser and a lot of back - end programming and it works ok, althoug it was very hard to make it, and will be harded to maintain, so I will try to look at the eXist project.
    Thanks everyone for the help.

  • Large XML file Loading

    I have a large XML file that I am converting to an
    ArrayCollection to use as a dataprovider for a datagrid. It takes
    sometime to fully load. Is there any way to load partial list while
    the rest of the list is loading?? or does anyone know a way speed
    up this process??
    Thanks

    I'd try to modify the autoComplete component.
    You could break this processing up into smaller chunks. For
    it to work, you need some outside counter or indexer that keeps
    track of where you are. Have the conversion function process say
    nodes 0-500, then end. Then using callLater, call that function
    again, to process ne next batch of nodes.
    This process will allow the UI to update between iteration
    batches. If you need more responsiveness, you could try monitoring
    mouse move, and stopping the conversion, until the mouse is
    inactive again. That is just brainstorming. I have not tried it
    (the mouse move part. I know the iterator method works to allow the
    UI to update.)
    Tracy

  • PBDOM : Exporting to XML file gives a single line

    I've been starting developing with PowerBuilder 12.5 a few weeks ago. I had to write some XML files, so I got familiar with the PBDOM library.
    I can build a lot of different things, it's very nice, but one thing still bothers me :
    In the output file, the whole XML is written on a single line.
    I use the SaveDocument function.
    For example, here is some code :
    PBDOM_Document doc
    PBDOM_Element noderoot, node1, node11, node12
    doc = CREATE PBDOM_Document
    doc.NewDocument("NodeRoot")
    noderoot = doc.GetRootElement()
    node1 = CREATE PBDOM_Element
    node1.SetName("Node1")
    noderoot.AddContent(node1)
    node1.SetAttribute("Attr", "AttrValue")
    node11 = CREATE PBDOM_Element
    node11.SetName("Node11")
    node11.AddContent("Here is a value")
    node1.AddContent(node11)
    node12 = CREATE PBDOM_ELEMENT
    node12.SetName("Node12")
    node12.AddContent("Here is another value")
    node1.AddContent(node12)
    doc.SaveDocument("myDoc.xml")
    Here is the result when I open it with notepad++
    <NodeRoot><Node1 Attr="AttrValue"><Node11>Here is a value</Node11><Node12>Here is another value</Node12></Node1></NodeRoot>
    Whereas I wanted :
    <NodeRoot> <Node1 Attr="AttrValue"> <Node11>Here is a value</Node11> <Node12>Here is another value</Node12> </Node1> </NodeRoot>
    With the notepad++ XML tools plugin, I can use the "pretty print" function to get this nice representation. But I would like my file to be formatted this way from the beginning. I noticed that the "line ending" was set to UNIX format (indicator on bottom right of the window), but I'm working on Windows. When I use the menu to convert it to Windows format (CR+LF), it changes this indicator, but the code stays on one single line.
    Is there a way to tell PBDOM to export the XML file with a nice output ?
    Thank you !
    Notes :
    - Opening the XML file with Internet Explorer or Google Chrome gives me a nice vizualisation, with indentation, line breaks...
    - Adding a <?xml version="1.0" encoding="ISO-8859-1" ?> does not help (I've been doing it while exporting some more complex files, but I still get the output on one line...)

    Here is a very simple powerbuilder example. It uses MSXML instead of PBDOM. See my comments for hints how to use it with PBDOM.
    oleobject lole_dom, lole_root, lole_element1, lole_element11, lole_element12
    oleobject lole_Writer, lole_saxreader
    string ls_result
    // build DOM (you don't need this if you use PBDOM)
    lole_dom = create oleobject
    lole_dom.ConnectToNewObject ("Msxml2.DOMDocument")
    lole_root = lole_dom.CreateElement ("NodeRoot")
    lole_dom.documentElement = lole_root
    lole_element1 = lole_dom.CreateElement("Node1")
    lole_root.AppendChild(lole_element1)
    lole_element1.SetAttribute("Attr", "AttrValue")
    lole_element11 = lole_dom.CreateElement("Node11")
    lole_element11.AppendChild (lole_dom.CreateTextNode("Here is a value"))
    lole_element1.AppendChild(lole_element11)
    lole_element12 = lole_dom.CreateElement("Node12")
    lole_element12.AppendChild (lole_dom.CreateTextNode("Here is another value"))
    lole_element1.AppendChild(lole_element12)
    // this saves the DOM without formatting
    //lole_dom.save ("d:\testxml.xml")
    // this part is for formating
    lole_Writer = create oleobject
    lole_saxreader = create oleobject
    lole_Writer.ConnectToNewObject ("MSXML2.MXXMLWriter")
    lole_saxreader.ConnectToNewObject ("MSXML2.SAXXMLReader")
    lole_Writer.omitXMLDeclaration = True
    lole_Writer.indent = True
    lole_saxreader.contentHandler = lole_Writer
    // instead of DOM you can also put a XML string to parser: lole_saxreader.parse ("<node>...</node>")
    // so you can also use the PBDOM output directly
    lole_saxreader.parse (lole_dom)
    // here is your formatted output
    ls_Result = lole_Writer.output
    MessageBox ("", ls_result)
    lole_writer.Disconnectobject( )
    lole_saxreader.Disconnectobject( )
    lole_dom.Disconnectobject( )
    DESTROY lole_writer
    DESTROY lole_saxreader
    DESTROY lole_dom

  • Best technology to navigate through a very large XML file in a web page

    Hi!
    I have a very large XML file that needs to be displayed in my web page, may be as a tree structure. Visitors should be able to go to any level depth nodes and access the children elements or text element of those nodes.
    I thought about using DOM parser with Java but dropped that idea as DOM would be stored in memory and hence its space consuming. Neither SAX works for me as every time there is a click on any of the nodes, my SAX parser parses the whole document for the node and its time consuming.
    Could anyone please tell me the best technology and best parser to be used for very large XML files?

    Thank you for your suggestion. I have a question,
    though. If I use a relational database and try to
    access it for EACH and EVERY click the user makes,
    wouldn't that take much time to populate the page with
    data?
    Isn't XML store more efficient here? Please reply me.You have the choice of reading a small number of records (10 children per element?) from a database, or parsing multiple megabytes. Reading 10 records from a database should take maybe 100 milliseconds (1/10 of a second). I have written a web application that reads several hundred records and returns them with acceptable response time, and I am no expert. To parse an XML file of many megabytes... you have already tried this, so you know how long it takes, right? If you haven't tried it then you should. It's possible to waste a lot of time considering alternatives -- the term is "analysis paralysis". Speculating on how fast something might be doesn't get you very far.

  • Transform Large XML files with XSL

    HELP, LARGE XML FILES
    I have got 30 - 50 MB large xml file, and i would like to transform it
    with xslt, i tried but i have got OutOfMemory Exception.
    I tried to find out the solution on JAVA site, but i didn't find it.
    I can not displit my xml file. I hope for some help.
    I tried really everything.
    Thanks a lot

    What is your machine configuration ?
    The above 2 suggestions would help, but it does depend on how your software is written.
    Please post more info about your environment and object design.
    Chintan

  • Efficient searching in a large XML file for specific elements

    Hi
    How can I search in a large XML file for a specific element efficiently (fast and memory savvy?) I have a large (approximately 32MB with about 140,000 main elements) XML file and I have to search through it for specific elements. What stable and production-ready open source tools are available for such tasks? I think PDOM is a solution but I can't find any well-known and stable implementations on the web.
    Thanks in advance,
    Behrang Saeedzadeh.

    The problem with DOM parsers is that the whole document needs to be parsed!
    So with large documents this uses up a lot of memory.
    I suggest you look at sometthing like a pull parser (Piccolo or MPX1) which is a fast parser that is program driven and not event driven like SAX. This has the advantage of not needing to remember your state between events.
    I have used Piccolo to extract events from large xml based log files.
    Carl.

  • Large XML File in JMS Message

    Hi Experts,
    I got to send larg XML files (+20MB) to the Message listener for processing. Would you please comment if it would be a good approach or suggest some tested and authentic solution for this purpose. Prompt response will be highly appreciated.
    Regards
    Shahzad mahmood

    You can use the following algo to send large messages
    - use BytesMessage to send the large message
    - break the message into chunks (chunk size can be configurable)
    - in the header of the first message add a header prioperty to notify that this is the first message
    - in the header add a property for the size of the each message
    - in the header of the last message add a property to notify that its the last message
    on the receiver side
    - see the header property of the message
    - if its first message, open a stream (may be file stream)
    - keep writing messages to this stream till you receive the last message (identified by the header property)
    - close the stream after processing the last message.
    BTW some JMS products like FioranoMQ (www.fiorano.com) provide built-in support for large messages.
    You might want to look into such products.
    Thanks
    Bhuvan

  • What are the best tools for opening very large XML files and examining the tree and confirming they are valid?

    I am generating some very large XML files (600,000+ lines, 50MB+ characters). I finally have them all being valid XML and valid UTF-8.
    But the files are so large Safari and Chrome will often not open them. FireFox will though.
    Instead of these browsers, I was wondering if there are there any other recommended apps for the Mac for opening and viewing the XML, getting an error message if they are not valid for some reason and examing the XML tree?
    I opened the file in the default app for XML which is Xcode, but that is just like opening it in a plain text editor. You can't expand/collapse the XML tree like you can with a browser, and it doesn't report errors.
    Thanks,
    Doug

    Hi Tom,
    I had not seen that list. I'll look it over.
    I'm also in touch with the developer of BBEdit (they are quite responsive) and they are willing to look at the file in question and see why it is not reporting UTF-8 errors while Chrome is.
    For now I have all the invalid characters quashed and things are working. But it would be useful in the future.
    By the by, some of those editors are quite pricey!
    doug

Maybe you are looking for