Loading large XML documents
Hi everyone again :) Just sitting here trying to load a large XML document (it's ~13Mb). I know that's a massive XML document, but that's the way it is. The problem that I am having is that when I try to load the document I get an out of memory exception.
Frankly, I'm not surprised, but is there a remedy? Any thoughts/ideas/solutions would be greatly appreciated. :)
Ben
Sounds like you are using DOM. The DOM parser you are using must be loading the whole Document tree right away. DOM really eats up memory. There are two possible solutions:
1. Look into using a SAX parser. I don't know what you are doing with the xml, so I can't say whether or not that will work for you.
2. Configure the DOM parser to defer loading nodes until they are requested, or if that option is not available with your parser, get a parser that will defer node loading.
If option 2 sounds like what you need, then I suggest looking into the Apache Xerces parser. I am pretty sure it defers loading. You shouldn't have to change your code to work with the Xerces parser, you just have to make sure you set the proper system properties so that Java will automatically use the Xerces parser.
Similar Messages
-
Bulk Loader Program to load large xml document
I am looking for a bulk loader database program that will load a very large xml document. The simple bulk loader application available on the oracle site will not load this document due to its size which is approximately 20MG. Please advise asap. Thank you.
From the above document:
Storing XML Data Across Tables
Question
Can XML- SQL Utility store XML data across tables?
Answer
Currently XML-SQL Utility (XSU) can only store to a single table. It maps a canonical representation of an XML document into any table/view. But of course there is a way to store XML with the XSU across table. One can do this using XSLT to transform any document into multiple documents and insert them separately. Another way is to define views over multiple tables (object views if needed) and then do the inserts ... into the view. If the view is inherently non-updatable (because of complex joins, ...), then one can use INSTEAD-OF triggers over the views to do the inserts.
-- I've tried this, works fine. -
To load an XML document into 40 tables
How do I load a large XML document into 40 tables. Most of the exmaples, I see only load one table into the Orcale database?
From the above document:
Storing XML Data Across Tables
Question
Can XML- SQL Utility store XML data across tables?
Answer
Currently XML-SQL Utility (XSU) can only store to a single table. It maps a canonical representation of an XML document into any table/view. But of course there is a way to store XML with the XSU across table. One can do this using XSLT to transform any document into multiple documents and insert them separately. Another way is to define views over multiple tables (object views if needed) and then do the inserts ... into the view. If the view is inherently non-updatable (because of complex joins, ...), then one can use INSTEAD-OF triggers over the views to do the inserts.
-- I've tried this, works fine. -
I'm running 10.2.0.3 on a Linux box and I'm having problems loading a large XML document (about 100 MB). In the past, I would simply load the XML file into a XMLType column like this:
INSERT INTO foo VALUES (XMLType(bfilename('XMLDIR', 'test.xml'). nls_charset_id('AL32UTF8')));
But when I try this with a large file, it runs for 10 minutes and then returns an ORA-03113. I'm assuming the file is just too large for this technique. I spoke to Mark Drake when I was at OpenWorld and he suggested I use Oracle XML DB, so I created and registered a schema and tried using sqlldr to load the doc, but it ran for 2 1/2 hours before returning:
Parse Error on row 1 in table FOO
OCI-31038: Invalid integer value: "129"
I tried simplifying both the XML file and schema to just the following:
<schedules>
<s s="2009-09-21T04:00:00" d="21600" p="335975" c="19672"/>
<s s="2009-09-21T04:00:00" d="21600" p="335975" c="15387"/>
<s s="2009-09-21T04:00:00" d="25200" p="335975" c="5256"/>
<s s="2009-09-21T04:00:00" d="86400" p="335975" c="26198">
<k id="5" v="2009-09-21 09:00:00.000"/>
<k id="6" v="2009-09-22 03:59:59.000"/>
<k id="26" v="0.00"/><k id="27" v="US"/>
</s>
<s s="2009-09-21T04:00:00" d="21600" p="335975" c="11678"/>
<s s="2009-09-21T04:00:00" d="21600" p="335975" c="26697"/>
<s s="2009-09-21T04:00:00" d="21600" p="335975" c="25343"/>
<s s="2009-09-21T04:00:00" d="21600" p="335975" c="25269"/>
<s s="2009-09-21T04:00:00" d="86400" p="335975" c="26200">
<k id="5" v="2009-09-21 09:00:00.000"/>
<k id="6" v="2009-09-22 03:59:59.000"/>
<k id="26" v="0.00"/><k id="27" v="US"/>
</s>
</schedules>
<?xml version="1.0" encoding="UTF-8"?>
<!--W3C Schema generated by XMLSpy v2008 sp1 (http://www.altova.com)-->
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:xdb="http://xmlns.oracle.com/xdb"
version="1.0"
xdb:storeVarrayAsTable="true">
<xs:element name="schedules" xdb:defaultTable="SCHEDULES">
<xs:complexType xdb:SQLType="SCHEDULES_T">
<xs:sequence>
<xs:element ref="s" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="s">
<xs:complexType>
<xs:choice minOccurs="0">
<xs:element ref="f" maxOccurs="unbounded"/>
<xs:element ref="k" maxOccurs="unbounded"/>
</xs:choice>
<xs:attribute name="s" use="required" type="xs:dateTime"/>
<xs:attribute name="p" use="required" type="xs:int"/>
<xs:attribute name="d" use="required" type="xs:int"/>
<xs:attribute name="c" use="required" type="xs:short"/>
</xs:complexType>
</xs:element>
<xs:element name="k">
<xs:complexType>
<xs:attribute name="v" use="required" type="xs:string"/>
<xs:attribute name="id" use="required" type="xs:byte"/>
</xs:complexType>
</xs:element>
<xs:element name="f">
<xs:complexType>
<xs:attribute name="id" use="required" type="xs:byte"/>
</xs:complexType>
</xs:element>
</xs:schema>
Keep in mind both the actual XML file and corresponding XSD is much more complex, but this particular section is about 70 MB so I wanted to see if I could just load that. I used the following sqlldr script:
LOAD DATA
INFILE *
INTO TABLE schedules_tmp TRUNCATE
XMLType(xml_doc)(
lobfn FILLER CHAR TERMINATED by ',',
xml_doc LOBFILE(lobfn) TERMINATED BY EOF
BEGINDATA
/tmp/schedules.xmlThis worked fine on a small doc - loaded correctly and I could query it fine - but when I tried using the 70 MB file it ran for a couple of hours before dying with a memory problem.
So what am I doing wrong? Is there a better way to load a large file?
Thanks for the help.
Pete
Edited by: mdrake on Nov 9, 2009 8:46 PMMark:
Answers to your questions:
Do you use direct load ? -> Yes. I tried again using UNRECOVERABLE LOAD DATA to try to speed up performance, but it still ran for a couple of hours before dying.
Which DB Release are you working with ? -> 10.2.0.3
Can you see if you can upload via FTP ? -> I added noNamespaceSchemaLocation to my XML file and ftp'd it to my XML directory, but it wasn't recognized. Is there something else I have to do?
The change for unsignedInt should have code rid of the issue with the value 129. Did it ? -> I didn't try it again on the whole XML file (I'm just working with the schedules section), so I haven't verified this.
I'm still stumped as to why sqlldr takes so long. I could write something to parse the XML file into a flat file and then use sqlldr to load it into a relational table, and the load would only take a few minutes. But then I wouldn't be using XML DB which I thought would be faster. What am I doing wrong?
Pete -
Error while loading an XML document using a structured application
Hi,
I try to load an XML document using a structured application defined in the default structapps.fm
My code is shown down, extracted from the FDK API code sample.
Problem, I always have the same message :
"Cannot find the file named e:\xml\AdobeFrameMaker10\file. Make sure that the file exists. "
Where "e:\xml\AdobeFrameMaker10\" is my install directory.
So I assume that frame try to find the structapps.fm file but does not find it.
What else can it be ?
Does anyone knowns how to achieve this simple task using extendScript ?
Thanks for any comments, Pierre
function openXMLFile(myLastFile) {
var filename = myLastFile.openDlg("Choose XML file ...", "*.xml", false);
if (filename != null) {
/* Get default open properties. Return if it can’t be allocated. */
var params = GetOpenDefaultParams();
/* Set properties to open an XML document*/
/*Specify XML as file type to open*/
var i = GetPropIndex(params, Constants.FS_OpenAsType)
params[i].propVal.ival = Constants.FV_TYPE_XML;
/* Specify the XML application to be used when opening the document.*/
i = GetPropIndex(params, Constants.FS_StructuredOpenApplication)
params[i].propVal.sval = "myApp";
i = GetPropIndex(params, Constants.FS_FileIsOldVersion)
params[i].propVal.ival = Constants.FV_DoOK
i = GetPropIndex(params, Constants.FS_FontNotFoundInDoc)
params[i].propVal.ival = Constants.FV_DoOK
i = GetPropIndex(params, Constants.FS_FileIsInUse)
params[i].propVal.ival = Constants.FV_DoCancel
i = GetPropIndex(params, Constants.FS_AlertUserAboutFailure)
params[i].propVal.ival = Constants.FV_DoCancel
/*The structapps.fm file containing the specified application must have
already been read. The default structapps.fm file is read when FrameMaker is
opened so this shouldn't be a problem if the application to be used is
listed in the structapps.fm file.*/
var retParm = new PropVals()
var fileObj = Open(filename, params, retParm);
return fileObj
} else {
return null;Pierre,
Depending on the object "myLastFile", the method openDlg might not even exist (if the myLastFile object is not a File object, for instance). And I do not see any need for the myLastFile anyhow, as you are presenting a dialog to select a file to open. I recommend using the global ChooseFile( ) method instead. This will give you a filename as string in full path notation, or null when no file was selected in the dialog. I am not sure what your ExtendScript documentation states about the return value for ChooseFile, but if that differs from what I am telling you here, the documentation is wrong. So, if you replace the first lines of your code with the following it should work:
function openXMLFile ( ) {
var filename = ChooseFile ( "Choose XML file ...", "", "*.xml", Constants.FV_ChooseSelect );
While writing this, I see that Russ has already given you the same advice. Use the symbolic constant value I indicated to use the ChooseFile dialog to select a single file (it can also be used to select a directory or open a file - but you want to control the opening process yourself). Note that this method allows you to set a start directory for the dialog (second parameter). The ESTK autocompletion also gives you a fifth parameter "helplink" which is undocumented and can safely be ignored.
Good luck
Jang -
Problem when trying to load an XML document with DTD or XML SCHEMA
Hello
I have tried to load an XML document in Data Services, and I created the xsd file and Dtd file. (With altova xml spy software automatically) to import into SAP Data Services 3.2. .
In Data Services I created the dtd import file DTD and then called the XML file from the DTD (the xml file is validated vs the dtd file), and I could not read the xml correctly because it tells me that an ELEMENT called <item> expected joiners who did not come in the structure of import (dtd), but if the xml.
I understand that the document root is the label: CUSTOMER_FULL_2014, the data flow is as follows:CARGA_XML_CUSTOMER |
Turns out the joiners element is used to separate the xml elements are repeated, but is used at different levels. My idea is that the design will dtd second, or something I'm missing or is incorrectly stated.
Thank you.
xml
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
<?xml version="1.0" encoding="utf-8"?>
<CUSTOMER_FULL_2014>
<item>
<CUST_NO>202222</CUST_NO>
<ADDRESS>
<item>
<SHIP_TO>202222</SHIP_TO>
<NAME1>Henley.</NAME1>
<STREET>Vitacura #40</STREET>
<CITY>LIMA</CITY>
</item>
</ADDRESS>
<EQUIPMENT>
<item>
<EQUI_NO>81623</EQUI_NO>
</item>
<item>
<EQUI_NO>81633</EQUI_NO>
</item>
<item>
<EQUI_NO>81993</EQUI_NO>
</item>
<item>
<EQUI_NO>82003</EQUI_NO>
</item>
<item>
<EQUI_NO>82013</EQUI_NO>
</item>
<item>
<EQUI_NO>82103</EQUI_NO>
</item>
<item>
<EQUI_NO>82113</EQUI_NO>
</item>
<item>
<EQUI_NO>581203</EQUI_NO>
</item>
<item>
<EQUI_NO>900003-EMER</EQUI_NO>
</item>
<item>
<EQUI_NO>9000033-STOCK</EQUI_NO>
</item>
</EQUIPMENT>
<STORAGE_LOC>
<item>
<STOR_LOC_NO>0001</STOR_LOC_NO>
<DESCRIPTION>01 Parts Center</DESCRIPTION>
</item>
<item>
<STOR_LOC_NO>0056</STOR_LOC_NO>
<DESCRIPTION>56 henley</DESCRIPTION>
</item>
</STORAGE_LOC>
</item>
<item>
<CUST_NO>2007933434343</CUST_NO>
<ADDRESS>
<item>
<SHIP_TO>2007933434343</SHIP_TO>
<NAME1>Campos de Almacenaje SA</NAME1>
<STREET>Calacoto2, Calle 1</STREET>
<HOUSE_NO>Piso 1</HOUSE_NO>
<CITY>La Paz</CITY>
</item>
</ADDRESS>
<EQUIPMENT>
<item>
<EQUI_NO>90000-EMER</EQUI_NO>
</item>
<item>
<EQUI_NO>90000333-STOCK</EQUI_NO>
</item>
</EQUIPMENT>
<STORAGE_LOC>
<item>
<STOR_LOC_NO>00012</STOR_LOC_NO>
<DESCRIPTION>01 Parts Center</DESCRIPTION>
</item>
<item>
<STOR_LOC_NO>0056</STOR_LOC_NO>
<DESCRIPTION>56 henley</DESCRIPTION>
</item>
</STORAGE_LOC>
</item>
<item>
<CUST_NO>200801333</CUST_NO>
<ADDRESS>
<item>
<SHIP_TO>200801333</SHIP_TO>
<NAME1>CONSTRUCTORA SA.</NAME1>
<STREET>Ruta Panamericana Km 100</STREET>
<CITY>San Antonio 23</CITY>
</item>
</ADDRESS>
<EQUIPMENT>
<item>
<EQUI_NO>1507933</EQUI_NO>
</item>
<item>
<EQUI_NO>1509733</EQUI_NO>
</item>
<item>
<EQUI_NO>90000-EMER</EQUI_NO>
</item>
<item>
<EQUI_NO>90000333-STOCK</EQUI_NO>
</item>
</EQUIPMENT>
<STORAGE_LOC>
<item>
<STOR_LOC_NO>0001</STOR_LOC_NO>
<DESCRIPTION>01 Parts Center</DESCRIPTION>
</item>
<item>
<STOR_LOC_NO>0056</STOR_LOC_NO>
<DESCRIPTION>56 henley</DESCRIPTION>
</item>
</STORAGE_LOC>
</item>
</CUSTOMER_FULL_2014>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ARCHIVO DTD CREADO (automáticamente con xml spy)
<?xml version="1.0" encoding="UTF-8"?>
<!-- DTD generado con XMLSpy v2014 rel. 2 (x64) (http://www.altova.com) -->
<!ELEMENT CITY (#PCDATA)>
<!ELEMENT item ((SHIP_TO, NAME1, STREET, HOUSE_NO?, CITY) | (CUST_NO, ADDRESS, EQUIPMENT, STORAGE_LOC) | (STOR_LOC_NO, DESCRIPTION) | EQUI_NO)>
<!ELEMENT NAME1 (#PCDATA)>
<!ELEMENT STREET (#PCDATA)>
<!ELEMENT ADDRESS (item)>
<!ELEMENT CUST_NO (#PCDATA)>
<!ELEMENT EQUI_NO (#PCDATA)>
<!ELEMENT SHIP_TO (#PCDATA)>
<!ELEMENT HOUSE_NO (#PCDATA)>
<!ELEMENT EQUIPMENT (item+)>
<!ELEMENT DESCRIPTION (#PCDATA)>
<!ELEMENT STORAGE_LOC (item+)>
<!ELEMENT STOR_LOC_NO (#PCDATA)>
<!ELEMENT CUSTOMER_FULL_2014 (item+)>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The error of MONITOR Data Services
11676
5184
XML-240108
11-04-2014 17:34:16
|Data flow CARGA_XML_CUSTOMER|Reader READ MESSAGE customer OUTPUT(customer)
11676
5184
XML-240108
11-04-2014 17:34:16
An element named <item> present in the XML data input does not exist in the XML format used to set up this XML source in data
11676
5184
XML-240108
11-04-2014 17:34:16
flow <CARGA_XML_CUSTOMER>. Please validate your XML data.
11676
5184
XML-240307
11-04-2014 17:34:16
|Data flow CARGA_XML_CUSTOMER|Reader READ MESSAGE customer OUTPUT(customer)
11676
5184
XML-240307
11-04-2014 17:34:16
XML parser failed: See previously displayed error message.
The Error from Monitor
The metadata DTD
Thanks
JuanHi Juan Juan,
I generated a new XSD file using MS Visual Studio based on your XML file.
The empty spaces in the source is replaced with <Null> in the target table.
Here is the XSD:
<?xml version="1.0" encoding="utf-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="CUSTOMER_FULL_20140207_033015_001">
<xs:complexType>
<xs:sequence>
<xs:choice maxOccurs="unbounded">
<xs:element name="CUST_NO" type="xs:unsignedInt" />
<xs:element name="ADDRESS">
<xs:complexType>
<xs:sequence>
<xs:element name="SHIP_TO" type="xs:unsignedInt" />
<xs:element name="NAME1" type="xs:string" />
<xs:element name="STREET" type="xs:string" />
<xs:element minOccurs="0" name="HOUSE_NO" type="xs:string" />
<xs:element name="CITY" type="xs:string" />
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="EQUIPMENT">
<xs:complexType>
<xs:sequence>
<xs:element maxOccurs="unbounded" name="EQUI_NO" type="xs:string" />
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="STORAGE_LOC">
<xs:complexType>
<xs:sequence>
<xs:choice maxOccurs="unbounded">
<xs:element name="STOR_LOC_NO" type="xs:unsignedByte" />
<xs:element name="DESCRIPTION" type="xs:string" />
</xs:choice>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:choice>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
For Source XML
<?xml version="1.0" encoding="utf-8"?>
<CUSTOMER_FULL_20140207_033015_001>
<CUST_NO>200530</CUST_NO>
<ADDRESS>
<SHIP_TO>903533</SHIP_TO>
<NAME1>HENLEY - PART MAQUINARIAS S.A.</NAME1>
<STREET>Dean Camilo # 148, San Carlos</STREET>
<CITY>LIMA</CITY>
</ADDRESS>
<EQUIPMENT>
<EQUI_NO>4442</EQUI_NO>
<EQUI_NO>8163</EQUI_NO>
<EQUI_NO>8199</EQUI_NO>
<EQUI_NO>8200</EQUI_NO>
<EQUI_NO>8201</EQUI_NO>
<EQUI_NO>8210</EQUI_NO>
<EQUI_NO>8211</EQUI_NO>
<EQUI_NO>58120</EQUI_NO>
<EQUI_NO>90000-EMERGENCY</EQUI_NO>
<EQUI_NO>90000-STOCK</EQUI_NO>
</EQUIPMENT>
<STORAGE_LOC>
<STOR_LOC_NO>0001</STOR_LOC_NO>
<DESCRIPTION>01 Parts Center</DESCRIPTION>
<STOR_LOC_NO>0056</STOR_LOC_NO>
<DESCRIPTION>56 HEN</DESCRIPTION>
</STORAGE_LOC>
<CUST_NO>200793</CUST_NO>
<ADDRESS>
<SHIP_TO>200793</SHIP_TO>
<NAME1>Minera San Cristobal S.A.</NAME1>
<STREET>Calacoto, Calle 90, Torre 2</STREET>
<HOUSE_NO>Piso 5</HOUSE_NO>
<CITY>La Paz</CITY>
</ADDRESS>
<EQUIPMENT>
<EQUI_NO>90000-EMERGENCY</EQUI_NO>
<EQUI_NO>90000-STOCK</EQUI_NO>
</EQUIPMENT>
<STORAGE_LOC>
<STOR_LOC_NO>0001</STOR_LOC_NO>
<DESCRIPTION>01 Parts Center</DESCRIPTION>
<STOR_LOC_NO>0056</STOR_LOC_NO>
<DESCRIPTION>56 HEN</DESCRIPTION>
</STORAGE_LOC>
<CUST_NO>200801</CUST_NO>
<ADDRESS>
<SHIP_TO>200801</SHIP_TO>
<NAME1>ISEMAR S.A.</NAME1>
<STREET>Ruta Km 28.45</STREET>
<CITY>Don Torcuato Paraguay</CITY>
</ADDRESS>
<EQUIPMENT>
<EQUI_NO>15079</EQUI_NO>
<EQUI_NO>15097</EQUI_NO>
<EQUI_NO>90000-EMERGENCY</EQUI_NO>
<EQUI_NO>90000-STOCK</EQUI_NO>
</EQUIPMENT>
<STORAGE_LOC>
<STOR_LOC_NO>0001</STOR_LOC_NO>
<DESCRIPTION>01 Parts Center</DESCRIPTION>
<STOR_LOC_NO>0056</STOR_LOC_NO>
<DESCRIPTION>56 HEN</DESCRIPTION>
</STORAGE_LOC>
</CUSTOMER_FULL_20140207_033015_001>
Output:
Regards,
Akhileshkiran. -
Large XML document performance
We are using XDB 9.2.0.4. I am seeing a severe performance degradation when attempting to extract larger XML documents from XDB (somewhere over 3M). Smaller documents appear to be working fine.
I have been reading in the forum that the problem I am running into is most likely related to the storage model being used. ie) There are several repeating elements within the schema.
I have added xdb:storeVarrayAsTable="true" statement to the schema and re-registered. I can see, based on user_nested_tables, that XDB appears to be storing the repeating elements as nested tables vs varrays.
The change to the storage model does not seem to have significantly changed the queries performance.
The schemas I am using can be found at http://www.sasked.gov.sk.ca/xsd/sl/1.x/SLMessage.xsd & http://www.sasked.gov.sk.ca/xsd/sl/1.x/SDSElements.xsd
The schema documentation can be found at http://www.sasked.gov.sk.ca/sds/xml/SchemaDocumentation/SLMessage.html
The element /SL_Message/SL_Event/SL_ObjectData/SL_EventObject is the primary repeating element
I am using a table with an XMLType column
CREATE TABLE XML_SL_MESSAGE
(XML_SL_MESSAGE_ID NUMBER(11) NOT NULL
,DTE_TIMESTAMP TIMESTAMP DEFAULT SYSTIMESTAMP NOT NULL
,ORIGINAL_XML_SL_MESSAGE_ID NUMBER(11)
,VALID_SL_MESSAGE_XML sys.XMLType
,INVALID_XML CLOB
,ERROR_MESSAGE VARCHAR2(4000)
) xmltype column valid_sl_message_xml XMLSCHEMA "http://www.sasked.gov.sk.ca/xsd/sl/1.x/SLMessage.xsd" element "SL_Message"
The SQL I am using is attempting to bring the XMLType back as a clob, the query seems to be intensive in both CPU and I/O. (looks like it is the getClobVal function)
select xsm.xml_sl_message_id
,xsm.dte_timestamp
,nvl(xsm.valid_sl_message_xml.getClobVal(),xsm.invalid_xml) as xml_clob
,xsm.error_message
,xsm.original_xml_sl_message_id
from xml_sl_message xsm
where xsm.dte_timestamp > sysdate 1
I guess what I am wondering is what are my options ? Changing the storage model ? Applying Indexes ?
On an unrelated topic, Are there many differences in XDB 9.2.0.5 and 9.2.0.4 ? (I dont believe 10g will be an option here yet)
Thanx in advance
TrentI have applied the 9.2.0.5.0 patches and created the relational table with the following attributes:
CREATE TABLE XML_SL_MESSAGE
(XML_SL_MESSAGE_ID NUMBER(11) NOT NULL
,DTE_TIMESTAMP TIMESTAMP DEFAULT SYSTIMESTAMP NOT NULL
,PSE_SYS_USR_ID NUMBER(11) NOT NULL
,ORIGINAL_XML_SL_MESSAGE_ID NUMBER(11)
,VALID_SL_MESSAGE_XML sys.XMLType
,INVALID_XML CLOB
,ERROR_MESSAGE VARCHAR2(4000)
xmltype column valid_sl_message_xml
STORE AS OBJECT RELATIONAL
XMLSCHEMA "http://www.sasked.gov.sk.ca/xsd/sl/1.x/SLMessage.xsd" ELEMENT "SL_Message"
-- 1:1 ----------------------
varray valid_sl_message_xml."XMLDATA"."SL_Event"."SL_ObjectData"."SL_EventObject"
store as table SL_EVENTOBJECT2_TB(
(constraint SL_EVENTOBJECT2_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 2:2 ----------------------
varray "SchoolTerm"
store as table SCHOOLTERM2_TB(
(constraint SCHOOLTERM2_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 3:2 ----------------------
varray "SchoolClass"
store as table SCHOOLCLASS3_TB(
(constraint SCHOOLCLASS3_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 6:2 ----------------------
varray "StudentCourseHistory"
store as table STUDENTCOURSEHISTORY6_TB(
(constraint STUDENTCOURSEHISTORY6_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 7:2 ----------------------
varray "StudentSupplementalMark"
store as table STUDENTSUPPLEMENTALMARK7_TB(
(constraint STUDENTSUPPLEMENTALMARK7_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 8:2 ----------------------
varray "StudentClassMark"
store as table STUDENTCLASSMARK8_TB(
(constraint STUDENTCLASSMARK8_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 9:2 ----------------------
varray "StudentExamRegistration"
store as table STUDENTEXAMREGISTRATION9_TB(
(constraint STUDENTEXAMREGISTRATION9_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 10:2 ----------------------
varray "StudentClassEnrollment"
store as table STUDENTCLASSENROLLMENT10_TB(
(constraint STUDENTCLASSENROLLMENT10_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 11:2 ----------------------
varray "StudentPersonal"
store as table STUDENTPERSONAL11_TB(
(constraint STUDENTPERSONAL11_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 18:2 ----------------------
varray "StudentProgramEnrollment"
store as table STUDENTPROGRAMENROLLMENT18_TB(
(constraint STUDENTPROGRAMENROLLMENT18_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 19:2 ----------------------
varray "StudentSchoolEnrollment"
store as table STUDENTSCHOOLENROLLMENT19_TB(
(constraint STUDENTSCHOOLENROLLMENT19_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 26:1 ----------------------
varray valid_sl_message_xml."XMLDATA"."SL_Response"."SL_ObjectData"."SL_EventObject"
store as table SL_EVENTOBJECT26_TB(
(constraint SL_EVENTOBJECT26_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 27:2 ----------------------
varray "SchoolTerm"
store as table SCHOOLTERM27_TB(
(constraint SCHOOLTERM27_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 28:2 ----------------------
varray "SchoolClass"
store as table SCHOOLCLASS28_TB(
(constraint SCHOOLCLASS28_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 31:2 ----------------------
varray "StudentProgramEnrollment"
store as table STUDENTPROGRAMENROLLMENT31_TB(
(constraint STUDENTPROGRAMENROLLMENT31_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 32:2 ----------------------
varray "StudentExamRegistration"
store as table STUDENTEXAMREGISTRATION32_TB(
(constraint STUDENTEXAMREGISTRATION32_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 33:2 ----------------------
varray "StudentClassEnrollment"
store as table STUDENTCLASSENROLLMENT33_TB(
(constraint STUDENTCLASSENROLLMENT33_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 34:2 ----------------------
varray "StudentPersonal"
store as table STUDENTPERSONAL34_TB(
(constraint STUDENTPERSONAL34_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 41:2 ----------------------
varray "StudentSchoolEnrollment"
store as table STUDENTSCHOOLENROLLMENT41_TB(
(constraint STUDENTSCHOOLENROLLMENT41_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 48:2 ----------------------
varray "StudentClassMark"
store as table STUDENTCLASSMARK48_TB(
(constraint STUDENTCLASSMARK48_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 49:2 ----------------------
varray "StudentCourseHistory"
store as table STUDENTCOURSEHISTORY49_TB(
(constraint STUDENTCOURSEHISTORY49_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 50:2 ----------------------
varray "StudentSupplementalMark"
store as table STUDENTSUPPLEMENTALMARK50_TB(
(constraint STUDENTSUPPLEMENTALMARK50_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 51:1 ----------------------
varray valid_sl_message_xml."XMLDATA"."SL_Response"."SL_Ack"."SL_Error"
store as table SL_ERROR51_TB(
(constraint SL_ERROR51_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 52:1 ----------------------
varray valid_sl_message_xml."XMLDATA"."SL_Request"."SL_Query"."SL_QueryObject"
store as table SL_QUERYOBJECT52_TB(
(constraint SL_QUERYOBJECT52_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
tablespace data
ALTER TABLE XML_SL_MESSAGE
ADD (CONSTRAINT XML_SL_MESSAGE_PK PRIMARY KEY
(XML_SL_MESSAGE_ID))
ALTER TABLE XML_SL_MESSAGE
ADD (CONSTRAINT XMLSLMSG_ORIGINAL_XMLSLMSG_UK UNIQUE
(ORIGINAL_XML_SL_MESSAGE_ID))
ALTER TABLE XML_SL_MESSAGE ADD (CONSTRAINT
XMLSLMSG_SYSUSR_FK FOREIGN KEY
(PSE_SYS_USR_ID) REFERENCES PSE_SYS_USR
(PSE_SYS_USR_ID))
ALTER TABLE XML_SL_MESSAGE ADD (CONSTRAINT
XMLSLMSG_ORIGINAL_XMLSLMSG_FK FOREIGN KEY
(ORIGINAL_XML_SL_MESSAGE_ID) REFERENCES XML_SL_MESSAGE
(XML_SL_MESSAGE_ID))
-- Create a unique index for the XML Message id
CREATE UNIQUE INDEX XMLSLMSG_MSGID_UNIQUE ON XML_SL_MESSAGE
((substr(extractValue(valid_sl_message_xml,'//SL_MsgId'),1,255)))
tablespace indx
COMPUTE STATISTICS
Here is the nested table structure of the XMLType table created during the schema registration:
select level
,parent_table_column
from user_nested_tables
connect by prior table_name = parent_table_name
start with parent_table_name = 'SL_Message4724_TAB'
LEVEL PARENT_TABLE_COLUMN
1 "XMLDATA"."SL_Event"."SL_ObjectData"."SL_EventObject"
2 SchoolTerm
2 StudentSchoolEnrollment
3 "StudentInfo"."Name"
3 "StudentInfo"."Demographics"."CountryOfCitizenship"
3 "StudentInfo"."StudentAddress"
3 "StudentInfo"."PhoneNumber"
3 "StudentInfo"."Demographics"."Language"
3 "StudentInfo"."Email"
2 SchoolClass
3 "ClassInfo"."DeptAssignedCourseId"
3 "ClassInfo"."EducatorCertificateNumber"
2 StudentProgramEnrollment
2 StudentClassEnrollment
2 StudentClassMark
2 StudentCourseHistory
2 StudentSupplementalMark
2 StudentExamRegistration
2 StudentPersonal
3 "StudentInfo"."Name"
3 "StudentInfo"."Email"
3 "StudentInfo"."Demographics"."CountryOfCitizenship"
3 "StudentInfo"."StudentAddress"
3 "StudentInfo"."PhoneNumber"
3 "StudentInfo"."Demographics"."Language"
1 "XMLDATA"."SL_Request"."SL_Query"."SL_QueryObject"
1 "XMLDATA"."SL_Response"."SL_ObjectData"."SL_EventObject"
2 SchoolTerm
2 SchoolClass
3 "ClassInfo"."DeptAssignedCourseId"
3 "ClassInfo"."EducatorCertificateNumber"
2 StudentProgramEnrollment
2 StudentClassEnrollment
2 StudentClassMark
2 StudentCourseHistory
2 StudentSupplementalMark
2 StudentExamRegistration
2 StudentPersonal
3 "StudentInfo"."Name"
3 "StudentInfo"."Email"
3 "StudentInfo"."Demographics"."Language"
3 "StudentInfo"."PhoneNumber"
3 "StudentInfo"."StudentAddress"
3 "StudentInfo"."Demographics"."CountryOfCitizenship"
2 StudentSchoolEnrollment
3 "StudentInfo"."Name"
3 "StudentInfo"."Demographics"."Language"
3 "StudentInfo"."PhoneNumber"
3 "StudentInfo"."StudentAddress"
3 "StudentInfo"."Demographics"."CountryOfCitizenship"
3 "StudentInfo"."Email"
1 "XMLDATA"."SL_Response"."SL_Ack"."SL_Error"
52 rows selected.
When I attempt to insert to previously valid XML documents I get a core dump Here are the insert statements:
insert into xml_sl_message (
xml_sl_message_id, dte_timestamp, pse_sys_usr_id, original_xml_sl_message_id, valid_sl_message_xml, invalid_xml,error_message
select xml_sl_message_id, dte_timestamp, pse_sys_usr_id, original_xml_sl_message_id, xmltype(valid_sl_message_clob), invalid_xml,error_message
from xml_sl_message_temp where xml_sl_message_id in (5154,5155)
Here are the details on the exception:
Oracle9i Enterprise Edition Release 9.2.0.5.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.5.0 - Production
ORACLE_HOME = /opt/app/oracle/product/9.2.0
System name: SunOS
Node name: *****
Release: 5.8
Version: Generic_117000-01
Machine: sun4u
Instance name: EDDSDS
Redo thread mounted by this instance: 1
Oracle process number: 76
Unix process pid: 11460, image: oracle@***** (TNS V1-V3)
*** 2004-06-22 09:24:34.603
*** SESSION ID:(29.159) 2004-06-22 09:24:34.602
Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x20, PC: [0x101e41830, 0000000101E41830]
*** 2004-06-22 09:24:34.606
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [0000000101E41830] [SIGSEGV] [Address not mapped to object] [0x000000020] [] []
Current SQL statement for this session:
insert into xml_sl_message (
xml_sl_message_id, dte_timestamp, pse_sys_usr_id, original_xml_sl_message_id, valid_sl_message_xml, invalid_xml,error_message
select xml_sl_message_id, dte_timestamp, pse_sys_usr_id, original_xml_sl_message_id, xmltype(valid_sl_message_clob), invalid_xml,error_message
from xml_sl_message_temp where xml_sl_message_id in (5154,5155)
I have tried a couple different storage sections (with different levels of nesting) and still the same problem ..
Is there something wrong with my storage section ?
What is the "return as LOCATOR" clause for? -
Date insertion from large XML document (clob) into relation table very slow
Hi Everybody!
I'm working with Oracle 9.2.0.5 on Microsoft Windows Server 2003 Enterprise Edition.
The server (a test server) is a Pentium 4 2.8 GHz, 1GB of RAM.
I use a procedure called PARITOP_TRAITERXMLRESULTMASSE to insert the data contained in the pXMLDOC clob parameter in the table pTABLENAME. (You can see the format of the XML document below). The first step on this procedure is to verify that the XML document is not empty. If not, the procedure needs to add a node in the document, in every <ROW> tag. This added node is named âRST_IDâ. Itâs the foreign key of each record. I can retrieve the value of each <RST_ID> node in an other table in which the data has been previously added (by the calling procedure). When each of the <ROW> elements has been treated, the PARITOP_INSERTXML procedure is called. This procedure uses DBMS_XMLSAVE.INSERTXML to insert the data contained in the XML document in the specified table.
(Below, you can see the code of my procedures.)
With this information, can you tell me why this treatment is very very very slow with a large XML document and how I can improve it?
Thank you for your help!
Anne-Marie
CREATE OR REPLACE PROCEDURE "PARITOP_TRAITERXMLRESULTMASSE" (
pPRC_ID IN PARITOP_PARC.PRC_ID%TYPE,
pRST_MONDE IN PARITOP_RESULTAT.RST_MONDE%TYPE,
pXMLDOC IN CLOB,
pTABLENAME IN VARCHAR2)
AS
Objectif :Insérer le contenu du XML passé en paramètre (pXMLDOC) à la table passée en paramètre (pTABLENAME)
La table passée en paramètre doit être une table ayant comme clé étrangère le champs "RST_ID" .
(Le noeud RST_ID est donc ajouté à tous les document XML. Ce rst_id est
déterminé à partir de la table PARITOP_RESULTAT grâce à pPRC_ID et
pRstMonde fournis en paramètre)
result_doc CLOB;
XMLDOMDOC XDB.DBMS_XMLDOM.DOMDOCUMENT;
NODE_ROWSET DBMS_XMLDOM.DOMNODE;
NODE_ROW DBMS_XMLDOM.DOMNODE;
vUE_ID PARITOP_RESULTAT.UE_ID%TYPE;
vRST_ID PARITOP_RESULTAT.RST_ID%TYPE;
nodeList DBMS_XMLDOM.DOMNODELIST;
BEGIN
BEGIN
vUE_ID := 0;
vRST_ID := 0;
XMLDOMDOC := DBMS_XMLDOM.NEWDOMDOCUMENT(pXMLDOC);
IF NOT GESTXML_PKG.FN_PARITOP_DOCUMENT_IS_NULL(XMLDOMDOC) THEN
NODE_ROWSET := DBMS_XMLDOM.item(DBMS_XMLDOM.GETCHILDNODES (DBMS_XMLDOM.MAKENODE(XMLDOMDOC)),0);
for i in 0..dbms_xmldom.getLength(DBMS_XMLDOM.getchildnodes(NODE_ROWSET))-1 loop
NODE_ROW := DBMS_XMLDOM.ITEM(DBMS_XMLDOM.GETCHILDNODES(NODE_ROWSET), i) ;
nodeList := DBMS_XMLDOM.GETELEMENTSBYTAGNAME(DBMS_XMLDOM.makeelement(NODE_ROW) , 'UE_ID');
IF vUE_ID <> DBMS_XMLDOM.GETNODEVALUE(DBMS_XMLDOM.GETFIRSTCHILD(DBMS_XMLDOM.ITEM(nodeList, 0))) THEN
vUE_ID := DBMS_XMLDOM.GETNODEVALUE(DBMS_XMLDOM.GETFIRSTCHILD(DBMS_XMLDOM.ITEM(nodeList, 0)));
--on ramasse le rst_id
SELECT RST_ID INTO vRST_ID
FROM PARITOP_RESULTAT RST
WHERE RST.PRC_ID = pPRC_ID
AND RST.UE_ID = vUE_ID
AND RST.RST_MONDE = pRST_MONDE
AND RST_A_SUPPRIMER = 0;
END IF;
GESTXML_PKG.PARITOP_ADDNODETOROW(XMLDOMDOC, NODE_ROW, 'RST_ID', vRST_ID);
end loop;
RESULT_DOC := ' '; --Ã garder, pour ne pas que ca fasse d'erreur lors du WriteToClob.
dbms_xmldom.writeToClob(DBMS_XMLDOM.MAKENODE(XMLDOMDOC), RESULT_DOC);
--Insertion du document XML dans la table "tableName"
GESTXML_PKG.PARITOP_INSERTXML(RESULT_DOC, pTABLENAME);
DBMS_XMLDOM.FREEDOCUMENT( XMLDOMDOC);
end if;
EXCEPTION
[â¦exception treatementâ¦]
END;
END;
The format of a XML clob is :
<ROWSET>
<ROW>
<PRC_ID>193</PRC_ID>
<UE_ID>8781</UE_ID>
<VEN_ID>6223</VEN_ID>
<RST_MONDE>0</RST_MONDE>
<CMP_SELMAN>0</CMP_SELMAN>
<CMP_INDICESELECTION>92.307692307692307</CMP_INDICESELECTION>
<CMP_PVRES>94900</CMP_PVRES>
<CMP_PVAJUSTE>72678.017699115066</CMP_PVAJUSTE>
<CMP_PVAJUSTEMIN>72678.017699115095</CMP_PVAJUSTEMIN>
<CMP_PVAJUSTEMAX>72678.017699115037</CMP_PVAJUSTEMAX>
<CMP_PV>148000</CMP_PV>
<CMP_VALROLE>129400</CMP_VALROLE>
<CMP_PVRESECART>4790</CMP_PVRESECART>
<CMP_PVRHAB>101778.01769911509</CMP_PVRHAB>
<CMP_UTILISE>1</CMP_UTILISE>
<CMP_TVM>1</CMP_TVM>
<CMP_PVA>148000</CMP_PVA>
</ROW>
<ROW>
<PRC_ID>193</PRC_ID>
<UE_ID>8781</UE_ID>
<VEN_ID>6235</VEN_ID>
<RST_MONDE>0</RST_MONDE>
<CMP_SELMAN>0</CMP_SELMAN>
<CMP_INDICESELECTION>76.92307692307692</CMP_INDICESELECTION>
<CMP_PVRES>117800</CMP_PVRES>
<CMP_PVAJUSTE>118080</CMP_PVAJUSTE>
<CMP_PVAJUSTEMIN>118080</CMP_PVAJUSTEMIN>
<CMP_PVAJUSTEMAX>118080</CMP_PVAJUSTEMAX>
<CMP_PV>172000</CMP_PV>
<CMP_VALROLE>134800</CMP_VALROLE>
<CMP_PVRESECART>0</CMP_PVRESECART>
<CMP_PVRHAB>147180</CMP_PVRHAB>
<CMP_UTILISE>1</CMP_UTILISE>
<CMP_TVM>1</CMP_TVM>
<CMP_PVA>172000</CMP_PVA>
</ROW>
</ROWSET>
PARITOP_COMPARABLE TABLE :
RST_ID NUMBER(10) NOT NULL,
VEN_ID NUMBER(10) NOT NULL,
CMP_SELMAN NUMBER(1) NOT NULL,
CMP_UTILISE NUMBER(1) NOT NULL,
CMP_INDICESELECTION FLOAT(53) NOT NULL,
CMP_PVRES FLOAT(53) NULL,
CMP_PVAJUSTE FLOAT(53) NULL,
CMP_PVRHAB FLOAT(53) NULL,
CMP_TVM FLOAT(53) NULL
ROCEDURE PARITOP_INSERTXML (xmlDoc IN clob, tableName IN VARCHAR2)
AS
insCtx DBMS_XMLSave.ctxType;
rowss number;
BEGIN
--permet d'insérer les champs du XML dans la table passée en paramètre.
--il suffit que les champs XML aient le même nom que les champs de la table
BEGIN
insCtx := DBMS_XMLSave.newContext(tableName); -- get context handle
DBMS_XMLSAVE.SETDATEFORMAT( insCtx, 'yyyy-MM-dd HH:mm:ss');--attention, case sensitive
DBMS_XMLSAVE.setIgnoreCase(insCtx, 1);
rowss := DBMS_XMLSAVE.INSERTXML(insCtx , xmlDoc);
DBMS_XMLSave.closeContext(insCtx);
EXCEPTION
[â¦]
END;
END;
PROCEDURE PARITOP_ADDNODETOROW (
XMLDOMDOC DBMS_XMLDOM.DOMDOCUMENT,
NODE_ROW dbms_xmldom.DOMNode,
NOM_NOEUD VARCHAR2,
VALEUR_NOEUD VARCHAR2)
AS
--PERMET D'AJOUTER UN NOEUD AVEC 1 SEULE VALEUR DANS une ROW D'UN XML.
--UTILE SURTOUT POUR LES CLÃS ÃTRANGÃRES
domElemAInserer DBMS_XMLDOM.DOMELEMENT;
NODE dbms_xmldom.DOMNode;
NODE_TMP dbms_xmldom.DOMNode;
BEGIN
domElemAInserer := DBMS_XMLDOM.createElement(XMLDOMDOC, NOM_NOEUD) ;
NODE := DBMS_XMLDOM.MAKENODE(domElemAInserer); --cast
NODE := DBMS_XMLDOM.APPENDCHILD(NODE_ROW,NODE);
NODE_TMP := DBMS_XMLDOM.MAKENODE(DBMS_XMLDOM.CREATETEXTNODE(XMLDOMDOC, VALEUR_NOEUD ) );
NODE := DBMS_XMLDOM.APPENDCHILD(NODE,NODE_TMP );
END;Hi Everybody!
I'm working with Oracle 9.2.0.5 on Microsoft Windows Server 2003 Enterprise Edition.
The server (a test server) is a Pentium 4 2.8 GHz, 1GB of RAM.
I use a procedure called PARITOP_TRAITERXMLRESULTMASSE to insert the data contained in the pXMLDOC clob parameter in the table pTABLENAME. (You can see the format of the XML document below). The first step on this procedure is to verify that the XML document is not empty. If not, the procedure needs to add a node in the document, in every <ROW> tag. This added node is named âRST_IDâ. Itâs the foreign key of each record. I can retrieve the value of each <RST_ID> node in an other table in which the data has been previously added (by the calling procedure). When each of the <ROW> elements has been treated, the PARITOP_INSERTXML procedure is called. This procedure uses DBMS_XMLSAVE.INSERTXML to insert the data contained in the XML document in the specified table.
(Below, you can see the code of my procedures.)
With this information, can you tell me why this treatment is very very very slow with a large XML document and how I can improve it?
Thank you for your help!
Anne-Marie
CREATE OR REPLACE PROCEDURE "PARITOP_TRAITERXMLRESULTMASSE" (
pPRC_ID IN PARITOP_PARC.PRC_ID%TYPE,
pRST_MONDE IN PARITOP_RESULTAT.RST_MONDE%TYPE,
pXMLDOC IN CLOB,
pTABLENAME IN VARCHAR2)
AS
Objectif :Insérer le contenu du XML passé en paramètre (pXMLDOC) à la table passée en paramètre (pTABLENAME)
La table passée en paramètre doit être une table ayant comme clé étrangère le champs "RST_ID" .
(Le noeud RST_ID est donc ajouté à tous les document XML. Ce rst_id est
déterminé à partir de la table PARITOP_RESULTAT grâce à pPRC_ID et
pRstMonde fournis en paramètre)
result_doc CLOB;
XMLDOMDOC XDB.DBMS_XMLDOM.DOMDOCUMENT;
NODE_ROWSET DBMS_XMLDOM.DOMNODE;
NODE_ROW DBMS_XMLDOM.DOMNODE;
vUE_ID PARITOP_RESULTAT.UE_ID%TYPE;
vRST_ID PARITOP_RESULTAT.RST_ID%TYPE;
nodeList DBMS_XMLDOM.DOMNODELIST;
BEGIN
BEGIN
vUE_ID := 0;
vRST_ID := 0;
XMLDOMDOC := DBMS_XMLDOM.NEWDOMDOCUMENT(pXMLDOC);
IF NOT GESTXML_PKG.FN_PARITOP_DOCUMENT_IS_NULL(XMLDOMDOC) THEN
NODE_ROWSET := DBMS_XMLDOM.item(DBMS_XMLDOM.GETCHILDNODES (DBMS_XMLDOM.MAKENODE(XMLDOMDOC)),0);
for i in 0..dbms_xmldom.getLength(DBMS_XMLDOM.getchildnodes(NODE_ROWSET))-1 loop
NODE_ROW := DBMS_XMLDOM.ITEM(DBMS_XMLDOM.GETCHILDNODES(NODE_ROWSET), i) ;
nodeList := DBMS_XMLDOM.GETELEMENTSBYTAGNAME(DBMS_XMLDOM.makeelement(NODE_ROW) , 'UE_ID');
IF vUE_ID <> DBMS_XMLDOM.GETNODEVALUE(DBMS_XMLDOM.GETFIRSTCHILD(DBMS_XMLDOM.ITEM(nodeList, 0))) THEN
vUE_ID := DBMS_XMLDOM.GETNODEVALUE(DBMS_XMLDOM.GETFIRSTCHILD(DBMS_XMLDOM.ITEM(nodeList, 0)));
--on ramasse le rst_id
SELECT RST_ID INTO vRST_ID
FROM PARITOP_RESULTAT RST
WHERE RST.PRC_ID = pPRC_ID
AND RST.UE_ID = vUE_ID
AND RST.RST_MONDE = pRST_MONDE
AND RST_A_SUPPRIMER = 0;
END IF;
GESTXML_PKG.PARITOP_ADDNODETOROW(XMLDOMDOC, NODE_ROW, 'RST_ID', vRST_ID);
end loop;
RESULT_DOC := ' '; --Ã garder, pour ne pas que ca fasse d'erreur lors du WriteToClob.
dbms_xmldom.writeToClob(DBMS_XMLDOM.MAKENODE(XMLDOMDOC), RESULT_DOC);
--Insertion du document XML dans la table "tableName"
GESTXML_PKG.PARITOP_INSERTXML(RESULT_DOC, pTABLENAME);
DBMS_XMLDOM.FREEDOCUMENT( XMLDOMDOC);
end if;
EXCEPTION
[â¦exception treatementâ¦]
END;
END;
The format of a XML clob is :
<ROWSET>
<ROW>
<PRC_ID>193</PRC_ID>
<UE_ID>8781</UE_ID>
<VEN_ID>6223</VEN_ID>
<RST_MONDE>0</RST_MONDE>
<CMP_SELMAN>0</CMP_SELMAN>
<CMP_INDICESELECTION>92.307692307692307</CMP_INDICESELECTION>
<CMP_PVRES>94900</CMP_PVRES>
<CMP_PVAJUSTE>72678.017699115066</CMP_PVAJUSTE>
<CMP_PVAJUSTEMIN>72678.017699115095</CMP_PVAJUSTEMIN>
<CMP_PVAJUSTEMAX>72678.017699115037</CMP_PVAJUSTEMAX>
<CMP_PV>148000</CMP_PV>
<CMP_VALROLE>129400</CMP_VALROLE>
<CMP_PVRESECART>4790</CMP_PVRESECART>
<CMP_PVRHAB>101778.01769911509</CMP_PVRHAB>
<CMP_UTILISE>1</CMP_UTILISE>
<CMP_TVM>1</CMP_TVM>
<CMP_PVA>148000</CMP_PVA>
</ROW>
<ROW>
<PRC_ID>193</PRC_ID>
<UE_ID>8781</UE_ID>
<VEN_ID>6235</VEN_ID>
<RST_MONDE>0</RST_MONDE>
<CMP_SELMAN>0</CMP_SELMAN>
<CMP_INDICESELECTION>76.92307692307692</CMP_INDICESELECTION>
<CMP_PVRES>117800</CMP_PVRES>
<CMP_PVAJUSTE>118080</CMP_PVAJUSTE>
<CMP_PVAJUSTEMIN>118080</CMP_PVAJUSTEMIN>
<CMP_PVAJUSTEMAX>118080</CMP_PVAJUSTEMAX>
<CMP_PV>172000</CMP_PV>
<CMP_VALROLE>134800</CMP_VALROLE>
<CMP_PVRESECART>0</CMP_PVRESECART>
<CMP_PVRHAB>147180</CMP_PVRHAB>
<CMP_UTILISE>1</CMP_UTILISE>
<CMP_TVM>1</CMP_TVM>
<CMP_PVA>172000</CMP_PVA>
</ROW>
</ROWSET>
PARITOP_COMPARABLE TABLE :
RST_ID NUMBER(10) NOT NULL,
VEN_ID NUMBER(10) NOT NULL,
CMP_SELMAN NUMBER(1) NOT NULL,
CMP_UTILISE NUMBER(1) NOT NULL,
CMP_INDICESELECTION FLOAT(53) NOT NULL,
CMP_PVRES FLOAT(53) NULL,
CMP_PVAJUSTE FLOAT(53) NULL,
CMP_PVRHAB FLOAT(53) NULL,
CMP_TVM FLOAT(53) NULL
ROCEDURE PARITOP_INSERTXML (xmlDoc IN clob, tableName IN VARCHAR2)
AS
insCtx DBMS_XMLSave.ctxType;
rowss number;
BEGIN
--permet d'insérer les champs du XML dans la table passée en paramètre.
--il suffit que les champs XML aient le même nom que les champs de la table
BEGIN
insCtx := DBMS_XMLSave.newContext(tableName); -- get context handle
DBMS_XMLSAVE.SETDATEFORMAT( insCtx, 'yyyy-MM-dd HH:mm:ss');--attention, case sensitive
DBMS_XMLSAVE.setIgnoreCase(insCtx, 1);
rowss := DBMS_XMLSAVE.INSERTXML(insCtx , xmlDoc);
DBMS_XMLSave.closeContext(insCtx);
EXCEPTION
[â¦]
END;
END;
PROCEDURE PARITOP_ADDNODETOROW (
XMLDOMDOC DBMS_XMLDOM.DOMDOCUMENT,
NODE_ROW dbms_xmldom.DOMNode,
NOM_NOEUD VARCHAR2,
VALEUR_NOEUD VARCHAR2)
AS
--PERMET D'AJOUTER UN NOEUD AVEC 1 SEULE VALEUR DANS une ROW D'UN XML.
--UTILE SURTOUT POUR LES CLÃS ÃTRANGÃRES
domElemAInserer DBMS_XMLDOM.DOMELEMENT;
NODE dbms_xmldom.DOMNode;
NODE_TMP dbms_xmldom.DOMNode;
BEGIN
domElemAInserer := DBMS_XMLDOM.createElement(XMLDOMDOC, NOM_NOEUD) ;
NODE := DBMS_XMLDOM.MAKENODE(domElemAInserer); --cast
NODE := DBMS_XMLDOM.APPENDCHILD(NODE_ROW,NODE);
NODE_TMP := DBMS_XMLDOM.MAKENODE(DBMS_XMLDOM.CREATETEXTNODE(XMLDOMDOC, VALEUR_NOEUD ) );
NODE := DBMS_XMLDOM.APPENDCHILD(NODE,NODE_TMP );
END; -
XMLTYPE as CLOB storage "inserting large xml document in xml type column"
Hi All,
i have a table containing an xml datatype(non schema based)
i would like to insert a large xml document in it
but an exception is thrown-->"string literal too long"
i tried to use bind variables as a solution"prepared statements as i write in java"
but it didn't work....as xml document is large
when i tried to change the column type to CLOB,it worked but without xml validataion,
although the xml type is mapped to a CLOB in storage, xml type couldn't insert the document
if anyone have a solution plz tell,i needed it urgently
thanks,in advance :-)thx it was very useful :-)
but i am not having any success getting the following statement working using a JDBC connection pool rather than a hard coded URL connection
tempClob = CLOB.createTemporary(conn, true, CLOB.DURATION_SESSION);
it works with:
"jdbc:oracle:thin:@server:port:dbname"
Does NOT work with:
datasource.getConnection()
if anyone colud help... -
Loading Large XML files using plsql
I have a process where there is a need to load large xml files (i.e. easily over 500k or more) into Oracle via an interface. Preference would be to use plsql or some plsql based utility if possible. I am looking for any suggestions on the best method to accomplish this. Currently running on 9.2.0.6. Thanks in advance.
I have a process where there is a need to load large xml files (i.e. easily over 500k or more) into Oracle via an interface. Preference would be to use plsql or some plsql based utility if possible. I am looking for any suggestions on the best method to accomplish this. Currently running on 9.2.0.6. Thanks in advance.
-
Loading Large XML into Oracle Database
Hi,
I am fairly new to XML DB. I have been successful in registering a schema to a table in the Database. Now, I have to load the appropriate XML into that table. I am using the Simple Bulk Loader program found on this oracle site, however, when I load my XML file I get the following error: ORA-21700: object does not exist or is marked for delete.
So, I figured maybe simple bulk loader cannot handle large files? So I reduced my XML file and loaded it with the program and it worked. However, does anyone know how I can load large files into my registered schema table.
Thanks,
Prerna :o)Did you specify genTables true or false when registering the XML Schema ?
Does you XML schema contain a recursive definition
Is it possible that after reducing the size of the document you no longer have nodes that contain recursive structures... -
Performance issues with FDK in large XML documents
In my current project with FrameMaker 8 I'm experiencing severe performance issues with some FDK API calls.
The documents are about 3-8 MBytes in size. Fortmatted they cover 150-250 pages.
When importing such an XML document I do some extensive "post-processing" using FDK. This processing happens in Sr_EventHandler() during the SR_EVT_END_READER event. I noticed that some FDK functions calls which modify the document's structure, like F_ApiSetAttribute() or F_ApiNewElementInHierarchy(), take several seconds, for the larger documents even minutes, to complete one single function call. I tried to move some of these calls to earlier events, mostly to SR_EVT_END_ELEM. There the calls work without a delay. Unfortunately I can't rewrite the FDK client to move all the calls that are lagging to earlier events.
Does anybody have a clue why such delays happen, and possibly can make a suggestion, how to solve this issue? Thank you in advance.
PS: I already thought of splitting such a document in smaller pieces by using the FrameMaker book function. But I don't think, the structure of the documents will permit such an automatic split, and it definitely isn't an option to change the document structure (the project is about migrating documents from Interleaf to XML with the constraint of keeping the document layout identical).FP_ApplyFormatRules sounds really good--I'll give it a try on Monday. Wonder how I could miss it, as I already tried FP_Reformatting and FP_Displaying at no avail?! By the way, what is actually meant with FP_Reformatting (when I used it I assumed it would do exactly what FP_ApplyFormatRules sounds to do), or is that one another of Lynne's well-kept secrets?
Thank's for all the helpful suggestions, guys. On Friday I already had my first improvements in a test version of my client: I did some (not all necessary) structural changes using XSLT pre-processing, and processing went down from 8 hours(!) to 1 hour--Yeappie! I was also playing with the idea of writing a wrapper to F_ApiNewElementInHierarchy() which actually pastes an appropriate element created in a small flow on the reference pages at the intended insertion location. But now, with FP_ApplyFormatRules on the horizon, I'm quite confident to get even the complicated stuff under control, which cannot be handled by the XSLT pre-processing, as it is based on the actual formatting of the document at run-time and cannot be anticipated in pre-processing.
--Franz -
Want to load an xml document directly into an Oracle table
From the Oracle XML Applications book pg: 106/107
there is an example where it inserts into a Oracle table from the xml_documents table. What I would like to do is insert directly from the xml document and was able to but not able to get rid of the inserting into xml_documents section.
CREATE OR REPLACE PROCEDURE insertXmlFile( dir VARCHAR2,
file VARCHAR2,
myTable varchar2,
name VARCHAR2 := NULL) IS
theBFile BFILE;
theCLob CLOB;
theDocName VARCHAR2(200) := NVL(name,file);
BEGIN
-- (1) Insert a new row into xml_documents with an empty CLOB, and
-- (2) Retrieve the empty CLOB into a variable with RETURNING..INTO
INSERT INTO xml_documents(XML_DOCUMENT_NM,XML_DOCUMENT_TX)
VALUES('DELETE ME',empty_clob())
RETURNING XML_DOCUMENT_TX INTO theCLob;
-- (3) Get a BFile handle to the external file
theBFile := BFileName(dir,file);
-- (4) Open the file
dbms_lob.fileOpen(theBFile);
-- (5) Copy the contents of the BFile into the empty CLOB
dbms_lob.loadFromFile(dest_lob => theCLob,
src_lob => theBFile,
amount => dbms_lob.getLength(theBFile));
-- (5a) Insert the rows into the proper table
insProc(theClob, myTable);
-- (6) Close the file and commit
dbms_lob.fileClose(theBFile);
delete from xml_documents
where xml_document_nm = 'DELETE ME';
COMMIT;
END;
execute insertXmlFile ('XMLFILES','emp.xml','emp')Yes I would like to insert directly from an emp.xml file and load it directly into an emp table and not have to load the document in the xml_documents table and then insert into the emp table like I am doing in this procedure.
-
We are using Oracle's XSQL servlet engine 9.0.2.0.0A to query an Oracle Database using xsql:include-owa. This xsql document uses a custom serializer to write the resulting XML to a file on the webserver. We have run into a problem with writing XML documents larger than approximatly 7MB. The file is written but just contains a Malformed XML Document error. We have tried tracking down the problem and it seems to be occuring before the custom serializer is given control of the document. Attached below is the xsql document and the Java source of our serializer if it would be of any help.
-Eric Dalquist
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="simple_transform_xml.xsl" serializer="XMLWriter"?>
<BASE_TAG connection="XXXXXXXX" xmlns:xsql="urn:oracle-xsql">
<xsql:set-page-param name="save-file" value="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/applInfo.xml"/>
<xsql:include-owa >
STUDEV.STUADMSDSET.P_GET_RECRUIT_INFO('A','C');
</xsql:include-owa>
</BASE_TAG>-----------------------------------------
* Author: Eric Dalquist
* E-Mail: [email protected]
* Date Created: 10-03-2001
* Last Modified: 10-03-2001
* Modified By: Eric Dalquist
* Description: This class writes out the contents of the final document produced by the xsql servlet.
package com.mtu.XSQLSerializers;
import java.io.FileOutputStream;
import oracle.xml.xsql.*;
import oracle.xml.parser.v2.XMLDocument;
import org.w3c.dom.Document;
* <P>
* XMLWriter is an implementation of the abstract XSQLDocumentSerializer class. It is passed a refrence to a DOM Document
* and the servlet environment. The XMLWriter takes the document and writes it out to the servlet's output stream. There
* is also the option that the saveFile xsql page parameter can be set to a fully qualified path and file name to write the
* document to.
* </P>
* <P>
* Here is an example of setting the saveFile page paramter. This line would be the first tag inside the root xsql servlet tag.
* </P>
* <PRE>
* <xsql:set-page-param name="save-file" value="/u01/oracle/ias10210/xdk/java/xsql/Repository/Course_List/XMLDocument.xml"/>
* </PRE>
* @author Eric B. Dalquist
* @version 1.0 10/03/2001
public class XMLWriter implements XSQLDocumentSerializer
/** the header we send if we are returning XML */
private static final String MIME_XML = "text/xml";
/** the name of the page parameter we look for to get the path and name to save the XML document to */
private static final String ATTR_FILE = "save-file";
* The serialize is called by the servlet engine and is passed the DOM Docuement result
* from the XSLT transformation if there was one and a refrence to the XSQL servlet environment
* @param doc The document representation of the XML output of the xsql servlet.
* @param env A refrence to the servlet's environment.
* @throws Throwable Any unhandled error is handled by the servlet by default.
public void serialize(Document doc, XSQLPageRequest env) throws Throwable
System.out.print("start ... ");
FileOutputStream fos = null;
String reqType = env.getRequestType();
String saveFile = env.getParameter(ATTR_FILE);
if (!reqType.equals("Command Line"))
env.setContentType(MIME_XML);
((XMLDocument)doc).print(env.getOutputStream());
env.getOutputStream().close();
if (saveFile != null && !saveFile.equals("") && saveFile.indexOf("..") == -1 && saveFile.indexOf("~") == -1)
fos = new FileOutputStream(saveFile);
((XMLDocument)doc).print(fos);
fos.close();
System.gc();
System.out.println("end");
}Technically DBMS_XMLSAVE and it's alter ego DBMS_XMLQUERY are not considered part of XML DB. DBMS_XMLSAVE and DBMS_XMLQUERY are legacy Java implementations.
In 9.2.x these routines were replaced by DBMS_XMLSTORE and DBMS_XMLGEN which are written in 'C' and should be much faster in most cases and address a number of limitations inherant in the older implementations. DBMS_XMLSTORE and DBMS_XMLGEN are part of XML DB and the minimum supported release for XML DB is 9.2.0.3.0.
DBMS_XMLSAVE issues are addressed in the TECH/XML forum
http://forums.oracle.com/forums/category.jspa?categoryID=51 -
ORA-04030: out of process memory Loading Large XML File
Experts: I am trying to load a 2.1G XML file into an Object Relational table . The xml schema document xsd is already registered successfully. It fails with the following error :
ORA-04030: out of process memory when trying to allocate 4032 bytes
(qmxtgCreateBuf,kghsseg: kolaslCreateCtx)
ORA-06512: at "SYS.XMLTYPE", line 296
ORA-06512: at line 1I am able to load the document successfully in SECUREFILE BINARY XML table but that would not work as I need to create relational view on top of this table which does not work with SECUREFILE BINARY XML storage.
Please suggest what may be the workaround here?
Thanks
KevinMDrake: I am trying to load like this:
insert into TEST_HUGE_XML
values(
xmltype(
bfilename('XMLDIR', 'huge_xmldoc.xml')
, nls_charset_id('AL32UTF8')
, 'huge_xmldoc.xsd'
db version: 11.2.0.3I saw an example of loading using createresource API here:
http://www.oracle-developer.net/display.php?id=416
SQL> DECLARE
2 v_return BOOLEAN;
3 BEGIN
4 v_return := DBMS_XDB.CREATERESOURCE(
5 abspath => '/public/demo/xml/db_objects.xml',
6 data => BFILENAME('XML_DIR', 'db_objects.xml')
7 );
8 COMMIT;
9 END;
10 /
PL/SQL procedure successfully completed.How do i load the huge xml document in my custom object relational table which was created like this:
CREATE TABLE HUGE_XML OF XMLTYPE
XMLTYPE STORE AS OBJECT RELATIONAL
XMLSCHEMA "huge_xmldoc.xsd"
ELEMENT "root_element"
Thanks
Edited by: Kevin_K on Feb 8, 2013 9:35 AM
Maybe you are looking for
-
How do I preserve an Existing ext4 Partition during a new install?
Hi, all! I need to do an install, but want to keep my existing /home partition which is on /dev/sda4. I've never done this with Arch. Is there anything I need to know, before I start? Cheers, Chris.
-
Adobe Photoshop Elements 12 won't just Install
Hi there, I purchased a licensed Adobe Photoshop elements 12 from the UK and installing it in Uganda. However when i tried installing it on my iMac running Mac 10.6.8 it failed at the 99% with an error message "the software your trying to install is
-
Hello. I am creating a simple breakout game. When the user hits a block, the block is removed from the display list and a new plane object is created in its place. The problem is the newly created plane movie is never in the same place as the removed
-
Dear Gurus, How can i make the Yellow request to green in the Real time cube, iam making the request green manually so that request can be used for the reporting is there any other means so that i can make yellow request availability for reporting. R
-
Hi folks. I'm a little confused in iDVD6 about what a drop zone is for. Is this to link to a submenu or movie or is it just a graphic as part of the menu? Super confused here. Trying to make my first DVD and it's not cooperating. Thanks.