Problem Loading Large XML Doc

Question

I'm running 10.2.0.3 on a Linux box and I'm having problems loading a large XML document (about 100 MB). In the past, I would simply load the XML file into a XMLType column like this:
INSERT INTO foo VALUES (XMLType(bfilename('XMLDIR', 'test.xml'). nls_charset_id('AL32UTF8')));
But when I try this with a large file, it runs for 10 minutes and then returns an ORA-03113. I'm assuming the file is just too large for this technique. I spoke to Mark Drake when I was at OpenWorld and he suggested I use Oracle XML DB, so I created and registered a schema and tried using sqlldr to load the doc, but it ran for 2 1/2 hours before returning:
Parse Error on row 1 in table FOO
OCI-31038: Invalid integer value: "129"
I tried simplifying both the XML file and schema to just the following:
<schedules>
<s s="2009-09-21T04:00:00" d="21600" p="335975" c="19672"/>
<s s="2009-09-21T04:00:00" d="21600" p="335975" c="15387"/>
<s s="2009-09-21T04:00:00" d="25200" p="335975" c="5256"/>
<s s="2009-09-21T04:00:00" d="86400" p="335975" c="26198">
<k id="5" v="2009-09-21 09:00:00.000"/>
<k id="6" v="2009-09-22 03:59:59.000"/>
<k id="26" v="0.00"/><k id="27" v="US"/>
</s>
<s s="2009-09-21T04:00:00" d="21600" p="335975" c="11678"/>
<s s="2009-09-21T04:00:00" d="21600" p="335975" c="26697"/>
<s s="2009-09-21T04:00:00" d="21600" p="335975" c="25343"/>
<s s="2009-09-21T04:00:00" d="21600" p="335975" c="25269"/>
<s s="2009-09-21T04:00:00" d="86400" p="335975" c="26200">
<k id="5" v="2009-09-21 09:00:00.000"/>
<k id="6" v="2009-09-22 03:59:59.000"/>
<k id="26" v="0.00"/><k id="27" v="US"/>
</s>
</schedules>
<?xml version="1.0" encoding="UTF-8"?>
<!--W3C Schema generated by XMLSpy v2008 sp1 (http://www.altova.com)-->
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
     xmlns:xdb="http://xmlns.oracle.com/xdb"
    version="1.0"
    xdb:storeVarrayAsTable="true">
     <xs:element name="schedules" xdb:defaultTable="SCHEDULES">
          <xs:complexType xdb:SQLType="SCHEDULES_T">
               <xs:sequence>
                    <xs:element ref="s" maxOccurs="unbounded"/>
               </xs:sequence>
          </xs:complexType>
     </xs:element>
     <xs:element name="s">
          <xs:complexType>
               <xs:choice minOccurs="0">
                    <xs:element ref="f" maxOccurs="unbounded"/>
                    <xs:element ref="k" maxOccurs="unbounded"/>
               </xs:choice>
               <xs:attribute name="s" use="required" type="xs:dateTime"/>
               <xs:attribute name="p" use="required" type="xs:int"/>
               <xs:attribute name="d" use="required" type="xs:int"/>
               <xs:attribute name="c" use="required" type="xs:short"/>
          </xs:complexType>
     </xs:element>
     <xs:element name="k">
          <xs:complexType>
               <xs:attribute name="v" use="required" type="xs:string"/>
               <xs:attribute name="id" use="required" type="xs:byte"/>
          </xs:complexType>
     </xs:element>
     <xs:element name="f">
          <xs:complexType>
               <xs:attribute name="id" use="required" type="xs:byte"/>
          </xs:complexType>
     </xs:element>
</xs:schema>
Keep in mind both the actual XML file and corresponding XSD is much more complex, but this particular section is about 70 MB so I wanted to see if I could just load that.  I used the following sqlldr script:
LOAD DATA
INFILE *
INTO TABLE schedules_tmp TRUNCATE
XMLType(xml_doc)(
     lobfn FILLER CHAR TERMINATED by ',',
     xml_doc LOBFILE(lobfn) TERMINATED BY EOF
BEGINDATA
/tmp/schedules.xmlThis worked fine on a small doc - loaded correctly and I could query it fine - but when I tried using the 70 MB file it ran for a couple of hours before dying with a memory problem.
So what am I doing wrong? Is there a better way to load a large file?
Thanks for the help.
Pete
Edited by: mdrake on Nov 9, 2009 8:46 PM

Answer 1

Mark:
Answers to your questions:
Do you use direct load ? -> Yes. I tried again using UNRECOVERABLE LOAD DATA to try to speed up performance, but it still ran for a couple of hours before dying.
Which DB Release are you working with ? -> 10.2.0.3
Can you see if you can upload via FTP ? -> I added noNamespaceSchemaLocation to my XML file and ftp'd it to my XML directory, but it wasn't recognized. Is there something else I have to do?
The change for unsignedInt should have code rid of the issue with the value 129. Did it ? -> I didn't try it again on the whole XML file (I'm just working with the schedules section), so I haven't verified this.
I'm still stumped as to why sqlldr takes so long. I could write something to parse the XML file into a flat file and then use sqlldr to load it into a relational table, and the load would only take a few minutes. But then I wouldn't be using XML DB which I thought would be faster. What am I doing wrong?
Pete