Insert really slow on 100Mb file

Question

Hello,
I have an XMLType table with a registered XSD. Performance on smaller files is good but 1 XML file which is 100 Mb takes 8+ hours only to load. I really need to speeds this up.
I detected the most time being consumed in the INSERT statement. This i didnt expect but also i dont know how to improve this. I hope someone can help me out here.
See following for detailed information.
Thanks in advance
Remy
Table definition
CREATE TABLE PHILIPS_XML of XMLType
                       XMLSCHEMA "http://localhost/philips_format.xsd" ELEMENT "Tree"First thing i tried is loading the file directly into the Table using
INSERT INTO PHILIPS_XML
  VALUES (XMLTYPE(bfilename('XMLDIR',:1 ),NLS_CHARSET_ID('AL32UTF8')))No performance so i tried loading using the sql*loader but no major succes on this one also
In order to find out where my performance loss is i tried loading into a CLOB and then insert into the XMLType table
CREATE TABLE TEMP_CLOB
                       (TEMP_DATA CLOB)
    dest_clob   CLOB;
    src_clob    BFILE;
    dst_offset  number := 1 ;
    src_offset  number := 1 ;
    lang_ctx    number := DBMS_LOB.DEFAULT_LANG_CTX;
    warning     number;
   BEGIN
    INSERT INTO temp_clob(temp_data)
        VALUES(empty_clob())
        RETURNING temp_data INTO dest_clob;
    -- OPENING THE SOURCE BFILE IS MANDATORY
    src_clob:=bfilename('XMLDIR',p_supplier_xml);
    DBMS_LOB.OPEN(src_clob, DBMS_LOB.LOB_READONLY);
    DBMS_LOB.LoadCLOBFromFile(
          DEST_LOB     => dest_clob
        , SRC_BFILE    => src_clob
        , AMOUNT       => DBMS_LOB.GETLENGTH(src_clob)
        , DEST_OFFSET  => dst_offset
        , SRC_OFFSET   => src_offset
        , BFILE_CSID   => DBMS_LOB.DEFAULT_CSID
        , LANG_CONTEXT => lang_ctx
        , WARNING      => warning
    DBMS_LOB.CLOSE(src_clob);
  I let the loading part run with a trace.
INSERT INTO PHILIPS_XML
        SELECT XMLTYPE(TEMP_DATA) FROM TEMP_CLOB;   Here are the trace results (please forget the first error .. i made a typo in the statement)
TKPROF: Release 10.1.0.2.0 - Production on Mon May 19 09:20:45 2008
Copyright (c) 1982, 2004, Oracle.  All rights reserved.
Trace file: ecd1d_ora_19063_REMY.trc
Sort options: default
count    = number of times OCI procedure was executed
cpu      = cpu time in seconds executing
elapsed  = elapsed time in seconds executing
disk     = number of physical reads of buffers from disk
query    = number of buffers gotten for consistent read
current  = number of buffers gotten in current mode (usually for update)
rows     = number of rows processed by the fetch or execute call
The following statement encountered a error during parse:
insert into philips_xml
(select xmltype(xml_data) from temp_table
Error encountered: ORA-00942
select metadata
from
kopm$  where name='DB_FDO'
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.00       0.00          0          0          0           0
Execute      2      0.01       0.00          0          0          0           0
Fetch        1      0.00       0.00          0          2          0           1
total        4      0.01       0.00          0          2          0           1
Misses in library cache during parse: 0
Optimizer mode: CHOOSE
Parsing user id: SYS   (recursive depth: 1)
Rows     Row Source Operation
      1  TABLE ACCESS BY INDEX ROWID KOPM$ (cr=2 pr=0 pw=0 time=59 us)
      1   INDEX UNIQUE SCAN I_KOPM1 (cr=1 pr=0 pw=0 time=31 us)(object id 365)
SELECT /* OPT_DYN_SAMP */ /*+ ALL_ROWS IGNORE_WHERE_CLAUSE
  NO_PARALLEL(SAMPLESUB) opt_param('parallel_execution_enabled', 'false')
  NO_PARALLEL_INDEX(SAMPLESUB) NO_SQL_TUNE */ NVL(SUM(C1),:"SYS_B_0"),
  NVL(SUM(C2),:"SYS_B_1")
FROM
(SELECT /*+ NO_PARALLEL("TEMP_CLOB") FULL("TEMP_CLOB")
  NO_PARALLEL_INDEX("TEMP_CLOB") */ :"SYS_B_2" AS C1, :"SYS_B_3" AS C2 FROM
  "TEMP_CLOB" "TEMP_CLOB") SAMPLESUB
call     count       cpu    elapsed       disk      query    current        rows
Parse        3      0.00       0.00          0          0          0           0
Execute      3      0.01       0.00          0          0          0           0
Fetch        3      0.00       0.00          0         23          0           3
total        9      0.01       0.00          0         23          0           3
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 46     (recursive depth: 1)
Rows     Row Source Operation
      1  SORT AGGREGATE (cr=9 pr=0 pw=0 time=601 us)
      0   TABLE ACCESS FULL TEMP_CLOB (cr=9 pr=0 pw=0 time=431 us)
insert into philips_xml
(select xmltype (temp_data) from temp_clob)
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.02       0.01          0        108          0           0
Execute      1      0.00       0.00          0          7          0           0
Fetch        0      0.00       0.00          0          0          0           0
total        2      0.02       0.01          0        115          0           0
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 87 
Rows     Row Source Operation
      0  TABLE ACCESS FULL TEMP_CLOB (cr=7 pr=0 pw=0 time=64 us)
(select xmltype (temp_data) from temp_clob)
call     count       cpu    elapsed       disk      query    current        rows
Parse        2      0.01       0.00          0          8          0           0
Execute      2      0.00       0.00          0          0          0           0
Fetch        2    100.80     260.23     263377    1756877          0           1
total        6    100.81     260.24     263377    1756885          0           1
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 87 
Rows     Row Source Operation
      1  TABLE ACCESS FULL TEMP_CLOB (cr=7 pr=0 pw=0 time=492 us)
select sys_nc_oid$
from
xdb.xdb$resource where rowid = :1
call     count       cpu    elapsed       disk      query    current        rows
Parse        2      0.00       0.00          0          0          0           0
Execute      2      0.00       0.00          0          0          0           0
Fetch        2      0.00       0.00          0          4          0           2
total        6      0.00       0.00          0          4          0           2
Misses in library cache during parse: 0
Optimizer mode: CHOOSE
Parsing user id: SYS   (recursive depth: 1)
Rows     Row Source Operation
      1  TABLE ACCESS BY USER ROWID XDB$RESOURCE (cr=2 pr=0 pw=0 time=40 us)
select value(p$)
from
"XDB"."XDB$RESOURCE"                 as of snapshot(:2) p$ where SYS_NC_OID$
  = :1
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.00       0.00          0          0          0           0
Execute      2      0.00       0.00          0          0          0           0
Fetch        2      0.00       0.00          0          4          0           2
total        5      0.00       0.00          0          4          0           2
Misses in library cache during parse: 0
Optimizer mode: CHOOSE
Parsing user id: SYS   (recursive depth: 1)
select obj#,type#,ctime,mtime,stime,status,dataobj#,flags,oid$, spare1,
  spare2
from
obj$ where owner#=:1 and name=:2 and namespace=:3 and remoteowner is null
  and linkname is null and subname is null
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.00       0.00          0          0          0           0
Execute      2      0.00       0.00          0          0          0           0
Fetch        2      0.00       0.00          0          4          0           0
total        5      0.00       0.00          0          4          0           0
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: CHOOSE
Parsing user id: SYS   (recursive depth: 1)
Rows     Row Source Operation
      0  TABLE ACCESS BY INDEX ROWID OBJ$ (cr=2 pr=0 pw=0 time=53 us)
      0   INDEX RANGE SCAN I_OBJ2 (cr=2 pr=0 pw=0 time=48 us)(object id 37)
insert into philips_xml
(select xmltype(temp_data) from temp_clob)
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.01       0.00          0         15          0           0
Execute      1  25003.01   30465.99     526759    3514263      29095           1
Fetch        0      0.00       0.00          0          0          0           0
total        2  25003.02   30465.99     526759    3514278      29095           1
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 87 
select file#
from
file$ where ts#=:1
call     count       cpu    elapsed       disk      query    current        rows
Parse       82      0.00       0.00          0          0          1           0
Execute     82      0.08       0.11          0          0          0           0
Fetch      164      0.03       0.01          0        328          0          82
total      328      0.11       0.13          0        328          1          82
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: CHOOSE
Parsing user id: SYS   (recursive depth: 1)
Rows     Row Source Operation
      1  TABLE ACCESS FULL FILE$ (cr=4 pr=0 pw=0 time=145 us)
update seg$ set type#=:4,blocks=:5,extents=:6,minexts=:7,maxexts=:8,extsize=
  :9,extpct=:10,user#=:11,iniexts=:12,lists=decode(:13, 65535, NULL, :13),
  groups=decode(:14, 65535, NULL, :14), cachehint=:15, hwmincr=:16, spare1=
  DECODE(:17,0,NULL,:17),scanhint=:18
where
ts#=:1 and file#=:2 and block#=:3
call     count       cpu    elapsed       disk      query    current        rows
Parse       82      0.00       0.00          0          0          2           0
Execute     82      0.03       0.06          0        410         82          82
Fetch        0      0.00       0.00          0          0          0           0
total      164      0.03       0.06          0        410         84          82
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: CHOOSE
Parsing user id: SYS   (recursive depth: 1)
Rows     Row Source Operation
      0  UPDATE  SEG$ (cr=5 pr=0 pw=0 time=537 us)
      1   TABLE ACCESS CLUSTER SEG$ (cr=5 pr=0 pw=0 time=175 us)
      1    INDEX UNIQUE SCAN I_FILE#_BLOCK# (cr=2 pr=0 pw=0 time=21 us)(object id 9)
select /*+ rule */ bucket_cnt, row_cnt, cache_cnt, null_cnt, timestamp#,
  sample_size, minimum, maximum, distcnt, lowval, hival, density, col#,
  spare1, spare2, avgcln
from
hist_head$ where obj#=:1 and intcol#=:2
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.00       0.00          0          0          6           0
Execute      7      0.01       0.01          0          0          0           0
Fetch        7      0.00       0.00          0         21          0           7
total       15      0.01       0.01          0         21          6           7
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: RULE
Parsing user id: SYS   (recursive depth: 2)
Rows     Row Source Operation
      1  TABLE ACCESS BY INDEX ROWID HIST_HEAD$ (cr=3 pr=0 pw=0 time=398 us)
      1   INDEX RANGE SCAN I_HH_OBJ#_INTCOL# (cr=2 pr=0 pw=0 time=374 us)(object id 257)
OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
call     count       cpu    elapsed       disk      query    current        rows
Parse        4      0.04       0.02          0        131          0           0
Execute      4  25003.01   30465.99     526759    3514270      29095           1
Fetch        2    100.80     260.23     263377    1756877          0           1
total       10  25103.85   30726.25     790136    5271278      29095           2
Misses in library cache during parse: 3
OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
call     count       cpu    elapsed       disk      query    current        rows
Parse      173      0.00       0.02          0          0          9           0
Execute    182      0.14       0.20          0        410         82          82
Fetch      181      0.03       0.01          0        386          0          97
total      536      0.17       0.24          0        796         91         179
Misses in library cache during parse: 5
Misses in library cache during execute: 5
    7  user  SQL statements in session.
  177  internal SQL statements in session.
  184  SQL statements in session.
Trace file: ecd1d_ora_19063_REMY.trc
Trace file compatibility: 10.01.00
Sort options: default
       0  session in tracefile.
       7  user  SQL statements in trace file.
     177  internal SQL statements in trace file.
     184  SQL statements in trace file.
      11  unique SQL statements in trace file.
    1653  lines in trace file.
   31142  elapsed seconds in trace file.XSD + register statement is here:
    DBMS_XMLSCHEMA.REGISTERSCHEMA(
       'http://localhost/philips_format.xsd'  ,
       '<?xml version="1.0" encoding="UTF-8"?>
       <!-- edited with XMLSPY v5 rel. 3 U (http://www.xmlspy.com) by Philips Consumer Electronics BV (Philips Consumer Electronics BV) -->
       <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">
        <xs:element name="Tree">
            <xs:complexType>
                <xs:sequence>
                    <xs:element ref="Node" minOccurs="0" maxOccurs="unbounded"/>
                </xs:sequence>
            </xs:complexType>
        </xs:element>
        <xs:element name="Node">
            <xs:annotation>
                <xs:documentation>This XSD is designed to have very similar structure as the XML v2 specification. Please see XMLv2Spec_rev_15.pdf for more information </xs:documentation>
            </xs:annotation>
            <xs:complexType>
                <xs:sequence>
                    <xs:element name="Id" type="xs:string"/>
                    <xs:element name="SuppliersId" type="xs:string" minOccurs="0">
                        <xs:annotation>
                            <xs:documentation>Is used to make a unique identifier that is readable. You dont have to worry about it, this key is unique across all service providers because the id that is stored in the DB will have the service nodes id prepended to it</xs:documentation>
                        </xs:annotation>
                    </xs:element>
                    <xs:element name="TopLevel" minOccurs="0">
                        <xs:annotation>
                            <xs:documentation>Add this tag if the node is to be the first node after the service node. </xs:documentation>
                        </xs:annotation>
                        <xs:complexType/>
                    </xs:element>
                    <xs:element name="Name" type="xs:string" minOccurs="0"/>
                    <xs:element name="ChildNodes" minOccurs="0">
                        <xs:annotation>
                            <xs:documentation>The list of child nodes. Note that this doesnt form a strict tree!! A child node can have multiple parent nodes.</xs:documentation>
                        </xs:annotation>
                        <xs:complexType>
                            <xs:sequence>
                                <xs:element name="ChildNode" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
                            </xs:sequence>
                        </xs:complexType>
                    </xs:element>
                    <xs:element name="DataItems" minOccurs="0">
                        <xs:annotation>
                            <xs:documentation>Exactly the same as the XMLv2 Spec</xs:documentation>
                        </xs:annotation>
                        <xs:complexType>
                            <xs:sequence>
                                <xs:element name="DataItem" type="DataItem" minOccurs="0" maxOccurs="unbounded"/>
                            </xs:sequence>
                        </xs:complexType>
                    </xs:element>
                    <xs:element name="NodeProperties" type="Properties" minOccurs="0">
                        <xs:annotation>
                            <xs:documentation>For internal use. It is used for as a point to add service specific code that is needed for that dataloader. </xs:documentation>
                        </xs:annotation>
                    </xs:element>
                    <xs:element name="Filters" minOccurs="0">
                        <xs:annotation>
                            <xs:documentation>For internal use. This is where a list of filters can be added. There are many filters in ECD if we need.</xs:documentation>
                        </xs:annotation>
                        <xs:complexType>
                            <xs:sequence maxOccurs="unbounded">
                                <xs:element name="Filter" type="xs:string" minOccurs="0"/>
                            </xs:sequence>
                        </xs:complexType>
                    </xs:element>
                    <xs:element name="DynamicNavigationNode" minOccurs="0">
                        <xs:annotation>
                            <xs:documentation>For internal use. A flag that indicates that there will be child nodes that are going to be created so this node should be a navigationNode and not a playableNode</xs:documentation>
                        </xs:annotation>
                        <xs:complexType/>
                    </xs:element>
                    <xs:element name="DynamicDataLoader" type="xs:string" minOccurs="0">
                        <xs:annotation>
                            <xs:documentation>For internal use. This points to an component</xs:documentation>
                        </xs:annotation>
                    </xs:element>
                    <xs:element name="MenuHelper" type="xs:string" minOccurs="0">
                        <xs:annotation>
                            <xs:documentation>For internal use. A menu helper is used to return how the node should be displayed and not what it actually is for example there is a DropThoughMenuHelper that will make the current node effectively invisible and will display the nodes children instead</xs:documentation>
                        </xs:annotation>
                    </xs:element>
                    <xs:element name="PlayableProperties" type="PlayableProperties" minOccurs="0"/>
                    <xs:element name="Type" type="xs:int" minOccurs="0">
                        <xs:annotation>
                            <xs:documentation>This allows you to override the default node type. In almost all cases the default node type is needed. The data loader will make nodes that have children a navigationNode and the ones that have no children, playableNodes</xs:documentation>
                        </xs:annotation>
                    </xs:element>
                </xs:sequence>
            </xs:complexType>
        </xs:element>
        <xs:complexType name="DataItem">
            <xs:all>
                <xs:element name="Title" type="xs:string" minOccurs="0"/>
                <xs:element name="Album" type="xs:string" minOccurs="0"/>
                <xs:element name="Artist" type="xs:string" minOccurs="0"/>
                <xs:element name="Playlength" type="xs:int" minOccurs="0"/>
                <xs:element name="MIMEType" type="xs:string" minOccurs="0"/>
                <xs:element name="Aspect" type="xs:string" minOccurs="0"/>
                <xs:element name="Bitrate" type="xs:string" minOccurs="0"/>
                <xs:element name="Description" type="xs:string" minOccurs="0"/>
                <xs:element name="Filesize" type="xs:string" minOccurs="0"/>
                <xs:element name="FramesPerSecond" type="xs:string" minOccurs="0"/>
                <xs:element name="Genre" type="xs:string" minOccurs="0"/>
                <xs:element name="Height" type="xs:int" minOccurs="0"/>
                <xs:element name="Quality" type="xs:string" minOccurs="0"/>
                <xs:element name="Samplerate" type="xs:string" minOccurs="0"/>
                <xs:element name="TrackNumber" type="xs:string" minOccurs="0"/>
                <xs:element name="Width" type="xs:int" minOccurs="0"/>
                <xs:element name="Year" type="xs:int" minOccurs="0"/>
                <xs:element ref="DetailSet" minOccurs="0"/>
                <xs:element ref="URLSet" minOccurs="0"/>
                <xs:element name="Usage" type="xs:string" minOccurs="0"/>
            </xs:all>
        </xs:complexType>
        <xs:complexType name="Properties">
            <xs:sequence>
                <xs:element name="Property" minOccurs="0" maxOccurs="unbounded">
                    <xs:complexType>
                        <xs:sequence>
                            <xs:element name="Key" type="xs:string"/>
                            <xs:element name="Value" type="xs:string"/>
                        </xs:sequence>
                    </xs:complexType>
                </xs:element>
            </xs:sequence>
        </xs:complexType>
        <xs:element name="DetailSet">
            <xs:complexType>
                <xs:sequence>
                    <xs:element name="Detail" minOccurs="0" maxOccurs="unbounded">
                        <xs:complexType>
                            <xs:sequence>
                                <xs:element name="Title" type="xs:string" minOccurs="0"/>
                                <xs:element name="Value" type="xs:string" minOccurs="0"/>
                            </xs:sequence>
                        </xs:complexType>
                    </xs:element>
                </xs:sequence>
            </xs:complexType>
        </xs:element>
        <xs:element name="URLSet">
            <xs:complexType>
                <xs:sequence>
                    <xs:element name="URL" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
                </xs:sequence>
            </xs:complexType>
        </xs:element>
        <xs:complexType name="PlayableProperties">
            <xs:sequence minOccurs="0">
                <xs:element name="QualityLevels" minOccurs="0">
                    <xs:complexType>
                        <xs:sequence minOccurs="0" maxOccurs="unbounded">
                            <xs:element name="Level" type="xs:string"/>
                        </xs:sequence>
                    </xs:complexType>
                </xs:element>
                <xs:element name="Recordable" type="xs:integer" minOccurs="0"/>
                <xs:element name="SkipForward" type="xs:integer" minOccurs="0"/>
                <xs:element name="SkipBackward" type="xs:integer" minOccurs="0"/>
                <xs:element name="Pauseable" type="xs:integer" minOccurs="0"/>
                <xs:element name="Seekable" type="xs:integer" minOccurs="0"/>
                <xs:element name="RepeatAll" type="xs:integer" minOccurs="0"/>
                <xs:element name="RepeatTrack" type="xs:integer" minOccurs="0"/>
                <xs:element name="MandatoryConnect" type="xs:integer" minOccurs="0"/>
                <xs:element name="RequestSingleSet" type="xs:integer" minOccurs="0"/>
                <xs:element name="InvalidateSetOnError" type="xs:integer" minOccurs="0"/>
                <xs:element name="ReconnectOnError" type="xs:integer" minOccurs="0"/>
                <xs:element name="Syncable" type="xs:integer" minOccurs="0"/>
                <xs:element name="ShowProgressBar" type="xs:integer" minOccurs="0"/>
                <xs:element name="AllowDigitalOut" type="xs:integer" minOccurs="0"/>
            </xs:sequence>
        </xs:complexType>
    </xs:schema>'
    , TRUE
    , TRUE
    , FALSE
    , TRUE);

Answer 1

You can "boost" via FTP by setting database initialization parameters:
- shared servers = 5
- large_pool_size at a minimum of 150 Mb (please increase ++ if you have the room / mem.)
- java_pool_size at a minimum of 150 Mb
- don't use sga_target and other "memory wizards" but set the parameters manually
- purge the dba_recyclebin and set the feature off.
- have a look a the tnsnames.ora and listener.ora files and / or local_listener and dispatcher settings and avoid TCP/IP name resolution. Use hard coded names, tcp/ip names (name resolution via the hosts file / NOT via a nameserver)
to name some pointers.