[9i] poor performance with XMLType.transform

Question

Hello,
I've got a problem with the Oracle function XMLType.transform.
When I try to apply a XSL to a big XML, it is very very slow, and it evens consumes all the CPU, and other users are not able to work until the processing is complete...
So I was wondering if my XSL was corrupted, but it does not seem so, because when i apply it with Internet Explorer (by just double-clicking on the .xml), it is immediately applied. I've also even tried with oraxsl, and the processing is quick and good.
So, i tried to use XDB, but it does not work, maybe I should upgrade to a newer version of XDB?
Please find the ZIP file here :
http://perso.modulonet.fr/~tleoutre/Oracle/samples.zip
Please find in this file :
1) The XSL file k_xsl.xsl
2) The "big" XML file big_xml.xml
Here you can try to apply the XSL on the XML with Internet Explorer : processing is very quick...
3) The batch file transform.bat
Here you can launch it, it calls oraxsl, and produces a result very quickly...
4) The SQL file test_xsl_with_xmltype_transform.sql.
You can try to launch it... First, it applies the same XSL with a little XML, and it's OK... And then, it applies the XSL to the same big XML as in point 1), and then, it takes a lot of time and CPU...
5) The SQL file test_xsl_with_xdb_1.sql ...
On my server, it fails... So I tried to change the XSL in the next point :
6) The SQL file test_xsl_with_xdb_2.sql with a "cleaned" XSL...
And then, it fails with exactly the same problem as in :
TransformerConfigurationException  (Unknown expression at EOF: *|/.)
Any help would be greatly appreciated!
Thank you!
P.S. : Sorry for my bad english, I'm a French man :-)

Answer 1

This is what I see...
Your tests are measuring the wrong thing. You are measuring the time to create the sample documents, which is being done very innefficiently, as well
as the time take to do the transform.
Below is the correct way to get mesasurements for each task..
Here's what I see on a PIV 2.4Ghz with 10.2.0.2.0 and 2GB of RAM
Fragments SourceSize  TargetSize createSource       Parse     Transform
        50      28014      104550  00:00:00.04 00:00:00.04   00:00:00.12
       100      55964      209100  00:00:00.03 00:00:00.05   00:00:00.23
       500     279564     1045500  00:00:00.16 00:00:00.23   00:00:01.76
      1000     559064     2091000  00:00:00.28 00:00:00.28   00:00:06.04
      2000    1118064     4182000  00:00:00.34 00:00:00.42   00:00:24.43
      5000    2795064    10455000  00:00:00.87 00:00:02.02   00:03:19.02I think this clearly shows the pattern.
Of course what this testing really shows is that you've clearly missed the point of performing XSLT transformation inside the database.
The idea behind database based transformation is to optimize XSLT processing by
(1), not having to parse the XML and build a DOM tree before commencing the XSLT processing. In this example this is not possible since the
XML is being created from a CLOB based XMLType, not a schema based XMLType.
(2) Leveraging the Lazily Loaded Virtual DOM when doing sparse transformation ( A Sparse transformation is one where there are large parts of the
source document that are not required to create the target document. Again in this case the XSL requires you to walk all the nodes to generate the
required output.
If is necessary to process all of the nodes in the source document to generate the entire output it probably makes more sense to use a midtier XSL engine.
Here's the code I used to generate the numbers in the above example
BTW in terms of BIG XML we've successully processed 12G documents with Schema Based Storage...So nothing you have hear comes any where our defintion of big.- 1 with Oracle 10g Express on Linux
Also, please remember that 9.2.0.1.0 is not a supported release for any XML DB related features.
SQL*Plus: Release 10.2.0.2.0 - Production on Fri Feb 10 07:44:59 2006
Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.
SQL> spool createDocument.log
SQL> --
SQL> connect &1/&2
Connected.
SQL> --
SQL> create or replace directory &1 as '&3'
  2  /
old   1: create or replace directory &1 as '&3'
new   1: create or replace directory SCOTT as '/home/mdrake/bugs/xslTest'
Directory created.
SQL> drop table source_data_table
  2  /
Table dropped.
SQL> create table source_data_table
  2  (
  3    fragCount number,
  4    xml_text  clob,
  5    xml       xmlType,
  6    result    clob
  7  )
  8  /
Table created.
SQL> create or replace procedure createDocument(fragmentCount number)
  2  as
  3    fragmentText clob :=
  4  '<AFL LIGNUM="1999">
  5    <mat>20000001683</mat>
  6    <name>DOE</name>
  7    <firstname>JOHN</firstname>
  8    <name2>JACK</name2>
  9    <SEX>MALE</SEX>
10    <birthday>1970-05-06</birthday>
11    <salary>5236</salary>
12    <code1>5</code1>
13    <code2>6</code2>
14    <code3>7</code3>
15    <date>2006-05-06</date>
16    <dsp>8.665</dsp>
17    <dsp_2000>455.45</dsp_2000>
18    <darr04>5.3</darr04>
19    <darvap04>6</darvap04>
20    <rcrr>8</rcrr>
21    <rcrvap>9</rcrvap>
22    <rcrvav>10</rcrvav>
23    <rinet>11.231</rinet>
24    <rmrr>12</rmrr>
25    <rmrvap>14</rmrvap>
26    <ro>15</ro>
27    <rr>189</rr>
28    <date2>2004-05-09</date2>
29  </AFL>';
30
31    xmlText CLOB;
32
33  begin
34    dbms_lob.createTemporary(xmlText,true,DBMS_LOB.CALL);
35    dbms_lob.write(xmlText,5,1,'<PRE>');
36    for i in 1..fragmentCount loop
37       dbms_lob.append(xmlText,fragmentText);
38    end loop;
39    dbms_lob.append(xmlText,xmlType('<STA><COD>TER</COD><MSG>Op?ation R?ssie</MSG></STA>').getClobVal());
40    dbms_lob.append(xmlText,'</PRE>');
41    insert into source_data_table (fragCount,xml_text) values (fragmentCount, xmlText);
42    commit;
43    dbms_lob.freeTemporary(xmlText);
44  end;
45  /
Procedure created.
SQL> show errors
No errors.
SQL> --
SQL> set timing on
SQL> --
SQL> call createDocument(50)
  2  /
Call completed.
Elapsed: 00:00:00.04
SQL> call createDocument(100)
  2  /
Call completed.
Elapsed: 00:00:00.03
SQL> call createDocument(500)
  2  /
Call completed.
Elapsed: 00:00:00.16
SQL> call createDocument(1000)
  2  /
Call completed.
Elapsed: 00:00:00.28
SQL> call createDocument(2000)
  2  /
Call completed.
Elapsed: 00:00:00.34
SQL> call createDocument(5000)
  2  /
Call completed.
Elapsed: 00:00:00.87
SQL> select fragCount dbms_lob.getLength(xmlText)
  2    from sample_data_table
  3  /
select fragCount dbms_lob.getLength(xmlText)
ERROR at line 1:
ORA-00923: FROM keyword not found where expected
Elapsed: 00:00:00.00
SQL> quit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
With the Partitioning, OLAP and Data Mining options
SQL*Plus: Release 10.2.0.2.0 - Production on Fri Feb 10 07:45:01 2006
Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.
SQL> spool testcase_&3..log
SQL> --
SQL> connect &1/&2
Connected.
SQL> --
SQL> set timing on
SQL> --
SQL> update source_data_table
  2     set xml = xmltype(xml_text)
  3   where fragCount = &3
  4  /
old   3:  where fragCount = &3
new   3:  where fragCount = 50
1 row updated.
Elapsed: 00:00:00.04
SQL> commit
  2  /
Commit complete.
Elapsed: 00:00:00.01
SQL> update source_data_table
  2     set result = xmltransform(xml,xmltype(bfilename(USER,'&4'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
  3   where fragCount = &3
  4  /
old   2:    set result = xmltransform(xml,xmltype(bfilename(USER,'&4'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
new   2:    set result = xmltransform(xml,xmltype(bfilename(USER,'k_xsl.xsl'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
old   3:  where fragCount = &3
new   3:  where fragCount = 50
1 row updated.
Elapsed: 00:00:00.12
SQL> commit
  2  /
Commit complete.
Elapsed: 00:00:00.01
SQL> select fragCount, dbms_lob.getLength(xml_text),dbms_lob.getLength(result)
  2    from source_data_table
  3  /
FRAGCOUNT DBMS_LOB.GETLENGTH(XML_TEXT) DBMS_LOB.GETLENGTH(RESULT)
        50                        28014                     104550
       100                        55964
       500                       279564
      1000                       559064
      2000                      1118064
      5000                      2795064
6 rows selected.
Elapsed: 00:00:00.01
SQL> quit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
With the Partitioning, OLAP and Data Mining options
SQL*Plus: Release 10.2.0.2.0 - Production on Fri Feb 10 07:45:02 2006
Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.
SQL> spool testcase_&3..log
SQL> --
SQL> connect &1/&2
Connected.
SQL> --
SQL> set timing on
SQL> --
SQL> update source_data_table
  2     set xml = xmltype(xml_text)
  3   where fragCount = &3
  4  /
old   3:  where fragCount = &3
new   3:  where fragCount = 100
1 row updated.
Elapsed: 00:00:00.05
SQL> commit
  2  /
Commit complete.
Elapsed: 00:00:00.01
SQL> update source_data_table
  2     set result = xmltransform(xml,xmltype(bfilename(USER,'&4'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
  3   where fragCount = &3
  4  /
old   2:    set result = xmltransform(xml,xmltype(bfilename(USER,'&4'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
new   2:    set result = xmltransform(xml,xmltype(bfilename(USER,'k_xsl.xsl'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
old   3:  where fragCount = &3
new   3:  where fragCount = 100
1 row updated.
Elapsed: 00:00:00.23
SQL> commit
  2  /
Commit complete.
Elapsed: 00:00:00.03
SQL> select fragCount, dbms_lob.getLength(xml_text),dbms_lob.getLength(result)
  2    from source_data_table
  3  /
FRAGCOUNT DBMS_LOB.GETLENGTH(XML_TEXT) DBMS_LOB.GETLENGTH(RESULT)
        50                        28014                     104550
       100                        55964                     209100
       500                       279564
      1000                       559064
      2000                      1118064
      5000                      2795064
6 rows selected.
Elapsed: 00:00:00.01
SQL> quit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
With the Partitioning, OLAP and Data Mining options
SQL*Plus: Release 10.2.0.2.0 - Production on Fri Feb 10 07:45:02 2006
Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.
SQL> spool testcase_&3..log
SQL> --
SQL> connect &1/&2
Connected.
SQL> --
SQL> set timing on
SQL> --
SQL> update source_data_table
  2     set xml = xmltype(xml_text)
  3   where fragCount = &3
  4  /
old   3:  where fragCount = &3
new   3:  where fragCount = 500
1 row updated.
Elapsed: 00:00:00.12
SQL> commit
  2  /
Commit complete.
Elapsed: 00:00:00.03
SQL> update source_data_table
  2     set result = xmltransform(xml,xmltype(bfilename(USER,'&4'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
  3   where fragCount = &3
  4  /
old   2:    set result = xmltransform(xml,xmltype(bfilename(USER,'&4'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
new   2:    set result = xmltransform(xml,xmltype(bfilename(USER,'k_xsl.xsl'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
old   3:  where fragCount = &3
new   3:  where fragCount = 500
1 row updated.
Elapsed: 00:00:01.76
SQL> commit
  2  /
Commit complete.
Elapsed: 00:00:00.00
SQL> select fragCount, dbms_lob.getLength(xml_text),dbms_lob.getLength(result)
  2    from source_data_table
  3  /
FRAGCOUNT DBMS_LOB.GETLENGTH(XML_TEXT) DBMS_LOB.GETLENGTH(RESULT)
        50                        28014                     104550
       100                        55964                     209100
       500                       279564                    1045500
      1000                       559064
      2000                      1118064
      5000                      2795064
6 rows selected.
Elapsed: 00:00:00.00
SQL> quit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
With the Partitioning, OLAP and Data Mining options
SQL*Plus: Release 10.2.0.2.0 - Production on Fri Feb 10 07:45:04 2006
Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.
SQL> spool testcase_&3..log
SQL> --
SQL> connect &1/&2
Connected.
SQL> --
SQL> set timing on
SQL> --
SQL> update source_data_table
  2     set xml = xmltype(xml_text)
  3   where fragCount = &3
  4  /
old   3:  where fragCount = &3
new   3:  where fragCount = 1000
1 row updated.
Elapsed: 00:00:00.28
SQL> commit
  2  /
Commit complete.
Elapsed: 00:00:00.01
SQL> update source_data_table
  2     set result = xmltransform(xml,xmltype(bfilename(USER,'&4'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
  3   where fragCount = &3
  4  /
old   2:    set result = xmltransform(xml,xmltype(bfilename(USER,'&4'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
new   2:    set result = xmltransform(xml,xmltype(bfilename(USER,'k_xsl.xsl'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
old   3:  where fragCount = &3
new   3:  where fragCount = 1000
1 row updated.
Elapsed: 00:00:06.04
SQL> commit
  2  /
Commit complete.
Elapsed: 00:00:00.00
SQL> select fragCount, dbms_lob.getLength(xml_text),dbms_lob.getLength(result)
  2    from source_data_table
  3  /
FRAGCOUNT DBMS_LOB.GETLENGTH(XML_TEXT) DBMS_LOB.GETLENGTH(RESULT)
        50                        28014                     104550
       100                        55964                     209100
       500                       279564                    1045500
      1000                       559064                    2091000
      2000                      1118064
      5000                      2795064
6 rows selected.
Elapsed: 00:00:00.00
SQL> quit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
With the Partitioning, OLAP and Data Mining options
SQL*Plus: Release 10.2.0.2.0 - Production on Fri Feb 10 07:45:11 2006
Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.
SQL> spool testcase_&3..log
SQL> --
SQL> connect &1/&2
Connected.
SQL> --
SQL> set timing on
SQL> --
SQL> update source_data_table
  2     set xml = xmltype(xml_text)
  3   where fragCount = &3
  4  /
old   3:  where fragCount = &3
new   3:  where fragCount = 2000
1 row updated.
Elapsed: 00:00:00.42
SQL> commit
  2  /
Commit complete.
Elapsed: 00:00:00.02
SQL> update source_data_table
  2     set result = xmltransform(xml,xmltype(bfilename(USER,'&4'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
  3   where fragCount = &3
  4  /
old   2:    set result = xmltransform(xml,xmltype(bfilename(USER,'&4'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
new   2:    set result = xmltransform(xml,xmltype(bfilename(USER,'k_xsl.xsl'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
old   3:  where fragCount = &3
new   3:  where fragCount = 2000
1 row updated.
Elapsed: 00:00:24.43
SQL> commit
  2  /
Commit complete.
Elapsed: 00:00:00.03
SQL> select fragCount, dbms_lob.getLength(xml_text),dbms_lob.getLength(result)
  2    from source_data_table
  3  /
FRAGCOUNT DBMS_LOB.GETLENGTH(XML_TEXT) DBMS_LOB.GETLENGTH(RESULT)
        50                        28014                     104550
       100                        55964                     209100
       500                       279564                    1045500
      1000                       559064                    2091000
      2000                      1118064                    4182000
      5000                      2795064
6 rows selected.
Elapsed: 00:00:00.00
SQL> quit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
With the Partitioning, OLAP and Data Mining options
SQL*Plus: Release 10.2.0.2.0 - Production on Fri Feb 10 07:45:36 2006
Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.
SQL> spool testcase_&3..log
SQL> --
SQL> connect &1/&2
Connected.
SQL> --
SQL> set timing on
SQL> --
SQL> update source_data_table
  2     set xml = xmltype(xml_text)
  3   where fragCount = &3
  4  /
old   3:  where fragCount = &3
new   3:  where fragCount = 5000
1 row updated.
Elapsed: 00:00:02.02
SQL> commit
  2  /
Commit complete.
Elapsed: 00:00:00.05
SQL> update source_data_table
  2     set result = xmltransform(xml,xmltype(bfilename(USER,'&4'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
  3   where fragCount = &3
  4  /
old   2:    set result = xmltransform(xml,xmltype(bfilename(USER,'&4'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
new   2:    set result = xmltransform(xml,xmltype(bfilename(USER,'k_xsl.xsl'),nls_charset_id('WE8ISO8859P1'))).getClobVal()
old   3:  where fragCount = &3
new   3:  where fragCount = 5000
1 row updated.
Elapsed: 00:03:19.02
SQL> commit
  2  /
Commit complete.
Elapsed: 00:00:00.01
SQL> select fragCount, dbms_lob.getLength(xml_text),dbms_lob.getLength(result)
  2    from source_data_table
  3  /
FRAGCOUNT DBMS_LOB.GETLENGTH(XML_TEXT) DBMS_LOB.GETLENGTH(RESULT)
        50                        28014                     104550
       100                        55964                     209100
       500                       279564                    1045500
      1000                       559064                    2091000
      2000                      1118064                    4182000
      5000                      2795064                   10455000
6 rows selected.
Elapsed: 00:00:00.04
SQL> quit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
With the Partitioning, OLAP and Data Mining options
bash-2.05b$