Querying XML File

Question

Hi,
Based on the logic provided in forum(Please see below), we have developed procedure to read XML file which we receive as CLOB input parameter. This XML file will contain 50 records which need to be read inside the procedure. To read these 50 records it is taking 2 seconds of time.
Now we got requirement to read up to 1500 records which will be supplied by CLOB field as input parameter to the procedure.
But we are facing really a big challenge over here. Initially in testing phase we are trying to read just 100 records for which the time taking from point 1 to point 2(Mentioned below, that is going in to the loop) is around 60 sec. As you know this is not acceptable. So can you please help me to reduce this time frame?
POINT 1
For I IN( select /*+ no_xml_query_rewrite */
x.firstname fnname,
x.LastName lstname,
x.BirthDate birthdate,
x.Gender gender,
x.usercode usercode,
x.opened opened,
x.time time,
x.TobaccoInd TobaccoInd,
x.Employmeeid Employmeeid,
x.Title Title,
x.zip zip,
x.riskclass riskclass,
x.policystatus policystatus,
x.NYReg60Ind NYReg60Ind,
x.Exchange1035Ind Exchange1035Ind,
x.ModalPremAmt ModalPremAmt,
x.FaceAmt FaceAmt,
x.vProductType vProductType,
x.vHoldingSysKey vHoldingSysKey,
x.InterestPercent InterestPercent,
x.vPolno vPolno,
/*x.DSGN_OPT_AMT DSGN_OPT_AMT,
x.dsgn_opt_pct dsgn_opt_pct,
x.ArrType ArrType,
x.ArrSubType ArrSubType,
x.ArrStatus ArrStatus,*/
x.Arrangement,
x.vattdate vattdate,
x.vatttime vatttime,
x.EVENT_NM EVENT_NM,
x.AttachmentData AttachmentData,
x.AttachmentSource AttachmentSource,
x.attusercode attusercode,
x.vHoldingid Holdingid
from (SELECT xData doc FROM dual) t,
xmltable(
'let $x := $d/TXLife/TXLifeRequest/OLifE
for $i in $x/Relation[RelationRoleCode/@tc=32]
return element r
$x/Holding[@id=$i/@OriginatingObjectID]/@id,
$x/Holding[@id=$i/@OriginatingObjectID]/Policy,
$x/Holding[@id=$i/@OriginatingObjectID]//Arrangement,
$x/Holding[@id=$i/@OriginatingObjectID]/HoldingSysKey,
$x/Holding[@id=$i/@OriginatingObjectID]/Attachment,
$x/Party[@id=$i/@RelatedObjectID]/Person,
$x/Party[@id=$i/@RelatedObjectID]/Risk,
$x/Party[@id=$i/@RelatedObjectID]/Employment,
$x/Party[@id=$i/@RelatedObjectID]/Address,
$x/Activity,
$x/Relation[@OriginatingObjectID=$x/Holding[@id=$i/@OriginatingObjectID]/@id][RelationRoleCode/@tc=32]/InterestPercent
passing doc as "d"
columns
vHoldingid varchar2(50) path '@id',
usercode varchar2(200) path 'Activity/UserCode',
opened varchar2(20) path 'Activity/Opened',
time varchar2(20) path 'Activity/OLifEExtension/ActivityExtension/OpenedTime',
InterestPercent varchar2(200) path 'InterestPercent',
vPolno number path 'Policy/PolNumber',
riskclass varchar2(20) path 'Policy/Life/Coverage/CovOption/UnderwritingClass/@tc',
policystatus varchar2(100) path 'Policy/PolicyStatus/@tc',
NYReg60Ind VARCHAR2(100) path 'Policy/OLifEExtension/PolicyExtension/NYReg60Ind/@tc',
Exchange1035Ind varchar2(10) path 'Policy/Life/LifeUSA/OLifEExtension/LifeUSAExtension/Exchange1035Ind/@tc',
ModalPremAmt varchar2(20) path 'Policy/Life/Coverage/ModalPremAmt',
FaceAmt varchar2(20) path 'Policy/Life/FaceAmt',
vProductType VARCHAR2(100) path 'Policy/ProductType/@tc',
firstname VARCHAR2(200) path 'Person/FirstName',
LastName VARCHAR2(200) path 'Person/LastName',
BirthDate VARCHAR2(20) PATH 'Person/BirthDate',
Gender varchar2(10) PATH 'Person/Gender/@tc',
TobaccoInd varchar2(5) path 'Risk/TobaccoInd/@tc',
Employmeeid varchar2(100) path 'Employment/EmployeeID',
Title varchar2(100) path 'Employment/Title',
zip varchar2(100) path 'Address/Zip',
vHoldingSysKey varchar2(100) path 'HoldingSysKey',
Arrangement XMLTYPE PATH 'Arrangement',
/*DSGN_OPT_AMT varchar2(100) path 'Arrangement/ArrSource[TransferAmtType/@tc=2]/TransferAmt',
dsgn_opt_pct varchar2(100) path 'Arrangement/ArrSource[TransferAmtType/@tc=3]/TransferPct',
ArrType VARCHAR2(20) path 'Arrangement/ArrType/@tc',
ArrSubType VARCHAR2(20) path 'Arrangement/OLifEExtension/ArrangementExtension/ArrSubType/@tc',
ArrStatus VARCHAR2(10) path 'Arrangement/ArrStatus/@tc',*/
EVENT_NM VARCHAR2(1000) path 'Attachment/Description',
AttachmentData VARCHAR2(1000) path 'Attachment/AttachmentData',
AttachmentSource VARCHAR2(1000) path 'Attachment/AttachmentSource',
attusercode vARCHAR2(1000) path 'Attachment/UserCode',
vattdate VARCHAR2(20) PATH 'Attachment/DateCreated',
vatttime VARCHAR2(20) path 'Attachment/OLifEExtension/AttachmentExtension/CreationTime'
) x
LOOP
     BEGIN
          POINT 2
          <Here comes the business logic>
     END;
END LOOP;
Logic posted in Forum:
I guess this should normally work without the hint (NO_XML_QUERY_REWRITE), but actually it doesn't :
SQL> select /*+ no_xml_query_rewrite */
2 x.*
3 from test_xml t,
4 xmltable(
5 'let $x := $d/TXLife/TXLifeRequest/OLifE
6 for $i in $x/Relation[RelationRoleCode/@tc="32"]
7 return element r
8 {
9 $x/Holding[@id=$i/@OriginatingObjectID]/Policy
10 , $x/Party[@id=$i/@RelatedObjectID]/Person
11 }'
12 passing t.object_value as "d"
13 columns PolNumber number path 'Policy/PolNumber'
14 , FirstName varchar2(30) path 'Person/FirstName'
15 , LastName varchar2(30) path 'Person/LastName'
16 ) x
17 ;
POLNUMBER FIRSTNAME LASTNAME
308221234 John Doe
308221235 Jane Doe
TEST_XML is an XMLType table (binary storage) where I loaded the XML document.
Edited by: 916962 on Feb 25, 2012 5:42 AM

Answer 1

Vamsi,
Please try the steps below :
1) Create an xmltype table (could be temporary if you want) :
create table tmp_xml of xmltype
xmltype store as securefile binary xml;2) At the beginning of your procedure, store the input CLOB in the table :
insert into tmp_xml values(xmltype(xData));3) Then in the procedure, query directly from the XMLType table.
Here's a modified and simplified (for this example) version of your actual query :
select x.ooid
     , x.roid
     , x.interestpercent
     , x1.*
     , x2.*
from tmp_xml t
   , xmltable(
       '/TXLife/TXLifeRequest/OLifE/Relation[RelationRoleCode/@tc="32"]'
       passing t.object_value
       columns OOID            varchar2(50)  path '@OriginatingObjectID'
             , ROID            varchar2(50)  path '@RelatedObjectID'
             , InterestPercent varchar2(200) path 'InterestPercent'
     ) x
   , xmltable(
       '/TXLife/TXLifeRequest/OLifE/Holding[@id=$ooid]'
       passing t.object_value, x.ooid as "ooid"
       columns
         vPolno          number        path 'Policy/PolNumber',
         vProductType    varchar2(100) path 'Policy/ProductType/@tc',
         policystatus    varchar2(100) path 'Policy/PolicyStatus/@tc',
         FaceAmt         varchar2(20)  path 'Policy/Life/FaceAmt',
         ModalPremAmt    varchar2(20)  path 'Policy/Life/Coverage/ModalPremAmt',
         riskclass       varchar2(20)  path 'Policy/Life/Coverage/CovOption/UnderwritingClass/@tc',
         Exchange1035Ind varchar2(10)  path 'Policy/Life/LifeUSA/OLifEExtension/LifeUSAExtension/Exchange1035Ind/@tc', 
         NYReg60Ind      varchar2(100) path 'Policy/OLifEExtension/PolicyExtension/NYReg60Ind/@tc'
     ) x1
   , xmltable(
       '/TXLife/TXLifeRequest/OLifE/Party[@id=$roid]'
       passing t.object_value, x.roid as "roid"
       columns
         firstname   varchar2(200) path 'Person/FirstName',
         LastName    varchar2(200) path 'Person/LastName',
         BirthDate   date          path 'Person/BirthDate',
         Gender      varchar2(10)  path 'Person/Gender/@tc',
         TobaccoInd  varchar2(5)   path 'Risk/TobaccoInd/@tc',
         Employmeeid varchar2(100) path 'Employment/EmployeeID',
         Title       varchar2(100) path 'Employment/Title',
         zip         varchar2(100) path 'Address/Zip'
     ) x2
OOID          ROID          INTERESTPERCENT        VPOLNO VPRODUCTTYPE  POLICYSTATUS   FACEAMT    MODALPREMAMT   RISKCLASS    EXCHANGE1035IND NYREG60IND   FIRSTNAME      LASTNAME      BIRTHDATE   GENDER     TOBACCOIND EMPLOYMEEID    TITLE            ZIP
Holding_1     Client_1      50                  308221234 1009800001    1              150000     500            1009800001   1               1            John           Doe           25/10/1960  1          0          12345678       Vice President   07310
Holding_3     Client_2                          308221235 1009800001    1              150000     500            1009800001   1               1            Jane           Doe           30/10/1970  2          0          98765432                        07310
The explain plan should show something like this ("streaming evaluation" instead of "collection iterator fetch") :
Execution Plan
Plan hash value: 201581907
| Id  | Operation             | Name    | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT      |         |  4451M|   551G|  1501M  (1)|999:59:59 |
|   1 |  NESTED LOOPS         |         |  4451M|   551G|  1501M  (1)|999:59:59 |
|   2 |   NESTED LOOPS        |         |    54M|  5976M|    18M  (1)| 61:17:09 |
|   3 |    NESTED LOOPS SEMI  |         |   667K|    61M|   222K  (1)| 00:44:34 |
|   4 |     NESTED LOOPS      |         |  8168 |   757K|    32   (0)| 00:00:01 |
|   5 |      TABLE ACCESS FULL| TMP_XML |     1 |    87 |     3   (0)| 00:00:01 |
|   6 |      XPATH EVALUATION |         |       |       |            |          |
|*  7 |     XPATH EVALUATION  |         |       |       |            |          |
|*  8 |    XPATH EVALUATION   |         |       |       |            |          |
|*  9 |   XPATH EVALUATION    |         |       |       |            |          |
Predicate Information (identified by operation id):
   7 - filter("P5"."C_01$"='32')
   8 - filter("P1"."C_01$"=CAST("P"."C_03$" AS varchar2(50)  ))
   9 - filter("P2"."C_01$"=CAST("P"."C_02$" AS varchar2(50)  ))How complex are the business rules you're implementing in the loop? Do you really need a FOR loop at all?
For instance, if you're just inserting into relational tables, then you don't have to loop.
Edited by: odie_63 on 19 mars 2012 15:23