Improve XML readability in Oracle 11g for binary XMLType storage for huge files

Question

I have one requirement in which I have to process huge XML files. That means there might be around 1000 xml files and the whole size of these files would be around 2GB.
What I need is to store all the data in these files to my Oracle DB. For this I have used sqlloader for bulk uploading of all my XML files to my DB and it is stored as binary XMLTYPE in my database.Now I need to query these files and store the data in relational tables.For this I have used XMLTable Xpath queries. Everything is fine when I try to query single xml file within my DB. But if it is trying to query all those files it is taking too much time which is not acceptable.
Here's my one sample xml content:
<ABCD>
  <EMPLOYEE id="11" date="25-Apr-1983">
    <NameDetails>
      <Name NameType="a">
        <NameValue>
          <FirstName>ABCD</FirstName>
          <Surname>PQR</Surname>
          <OriginalName>TEST1</OriginalName>
          <OriginalName>TEST2</OriginalName>
        </NameValue>
      </Name>
      <Name NameType="b">
        <NameValue>
          <FirstName>TEST3</FirstName>
          <Surname>TEST3</Surname>
        </NameValue>
        <NameValue>
          <FirstName>TEST5</FirstName>
          <MiddleName>TEST6</MiddleName>
          <Surname>TEST7</Surname>
          <OriginalName>JAB1</OriginalName>
        </NameValue>
        <NameValue>
          <FirstName>HER</FirstName>
          <MiddleName>HIS</MiddleName>
          <Surname>LOO</Surname>
        </NameValue>
      </Name>
      <Name NameType="c">
        <NameValue>
          <FirstName>CDS</FirstName>
          <MiddleName>DRE</MiddleName>
          <Surname>QWE</Surname>
        </NameValue>
        <NameValue>
          <FirstName>CCD</FirstName>
          <MiddleName>YTD</MiddleName>
          <Surname>QQA</Surname>
        </NameValue>
        <NameValue>
          <FirstName>DS</FirstName>
          <Surname>AzDFz</Surname>
        </NameValue>
      </Name>
    </NameDetails>
  </EMPLOYEE >
</ABCD>
Please note that this is just one small record inside one big xml.Each xml would contain similar records around 5000 in number.Similarly there are more than 400 files each ranging about 4MB size approx.
My xmltable query :
SELECT t.personid,n.nametypeid,t.titlehonorofic,t.firstname,
        t.middlename,
        t.surname,
        replace(replace(t.maidenname, '<MaidenName>'),'</MaidenName>', '#@#') maidenname,
        replace(replace(t.suffix, '<Suffix>'),'</Suffix>', '#@#') suffix,
        replace(replace(t.singleStringName, '<SingleStringName>'),'</SingleStringName>', '#@#') singleStringName,
        replace(replace(t.entityname, '<EntityName>'),'</EntityName>', '#@#') entityname,
        replace(replace(t.originalName, '<OriginalName>'),'</OriginalName>', '#@#') originalName
FROM xmlperson p,master_nametypes n,
         XMLTABLE (
          --'ABCD/EMPLOYEE/NameDetails/Name/NameValue'
          'for $i in ABCD/EMPLOYEE/NameDetails/Name/NameValue        
           return <row>
                    {$i/../../../@id}
                     {$i/../@NameType}
                     {$i/TitleHonorific}{$i/Suffix}{$i/SingleStringName}
                    {$i/FirstName}{$i/MiddleName}{$i/OriginalName}
                    {$i/Surname}{$i/MaidenName}{$i/EntityName}
                </row>'
        PASSING p.filecontent
        COLUMNS
                personid     NUMBER         PATH '@id',
                nametypeid   VARCHAR2(255)  PATH '@NameType',
                titlehonorofic VARCHAR2(4000) PATH 'TitleHonorific',
                 firstname    VARCHAR2(4000) PATH 'FirstName',
                 middlename  VARCHAR2(4000) PATH 'MiddleName',
                surname     VARCHAR2(4000) PATH 'Surname',
                 maidenname   XMLTYPE PATH 'MaidenName',
                 suffix XMLTYPE PATH 'Suffix',
                 singleStringName XMLTYPE PATH 'SingleStringName',
                 entityname XMLTYPE PATH 'EntityName',
                originalName XMLTYPE        PATH 'OriginalName'
                ) t where t.nametypeid = n.nametype and n.recordtype = 'Person'
But this is taking too much time to query all those huge data. The resultset of this query would return about millions of rows. I tried to index the table using this query :
CREATE INDEX myindex_xmlperson on xml_files(filecontent) indextype is xdb.xmlindex parameters ('paths(include(ABCD/EMPLOYEE//*))');
My Database version :
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
"CORE 11.2.0.2.0 Production"
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production
Index is created but still no improvement with the performance though. It is taking more than 20 minutes to query even a set of 10 similar xml files.Now you can imagine how much will it take to query all those 1000 xml files.
Could someone please suggest me how to improve the performance of my database.Since I am new to this I am not sure whether I am doing it in proper way. If there is a better solution please suggest. Your help will be greatly appreciated.

Answer 1

Hi Odie..
I tried to run your code through all the xml files but it is taking too much time. It has not ended even after 3hours.
When I tried to do a single insert select statement  for one single xml it is working.But stilli ts in the range of ~10sec.
Please find my execution plan for one single xml file with your code.
"PLAN_TABLE_OUTPUT"
"Plan hash value: 2771779566"
"| Id  | Operation                                     | Name                                     | Rows   | Bytes | Cost (%CPU)| Time     |"
"|   0 | INSERT STATEMENT                   |                                              |   499G |   121T |   434M  (2) |999:59:59  |"
"|   1 |  LOAD TABLE CONVENTIONAL    | WATCHLIST_NAMEDETAILS  |            |           |                 |                 |"
"|   2 |   SORT AGGREGATE                   |                                             |     1      |     2    |                 |          |"
"|   3 |    XPATH EVALUATION                 |                                             |             |          |                 |          |"
"|   4 |   SORT AGGREGATE                   |                                             |     1      |     2    |                 |          |"
"|   5 |    XPATH EVALUATION                 |                                             |             |          |                 |          |"
"|   6 |   SORT AGGREGATE                   |                                             |     1       |     2   |                 |          |"
"|   7 |    XPATH EVALUATION                 |                                             |              |         |                 |          |"
"|   8 |   SORT AGGREGATE                   |                                             |     1        |     2  |                 |          |"
"|   9 |    XPATH EVALUATION                 |                                             |              |         |                 |          |"
"|  10 |   NESTED LOOPS                       |                                             |   499G    | 121T |   434M (2) | 999:59:59 |"
"|  11 |    NESTED LOOPS                      |                                             |    61M     |  14G |  1222K (1) | 04:04:28 |"
"|  12 |     NESTED LOOPS                     |                                             | 44924      |  10M |    61   (2) | 00:00:01 |"
"|  13 |      MERGE JOIN CARTESIAN      |                                             |     5         | 1235 |     6   (0) | 00:00:01 |"
"|* 14 |       TABLE ACCESS FULL          | XMLPERSON                        |     1          |  221 |     2   (0) | 00:00:01 |"
"|  15 |       BUFFER SORT                     |                                             |     6          |  156 |     4   (0) | 00:00:01 |"
"|* 16 |        TABLE ACCESS FULL         | MASTER_NAMETYPES        |     6          |  156 |     3   (0) | 00:00:01 |"
"|  17 |      XPATH EVALUATION             |                                             |                |         |               |          |"
"|* 18 |     XPATH EVALUATION              |                                             |               |          |               |          |"
"|  19 |    XPATH EVALUATION               |                                              |               |         |              |          |"
"Predicate Information (identified by operation id):"
"  14 - filter(""P"".""FILENAME""='PFA2_95001_100000_F.xml')"
"  16 - filter(""N"".""RECORDTYPE""='Person')"
"  18 - filter(""N"".""NAMETYPE""=CAST(""P1"".""C_01$"" AS VARCHAR2(255) ))"
"Note"
"   - Unoptimized XML construct detected (enable XMLOptimizationCheck for more information)"
Please note that this is for a single xml file. I have like more than 400 similar files in the same table.
And for your's as well as Jason's Question:
What are you trying to accomplish with
replace(replace(t.originalName, '<OriginalName>'),'</OriginalName>', '#@#') originalName 
originalName XMLTYPE PATH 'OriginalName'
Like Jason, I also wonder what's the purpose of all those XMLType projections and strange replaces in the SELECT clause
What I was trying to achieve was to create a table containing separate rows for all the multi item child nodes for this particular xml.
But since there was an error beacuse of multiple child nodes like 'ORIGINALNAME' under 'NAMEVALUE' node, I tried this script to insert those values by providing a delimiter and replacing the tag names.
Please see the link for more details - http://stackoverflow.com/questions/16835323/construct-xmltype-query-to-store-data-in-oracle11g
This was the execution plan for one single xml file with my code :
Plan hash value: 2851325155
| Id  | Operation                                                     | Name                                         | Rows  | Bytes   | Cost (%CPU)  | Time       |    TQ  | IN-OUT | PQ Distrib |
|   0 | SELECT STATEMENT                                   |                                                 |  7487   |  1820K |    37   (3)        | 00:00:01 |           |             |            |
|*  1 |  HASH JOIN                                                 |                                                 |  7487   |  1820K  |    37   (3)        | 00:00:01 |           |             |            |
|*  2 |   TABLE ACCESS FULL                                | MASTER_NAMETYPES            |     6     |   156     |     3   (0)         | 00:00:01 |           |             |            |
|   3 |   NESTED LOOPS                                        |                                                 |  8168   |  1778K  |    33   (0)        | 00:00:01 |           |             |            |
|   4 |    PX COORDINATOR                                    |                                                 |            |             |                      |               |           |             |            |
|   5 |     PX SEND QC (RANDOM)                           | :TQ10000                                  |     1    |   221      |     2   (0)        | 00:00:01 |  Q1,00 | P->S     | QC (RAND)  |
|   6 |      PX BLOCK ITERATOR                              |                                                 |     1    |   221      |     2   (0)        | 00:00:01 |  Q1,00 | PCWC   |            |
|*  7 |       TABLE ACCESS FULL                            | XMLPERSON                            |     1    |   221      |     2   (0)        | 00:00:01 |  Q1,00 | PCWP   |            |
|   8 |    COLLECTION ITERATOR PICKLER FETCH  | XQSEQUENCEFROMXMLTYPE |  8168  | 16336    |    29   (0)       | 00:00:01  |           |               |            |
Predicate Information (identified by operation id):
   1 - access("N"."NAMETYPE"=CAST(SYS_XQ_UPKXML2SQL(SYS_XQEXVAL(SYS_XQEXTRACT(VALUE(KOKBF$),'/*/@NameType'),0,0,20971520,0),50,1,2
              ) AS VARCHAR2(255)  ))
   2 - filter("N"."RECORDTYPE"='Person')
   7 - filter("P"."FILENAME"='PFA2_95001_100000_F.xml')
Note
   - Unoptimized XML construct detected (enable XMLOptimizationCheck for more information)
Please let me know whether this has helped.
My intention is to save the details in the xml to different relational tables so that I can easily query it from my application. I have similarly many queries which inserts the xml values to different tables like the one which I have mentioned here. I was thinking of creating a stored procedure to insert all these values in the relational tables once I receive the xml files. But even a single query is taking too much time to complete. Could you please help me in this regard. Waiting for your valuable feedback.