Large XML and Query performance

This problem came to me from a Developer and she claims XML query on XMLType field is very slow when using large XML and is there any alternates. Details are below:
=============
Query:
select attributepool_id, attributepool_name, vintage , p.attributepool.extract('//attributepool/segmentationsystem/id/text()').getStringVal() ,
p.attributepool.extract('//attributepool/datasource/id/text()').getStringVal()
from saved_attributepools p
where user_id = 'CLPROFILE2' and vintage = 'SPRING_2003' order by attributepool_name
Table name:                saved_attributepools
Space:                    ecommerce
A Column Name:                attributepool
attributepool Column Type:      XmlType
One of xml contains 4Mbytes:     CORE LIFESTLY
When we try to get the data against this row, query is taking longer.
conn ecommerce@ecom3 --> 82 seconds (table has 65 rows)
conn ecommerce@oradev--> 34 seconds (table has only 4 rows)
We think that;
Oracle parse the entire XML document and load this document into an 'in-memory' DOM structure before executing the specified xpaths.
Adding INDEX into XmlType won't help as we don't use whereclasue against XmlType for this case.
We don't know 10g has solution for this or not.
Any suggestion will be greatly appreciated.

This problem came to me from a Developer and she claims XML query on XMLType field is very slow when using large XML and is there any alternates. Details are below:
=============
Query:
select attributepool_id, attributepool_name, vintage , p.attributepool.extract('//attributepool/segmentationsystem/id/text()').getStringVal() ,
p.attributepool.extract('//attributepool/datasource/id/text()').getStringVal()
from saved_attributepools p
where user_id = 'CLPROFILE2' and vintage = 'SPRING_2003' order by attributepool_name
Table name:                saved_attributepools
Space:                    ecommerce
A Column Name:                attributepool
attributepool Column Type:      XmlType
One of xml contains 4Mbytes:     CORE LIFESTLY
When we try to get the data against this row, query is taking longer.
conn ecommerce@ecom3 --> 82 seconds (table has 65 rows)
conn ecommerce@oradev--> 34 seconds (table has only 4 rows)
We think that;
Oracle parse the entire XML document and load this document into an 'in-memory' DOM structure before executing the specified xpaths.
Adding INDEX into XmlType won't help as we don't use whereclasue against XmlType for this case.
We don't know 10g has solution for this or not.
Any suggestion will be greatly appreciated.

Similar Messages

  • Large XML and DBMS_XMLSTORE

    Oracle DB = 10.2.0.3
    I have XML messages (size range 300KB - 15MB) that are inserted into a table with an XMLType column throughout the day.
    The volume will vary...say 10 transactions per minute. A single transaction can contain any number of entities/objects.
    I have no need to preserve the XML - aside from assigning it a sequence number, date etc. and storing it for debugging (if needed).
    Currently, I have a trigger defined on the table that calls various database packages
    that shred the data via XMLTable (i.e. insert into tab1...select...xmltable..).
    The problem is that I potentially need to insert into 50+ tables out of 200+ tables per transaction. So, I'm passing through the single instance document 200+ times.
    The number of tables that are needed to represent the XML will continue to grow.
    A 300KB XML message takes approx 10 seconds to shred. Obviously, I'm concerned about the larger messages.
         So, I was wondering if doing the following would be any better:
    1. Have a XSLT transform the original message into "pseudo" Oracle canonical format.
    At this point, all of the data contained in the message would need to be shredded and all unwanted data from the original message would be filtered..i.e. lowering the message size a little.
    The message to be inserted would then look something like this:
    <Transaction>
         <Table>
         <Name>TAB1</Name>
         <ROWSET>
              <ROW>
              <COL1>
              <COLn>
              </ROW>
    </ROWSET>
    </Table>
         <Table>
         <Name>TAB2</Name>
         <ROWSET>
              <ROW>
              <COL1>
              <COLn>
              </ROW>
    </ROWSET>
    </Table>
    </Transaction>
    2. Next, define a trigger on the table like this:
    CREATE OR REPLACE TRIGGER tr_ai_my_table
    AFTER INSERT
    ON my_table
    FOR EACH ROW
    DECLARE
    CURSOR table_cur
    IS
    SELECT a.table_nm
    ,a.xml_fragment
    FROM XMLTABLE ('/Transaction/Table[Name]'
    PASSING :NEW.xml_document
    COLUMNS
    table_nm VARCHAR2(30) PATH 'Name'
    ,xml_fragment XMLTYPE PATH 'ROWSET'
    ) a;
    v_context DBMS_XMLSTORE.ctxtype;
    v_rows NUMBER;
    -- XML Date format: 2007-08-02
    -- XML Datetime format: 2007-08-02T11:58:28.229-04:00
    -- ALTER SESSION SET NLS_DATE_FORMAT = 'YYYY-MM-DD';
    -- ALTER SESSION SET NLS_TIMESTAMP_FORMAT = 'YYYY-MM-DD-HH24.MI.SS.FF';
    -- ALTER SESSION SET NLS_TIMESTAMP_TZ_FORMAT = 'YYYY-MM-DD"T"HH24:MI:SS.FFTZH:TZM';
    BEGIN
    FOR table_rec IN table_cur
    LOOP
    v_context := DBMS_XMLSTORE.newcontext (table_rec.table_nm);
    v_rows := DBMS_XMLSTORE.insertxml (v_context, table_rec.xml_fragment);
    DBMS_XMLSTORE.closecontext (v_context);
    END LOOP;
    END;
         I know this will get me out of maintaining 200+ insert statements/xpaths. Also, I wouldn't be coding the XSLT either and the XSLT may end up running on some specialized hardware for optimized performance.
    But do I gain a significant performance boost using the Oracle canonical format
    and DBMS_XMLSTORE even though I'm still building the DOM tree to build the cursor ?

    i saw a problem with the xmlbeans implementation with wli 8.1
    we had memeory issues with large xmls messages as they seemded to be in memory even through the JPD had finished.
    to solve this we inserted code at the end of a process to assign the top level xmlbean document class to a tiny version of the same message type. this is valid when the document object is a top level variable of the jpd.
    this was done to stop xmlbeans caching the large message behind the scenes.

  • Import  xml and query

    Hi,
    Thanks in advance for reading this.
    I am new to this XML and import into database.
    I have the xml file, i need to import this into database and search on the database.
    I have around 2.2 million records in the database.
    Please explain me what is best method of storing in database, like store the file in clob column and search on the clob column.How the search is going to be on 2.2 million records.
    Please send me any if any sample code you have to [email protected]

    The answer on this question also depends on the version of the database you are using.
    With 9i/10g i would probably use structured storage of XML (based on the XMLschema).
    But the answer also depends on another question: does the XML document contains "2.2 million records" or will you be storing 2.2 million XML documents?

  • Xml select query performance

    the table i have
    create table test of xmltype;
    and the xml that i have loaded is
    <root>
         <company>
    <department>
    <id>10</id>
    <name>Accounting</name>
    </department>
    <department>
    <id>11</id>
    <name>Billing</name>
    </department>
    </company>
         </root>
    select query using xmltable is
    select id,name from test,xmltable('/root/company/department'
    passing object_value
    columns
    id number path 'id',
    name varchar2(20) 'name');
    the query is working fine but issue is performance
    i have imlplemented index using extract() and extractstringval() functions but as i have multiple
    occurance of data ther two are not working. I have non-schema-based xmltype table.
    I need help for creating index on multiple occurance element
    Any help is appreciated

    First of all "XMLOptimizationCheck" AFAIK is not yet explained. Haven't checked support.oracle.com for a while though.
    It's more or less currently an internal used, but for the public a fast method to detect, that Oracle internal XQuery / XPath optimization rewrites towards SQL methods (shortcuts) are not properly working. SYS_XQEXVAL probably means something like XQuery Element XML Value/validation (??? towards SQL value) isn't producing a simple construct with a predicate validation. The reasons section gives insight, just like a 10053 trace, on what attempts/rules where applied and failed or worked. I am guessing that the overall cost for the use of the normal PK index is so high because it can not be properly matched and/or optimized against the global index structure supporting the partitions.
    In all, a bit more info regarding the table/partition structure and its used index regime/structure would be helpful.
    Beside that. THIS IS A BUG and should be reported, request for help, via support.oracle.com
    Edited by: Marco Gralike on Mar 23, 2011 9:35 PM

  • Compression and query performance in data warehouses

    Hi,
    Using Oracle 11.2.0.3 have a large fact table with bitmap indexes to the asscoiated dimensions.
    Understand bitmap indexes are compressed by default so assume cannot further compress them.
    Is this correct?
    Wish to try compress the large fact table to see if this will reduce the i/o on reads and therfore give performance benefits.
    ETL speed fine just want to increase the report performance.
    Thoughts - anyone seen significant gains in data warehouse report performance with compression.
    Also, current PCTFREE on table 10%.
    As only insert into tabel considering making this 1% to imporve report performance.
    Thoughts?
    Thanks

    First of all:
    Table Compression and Bitmap Indexes
    To use table compression on partitioned tables with bitmap indexes, you must do the following before you introduce the compression attribute for the first time:
    Mark bitmap indexes unusable.
    Set the compression attribute.
    Rebuild the indexes.
    The first time you make a compressed partition part of an existing, fully uncompressed partitioned table, you must either drop all existing bitmap indexes or mark them UNUSABLE before adding a compressed partition. This must be done irrespective of whether any partition contains any data. It is also independent of the operation that causes one or more compressed partitions to become part of the table. This does not apply to a partitioned table having B-tree indexes only.
    This rebuilding of the bitmap index structures is necessary to accommodate the potentially higher number of rows stored for each data block with table compression enabled. Enabling table compression must be done only for the first time. All subsequent operations, whether they affect compressed or uncompressed partitions, or change the compression attribute, behave identically for uncompressed, partially compressed, or fully compressed partitioned tables.
    To avoid the recreation of any bitmap index structure, Oracle recommends creating every partitioned table with at least one compressed partition whenever you plan to partially or fully compress the partitioned table in the future. This compressed partition can stay empty or even can be dropped after the partition table creation.
    Having a partitioned table with compressed partitions can lead to slightly larger bitmap index structures for the uncompressed partitions. The bitmap index structures for the compressed partitions, however, are usually smaller than the appropriate bitmap index structure before table compression. This highly depends on the achieved compression rates.

  • How to make use of XMLDB to process large XML and create spatial objects

    For someone new to XMLDB I find it hard to make sense of the enormous amount of information (and easy to get lost in it). So I come here to ask for directions.
    I have to build a procedure that fills a table of spatial objects based on XML input. Basically the XML contains a large amount of elements that describe the geometry type and contain the geometries coordinates. The XML can get quite large (200-300Mb).
    Somehow I have to process each element and create an sdo_geometry object.
    Now let me ask a very broad question: What would be a good way to handle this?

    I have the same question. Any news on this?
    Wijnand

  • Distinct clause and query performance

    Friends,
    I have a query which returns results in 40 seconds without distinct clause and when I add distinct clause it takes over 2 hours.
    I have verified following -
    1. indexes/table statistics are up to date.
    2. columns that are used in where clause but are not indexed have upto date column statistics
    Any idea what could be the reason, explain plan shows that distinct clause has a very expensive cost.
    Thanks
    Query and explain plan is below
    SELECT
    DISTINCT -- with distinct 2hrs + and without 40 seconds
    quote_by_dst.qte_hdr_stat_cd, quote_by_dst.qte_ln_cond_cd,
                    product.prod_nm, product.prod_id,
                    cs_ship_by_dst.bto_ds_cac_ownr_ud,
                    quote_by_dst.qte_csup_csup_am, cs_ship_by_dst.bto_ds_cac_nm,
                    product.spl_sht_nm,
                       product.prod_blg_un_fac_um
                    || ' '
                    || product.prod_blg_um
                    || ' '
                    || product.prod_stck_um,
                    product.prod_blg_um, quote_by_dst.qte_ln_brk_1_blg_uom_am,
                    quote_by_dst.qte_csup_avg_cst_am,
                    quote_by_dst.qte_csup_rev_gm_pct_am,
                    quote_by_dst.qte_csup_avg_cst_am, cs_ship_by_dst.bto_id,
                    cs_ship_by_dst.bto_ds_cac_cd,
                       cs_ship_by_dst.bto_ds_cac_cd
                    || product.prod_id
                    || cs_ship_by_dst.bto_id
               FROM infowhse.quote_by_dst4 quote_by_dst,
                    infowhse.product,
                    infowhse.cs_ship_by_dst4 cs_ship_by_dst,
                    infowhse.department
              WHERE (quote_by_dst.dpt_cd = department.dpt_cd)
                AND (quote_by_dst.cus_dpt_id = cs_ship_by_dst.cus_dpt_id)
                AND (product.prod_id = quote_by_dst.prod_id)
                AND (    (   quote_by_dst.qte_ln_cond_cd = 'E'
                          OR quote_by_dst.qte_ln_cond_cd = 'C'
                     AND quote_by_dst.qte_hdr_stat_cd = 'A'
                     AND ((cs_ship_by_dst.bto_cust_type_cd) = '01')
                     AND cs_ship_by_dst.bto_ds_cac_ownr_ud = 'EHOC'
                     AND department.dpt_cd > '0.00'
                    )Explain plan
    Plan
    SELECT STATEMENT  CHOOSECost: 911,832,256  Bytes: 433,941,639,459  Cardinality: 2,729,192,701                                               
         15 SORT UNIQUE  Cost: 911,832,256  Bytes: 433,941,639,459  Cardinality: 2,729,192,701                                          
              14 NESTED LOOPS  Cost: 68,705  Bytes: 433,941,639,459  Cardinality: 2,729,192,701                                     
                   12 HASH JOIN  Cost: 68,705  Bytes: 425,754,061,356  Cardinality: 2,729,192,701                                
                        1 INDEX FAST FULL SCAN NON-UNIQUE INFOWHSE.DST_SEC_DST_SEC_DST_CD_IX Cost: 25  Bytes: 922,700  Cardinality: 184,540                           
                        11 HASH JOIN  Cost: 16,179  Bytes: 1,199,209,082  Cardinality: 7,941,782                           
                             2 INDEX FAST FULL SCAN NON-UNIQUE INFOWHSE.DST_SEC_DST_SEC_DST_CD_IX Cost: 25  Bytes: 922,700  Cardinality: 184,540                      
                             10 HASH JOIN  Cost: 15,879  Bytes: 3,374,060  Cardinality: 23,110                      
                                  8 HASH JOIN  Cost: 15,200  Bytes: 2,981,190  Cardinality: 23,110                 
                                       6 HASH JOIN  Cost: 13,113  Bytes: 1,779,470  Cardinality: 23,110            
                                            3 TABLE ACCESS FULL INFOWHSE.CUSTOMER_SHIP Cost: 5,640  Bytes: 42,372  Cardinality: 1,177       
                                            5 PARTITION RANGE ALL  Partition #: 11  Partitions accessed #1 - #12     
                                                 4 TABLE ACCESS FULL INFOWHSE.QUOTE Cost: 7,328  Bytes: 38,826,590  Cardinality: 946,990  Partition #: 11  Partitions accessed #1 - #12
                                       7 TABLE ACCESS FULL INFOWHSE.PRODUCT Cost: 1,542  Bytes: 9,246,640  Cardinality: 177,820            
                                  9 INDEX FAST FULL SCAN NON-UNIQUE INFOWHSE.CUST_SHIP_SLSDST_DTP_SICALL_IX Cost: 185  Bytes: 9,878,411  Cardinality: 581,083                 
                   13 INDEX UNIQUE SCAN UNIQUE INFOWHSE.DEPARTMENT_PK Bytes: 3  Cardinality: 1                                

    This might be more useful.
    Query is still running.
    There is heavy wait time for scattered file read.
    Results from
    SELECT * FROM V$SESSION_WAIT WHERE SID = 48;
    SID   SEQ#  EVENT                           P1TEXT                          P1    P1RAW            P2TEXT                          P2    P2RAW            P3TEXT                          P3    P3RAW            WAIT_TIME                              SECONDS_IN_WAIT                        STATE              
    48    6865  db file scattered read          file#                           108   000000000000006C block#                          1593370000000000026E69 blocks                          32    0000000000000020 2                                      30                                      WAITED KNOWN TIME  
    SELECT * FROM V$SESSION_EVENT WHERE SID = 48;
    SID                                    EVENT                                                            TOTAL_WAITS                            TOTAL_TIMEOUTS                         TIME_WAITED                            AVERAGE_WAIT                           MAX_WAIT                               TIME_WAITED_MICRO                     
    48                                     log file sync                                                    1                                      0                                      0                                      0                                      0                                      563                                   
    48                                     db file sequential read                                          11                                     0                                      0                                      0                                      0                                      243                                   
    48                                     db file scattered read                                           6820                                   0                                      330                                    0                                      7                                      3296557                               
    48                                     SQL*Net message to client                                        19                                     0                                      0                                      0                                      0                                      23                                    
    48                                     SQL*Net message from client                                      18                                     0                                      128                                    7                                      127                                    1281912                                Sorry for long post.

  • Large XML document performance

    We are using XDB 9.2.0.4. I am seeing a severe performance degradation when attempting to extract larger XML documents from XDB (somewhere over 3M). Smaller documents appear to be working fine.
    I have been reading in the forum that the problem I am running into is most likely related to the storage model being used. ie) There are several repeating elements within the schema.
    I have added xdb:storeVarrayAsTable="true" statement to the schema and re-registered. I can see, based on user_nested_tables, that XDB appears to be storing the repeating elements as nested tables vs varrays.
    The change to the storage model does not seem to have significantly changed the queries performance.
    The schemas I am using can be found at http://www.sasked.gov.sk.ca/xsd/sl/1.x/SLMessage.xsd & http://www.sasked.gov.sk.ca/xsd/sl/1.x/SDSElements.xsd
    The schema documentation can be found at http://www.sasked.gov.sk.ca/sds/xml/SchemaDocumentation/SLMessage.html
    The element /SL_Message/SL_Event/SL_ObjectData/SL_EventObject is the primary repeating element
    I am using a table with an XMLType column
    CREATE TABLE XML_SL_MESSAGE
    (XML_SL_MESSAGE_ID NUMBER(11) NOT NULL
    ,DTE_TIMESTAMP TIMESTAMP DEFAULT SYSTIMESTAMP NOT NULL
    ,ORIGINAL_XML_SL_MESSAGE_ID NUMBER(11)
    ,VALID_SL_MESSAGE_XML sys.XMLType
    ,INVALID_XML CLOB
    ,ERROR_MESSAGE VARCHAR2(4000)
    ) xmltype column valid_sl_message_xml XMLSCHEMA "http://www.sasked.gov.sk.ca/xsd/sl/1.x/SLMessage.xsd" element "SL_Message"
    The SQL I am using is attempting to bring the XMLType back as a clob, the query seems to be intensive in both CPU and I/O. (looks like it is the getClobVal function)
    select xsm.xml_sl_message_id
    ,xsm.dte_timestamp
    ,nvl(xsm.valid_sl_message_xml.getClobVal(),xsm.invalid_xml) as xml_clob
    ,xsm.error_message
    ,xsm.original_xml_sl_message_id
    from xml_sl_message xsm
    where xsm.dte_timestamp &gt; sysdate –1
    I guess what I am wondering is what are my options ? Changing the storage model ? Applying Indexes ?
    On an unrelated topic, Are there many differences in XDB 9.2.0.5 and 9.2.0.4 ? (I don’t believe 10g will be an option here … yet)
    Thanx in advance
    Trent

    I have applied the 9.2.0.5.0 patches and created the relational table with the following attributes:
    CREATE TABLE XML_SL_MESSAGE
    (XML_SL_MESSAGE_ID NUMBER(11) NOT NULL
    ,DTE_TIMESTAMP TIMESTAMP DEFAULT SYSTIMESTAMP NOT NULL
    ,PSE_SYS_USR_ID NUMBER(11) NOT NULL
    ,ORIGINAL_XML_SL_MESSAGE_ID NUMBER(11)
    ,VALID_SL_MESSAGE_XML sys.XMLType
    ,INVALID_XML CLOB
    ,ERROR_MESSAGE VARCHAR2(4000)
    xmltype column valid_sl_message_xml
    STORE AS OBJECT RELATIONAL
    XMLSCHEMA "http://www.sasked.gov.sk.ca/xsd/sl/1.x/SLMessage.xsd" ELEMENT "SL_Message"
    -- 1:1 ----------------------
    varray valid_sl_message_xml."XMLDATA"."SL_Event"."SL_ObjectData"."SL_EventObject"
    store as table SL_EVENTOBJECT2_TB(
    (constraint SL_EVENTOBJECT2_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 2:2 ----------------------
    varray "SchoolTerm"
    store as table SCHOOLTERM2_TB(
    (constraint SCHOOLTERM2_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 3:2 ----------------------
    varray "SchoolClass"
    store as table SCHOOLCLASS3_TB(
    (constraint SCHOOLCLASS3_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 6:2 ----------------------
    varray "StudentCourseHistory"
    store as table STUDENTCOURSEHISTORY6_TB(
    (constraint STUDENTCOURSEHISTORY6_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 7:2 ----------------------
    varray "StudentSupplementalMark"
    store as table STUDENTSUPPLEMENTALMARK7_TB(
    (constraint STUDENTSUPPLEMENTALMARK7_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 8:2 ----------------------
    varray "StudentClassMark"
    store as table STUDENTCLASSMARK8_TB(
    (constraint STUDENTCLASSMARK8_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 9:2 ----------------------
    varray "StudentExamRegistration"
    store as table STUDENTEXAMREGISTRATION9_TB(
    (constraint STUDENTEXAMREGISTRATION9_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 10:2 ----------------------
    varray "StudentClassEnrollment"
    store as table STUDENTCLASSENROLLMENT10_TB(
    (constraint STUDENTCLASSENROLLMENT10_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 11:2 ----------------------
    varray "StudentPersonal"
    store as table STUDENTPERSONAL11_TB(
    (constraint STUDENTPERSONAL11_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 18:2 ----------------------
    varray "StudentProgramEnrollment"
    store as table STUDENTPROGRAMENROLLMENT18_TB(
    (constraint STUDENTPROGRAMENROLLMENT18_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 19:2 ----------------------
    varray "StudentSchoolEnrollment"
    store as table STUDENTSCHOOLENROLLMENT19_TB(
    (constraint STUDENTSCHOOLENROLLMENT19_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 26:1 ----------------------
    varray valid_sl_message_xml."XMLDATA"."SL_Response"."SL_ObjectData"."SL_EventObject"
    store as table SL_EVENTOBJECT26_TB(
    (constraint SL_EVENTOBJECT26_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 27:2 ----------------------
    varray "SchoolTerm"
    store as table SCHOOLTERM27_TB(
    (constraint SCHOOLTERM27_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 28:2 ----------------------
    varray "SchoolClass"
    store as table SCHOOLCLASS28_TB(
    (constraint SCHOOLCLASS28_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 31:2 ----------------------
    varray "StudentProgramEnrollment"
    store as table STUDENTPROGRAMENROLLMENT31_TB(
    (constraint STUDENTPROGRAMENROLLMENT31_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 32:2 ----------------------
    varray "StudentExamRegistration"
    store as table STUDENTEXAMREGISTRATION32_TB(
    (constraint STUDENTEXAMREGISTRATION32_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 33:2 ----------------------
    varray "StudentClassEnrollment"
    store as table STUDENTCLASSENROLLMENT33_TB(
    (constraint STUDENTCLASSENROLLMENT33_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 34:2 ----------------------
    varray "StudentPersonal"
    store as table STUDENTPERSONAL34_TB(
    (constraint STUDENTPERSONAL34_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 41:2 ----------------------
    varray "StudentSchoolEnrollment"
    store as table STUDENTSCHOOLENROLLMENT41_TB(
    (constraint STUDENTSCHOOLENROLLMENT41_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 48:2 ----------------------
    varray "StudentClassMark"
    store as table STUDENTCLASSMARK48_TB(
    (constraint STUDENTCLASSMARK48_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 49:2 ----------------------
    varray "StudentCourseHistory"
    store as table STUDENTCOURSEHISTORY49_TB(
    (constraint STUDENTCOURSEHISTORY49_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 50:2 ----------------------
    varray "StudentSupplementalMark"
    store as table STUDENTSUPPLEMENTALMARK50_TB(
    (constraint STUDENTSUPPLEMENTALMARK50_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 51:1 ----------------------
    varray valid_sl_message_xml."XMLDATA"."SL_Response"."SL_Ack"."SL_Error"
    store as table SL_ERROR51_TB(
    (constraint SL_ERROR51_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    -- 52:1 ----------------------
    varray valid_sl_message_xml."XMLDATA"."SL_Request"."SL_Query"."SL_QueryObject"
    store as table SL_QUERYOBJECT52_TB(
    (constraint SL_QUERYOBJECT52_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
    tablespace data
    ALTER TABLE XML_SL_MESSAGE
    ADD (CONSTRAINT XML_SL_MESSAGE_PK PRIMARY KEY
    (XML_SL_MESSAGE_ID))
    ALTER TABLE XML_SL_MESSAGE
    ADD (CONSTRAINT XMLSLMSG_ORIGINAL_XMLSLMSG_UK UNIQUE
    (ORIGINAL_XML_SL_MESSAGE_ID))
    ALTER TABLE XML_SL_MESSAGE ADD (CONSTRAINT
    XMLSLMSG_SYSUSR_FK FOREIGN KEY
    (PSE_SYS_USR_ID) REFERENCES PSE_SYS_USR
    (PSE_SYS_USR_ID))
    ALTER TABLE XML_SL_MESSAGE ADD (CONSTRAINT
    XMLSLMSG_ORIGINAL_XMLSLMSG_FK FOREIGN KEY
    (ORIGINAL_XML_SL_MESSAGE_ID) REFERENCES XML_SL_MESSAGE
    (XML_SL_MESSAGE_ID))
    -- Create a unique index for the XML Message id
    CREATE UNIQUE INDEX XMLSLMSG_MSGID_UNIQUE ON XML_SL_MESSAGE
    ((substr(extractValue(valid_sl_message_xml,'//SL_MsgId'),1,255)))
    tablespace indx
    COMPUTE STATISTICS
    Here is the nested table structure of the XMLType table created during the schema registration:
    select level
    ,parent_table_column
    from user_nested_tables
    connect by prior table_name = parent_table_name
    start with parent_table_name = 'SL_Message4724_TAB'
    LEVEL PARENT_TABLE_COLUMN
    1 "XMLDATA"."SL_Event"."SL_ObjectData"."SL_EventObject"
    2 SchoolTerm
    2 StudentSchoolEnrollment
    3 "StudentInfo"."Name"
    3 "StudentInfo"."Demographics"."CountryOfCitizenship"
    3 "StudentInfo"."StudentAddress"
    3 "StudentInfo"."PhoneNumber"
    3 "StudentInfo"."Demographics"."Language"
    3 "StudentInfo"."Email"
    2 SchoolClass
    3 "ClassInfo"."DeptAssignedCourseId"
    3 "ClassInfo"."EducatorCertificateNumber"
    2 StudentProgramEnrollment
    2 StudentClassEnrollment
    2 StudentClassMark
    2 StudentCourseHistory
    2 StudentSupplementalMark
    2 StudentExamRegistration
    2 StudentPersonal
    3 "StudentInfo"."Name"
    3 "StudentInfo"."Email"
    3 "StudentInfo"."Demographics"."CountryOfCitizenship"
    3 "StudentInfo"."StudentAddress"
    3 "StudentInfo"."PhoneNumber"
    3 "StudentInfo"."Demographics"."Language"
    1 "XMLDATA"."SL_Request"."SL_Query"."SL_QueryObject"
    1 "XMLDATA"."SL_Response"."SL_ObjectData"."SL_EventObject"
    2 SchoolTerm
    2 SchoolClass
    3 "ClassInfo"."DeptAssignedCourseId"
    3 "ClassInfo"."EducatorCertificateNumber"
    2 StudentProgramEnrollment
    2 StudentClassEnrollment
    2 StudentClassMark
    2 StudentCourseHistory
    2 StudentSupplementalMark
    2 StudentExamRegistration
    2 StudentPersonal
    3 "StudentInfo"."Name"
    3 "StudentInfo"."Email"
    3 "StudentInfo"."Demographics"."Language"
    3 "StudentInfo"."PhoneNumber"
    3 "StudentInfo"."StudentAddress"
    3 "StudentInfo"."Demographics"."CountryOfCitizenship"
    2 StudentSchoolEnrollment
    3 "StudentInfo"."Name"
    3 "StudentInfo"."Demographics"."Language"
    3 "StudentInfo"."PhoneNumber"
    3 "StudentInfo"."StudentAddress"
    3 "StudentInfo"."Demographics"."CountryOfCitizenship"
    3 "StudentInfo"."Email"
    1 "XMLDATA"."SL_Response"."SL_Ack"."SL_Error"
    52 rows selected.
    When I attempt to insert to previously valid XML documents I get a core dump … Here are the insert statements:
    insert into xml_sl_message (
    xml_sl_message_id, dte_timestamp, pse_sys_usr_id, original_xml_sl_message_id, valid_sl_message_xml, invalid_xml,error_message
    select xml_sl_message_id, dte_timestamp, pse_sys_usr_id, original_xml_sl_message_id, xmltype(valid_sl_message_clob), invalid_xml,error_message
    from xml_sl_message_temp where xml_sl_message_id in (5154,5155)
    Here are the details on the exception:
    Oracle9i Enterprise Edition Release 9.2.0.5.0 - 64bit Production
    With the Partitioning, OLAP and Oracle Data Mining options
    JServer Release 9.2.0.5.0 - Production
    ORACLE_HOME = /opt/app/oracle/product/9.2.0
    System name: SunOS
    Node name: *****
    Release: 5.8
    Version: Generic_117000-01
    Machine: sun4u
    Instance name: EDDSDS
    Redo thread mounted by this instance: 1
    Oracle process number: 76
    Unix process pid: 11460, image: oracle@***** (TNS V1-V3)
    *** 2004-06-22 09:24:34.603
    *** SESSION ID:(29.159) 2004-06-22 09:24:34.602
    Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x20, PC: [0x101e41830, 0000000101E41830]
    *** 2004-06-22 09:24:34.606
    ksedmp: internal or fatal error
    ORA-07445: exception encountered: core dump [0000000101E41830] [SIGSEGV] [Address not mapped to object] [0x000000020] [] []
    Current SQL statement for this session:
    insert into xml_sl_message (
    xml_sl_message_id, dte_timestamp, pse_sys_usr_id, original_xml_sl_message_id, valid_sl_message_xml, invalid_xml,error_message
    select xml_sl_message_id, dte_timestamp, pse_sys_usr_id, original_xml_sl_message_id, xmltype(valid_sl_message_clob), invalid_xml,error_message
    from xml_sl_message_temp where xml_sl_message_id in (5154,5155)
    I have tried a couple different storage sections (with different levels of nesting) and still the same problem ..
    Is there something wrong with my storage section ?
    What is the "return as LOCATOR" clause for?

  • Query performance - A single large document VS multiple small documents

    Hi all,
    What are the performance trade offs when using a single large document VS multiple small documents ?
    I want to store xml snippets with similar structure in a container. Is there any benefit while querying, if I use a single large document to store all these snippets against adding each snippet as a different document. Would it degrade the performance when adding an xml snippet each time by modifying an existing document?
    How could we decide whether to use a single large document VS multiple small documents?
    Thanks,
    Anoop

    Hello Anoop,
    In case you wanted to get a comparison between the storage types for containers, wholedoc and node, let us know.
    What are the performance trade offs when using a
    single large document VS multiple small documents ?Depends on what is more important to you, performance when creating the container, and inserting the document(s) or performance when retrieving data.
    For querying the best option is to go with smaller documents, as node indexes would help in improving query performance.
    For inserting initial data, you can construct your large document composed of smaller xml snippets and insert the document as a whole.
    If you further want to modify this document changing its structure implies performance penalties; more indicated is to store the xml snippets as documents.
    Overall, I see no point in using a large document that will hold all of your xml snippets, so I strongly recommend going with multiple smaller documents.
    Regards,
    Andrei Costache
    Oracle Support Services

  • MII Performance with Large XML

    HI,
    We are facing performance issue while parsing XML of large size of around 50MB.
    Initially XMII was crashing due to "out of heap memory error". However, with some changes the problem has been removed.But
    now, we are facing an issue with time taken by the Payload to execute. Its taking more than half-an-hour to execute the transaction.
    The solution tried so far has decreased the time to half-an-hour. earlier it was taking more than one and half hour to process.
    We have tried parallel processing by asynchronous call to the transactions
    Is there any way to further reduce the time of processing?
    Further, is it possible to find out which parser is used by MII internally for XML SAX or DOM Parser.
    Thanks
    Amit

    Hi Amit,
    Just some tips to improve performance of BLS.
    1. Use Xpath whenever possible.
    2. Remove unnecessary repeaters performing on BLS.
    3. Check the Like to Optimizing BLS Performance.
    Optimizing BLS Performance for XML Handling in SAP MII
    If you are storing the data in database. just pass whole xml to query and insert data using bulk insert.
    Thanks
    Anshul

  • Performance Problem in parsing large XML file (15MB)

    Hi,
    I'm trying to parse a large XML file(15 MB) and facing a clear performance problem. A Simple XML Validation using the following code snippet:
    DBMS_LOB.fileopen(targetFile, DBMS_LOB.file_readonly);
    DBMS_LOB.loadClobfromFile
    tempCLOB,
    targetFile,
    DBMS_LOB.getLength(targetFile),
    dest_offset,
    src_offset,
    nls_charset_id(CONSTANT_CHARSET),
    lang_context,
    conv_warning
    DBMS_LOB.fileclose(targetFile);
    p_xml_document := XMLType(tempCLOB, p_schema_url, 0, 0);
    p_xml_document.schemaValidate();
    is taking 30 mins on a HP-UX (4GB ram, 2 CPU) machine (Oracle version : 9.2.0.4).
    Please explain what could be going wrong.
    Thanks In Advance,
    Vineet

    Thanks Mark,
    I'll open a TAR and also upload the schema and instance XML.
    If i'm not changing the track too much :-) one more thing in continuation:
    If i skip the Schema Validation step and directly insert the instance document into a Schema linked XMLType table, what does OracleXDB do in such a case?
    i'm getting a severe performance hit here too... the same file as above takes almost 40 mins to Insert.
    code snippet:
    DBMS_LOB.fileopen(targetFile, DBMS_LOB.file_readonly);
    DBMS_LOB.loadClobfromFile
    tempCLOB,
    targetFile,
    DBMS_LOB.getLength(targetFile),
    dest_offset,
    src_offset,
    nls_charset_id(CONSTANT_CHARSET),
    lang_context,
    conv_warning
    DBMS_LOB.fileclose(targetFile);
    p_xml_document := XMLType(tempCLOB, p_schema_url, 0, 0);
    -- p_xml_document.schemaValidate();
    insert into INCOMING_XML values(p_xml_document);
    Here table INCOMING_XML is :
    TABLE of SYS.XMLTYPE(XMLSchema "http://INCOMING_XML.xsd" Element "MatchingResponse") STORAGE Object-
    relational TYPE "XDBTYPE_MATCHING_RESPONSE"
    This table and type XDBTYPE_MATCHING_RESPONSE were created using the mapping provided in the registered XML Schema.
    Thanks,
    Vineet

  • Query in a large xml file

    Hello,
    I'm trying to work with very large xml files which are created from csv files. These files may be very large - up to 1 GB ! Untill now I have managed to do several validations on these big xml files, and the only thing that works for me is SAX parser, DOM is out of the question because it fills up memory.
    My next task is to do queries on these files, smth like:
    select field1,field2 from file.xml
    where field3 = 'A'
    and (fileld4>'B' or field1='C')
    order by field2.
    I searched the net about finding out how to make queries on xml files (since I have never done queries on xml before), but I couldn't find which "query language" is best for large files. If I use XPath (XSLT) will that not cause me memory problems because XSLT represents the file as a memory object?
    My idea is to parse the file with SAX and check every row if it fits the where condition and then write it immediately to a result xml file. But validating the where statement can be very complicated without using some tool. Also the order by statement is another problematic issue.
    Does anyone have some more intelegent ideas about how I can do this? Please help! :(
    The xml file looks like this:
    <doc>
    <row id ="1">
    <column id="1" name="column1">value</column>
    <column id="N" name="columnN">value</column>
    </row>
    <row id ="M">
    <column id="1" name="column1">value</column>
    <column id="N" name="columnN">value</column>
    </row>
    </doc>

    Hi all,
    Thank you very much for your replies.
    First, saxon didn't work because it uses an in-memory parser, and that is what I was trying to avoid.
    Different database is also out of the question, because the customer insist on XML, and also there are some files that can never be converted to a database table, because eventually with some transformations thay are changed and are not completely like the standard csv format.
    I think that maybe http://exist.sourceforge.net is the rigth solution for me, but I will probably try it in the next version of my project.
    For now I have managed to make the project with only SAXParser and a lot of back - end programming and it works ok, althoug it was very hard to make it, and will be harded to maintain, so I will try to look at the eXist project.
    Thanks everyone for the help.

  • How to improve Query performance on large table in MS SQL Server 2008 R2

    I have a table with 20 million records. What is the best option to improve query performance on this table. Is partitioning the table into filegroups  is a best option or splitting the table into multiple smaller tables? 

    Hi bala197164,
    First, I want to inform that both to partition the table into filegroups and split the table into multiple smaller tables can improve the table query performance, and they are fit for different situation. For example, our table have one hundred columns and
    some columns are not related to this table object directly (for example, there is a table named userinfo to store user information, it has columns address_street, address_zip,address_ province columns, at this time, we can create a new table named as Address,
    and add a foreign key in userinfo table references Address table), under this situation, by splitting a large table into smaller, individual tables, queries that access only a fraction of the data can run faster because there is less data to scan. Another
    situation is our table records can be grouped easily, for example, there is a column named year to store information about product release date, at this time, we can partition the table into filegroups to improve the query performance. Usually, we perform
    both of methods together. Additionally, we can add index to table to improve the query performance. For more detail information, please refer to the following document:
    Partitioning:
    http://msdn.microsoft.com/en-us/library/ms178148.aspx
    CREATE INDEX (Transact-SQL):
    http://msdn.microsoft.com/en-us/library/ms188783.aspx
    TechNet
    Subscriber Support 
    If you are
    TechNet Subscription user and have any feedback on our support quality, please send your feedback
    here.
    Allen Li
    TechNet Community Support

  • Performance problem - XML and Hashtable

    Hello!
    I have a strange (or not strange, we don't know) performance problem. The situation is simple: our application runs on Oracle IAS, and gets XML messages from an MQ, and we have to process them.
    Sample message:
    <RequestDeal RequestId="RequestDeal-20090209200010-998493885" CurrentFragment="0" EndFragment="51" Expires="2009-03-7T08:00:10">
         <Deal>
              <Id>385011</Id>
              <ModifyDate>2009-02-09</ModifyDate>
              <PastDuesOnly>false</PastDuesOnly>
         </Deal>
         <Deal>
              <Id>385015</Id>
              <ModifyDate>2009-02-09</ModifyDate>
              <PastDuesOnly>false</PastDuesOnly>
         </Deal>
    </RequestDeal>(there's an average of 50000-80000 deals in the whole message)
    After the application receives the whole message, we'd like to parse it, and put the values into a hashtable for further processing:
        Hashtable ht = new Hashtable();
        Node eDeals = getElementNode(docReq, "/RequestDeal");
        Node eDeal = (Element)(eDeals.getFirstChild());
        long start = System.currentTimeMillis();
        while (eDeal != null) {
          String id = getElementText((Element)eDeal, "Id");
          Date modifyDate = getElementDateValue((Element)eDeal, "ModifyDate");
          boolean szukitett = getElementBool((Element)eDeal, "PastDuesOnly", false);
          ht.put(id, new UgyletStatusz(id, modifyDate, szukitett));
          eDeal = eDeal.getNextSibling();
        logger.info("Request adatok betöltve.");
        long end = System.currentTimeMillis();
        logger.info("Hosszu muvelet ideje: " + (end - start) + "ms");The problem is the while statement. On my PC it runs for 15-25 seconds, depends on the number of the deals; but on our customer's server, it runs for 2-5 hours with very high CPU load. The application uses oracle xml parser.
    On my PC there are Websphere MQ 5.3 and Oracle 10ias with 1.5 JVM.
    On our customers's server there are Websphere MQ 5.3 and Oracle 9ias with 1.4 HotSpot JVM.
    Do you have any idea, what can be the cause of the problem?

    gyulaur wrote:
    Hello!
    The problem is the while statement. On my PC it runs for 15-25 seconds, depends on the number of the deals; but on our customer's server, it runs for 2-5 hours with very high CPU load. The application uses oracle xml parser.
    On my PC there are Websphere MQ 5.3 and Oracle 10ias with 1.5 JVM.
    On our customers's server there are Websphere MQ 5.3 and Oracle 9ias with 1.4 HotSpot JVM.
    Do you have any idea, what can be the cause of the problem?All sorts of things are possible.
    - MQ is configured differently. For instance transactional versus non-transactional.
    - The customer uses a network (multiple computers) and you use a box (singlur) and something is using a lot of bandwidth on the network. Could be your process, one of the dependent processes or something else that has nothing to do with your system
    - The processing computer is loaded with something else or something is artificially limiting the processing time of your application (there is a app that allows one to limit another app to one CPU.)
    - At least one version of 1.4 had a xml bug that consumed massive amounts of memory when processing large xml. Sound familar? If the physical memory is not up to it then it will be thrashing the hard drive as it swaps virtual in and out.
    - Of course virtual memory causing swaps would impact it regardless of the cause.
    - The database is loaded.
    You can of course at least get the same version of java that they are using. Actually that would seem like a rather good idea to me regardless.

  • Xml query hungs up with large xml response from utl_http request

    We are having very sensitive problem in Production environment with xmlquery.
    When receive a small or medium size xml, the query shown below works. But when the xml is large or very large its hung and slow down all the database.
    We are with Oracle 11gR2
    We are using clob to parse the response from the http request.
    What could be the problem or the solutions?. Please help. Its urgent...
    SELECT opciones_obj (x.strindice,
    x.nombrecompleto,
    x.nombre,
    x.strtipodato,
    x.codigoopcion,
    x.floatval,
    x.strtipo,
    x.strval)
    BULK COLLECT INTO t_opciones
    FROM XMLTABLE (
    xmlnamespaces (
    'http://schemas.xmlsoap.org/soap/envelope/' AS "env",
    'http://wsevaluarreglacondicioncomercial/' AS "ns0",
    'http://wsevaluarreglacondicioncomercial/types/' AS "ns1",
    'http://www.oracle.com/webservices/internal/literal' AS "ns2"),
    '/env:Envelope/env:Body/ns0:listarOpcionesAtributoEventoResponseElement/ns0:result/ns1:listaVariables/ns2:item/ns2:item'
    PASSING rsp_xml
    COLUMNS strindice VARCHAR2 (4000)
    PATH 'ns2:mapEntry[ns2:key="strIndice"]/ns2:value',
    nombrecompleto VARCHAR2 (4000)
    PATH 'ns2:mapEntry[ns2:key="nombreCompleto"]/ns2:value',
    nombre VARCHAR2 (4000)
    PATH 'ns2:mapEntry[ns2:key="nombre"]/ns2:value',
    strtipodato VARCHAR2 (4000)
    PATH 'ns2:mapEntry[ns2:key="strTipoDato"]/ns2:value',
    codigoopcion NUMBER
    PATH 'ns2:mapEntry[ns2:key="codigoOpcion"]/ns2:value',
    floatval FLOAT
    PATH 'ns2:mapEntry[ns2:key="floatVal"]/ns2:value',
    strtipo VARCHAR2 (4000)
    PATH 'ns2:mapEntry[ns2:key="strTipo"]/ns2:value',
    strval VARCHAR2 (4000)
    PATH 'ns2:mapEntry[ns2:key="strVal"]/ns2:value') x;

    What could be the problem or the solutions?1) Create an XMLType table (could be temporary) using binary XML storage :
    create table tmp_xml of xmltype
    xmltype store as securefile binary xml;2) In your procedure, load the XMLType containing the response (rsp_xml) into the table :
    insert into tmp_xml values (rsp_xml);3) Then, execute the query directly from the table :
    SELECT opciones_obj ( ... )
    BULK COLLECT INTO t_opciones
    FROM tmp_xml t
       , XMLTABLE (
             xmlnamespaces ( ... ),
             '/env:Envelope/env:Body/...'
             PASSING t.object_value
             COLUMNS ...4) At the end of the procedure, delete (or truncate) the table or simply let the table delete itself when the session ends (in case you created it TEMPORARY)

Maybe you are looking for

  • Reversal of Vendor Excise Invoice

    Dear Experts,    After Vendor Excise invoice is posted through T-Code J1IEX (part 2 Postings), How to reverse the posted Document ,(literally speaking we need to reverse the postings made in that transaction similar to Movement 102 for 101) With rega

  • Switching my mac from airport to ethernet

    I noticed that plugging in the LAN cable directly to my powerbook the internet is much faster, so I started unplugging the cable from the airport express to the computer when I need to go faster. The problem is I can't understand how to set this in t

  • Album shuffle doesn't work

    For some reason, selecting shuffle by album doesn't work for me. In fact, all of the three shuffle by options seem have the same effect; shuffling by song. whats going on? shuffle worked fine in 4.9. what did apple do to mess up a perfectly working s

  • Facebook log in opens after I clear my search history

    When I click on tab of the browser homepage to clear history, Facebook log in page opens.

  • Newbe needs help

    I will aplogize in advance. I hate to ask someone to hold my hand in the process but I really need the help. I am currently using Windows Media Server to stream my audio and video files. I like the idea of using Flash since it has no problems streami