Large XML and Query performance
This problem came to me from a Developer and she claims XML query on XMLType field is very slow when using large XML and is there any alternates. Details are below:
=============
Query:
select attributepool_id, attributepool_name, vintage , p.attributepool.extract('//attributepool/segmentationsystem/id/text()').getStringVal() ,
p.attributepool.extract('//attributepool/datasource/id/text()').getStringVal()
from saved_attributepools p
where user_id = 'CLPROFILE2' and vintage = 'SPRING_2003' order by attributepool_name
Table name: saved_attributepools
Space: ecommerce
A Column Name: attributepool
attributepool Column Type: XmlType
One of xml contains 4Mbytes: CORE LIFESTLY
When we try to get the data against this row, query is taking longer.
conn ecommerce@ecom3 --> 82 seconds (table has 65 rows)
conn ecommerce@oradev--> 34 seconds (table has only 4 rows)
We think that;
Oracle parse the entire XML document and load this document into an 'in-memory' DOM structure before executing the specified xpaths.
Adding INDEX into XmlType won't help as we don't use whereclasue against XmlType for this case.
We don't know 10g has solution for this or not.
Any suggestion will be greatly appreciated.
This problem came to me from a Developer and she claims XML query on XMLType field is very slow when using large XML and is there any alternates. Details are below:
=============
Query:
select attributepool_id, attributepool_name, vintage , p.attributepool.extract('//attributepool/segmentationsystem/id/text()').getStringVal() ,
p.attributepool.extract('//attributepool/datasource/id/text()').getStringVal()
from saved_attributepools p
where user_id = 'CLPROFILE2' and vintage = 'SPRING_2003' order by attributepool_name
Table name: saved_attributepools
Space: ecommerce
A Column Name: attributepool
attributepool Column Type: XmlType
One of xml contains 4Mbytes: CORE LIFESTLY
When we try to get the data against this row, query is taking longer.
conn ecommerce@ecom3 --> 82 seconds (table has 65 rows)
conn ecommerce@oradev--> 34 seconds (table has only 4 rows)
We think that;
Oracle parse the entire XML document and load this document into an 'in-memory' DOM structure before executing the specified xpaths.
Adding INDEX into XmlType won't help as we don't use whereclasue against XmlType for this case.
We don't know 10g has solution for this or not.
Any suggestion will be greatly appreciated.
Similar Messages
-
Large XML and DBMS_XMLSTORE
Oracle DB = 10.2.0.3
I have XML messages (size range 300KB - 15MB) that are inserted into a table with an XMLType column throughout the day.
The volume will vary...say 10 transactions per minute. A single transaction can contain any number of entities/objects.
I have no need to preserve the XML - aside from assigning it a sequence number, date etc. and storing it for debugging (if needed).
Currently, I have a trigger defined on the table that calls various database packages
that shred the data via XMLTable (i.e. insert into tab1...select...xmltable..).
The problem is that I potentially need to insert into 50+ tables out of 200+ tables per transaction. So, I'm passing through the single instance document 200+ times.
The number of tables that are needed to represent the XML will continue to grow.
A 300KB XML message takes approx 10 seconds to shred. Obviously, I'm concerned about the larger messages.
So, I was wondering if doing the following would be any better:
1. Have a XSLT transform the original message into "pseudo" Oracle canonical format.
At this point, all of the data contained in the message would need to be shredded and all unwanted data from the original message would be filtered..i.e. lowering the message size a little.
The message to be inserted would then look something like this:
<Transaction>
<Table>
<Name>TAB1</Name>
<ROWSET>
<ROW>
<COL1>
<COLn>
</ROW>
</ROWSET>
</Table>
<Table>
<Name>TAB2</Name>
<ROWSET>
<ROW>
<COL1>
<COLn>
</ROW>
</ROWSET>
</Table>
</Transaction>
2. Next, define a trigger on the table like this:
CREATE OR REPLACE TRIGGER tr_ai_my_table
AFTER INSERT
ON my_table
FOR EACH ROW
DECLARE
CURSOR table_cur
IS
SELECT a.table_nm
,a.xml_fragment
FROM XMLTABLE ('/Transaction/Table[Name]'
PASSING :NEW.xml_document
COLUMNS
table_nm VARCHAR2(30) PATH 'Name'
,xml_fragment XMLTYPE PATH 'ROWSET'
) a;
v_context DBMS_XMLSTORE.ctxtype;
v_rows NUMBER;
-- XML Date format: 2007-08-02
-- XML Datetime format: 2007-08-02T11:58:28.229-04:00
-- ALTER SESSION SET NLS_DATE_FORMAT = 'YYYY-MM-DD';
-- ALTER SESSION SET NLS_TIMESTAMP_FORMAT = 'YYYY-MM-DD-HH24.MI.SS.FF';
-- ALTER SESSION SET NLS_TIMESTAMP_TZ_FORMAT = 'YYYY-MM-DD"T"HH24:MI:SS.FFTZH:TZM';
BEGIN
FOR table_rec IN table_cur
LOOP
v_context := DBMS_XMLSTORE.newcontext (table_rec.table_nm);
v_rows := DBMS_XMLSTORE.insertxml (v_context, table_rec.xml_fragment);
DBMS_XMLSTORE.closecontext (v_context);
END LOOP;
END;
I know this will get me out of maintaining 200+ insert statements/xpaths. Also, I wouldn't be coding the XSLT either and the XSLT may end up running on some specialized hardware for optimized performance.
But do I gain a significant performance boost using the Oracle canonical format
and DBMS_XMLSTORE even though I'm still building the DOM tree to build the cursor ?i saw a problem with the xmlbeans implementation with wli 8.1
we had memeory issues with large xmls messages as they seemded to be in memory even through the JPD had finished.
to solve this we inserted code at the end of a process to assign the top level xmlbean document class to a tiny version of the same message type. this is valid when the document object is a top level variable of the jpd.
this was done to stop xmlbeans caching the large message behind the scenes. -
Hi,
Thanks in advance for reading this.
I am new to this XML and import into database.
I have the xml file, i need to import this into database and search on the database.
I have around 2.2 million records in the database.
Please explain me what is best method of storing in database, like store the file in clob column and search on the clob column.How the search is going to be on 2.2 million records.
Please send me any if any sample code you have to [email protected]The answer on this question also depends on the version of the database you are using.
With 9i/10g i would probably use structured storage of XML (based on the XMLschema).
But the answer also depends on another question: does the XML document contains "2.2 million records" or will you be storing 2.2 million XML documents? -
the table i have
create table test of xmltype;
and the xml that i have loaded is
<root>
<company>
<department>
<id>10</id>
<name>Accounting</name>
</department>
<department>
<id>11</id>
<name>Billing</name>
</department>
</company>
</root>
select query using xmltable is
select id,name from test,xmltable('/root/company/department'
passing object_value
columns
id number path 'id',
name varchar2(20) 'name');
the query is working fine but issue is performance
i have imlplemented index using extract() and extractstringval() functions but as i have multiple
occurance of data ther two are not working. I have non-schema-based xmltype table.
I need help for creating index on multiple occurance element
Any help is appreciatedFirst of all "XMLOptimizationCheck" AFAIK is not yet explained. Haven't checked support.oracle.com for a while though.
It's more or less currently an internal used, but for the public a fast method to detect, that Oracle internal XQuery / XPath optimization rewrites towards SQL methods (shortcuts) are not properly working. SYS_XQEXVAL probably means something like XQuery Element XML Value/validation (??? towards SQL value) isn't producing a simple construct with a predicate validation. The reasons section gives insight, just like a 10053 trace, on what attempts/rules where applied and failed or worked. I am guessing that the overall cost for the use of the normal PK index is so high because it can not be properly matched and/or optimized against the global index structure supporting the partitions.
In all, a bit more info regarding the table/partition structure and its used index regime/structure would be helpful.
Beside that. THIS IS A BUG and should be reported, request for help, via support.oracle.com
Edited by: Marco Gralike on Mar 23, 2011 9:35 PM -
Compression and query performance in data warehouses
Hi,
Using Oracle 11.2.0.3 have a large fact table with bitmap indexes to the asscoiated dimensions.
Understand bitmap indexes are compressed by default so assume cannot further compress them.
Is this correct?
Wish to try compress the large fact table to see if this will reduce the i/o on reads and therfore give performance benefits.
ETL speed fine just want to increase the report performance.
Thoughts - anyone seen significant gains in data warehouse report performance with compression.
Also, current PCTFREE on table 10%.
As only insert into tabel considering making this 1% to imporve report performance.
Thoughts?
ThanksFirst of all:
Table Compression and Bitmap Indexes
To use table compression on partitioned tables with bitmap indexes, you must do the following before you introduce the compression attribute for the first time:
Mark bitmap indexes unusable.
Set the compression attribute.
Rebuild the indexes.
The first time you make a compressed partition part of an existing, fully uncompressed partitioned table, you must either drop all existing bitmap indexes or mark them UNUSABLE before adding a compressed partition. This must be done irrespective of whether any partition contains any data. It is also independent of the operation that causes one or more compressed partitions to become part of the table. This does not apply to a partitioned table having B-tree indexes only.
This rebuilding of the bitmap index structures is necessary to accommodate the potentially higher number of rows stored for each data block with table compression enabled. Enabling table compression must be done only for the first time. All subsequent operations, whether they affect compressed or uncompressed partitions, or change the compression attribute, behave identically for uncompressed, partially compressed, or fully compressed partitioned tables.
To avoid the recreation of any bitmap index structure, Oracle recommends creating every partitioned table with at least one compressed partition whenever you plan to partially or fully compress the partitioned table in the future. This compressed partition can stay empty or even can be dropped after the partition table creation.
Having a partitioned table with compressed partitions can lead to slightly larger bitmap index structures for the uncompressed partitions. The bitmap index structures for the compressed partitions, however, are usually smaller than the appropriate bitmap index structure before table compression. This highly depends on the achieved compression rates. -
How to make use of XMLDB to process large XML and create spatial objects
For someone new to XMLDB I find it hard to make sense of the enormous amount of information (and easy to get lost in it). So I come here to ask for directions.
I have to build a procedure that fills a table of spatial objects based on XML input. Basically the XML contains a large amount of elements that describe the geometry type and contain the geometries coordinates. The XML can get quite large (200-300Mb).
Somehow I have to process each element and create an sdo_geometry object.
Now let me ask a very broad question: What would be a good way to handle this?I have the same question. Any news on this?
Wijnand -
Distinct clause and query performance
Friends,
I have a query which returns results in 40 seconds without distinct clause and when I add distinct clause it takes over 2 hours.
I have verified following -
1. indexes/table statistics are up to date.
2. columns that are used in where clause but are not indexed have upto date column statistics
Any idea what could be the reason, explain plan shows that distinct clause has a very expensive cost.
Thanks
Query and explain plan is below
SELECT
DISTINCT -- with distinct 2hrs + and without 40 seconds
quote_by_dst.qte_hdr_stat_cd, quote_by_dst.qte_ln_cond_cd,
product.prod_nm, product.prod_id,
cs_ship_by_dst.bto_ds_cac_ownr_ud,
quote_by_dst.qte_csup_csup_am, cs_ship_by_dst.bto_ds_cac_nm,
product.spl_sht_nm,
product.prod_blg_un_fac_um
|| ' '
|| product.prod_blg_um
|| ' '
|| product.prod_stck_um,
product.prod_blg_um, quote_by_dst.qte_ln_brk_1_blg_uom_am,
quote_by_dst.qte_csup_avg_cst_am,
quote_by_dst.qte_csup_rev_gm_pct_am,
quote_by_dst.qte_csup_avg_cst_am, cs_ship_by_dst.bto_id,
cs_ship_by_dst.bto_ds_cac_cd,
cs_ship_by_dst.bto_ds_cac_cd
|| product.prod_id
|| cs_ship_by_dst.bto_id
FROM infowhse.quote_by_dst4 quote_by_dst,
infowhse.product,
infowhse.cs_ship_by_dst4 cs_ship_by_dst,
infowhse.department
WHERE (quote_by_dst.dpt_cd = department.dpt_cd)
AND (quote_by_dst.cus_dpt_id = cs_ship_by_dst.cus_dpt_id)
AND (product.prod_id = quote_by_dst.prod_id)
AND ( ( quote_by_dst.qte_ln_cond_cd = 'E'
OR quote_by_dst.qte_ln_cond_cd = 'C'
AND quote_by_dst.qte_hdr_stat_cd = 'A'
AND ((cs_ship_by_dst.bto_cust_type_cd) = '01')
AND cs_ship_by_dst.bto_ds_cac_ownr_ud = 'EHOC'
AND department.dpt_cd > '0.00'
)Explain plan
Plan
SELECT STATEMENT CHOOSECost: 911,832,256 Bytes: 433,941,639,459 Cardinality: 2,729,192,701
15 SORT UNIQUE Cost: 911,832,256 Bytes: 433,941,639,459 Cardinality: 2,729,192,701
14 NESTED LOOPS Cost: 68,705 Bytes: 433,941,639,459 Cardinality: 2,729,192,701
12 HASH JOIN Cost: 68,705 Bytes: 425,754,061,356 Cardinality: 2,729,192,701
1 INDEX FAST FULL SCAN NON-UNIQUE INFOWHSE.DST_SEC_DST_SEC_DST_CD_IX Cost: 25 Bytes: 922,700 Cardinality: 184,540
11 HASH JOIN Cost: 16,179 Bytes: 1,199,209,082 Cardinality: 7,941,782
2 INDEX FAST FULL SCAN NON-UNIQUE INFOWHSE.DST_SEC_DST_SEC_DST_CD_IX Cost: 25 Bytes: 922,700 Cardinality: 184,540
10 HASH JOIN Cost: 15,879 Bytes: 3,374,060 Cardinality: 23,110
8 HASH JOIN Cost: 15,200 Bytes: 2,981,190 Cardinality: 23,110
6 HASH JOIN Cost: 13,113 Bytes: 1,779,470 Cardinality: 23,110
3 TABLE ACCESS FULL INFOWHSE.CUSTOMER_SHIP Cost: 5,640 Bytes: 42,372 Cardinality: 1,177
5 PARTITION RANGE ALL Partition #: 11 Partitions accessed #1 - #12
4 TABLE ACCESS FULL INFOWHSE.QUOTE Cost: 7,328 Bytes: 38,826,590 Cardinality: 946,990 Partition #: 11 Partitions accessed #1 - #12
7 TABLE ACCESS FULL INFOWHSE.PRODUCT Cost: 1,542 Bytes: 9,246,640 Cardinality: 177,820
9 INDEX FAST FULL SCAN NON-UNIQUE INFOWHSE.CUST_SHIP_SLSDST_DTP_SICALL_IX Cost: 185 Bytes: 9,878,411 Cardinality: 581,083
13 INDEX UNIQUE SCAN UNIQUE INFOWHSE.DEPARTMENT_PK Bytes: 3 Cardinality: 1This might be more useful.
Query is still running.
There is heavy wait time for scattered file read.
Results from
SELECT * FROM V$SESSION_WAIT WHERE SID = 48;
SID SEQ# EVENT P1TEXT P1 P1RAW P2TEXT P2 P2RAW P3TEXT P3 P3RAW WAIT_TIME SECONDS_IN_WAIT STATE
48 6865 db file scattered read file# 108 000000000000006C block# 1593370000000000026E69 blocks 32 0000000000000020 2 30 WAITED KNOWN TIME
SELECT * FROM V$SESSION_EVENT WHERE SID = 48;
SID EVENT TOTAL_WAITS TOTAL_TIMEOUTS TIME_WAITED AVERAGE_WAIT MAX_WAIT TIME_WAITED_MICRO
48 log file sync 1 0 0 0 0 563
48 db file sequential read 11 0 0 0 0 243
48 db file scattered read 6820 0 330 0 7 3296557
48 SQL*Net message to client 19 0 0 0 0 23
48 SQL*Net message from client 18 0 128 7 127 1281912 Sorry for long post. -
Large XML document performance
We are using XDB 9.2.0.4. I am seeing a severe performance degradation when attempting to extract larger XML documents from XDB (somewhere over 3M). Smaller documents appear to be working fine.
I have been reading in the forum that the problem I am running into is most likely related to the storage model being used. ie) There are several repeating elements within the schema.
I have added xdb:storeVarrayAsTable="true" statement to the schema and re-registered. I can see, based on user_nested_tables, that XDB appears to be storing the repeating elements as nested tables vs varrays.
The change to the storage model does not seem to have significantly changed the queries performance.
The schemas I am using can be found at http://www.sasked.gov.sk.ca/xsd/sl/1.x/SLMessage.xsd & http://www.sasked.gov.sk.ca/xsd/sl/1.x/SDSElements.xsd
The schema documentation can be found at http://www.sasked.gov.sk.ca/sds/xml/SchemaDocumentation/SLMessage.html
The element /SL_Message/SL_Event/SL_ObjectData/SL_EventObject is the primary repeating element
I am using a table with an XMLType column
CREATE TABLE XML_SL_MESSAGE
(XML_SL_MESSAGE_ID NUMBER(11) NOT NULL
,DTE_TIMESTAMP TIMESTAMP DEFAULT SYSTIMESTAMP NOT NULL
,ORIGINAL_XML_SL_MESSAGE_ID NUMBER(11)
,VALID_SL_MESSAGE_XML sys.XMLType
,INVALID_XML CLOB
,ERROR_MESSAGE VARCHAR2(4000)
) xmltype column valid_sl_message_xml XMLSCHEMA "http://www.sasked.gov.sk.ca/xsd/sl/1.x/SLMessage.xsd" element "SL_Message"
The SQL I am using is attempting to bring the XMLType back as a clob, the query seems to be intensive in both CPU and I/O. (looks like it is the getClobVal function)
select xsm.xml_sl_message_id
,xsm.dte_timestamp
,nvl(xsm.valid_sl_message_xml.getClobVal(),xsm.invalid_xml) as xml_clob
,xsm.error_message
,xsm.original_xml_sl_message_id
from xml_sl_message xsm
where xsm.dte_timestamp > sysdate 1
I guess what I am wondering is what are my options ? Changing the storage model ? Applying Indexes ?
On an unrelated topic, Are there many differences in XDB 9.2.0.5 and 9.2.0.4 ? (I dont believe 10g will be an option here yet)
Thanx in advance
TrentI have applied the 9.2.0.5.0 patches and created the relational table with the following attributes:
CREATE TABLE XML_SL_MESSAGE
(XML_SL_MESSAGE_ID NUMBER(11) NOT NULL
,DTE_TIMESTAMP TIMESTAMP DEFAULT SYSTIMESTAMP NOT NULL
,PSE_SYS_USR_ID NUMBER(11) NOT NULL
,ORIGINAL_XML_SL_MESSAGE_ID NUMBER(11)
,VALID_SL_MESSAGE_XML sys.XMLType
,INVALID_XML CLOB
,ERROR_MESSAGE VARCHAR2(4000)
xmltype column valid_sl_message_xml
STORE AS OBJECT RELATIONAL
XMLSCHEMA "http://www.sasked.gov.sk.ca/xsd/sl/1.x/SLMessage.xsd" ELEMENT "SL_Message"
-- 1:1 ----------------------
varray valid_sl_message_xml."XMLDATA"."SL_Event"."SL_ObjectData"."SL_EventObject"
store as table SL_EVENTOBJECT2_TB(
(constraint SL_EVENTOBJECT2_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 2:2 ----------------------
varray "SchoolTerm"
store as table SCHOOLTERM2_TB(
(constraint SCHOOLTERM2_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 3:2 ----------------------
varray "SchoolClass"
store as table SCHOOLCLASS3_TB(
(constraint SCHOOLCLASS3_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 6:2 ----------------------
varray "StudentCourseHistory"
store as table STUDENTCOURSEHISTORY6_TB(
(constraint STUDENTCOURSEHISTORY6_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 7:2 ----------------------
varray "StudentSupplementalMark"
store as table STUDENTSUPPLEMENTALMARK7_TB(
(constraint STUDENTSUPPLEMENTALMARK7_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 8:2 ----------------------
varray "StudentClassMark"
store as table STUDENTCLASSMARK8_TB(
(constraint STUDENTCLASSMARK8_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 9:2 ----------------------
varray "StudentExamRegistration"
store as table STUDENTEXAMREGISTRATION9_TB(
(constraint STUDENTEXAMREGISTRATION9_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 10:2 ----------------------
varray "StudentClassEnrollment"
store as table STUDENTCLASSENROLLMENT10_TB(
(constraint STUDENTCLASSENROLLMENT10_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 11:2 ----------------------
varray "StudentPersonal"
store as table STUDENTPERSONAL11_TB(
(constraint STUDENTPERSONAL11_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 18:2 ----------------------
varray "StudentProgramEnrollment"
store as table STUDENTPROGRAMENROLLMENT18_TB(
(constraint STUDENTPROGRAMENROLLMENT18_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 19:2 ----------------------
varray "StudentSchoolEnrollment"
store as table STUDENTSCHOOLENROLLMENT19_TB(
(constraint STUDENTSCHOOLENROLLMENT19_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 26:1 ----------------------
varray valid_sl_message_xml."XMLDATA"."SL_Response"."SL_ObjectData"."SL_EventObject"
store as table SL_EVENTOBJECT26_TB(
(constraint SL_EVENTOBJECT26_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 27:2 ----------------------
varray "SchoolTerm"
store as table SCHOOLTERM27_TB(
(constraint SCHOOLTERM27_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 28:2 ----------------------
varray "SchoolClass"
store as table SCHOOLCLASS28_TB(
(constraint SCHOOLCLASS28_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 31:2 ----------------------
varray "StudentProgramEnrollment"
store as table STUDENTPROGRAMENROLLMENT31_TB(
(constraint STUDENTPROGRAMENROLLMENT31_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 32:2 ----------------------
varray "StudentExamRegistration"
store as table STUDENTEXAMREGISTRATION32_TB(
(constraint STUDENTEXAMREGISTRATION32_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 33:2 ----------------------
varray "StudentClassEnrollment"
store as table STUDENTCLASSENROLLMENT33_TB(
(constraint STUDENTCLASSENROLLMENT33_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 34:2 ----------------------
varray "StudentPersonal"
store as table STUDENTPERSONAL34_TB(
(constraint STUDENTPERSONAL34_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 41:2 ----------------------
varray "StudentSchoolEnrollment"
store as table STUDENTSCHOOLENROLLMENT41_TB(
(constraint STUDENTSCHOOLENROLLMENT41_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 48:2 ----------------------
varray "StudentClassMark"
store as table STUDENTCLASSMARK48_TB(
(constraint STUDENTCLASSMARK48_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 49:2 ----------------------
varray "StudentCourseHistory"
store as table STUDENTCOURSEHISTORY49_TB(
(constraint STUDENTCOURSEHISTORY49_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 50:2 ----------------------
varray "StudentSupplementalMark"
store as table STUDENTSUPPLEMENTALMARK50_TB(
(constraint STUDENTSUPPLEMENTALMARK50_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 51:1 ----------------------
varray valid_sl_message_xml."XMLDATA"."SL_Response"."SL_Ack"."SL_Error"
store as table SL_ERROR51_TB(
(constraint SL_ERROR51_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
-- 52:1 ----------------------
varray valid_sl_message_xml."XMLDATA"."SL_Request"."SL_Query"."SL_QueryObject"
store as table SL_QUERYOBJECT52_TB(
(constraint SL_QUERYOBJECT52_PK primary key (NESTED_TABLE_ID,ARRAY_INDEX))
tablespace data
ALTER TABLE XML_SL_MESSAGE
ADD (CONSTRAINT XML_SL_MESSAGE_PK PRIMARY KEY
(XML_SL_MESSAGE_ID))
ALTER TABLE XML_SL_MESSAGE
ADD (CONSTRAINT XMLSLMSG_ORIGINAL_XMLSLMSG_UK UNIQUE
(ORIGINAL_XML_SL_MESSAGE_ID))
ALTER TABLE XML_SL_MESSAGE ADD (CONSTRAINT
XMLSLMSG_SYSUSR_FK FOREIGN KEY
(PSE_SYS_USR_ID) REFERENCES PSE_SYS_USR
(PSE_SYS_USR_ID))
ALTER TABLE XML_SL_MESSAGE ADD (CONSTRAINT
XMLSLMSG_ORIGINAL_XMLSLMSG_FK FOREIGN KEY
(ORIGINAL_XML_SL_MESSAGE_ID) REFERENCES XML_SL_MESSAGE
(XML_SL_MESSAGE_ID))
-- Create a unique index for the XML Message id
CREATE UNIQUE INDEX XMLSLMSG_MSGID_UNIQUE ON XML_SL_MESSAGE
((substr(extractValue(valid_sl_message_xml,'//SL_MsgId'),1,255)))
tablespace indx
COMPUTE STATISTICS
Here is the nested table structure of the XMLType table created during the schema registration:
select level
,parent_table_column
from user_nested_tables
connect by prior table_name = parent_table_name
start with parent_table_name = 'SL_Message4724_TAB'
LEVEL PARENT_TABLE_COLUMN
1 "XMLDATA"."SL_Event"."SL_ObjectData"."SL_EventObject"
2 SchoolTerm
2 StudentSchoolEnrollment
3 "StudentInfo"."Name"
3 "StudentInfo"."Demographics"."CountryOfCitizenship"
3 "StudentInfo"."StudentAddress"
3 "StudentInfo"."PhoneNumber"
3 "StudentInfo"."Demographics"."Language"
3 "StudentInfo"."Email"
2 SchoolClass
3 "ClassInfo"."DeptAssignedCourseId"
3 "ClassInfo"."EducatorCertificateNumber"
2 StudentProgramEnrollment
2 StudentClassEnrollment
2 StudentClassMark
2 StudentCourseHistory
2 StudentSupplementalMark
2 StudentExamRegistration
2 StudentPersonal
3 "StudentInfo"."Name"
3 "StudentInfo"."Email"
3 "StudentInfo"."Demographics"."CountryOfCitizenship"
3 "StudentInfo"."StudentAddress"
3 "StudentInfo"."PhoneNumber"
3 "StudentInfo"."Demographics"."Language"
1 "XMLDATA"."SL_Request"."SL_Query"."SL_QueryObject"
1 "XMLDATA"."SL_Response"."SL_ObjectData"."SL_EventObject"
2 SchoolTerm
2 SchoolClass
3 "ClassInfo"."DeptAssignedCourseId"
3 "ClassInfo"."EducatorCertificateNumber"
2 StudentProgramEnrollment
2 StudentClassEnrollment
2 StudentClassMark
2 StudentCourseHistory
2 StudentSupplementalMark
2 StudentExamRegistration
2 StudentPersonal
3 "StudentInfo"."Name"
3 "StudentInfo"."Email"
3 "StudentInfo"."Demographics"."Language"
3 "StudentInfo"."PhoneNumber"
3 "StudentInfo"."StudentAddress"
3 "StudentInfo"."Demographics"."CountryOfCitizenship"
2 StudentSchoolEnrollment
3 "StudentInfo"."Name"
3 "StudentInfo"."Demographics"."Language"
3 "StudentInfo"."PhoneNumber"
3 "StudentInfo"."StudentAddress"
3 "StudentInfo"."Demographics"."CountryOfCitizenship"
3 "StudentInfo"."Email"
1 "XMLDATA"."SL_Response"."SL_Ack"."SL_Error"
52 rows selected.
When I attempt to insert to previously valid XML documents I get a core dump Here are the insert statements:
insert into xml_sl_message (
xml_sl_message_id, dte_timestamp, pse_sys_usr_id, original_xml_sl_message_id, valid_sl_message_xml, invalid_xml,error_message
select xml_sl_message_id, dte_timestamp, pse_sys_usr_id, original_xml_sl_message_id, xmltype(valid_sl_message_clob), invalid_xml,error_message
from xml_sl_message_temp where xml_sl_message_id in (5154,5155)
Here are the details on the exception:
Oracle9i Enterprise Edition Release 9.2.0.5.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.5.0 - Production
ORACLE_HOME = /opt/app/oracle/product/9.2.0
System name: SunOS
Node name: *****
Release: 5.8
Version: Generic_117000-01
Machine: sun4u
Instance name: EDDSDS
Redo thread mounted by this instance: 1
Oracle process number: 76
Unix process pid: 11460, image: oracle@***** (TNS V1-V3)
*** 2004-06-22 09:24:34.603
*** SESSION ID:(29.159) 2004-06-22 09:24:34.602
Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x20, PC: [0x101e41830, 0000000101E41830]
*** 2004-06-22 09:24:34.606
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [0000000101E41830] [SIGSEGV] [Address not mapped to object] [0x000000020] [] []
Current SQL statement for this session:
insert into xml_sl_message (
xml_sl_message_id, dte_timestamp, pse_sys_usr_id, original_xml_sl_message_id, valid_sl_message_xml, invalid_xml,error_message
select xml_sl_message_id, dte_timestamp, pse_sys_usr_id, original_xml_sl_message_id, xmltype(valid_sl_message_clob), invalid_xml,error_message
from xml_sl_message_temp where xml_sl_message_id in (5154,5155)
I have tried a couple different storage sections (with different levels of nesting) and still the same problem ..
Is there something wrong with my storage section ?
What is the "return as LOCATOR" clause for? -
Query performance - A single large document VS multiple small documents
Hi all,
What are the performance trade offs when using a single large document VS multiple small documents ?
I want to store xml snippets with similar structure in a container. Is there any benefit while querying, if I use a single large document to store all these snippets against adding each snippet as a different document. Would it degrade the performance when adding an xml snippet each time by modifying an existing document?
How could we decide whether to use a single large document VS multiple small documents?
Thanks,
AnoopHello Anoop,
In case you wanted to get a comparison between the storage types for containers, wholedoc and node, let us know.
What are the performance trade offs when using a
single large document VS multiple small documents ?Depends on what is more important to you, performance when creating the container, and inserting the document(s) or performance when retrieving data.
For querying the best option is to go with smaller documents, as node indexes would help in improving query performance.
For inserting initial data, you can construct your large document composed of smaller xml snippets and insert the document as a whole.
If you further want to modify this document changing its structure implies performance penalties; more indicated is to store the xml snippets as documents.
Overall, I see no point in using a large document that will hold all of your xml snippets, so I strongly recommend going with multiple smaller documents.
Regards,
Andrei Costache
Oracle Support Services -
MII Performance with Large XML
HI,
We are facing performance issue while parsing XML of large size of around 50MB.
Initially XMII was crashing due to "out of heap memory error". However, with some changes the problem has been removed.But
now, we are facing an issue with time taken by the Payload to execute. Its taking more than half-an-hour to execute the transaction.
The solution tried so far has decreased the time to half-an-hour. earlier it was taking more than one and half hour to process.
We have tried parallel processing by asynchronous call to the transactions
Is there any way to further reduce the time of processing?
Further, is it possible to find out which parser is used by MII internally for XML SAX or DOM Parser.
Thanks
AmitHi Amit,
Just some tips to improve performance of BLS.
1. Use Xpath whenever possible.
2. Remove unnecessary repeaters performing on BLS.
3. Check the Like to Optimizing BLS Performance.
Optimizing BLS Performance for XML Handling in SAP MII
If you are storing the data in database. just pass whole xml to query and insert data using bulk insert.
Thanks
Anshul -
Performance Problem in parsing large XML file (15MB)
Hi,
I'm trying to parse a large XML file(15 MB) and facing a clear performance problem. A Simple XML Validation using the following code snippet:
DBMS_LOB.fileopen(targetFile, DBMS_LOB.file_readonly);
DBMS_LOB.loadClobfromFile
tempCLOB,
targetFile,
DBMS_LOB.getLength(targetFile),
dest_offset,
src_offset,
nls_charset_id(CONSTANT_CHARSET),
lang_context,
conv_warning
DBMS_LOB.fileclose(targetFile);
p_xml_document := XMLType(tempCLOB, p_schema_url, 0, 0);
p_xml_document.schemaValidate();
is taking 30 mins on a HP-UX (4GB ram, 2 CPU) machine (Oracle version : 9.2.0.4).
Please explain what could be going wrong.
Thanks In Advance,
VineetThanks Mark,
I'll open a TAR and also upload the schema and instance XML.
If i'm not changing the track too much :-) one more thing in continuation:
If i skip the Schema Validation step and directly insert the instance document into a Schema linked XMLType table, what does OracleXDB do in such a case?
i'm getting a severe performance hit here too... the same file as above takes almost 40 mins to Insert.
code snippet:
DBMS_LOB.fileopen(targetFile, DBMS_LOB.file_readonly);
DBMS_LOB.loadClobfromFile
tempCLOB,
targetFile,
DBMS_LOB.getLength(targetFile),
dest_offset,
src_offset,
nls_charset_id(CONSTANT_CHARSET),
lang_context,
conv_warning
DBMS_LOB.fileclose(targetFile);
p_xml_document := XMLType(tempCLOB, p_schema_url, 0, 0);
-- p_xml_document.schemaValidate();
insert into INCOMING_XML values(p_xml_document);
Here table INCOMING_XML is :
TABLE of SYS.XMLTYPE(XMLSchema "http://INCOMING_XML.xsd" Element "MatchingResponse") STORAGE Object-
relational TYPE "XDBTYPE_MATCHING_RESPONSE"
This table and type XDBTYPE_MATCHING_RESPONSE were created using the mapping provided in the registered XML Schema.
Thanks,
Vineet -
Hello,
I'm trying to work with very large xml files which are created from csv files. These files may be very large - up to 1 GB ! Untill now I have managed to do several validations on these big xml files, and the only thing that works for me is SAX parser, DOM is out of the question because it fills up memory.
My next task is to do queries on these files, smth like:
select field1,field2 from file.xml
where field3 = 'A'
and (fileld4>'B' or field1='C')
order by field2.
I searched the net about finding out how to make queries on xml files (since I have never done queries on xml before), but I couldn't find which "query language" is best for large files. If I use XPath (XSLT) will that not cause me memory problems because XSLT represents the file as a memory object?
My idea is to parse the file with SAX and check every row if it fits the where condition and then write it immediately to a result xml file. But validating the where statement can be very complicated without using some tool. Also the order by statement is another problematic issue.
Does anyone have some more intelegent ideas about how I can do this? Please help! :(
The xml file looks like this:
<doc>
<row id ="1">
<column id="1" name="column1">value</column>
<column id="N" name="columnN">value</column>
</row>
<row id ="M">
<column id="1" name="column1">value</column>
<column id="N" name="columnN">value</column>
</row>
</doc>Hi all,
Thank you very much for your replies.
First, saxon didn't work because it uses an in-memory parser, and that is what I was trying to avoid.
Different database is also out of the question, because the customer insist on XML, and also there are some files that can never be converted to a database table, because eventually with some transformations thay are changed and are not completely like the standard csv format.
I think that maybe http://exist.sourceforge.net is the rigth solution for me, but I will probably try it in the next version of my project.
For now I have managed to make the project with only SAXParser and a lot of back - end programming and it works ok, althoug it was very hard to make it, and will be harded to maintain, so I will try to look at the eXist project.
Thanks everyone for the help. -
How to improve Query performance on large table in MS SQL Server 2008 R2
I have a table with 20 million records. What is the best option to improve query performance on this table. Is partitioning the table into filegroups is a best option or splitting the table into multiple smaller tables?
Hi bala197164,
First, I want to inform that both to partition the table into filegroups and split the table into multiple smaller tables can improve the table query performance, and they are fit for different situation. For example, our table have one hundred columns and
some columns are not related to this table object directly (for example, there is a table named userinfo to store user information, it has columns address_street, address_zip,address_ province columns, at this time, we can create a new table named as Address,
and add a foreign key in userinfo table references Address table), under this situation, by splitting a large table into smaller, individual tables, queries that access only a fraction of the data can run faster because there is less data to scan. Another
situation is our table records can be grouped easily, for example, there is a column named year to store information about product release date, at this time, we can partition the table into filegroups to improve the query performance. Usually, we perform
both of methods together. Additionally, we can add index to table to improve the query performance. For more detail information, please refer to the following document:
Partitioning:
http://msdn.microsoft.com/en-us/library/ms178148.aspx
CREATE INDEX (Transact-SQL):
http://msdn.microsoft.com/en-us/library/ms188783.aspx
TechNet
Subscriber Support
If you are
TechNet Subscription user and have any feedback on our support quality, please send your feedback
here.
Allen Li
TechNet Community Support -
Performance problem - XML and Hashtable
Hello!
I have a strange (or not strange, we don't know) performance problem. The situation is simple: our application runs on Oracle IAS, and gets XML messages from an MQ, and we have to process them.
Sample message:
<RequestDeal RequestId="RequestDeal-20090209200010-998493885" CurrentFragment="0" EndFragment="51" Expires="2009-03-7T08:00:10">
<Deal>
<Id>385011</Id>
<ModifyDate>2009-02-09</ModifyDate>
<PastDuesOnly>false</PastDuesOnly>
</Deal>
<Deal>
<Id>385015</Id>
<ModifyDate>2009-02-09</ModifyDate>
<PastDuesOnly>false</PastDuesOnly>
</Deal>
</RequestDeal>(there's an average of 50000-80000 deals in the whole message)
After the application receives the whole message, we'd like to parse it, and put the values into a hashtable for further processing:
Hashtable ht = new Hashtable();
Node eDeals = getElementNode(docReq, "/RequestDeal");
Node eDeal = (Element)(eDeals.getFirstChild());
long start = System.currentTimeMillis();
while (eDeal != null) {
String id = getElementText((Element)eDeal, "Id");
Date modifyDate = getElementDateValue((Element)eDeal, "ModifyDate");
boolean szukitett = getElementBool((Element)eDeal, "PastDuesOnly", false);
ht.put(id, new UgyletStatusz(id, modifyDate, szukitett));
eDeal = eDeal.getNextSibling();
logger.info("Request adatok betöltve.");
long end = System.currentTimeMillis();
logger.info("Hosszu muvelet ideje: " + (end - start) + "ms");The problem is the while statement. On my PC it runs for 15-25 seconds, depends on the number of the deals; but on our customer's server, it runs for 2-5 hours with very high CPU load. The application uses oracle xml parser.
On my PC there are Websphere MQ 5.3 and Oracle 10ias with 1.5 JVM.
On our customers's server there are Websphere MQ 5.3 and Oracle 9ias with 1.4 HotSpot JVM.
Do you have any idea, what can be the cause of the problem?gyulaur wrote:
Hello!
The problem is the while statement. On my PC it runs for 15-25 seconds, depends on the number of the deals; but on our customer's server, it runs for 2-5 hours with very high CPU load. The application uses oracle xml parser.
On my PC there are Websphere MQ 5.3 and Oracle 10ias with 1.5 JVM.
On our customers's server there are Websphere MQ 5.3 and Oracle 9ias with 1.4 HotSpot JVM.
Do you have any idea, what can be the cause of the problem?All sorts of things are possible.
- MQ is configured differently. For instance transactional versus non-transactional.
- The customer uses a network (multiple computers) and you use a box (singlur) and something is using a lot of bandwidth on the network. Could be your process, one of the dependent processes or something else that has nothing to do with your system
- The processing computer is loaded with something else or something is artificially limiting the processing time of your application (there is a app that allows one to limit another app to one CPU.)
- At least one version of 1.4 had a xml bug that consumed massive amounts of memory when processing large xml. Sound familar? If the physical memory is not up to it then it will be thrashing the hard drive as it swaps virtual in and out.
- Of course virtual memory causing swaps would impact it regardless of the cause.
- The database is loaded.
You can of course at least get the same version of java that they are using. Actually that would seem like a rather good idea to me regardless. -
Xml query hungs up with large xml response from utl_http request
We are having very sensitive problem in Production environment with xmlquery.
When receive a small or medium size xml, the query shown below works. But when the xml is large or very large its hung and slow down all the database.
We are with Oracle 11gR2
We are using clob to parse the response from the http request.
What could be the problem or the solutions?. Please help. Its urgent...
SELECT opciones_obj (x.strindice,
x.nombrecompleto,
x.nombre,
x.strtipodato,
x.codigoopcion,
x.floatval,
x.strtipo,
x.strval)
BULK COLLECT INTO t_opciones
FROM XMLTABLE (
xmlnamespaces (
'http://schemas.xmlsoap.org/soap/envelope/' AS "env",
'http://wsevaluarreglacondicioncomercial/' AS "ns0",
'http://wsevaluarreglacondicioncomercial/types/' AS "ns1",
'http://www.oracle.com/webservices/internal/literal' AS "ns2"),
'/env:Envelope/env:Body/ns0:listarOpcionesAtributoEventoResponseElement/ns0:result/ns1:listaVariables/ns2:item/ns2:item'
PASSING rsp_xml
COLUMNS strindice VARCHAR2 (4000)
PATH 'ns2:mapEntry[ns2:key="strIndice"]/ns2:value',
nombrecompleto VARCHAR2 (4000)
PATH 'ns2:mapEntry[ns2:key="nombreCompleto"]/ns2:value',
nombre VARCHAR2 (4000)
PATH 'ns2:mapEntry[ns2:key="nombre"]/ns2:value',
strtipodato VARCHAR2 (4000)
PATH 'ns2:mapEntry[ns2:key="strTipoDato"]/ns2:value',
codigoopcion NUMBER
PATH 'ns2:mapEntry[ns2:key="codigoOpcion"]/ns2:value',
floatval FLOAT
PATH 'ns2:mapEntry[ns2:key="floatVal"]/ns2:value',
strtipo VARCHAR2 (4000)
PATH 'ns2:mapEntry[ns2:key="strTipo"]/ns2:value',
strval VARCHAR2 (4000)
PATH 'ns2:mapEntry[ns2:key="strVal"]/ns2:value') x;What could be the problem or the solutions?1) Create an XMLType table (could be temporary) using binary XML storage :
create table tmp_xml of xmltype
xmltype store as securefile binary xml;2) In your procedure, load the XMLType containing the response (rsp_xml) into the table :
insert into tmp_xml values (rsp_xml);3) Then, execute the query directly from the table :
SELECT opciones_obj ( ... )
BULK COLLECT INTO t_opciones
FROM tmp_xml t
, XMLTABLE (
xmlnamespaces ( ... ),
'/env:Envelope/env:Body/...'
PASSING t.object_value
COLUMNS ...4) At the end of the procedure, delete (or truncate) the table or simply let the table delete itself when the session ends (in case you created it TEMPORARY)
Maybe you are looking for
-
Reversal of Vendor Excise Invoice
Dear Experts, After Vendor Excise invoice is posted through T-Code J1IEX (part 2 Postings), How to reverse the posted Document ,(literally speaking we need to reverse the postings made in that transaction similar to Movement 102 for 101) With rega
-
Switching my mac from airport to ethernet
I noticed that plugging in the LAN cable directly to my powerbook the internet is much faster, so I started unplugging the cable from the airport express to the computer when I need to go faster. The problem is I can't understand how to set this in t
-
For some reason, selecting shuffle by album doesn't work for me. In fact, all of the three shuffle by options seem have the same effect; shuffling by song. whats going on? shuffle worked fine in 4.9. what did apple do to mess up a perfectly working s
-
Facebook log in opens after I clear my search history
When I click on tab of the browser homepage to clear history, Facebook log in page opens.
-
I will aplogize in advance. I hate to ask someone to hold my hand in the process but I really need the help. I am currently using Windows Media Server to stream my audio and video files. I like the idea of using Flash since it has no problems streami