Poor performance of inserts w/ table indexed
I have a java application that inserts approximately 2M records
into 2 different tables in an Oracle 8i database (NT). If I
place indexes on the tables, the whole process is drastically
slower than when I don't have indexes (18 hrs vs. 1.5 hrs). I
can understand some factor of slowdown, but I would have
expected the DB to utilize the CPU more intensively. My server
is a dual-p3/450 w/ 256M and U2W scsi discs in a SW RAID
config. Average CPU utilization of oracle is 2-3%. Is there
anyway to keep my indexes but have the data go into the db in a
reasonable amount of time? Thanks.
null
Brian Lelek (guest) wrote:
: I suggest that the process that does the inserts should do the
: following:
: 1. Drop the indexes
: 2. Insert the rows
: 3. Recreate the indexes
: This may be faster than leaving the indexes during the inserts.
: Brian
: Brian Krahmer (guest) wrote:
: : I have a java application that inserts approximately 2M
: records
: : into 2 different tables in an Oracle 8i database (NT). If I
: : place indexes on the tables, the whole process is
drastically
: : slower than when I don't have indexes (18 hrs vs. 1.5 hrs).
I
: : can understand some factor of slowdown, but I would have
: : expected the DB to utilize the CPU more intensively. My
: server
: : is a dual-p3/450 w/ 256M and U2W scsi discs in a SW RAID
: : config. Average CPU utilization of oracle is 2-3%. Is
there
: : anyway to keep my indexes but have the data go into the db
in
: a
: : reasonable amount of time? Thanks.
Need to see ur script which insert the records.
I guss , records are inserted one by one. Is it
possible to insert in bunch. Second analyse the
indexes are really required ? rebuilding of indexes
will be faster than recreating it .
null
Similar Messages
-
Performance of insert with spatial index
I'm writing a test that inserts (using OCI) 10,000 2D point geometries (gtype=2001) into a table with a single SDO_GEOMETRY column. I wrote the code doing the insert before setting up the index on the spatial column, thus I was aware of the insert speed (almost instantaneous) without a spatial index (with layer_gtype=POINT), and noticed immediately the performance drop with the index (> 10 seconds).
Here's the raw timing data of 3 runs in each 3 configuration (the clock ticks every 14 or 15 or 16 ms, thus the zero when it completes before the next tick):
truncate execute commit
no spatial index 0.016 0.171 0.016
no spatial index 0.031 0.172 0.000
no spatial index 0.031 0.204 0.000
index (1000 default for batch size) 0.141 10.937 1.547
index (1000 default for batch size) 0.094 11.125 1.531
index (1000 default for batch size) 0.094 10.937 1.610
index SDO_DML_BATCH_SIZE=10000 0.203 11.234 0.359
index SDO_DML_BATCH_SIZE=10000 0.094 10.828 0.344
index SDO_DML_BATCH_SIZE=10000 0.078 10.844 0.359As you can see, I played with SDO_DML_BATCH_SIZE to change the default of 1,000 to 10,000, which does improve the commit speed a bit from 1.5s to 0.35s (pretty good when you only look at these numbers...), but the shocking part of the almost 11s the inserts are now taking, compared to 0.2s without an index: that's a 50x drop in peformance!!!
I've looked at my table in SQL Developer, and it has no triggers associated, although there has to be something to mark the index as dirty so that it updates itself on commit.
So where is coming the huge overhead during the insert???
(by insert I mean the time OCIStmtExecute takes to run the array-bind of 10,000 points. It's exactly the same code with or without an index).
Can anyone explain the 50x insert performance drop?
Any suggestion on how to improve the performance of this scenario?
To provide another data point, creating the index itself on a populated table (with the same 10,000 points) takes less than 1 second, which is consistent with the commit speeds I'm seeing, and thus puzzles me all the more regarding this 10s insert overhead...
SQL> set timing on
SQL> select count(*) from within_point_distance_tab;
COUNT(*)
10000
Elapsed: 00:00:00.01
SQL> CREATE INDEX with6CDF1526$point$idx
2 ON within_point_distance_tab(point)
3 INDEXTYPE IS MDSYS.SPATIAL_INDEX
4 PARAMETERS ('layer_gtype=POINT');
Index created.
Elapsed: 00:00:00.96
SQL> drop index WITH6CDF1526$POINT$IDX force;
Index dropped.
Elapsed: 00:00:00.57
SQL> CREATE INDEX with6CDF1526$point$idx
2 ON within_point_distance_tab(point)
3 INDEXTYPE IS MDSYS.SPATIAL_INDEX
4 PARAMETERS ('layer_gtype=POINT SDO_DML_BATCH_SIZE=10000');
Index created.
Elapsed: 00:00:00.98
SQL>Thanks for your input. We are likely to use partioning down the line, but what you are describing (partition exchange) is currently beyond my abilities in plain SQL, and how this could be accomplished from an OCI client application without affecting other users and keep the transaction boundaries sounds far from trivial. (i.e. can it made transparent to the client application, and does it require privileges the client does have???). I'll have to investigate this further though, and this technique sounds like one accessible to a DBA only, not from a plain client app with non-privileged credentials.
The thing that I fail to understand though, despite your explanation, is why the slow down is not entirely on the commit. After all, documentation for the SDO_DML_BATCH_SIZE parameter of the Spatial index implies that the index is updated on commit only, where new rows are fed 1,000 or 10,000 at a time to the indexing engine, and I do see time being spent during commit, but it's the geometry insert that slow down the most, and that to me looks quite strange.
It's so much slower that it's as if each geometry was indexed one at a time, when I'm doing a single insert with an array bind (i.e. equivalent to a bulk operation in PL/SQL), and if so much time is spend during the insert, then why is any time spent during the commit. In my opinion it's one or the other, but not both. What am I missing? --DD -
Performance issue when inserting into spatial indexed table with JDBC
We have a table named 'feature' which has a "sdo_geometry" column, and we created spatial index on that column,
CREATE TABLE feature ( id number, desc varchar, oshape sdo_gemotry)
CREATE INDEX feature_sp_idx ON feature(oshape) INDEXTYPE IS MDSYS.SPATIAL_INDEX;
Then we executed the following SQL to insert about 800 records into that table(We tried this by using DB visualizer and
our Java application, both of them were using JDBC driver to connect to the oracle 11gR2 database) .
insert into feature(id,desc,oshape) values (1001,xxx,xxxxx);
insert into feature (id,desc,oshape) values (1002,xxx,xxxxx);
insert into feature (id,desc,oshape) values (1800,xxx,xxxxx);
We encoutered the same problem as this topic
Performance of insert with spatial index
It takes nearly 1 secs for inserting one record,compare to 50 records inserted per sec without spatial index,
which is 50x drop in peformance when doing insertion with spatial index.
However, when we copy and paste those insertion scripts into Oracle Client(same test and same table with spatial index), we got a totally different performance result:
more than 50 records inserted in 1 secs, just as fast as the insertion without building spatial index.
Is it because Oracle Client is not using JDBC? Perhaps JDBC was got something wrong when updating those spatial indexed tables.
Edited by: 860605 on 19/09/2011 18:57
Edited by: 860605 on 19/09/2011 18:58
Edited by: 860605 on 19/09/2011 19:00Normally JDBC use auto-commit. So each insert can causes a commit.
I don't know about Oracle Client. In sqlplus, insert is just a insert,
and you execute "commit" to explicitly commit your changes.
So maybe this is the reason. -
Poor performance and high number of gets on seemingly simple insert/select
Versions & config:
Database : 10.2.0.4.0
Application : Oracle E-Business Suite 11.5.10.2
2 node RAC, IBM AIX 5.3Here's the insert / select which I'm struggling to explain why it's taking 6 seconds, and why it needs to get > 24,000 blocks:
INSERT INTO WF_ITEM_ATTRIBUTE_VALUES ( ITEM_TYPE, ITEM_KEY, NAME, TEXT_VALUE,
NUMBER_VALUE, DATE_VALUE ) SELECT :B1 , :B2 , WIA.NAME, WIA.TEXT_DEFAULT,
WIA.NUMBER_DEFAULT, WIA.DATE_DEFAULT FROM WF_ITEM_ATTRIBUTES WIA WHERE
WIA.ITEM_TYPE = :B1
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 4 0
Execute 2 3.44 6.36 2 24297 198 36
Fetch 0 0.00 0.00 0 0 0 0
total 3 3.44 6.36 2 24297 202 36
Misses in library cache during parse: 1
Misses in library cache during execute: 2Also from the tkprof output, the explain plan and waits - virtually zero waits:
Rows Execution Plan
0 INSERT STATEMENT MODE: ALL_ROWS
0 TABLE ACCESS MODE: ANALYZED (BY INDEX ROWID) OF 'WF_ITEM_ATTRIBUTES' (TABLE)
0 INDEX MODE: ANALYZED (RANGE SCAN) OF 'WF_ITEM_ATTRIBUTES_PK' (INDEX (UNIQUE))
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
library cache lock 12 0.00 0.00
gc current block 2-way 14 0.00 0.00
db file sequential read 2 0.01 0.01
row cache lock 24 0.00 0.01
library cache pin 2 0.00 0.00
rdbms ipc reply 1 0.00 0.00
gc cr block 2-way 4 0.00 0.00
gc current grant busy 1 0.00 0.00
********************************************************************************The statement was executed 2 times. I know from slicing up the trc file that :
exe #1 : elapsed = 0.02s, query = 25, current = 47, rows = 11
exe #2 : elapsed = 6.34s, query = 24272, current = 151, rows = 25
If I run just the select portion of the statement, using bind values from exe #2, I get small number of gets (< 10), and < 0.1 secs elapsed.
If I make the insert into an empty, non-partitioned table, I get :
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.01 0.08 0 137 53 25
Fetch 0 0.00 0.00 0 0 0 0
total 2 0.01 0.08 0 137 53 25and same explain plan - using index range scan on WF_Item_Attributes_PK.
This problem is part of testing of a database upgrade and country go-live. On a 10.2.0.3 test system (non-RAC), the same insert/select - using the real WF_Item_Attributes_Value table takes :
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.10 10 27 136 25
Fetch 0 0.00 0.00 0 0 0 0
total 2 0.00 0.10 10 27 136 25So I'm struggling to understand why the performance on the 10.2.0.4 RAC system is so much worse for this query, and why it's doing so many gets. Suggestions, thoughts, ideas welcomed.
I've verified system level things - CPUs weren't/aren't max'd out, no significant paging/swapping activity, run queue not long. AWR report for the time period shows nothing unusual.
further info on the objects concerned:
query source table :
WF_Item_Attributes_PK : unique index on Item_Type, Name. Index has 144 blocks, non-partitioned
WF_Item_Attributes tbl : non-partitioned, 160 blocks
insert destination table:
WF_Item_Attribute_Values:
range partitioned on Item_Type, and hash sub-partitioned on Item_Key
both executions of the insert hit the partition with the most data : 127,691 blocks total ; 8 sub-partitions with 15,896 to 16,055 blocks per sub-partition.
WF_Item_Attribute_Values_PK : unique index on columns Item_Type, Item_Key, Name. Range/hash partitioned as per table.
Bind values:
exe #1 : Item_Type (:B1) = OEOH, Item_Key (:B2) = 1048671
exe #2 : Item_Type (:B1) = OEOL, Item_Key (:B2) = 4253168
number of rows in WF_Item_Attribute_Values for Item_Type = OEOH : 1132587
number of rows in WF_Item_Attribute_Values for Item_Type = OEOL : 18763670
The non-RAC 10.2.0.3 test system (clone of Production from last night) has higher row counts for these 2.
thanks and regards
Ivanhi Sven,
Thanks for your input.
1) I guess so, but I haven't lifted the lid to delve inside the form as to which one. I don't think it's the cause though, as I got poor performance running the insert statement with my own value (same statement, using my own bind value).
2) In every execution plan I've seen, checked, re-checked, it uses a range scan on the primary key. It is the most efficient I think, but the source table is small in any case - table 160 blocks, PK index 144 blocks. So I think it's the partitioned destination table that's the problem - but we only see this issue on the 10.2.0.4 pre-production (RAC) system. The 10.2.0.3 (RAC) Production system doesn't have it. This is why it's so puzzling to me - the source table read is fast, and does few gets.
3) table storage details below - the Item_Types being used were 'OEOH' (fast execution) and 'OEOL' (slow execution). Both hit partition WF_ITEM49, hence I've only expanded the subpartition info for that one (there are over 600 sub-partitions).
============= From DBA_Part_Tables : Partition Type / Count =============
PARTITI SUBPART PARTITION_COUNT DEF_TABLESPACE_NAME
RANGE HASH 77 APPS_TS_TX_DATA
1 row selected.
============= From DBA_Tab_Partitions : Partition Names / Tablespaces =============
Partition Name TS Name High Value High Val Len
WF_ITEM1 APPS_TS_TX_DATA 'A1' 4
WF_ITEM2 APPS_TS_TX_DATA 'AM' 4
WF_ITEM3 APPS_TS_TX_DATA 'AP' 4
WF_ITEM47 APPS_TS_TX_DATA 'OB' 4
WF_ITEM48 APPS_TS_TX_DATA 'OE' 4
WF_ITEM49 APPS_TS_TX_DATA 'OF' 4
WF_ITEM50 APPS_TS_TX_DATA 'OK' 4
WF_ITEM75 APPS_TS_TX_DATA 'WI' 4
WF_ITEM76 APPS_TS_TX_DATA 'WS' 4
WF_ITEM77 APPS_TS_TX_DATA MAXVALUE 8
77 rows selected.
============= From dba_part_key_columns : Partition Columns =============
NAME OBJEC Column Name COLUMN_POSITION
WF_ITEM_ATTRIBUTE_VALUES TABLE ITEM_TYPE 1
1 row selected.
PPR1 sql> @q_tabsubpart wf_item_attribute_values WF_ITEM49
============= From DBA_Tab_SubPartitions : SubPartition Names / Tablespaces =============
Partition Name SUBPARTITION_NAME TS Name High Value High Val Len
WF_ITEM49 SYS_SUBP3326 APPS_TS_TX_DATA 0
WF_ITEM49 SYS_SUBP3328 APPS_TS_TX_DATA 0
WF_ITEM49 SYS_SUBP3332 APPS_TS_TX_DATA 0
WF_ITEM49 SYS_SUBP3331 APPS_TS_TX_DATA 0
WF_ITEM49 SYS_SUBP3330 APPS_TS_TX_DATA 0
WF_ITEM49 SYS_SUBP3329 APPS_TS_TX_DATA 0
WF_ITEM49 SYS_SUBP3327 APPS_TS_TX_DATA 0
WF_ITEM49 SYS_SUBP3325 APPS_TS_TX_DATA 0
8 rows selected.
============= From dba_part_key_columns : Partition Columns =============
NAME OBJEC Column Name COLUMN_POSITION
WF_ITEM_ATTRIBUTE_VALUES TABLE ITEM_KEY 1
1 row selected.
from DBA_Segments - just for partition WF_ITEM49 :
Segment Name TSname Partition Name Segment Type BLOCKS Mbytes EXTENTS Next Ext(Mb)
WF_ITEM_ATTRIBUTE_VALUES @TX_DATA SYS_SUBP3332 TblSubPart 16096 125.75 1006 .125
WF_ITEM_ATTRIBUTE_VALUES @TX_DATA SYS_SUBP3331 TblSubPart 16160 126.25 1010 .125
WF_ITEM_ATTRIBUTE_VALUES @TX_DATA SYS_SUBP3330 TblSubPart 16160 126.25 1010 .125
WF_ITEM_ATTRIBUTE_VALUES @TX_DATA SYS_SUBP3329 TblSubPart 16112 125.875 1007 .125
WF_ITEM_ATTRIBUTE_VALUES @TX_DATA SYS_SUBP3328 TblSubPart 16096 125.75 1006 .125
WF_ITEM_ATTRIBUTE_VALUES @TX_DATA SYS_SUBP3327 TblSubPart 16224 126.75 1014 .125
WF_ITEM_ATTRIBUTE_VALUES @TX_DATA SYS_SUBP3326 TblSubPart 16208 126.625 1013 .125
WF_ITEM_ATTRIBUTE_VALUES @TX_DATA SYS_SUBP3325 TblSubPart 16128 126 1008 .125
WF_ITEM_ATTRIBUTE_VALUES_PK @TX_IDX SYS_SUBP3332 IdxSubPart 59424 464.25 3714 .125
WF_ITEM_ATTRIBUTE_VALUES_PK @TX_IDX SYS_SUBP3331 IdxSubPart 59296 463.25 3706 .125
WF_ITEM_ATTRIBUTE_VALUES_PK @TX_IDX SYS_SUBP3330 IdxSubPart 59520 465 3720 .125
WF_ITEM_ATTRIBUTE_VALUES_PK @TX_IDX SYS_SUBP3329 IdxSubPart 59104 461.75 3694 .125
WF_ITEM_ATTRIBUTE_VALUES_PK @TX_IDX SYS_SUBP3328 IdxSubPart 59456 464.5 3716 .125
WF_ITEM_ATTRIBUTE_VALUES_PK @TX_IDX SYS_SUBP3327 IdxSubPart 60016 468.875 3751 .125
WF_ITEM_ATTRIBUTE_VALUES_PK @TX_IDX SYS_SUBP3326 IdxSubPart 59616 465.75 3726 .125
WF_ITEM_ATTRIBUTE_VALUES_PK @TX_IDX SYS_SUBP3325 IdxSubPart 59376 463.875 3711 .125
sum 4726.5
[the @ in the TS Name is my shortcode, as Apps stupidly prefixes every ts with "APPS_TS_"]
The Tablespaces used for all subpartitions are UNIFORM extent mgmt, AUTO segment_space_management ; LOCAL extent mgmt.regards
Ivan -
Performance problem at bulk insert with spatial index
Hi,
I have a table with SDO_GEOMETRY.
Insert without spatial index is very fast, but with active spatial index it's very slow.
So for the first big import of data, I can drop the index, import the data and again create the index. Thats 10 times faster!
But for an already very big table that is no option.
The 10g1-Users Guide (1) says at 4.1.3 that the spatial index should be set to 'deferred', the data should be inserted and than the index should be synchronized again. That sounds very good, but I can't find this at the 11g1-Users Guide.
I tried it (11g1), but the performance is even worse than with active index!
What could be my mistake? Any hints?
Thank you,
Bjoern Weitzig
create table sggeoptcollection (pt SDO_GEOMETRY);
CREATE INDEX myIdx ON sggeoptcollection (pt) INDEXTYPE IS MDSYS.SPATIAL_INDEX PARAMETERS('sdo_indx_dims=2, layer_gtype=point, sdo_rtr_pctfree=50');
ALTER INDEX myIdx PARAMETERS ('index_status=deferred');
Big import with batch'ed PreparedStatements
ALTER INDEX myIdx PARAMETERS ('index_status=synchronize sdo_batch_size=500');
1) http://download.oracle.com/docs/html/B10826_01/sdo_index_query.htm#g1010227Hi,
I have a table with SDO_GEOMETRY.
Insert without spatial index is very fast, but with active spatial index it's very slow.
So for the first big import of data, I can drop the index, import the data and again create the index. Thats 10 times faster!
But for an already very big table that is no option.
The 10g1-Users Guide (1) says at 4.1.3 that the spatial index should be set to 'deferred', the data should be inserted and than the index should be synchronized again. That sounds very good, but I can't find this at the 11g1-Users Guide.
I tried it (11g1), but the performance is even worse than with active index!
What could be my mistake? Any hints?
Thank you,
Bjoern Weitzig
create table sggeoptcollection (pt SDO_GEOMETRY);
CREATE INDEX myIdx ON sggeoptcollection (pt) INDEXTYPE IS MDSYS.SPATIAL_INDEX PARAMETERS('sdo_indx_dims=2, layer_gtype=point, sdo_rtr_pctfree=50');
ALTER INDEX myIdx PARAMETERS ('index_status=deferred');
Big import with batch'ed PreparedStatements
ALTER INDEX myIdx PARAMETERS ('index_status=synchronize sdo_batch_size=500');
1) http://download.oracle.com/docs/html/B10826_01/sdo_index_query.htm#g1010227 -
DML operations performance on table indexed with CTXCAT
Hi,
I have a table with 2M records. The table is batch updated once a day, and the number of record movements (update/delete/insert) should be around 100K.
The table is indexed with CTXCAT.
If I create the index from scratch, it takes 5minutes.
If I perform delete/insert/update operations involving 40K records, it takes a lot more (especially for delete and update operations, something like 30 minutes).
In this particular case I can drop the index and recreate it from scratch every night. The problem is that the 2M records table is only the first step in adoption of Oracle Text. The next step will be a 40M records table, on which the initial index creation takes something like 2hours (so I can't rebuild it every night).
Do you have any suggest?
Thanks.
-- table DDL
CREATE TABLE TAHZVCON_TEXT
CONSUMER_ID NUMBER(10) NOT NULL,
COMPANY_NAME VARCHAR2(4000 CHAR),
CITY VARCHAR2(30 BYTE),
PROVINCE VARCHAR2(3 CHAR),
POST_CODE VARCHAR2(10 BYTE)
CREATE UNIQUE INDEX TAHZVCON_TEXT_PK ON TAHZVCON_TEXT (CONSUMER_ID);
begin
ctx_ddl.drop_preference('mylex');
ctx_ddl.create_preference('mylex', 'BASIC_LEXER');
ctx_ddl.set_attribute('mylex', 'printjoins', '.#');
ctx_ddl.set_attribute('mylex', 'base_letter', 'YES');
ctx_ddl.set_attribute('mylex', 'index_themes','NO');
ctx_ddl.set_attribute('mylex', 'index_text','YES');
ctx_ddl.set_attribute('mylex', 'prove_themes','NO');
ctx_ddl.drop_preference('mywordlist');
ctx_ddl.create_preference('mywordlist', 'BASIC_WORDLIST');
ctx_ddl.set_attribute('mywordlist','stemmer','NULL');
ctx_ddl.set_attribute('mywordlist','SUBSTRING_INDEX', 'NO');
ctx_ddl.set_attribute('mywordlist','PREFIX_INDEX','NO');
ctx_ddl.drop_index_set('tahzvcon_iset');
ctx_ddl.create_index_set('tahzvcon_iset');
ctx_ddl.add_index('tahzvcon_iset','city');
ctx_ddl.add_index('tahzvcon_iset','province');
ctx_ddl.add_index('tahzvcon_iset','post_code');
end;
CREATE INDEX TAHZVCON_TEXT_TI01 ON TAHZVCON_TEXT(COMPANY_NAME)
INDEXTYPE IS CTXSYS.CTXCAT
PARAMETERS ('lexer mylex wordlist mywordlist index set tahzvcon_iset')
PARALLEL 8;
AndreaHi kevinUCB,
I've decided to use CTXCAT indexes because I had to perform queries involving different columns (company name, city, region, etc.). So I thought CTXCAT was the right index for me.
Now I've discovered that if I index an XML with CONTEXT, I can perform a search on single XML fields, so CONTEXT is suitable for my needs.
Preliminary test on the 2M record table looks very good.
Bye,
Andrea -
Tip: Improving The Performance Of OLAP DML Table Inserts
Quick Oracle OLAP Tip:
If you need to write the contents of a variable, or a group of variables, to a relational table, you would normally use the SQL INSERT command. Normal practice is to loop round all of the variables' dimension values, inserting the variable values into the relational tables one by one, until the variable has been completely loaded into your database table.
Oracle 9i OLAP however introduces two new commands, SQL PREPARE and SQL EXECUTE (http://download-west.oracle.com/docs/cd/B10501_01/olap.920/a95298/sql5.htm#1027902) , that allow us to prepare our INSERT statement in such a way that it uses bind variables to pass values to the Oracle tables. Bind variables are generally a 'good thing' and reduce the amount of time Oracle has to spend parsing your SQL insert statements. In addition, you can specify additional options with SQL PREPARE to specify 'direct path' insertions (quicker as they bypass the normal SQL engine and directly load data into Oracle blocks), nologging (to eliminate redo log generation), and to nominate individual partitions to load data in to. It's worth noting that there's an error in the current OLAP DML documentation that suggest that any OLAP DML insert operation into an Oracle table locks the entire table, preventing other AW processes from inserting into the table until you commit. This is actually incorrect, and full-table locking only occurs if you use the DIRECT=YES option, which locks the table in the same way that SQL*Loader locks the table as they both use the Direct Path API.
However, an even better solution than using SQL EXECUTE and SQL PREPARE is to use the OLAP_TABLE feature in Oracle 9i (http://download-west.oracle.com/docs/cd/B10501_01/olap.920/a95295/olap_tab.htm#73729) to create a view against your AW variable, then use this view as the source for a "INSERT INTO table SELECT * FROM source" SQL statement, optionally using the /*+ INSERT APPEND */ option if you want to carry out direct path insertions. By using OLAP TABLE and having the SQL engine insert multiple variable values into our target table, rather than having an OLAP DML program loop through the variable and carry out multiple single-row insertions, we were able to increase our write performance by an order of magnitude compared to our earlier SQL INSERT command. One thing to bear in mind though is that, if you are running many copies of the program concurrently, using direct path insertions may well cause lock contention, as each process will obtain an exclusive table lock while the direct insertion takes place. In the case of concurrent processes, it may be better to use conventional path insertions (but still use SQL PREPARE and EXECUTE, or OLAP TABLE) as these only require row-level exclusive locks.you can use Execution plain
http://stackoverflow.com/questions/7359702/how-do-i-obtain-a-query-execution-plan
and add index.
Index according to the fields you ask queries can improve performance greatly larger!
You can use the statistics for building indexes:
http://www.mssqltips.com/sqlservertip/2979/querying-sql-server-index-statistics/
Tzuri Ben Ezra | My Certifications:
CompTIA A+ ,Microsoft MCP, MCTS, MCSA, MCITP |
FaceBook: Tzuri FaceBook | vCard:
Tzuri vCard |
Microsoft ID:
Microsoft Transcript
| -
Insert performance issue with Partitioned Table.....
Hi All,
I have a performance issue during with a table which is partitioned. without table being partitioned
it ran in less time but after partition it took more than double.
1) The table was created initially without any partition and the below insert took only 27 minuts.
Total Rec Inserted :- 2424233
PL/SQL procedure successfully completed.
Elapsed: 00:27:35.20
2) Now I re-created the table with partition(range yearly - below) and the same insert took 59 minuts.
Is there anyway i can achive the better performance during insert on this partitioned table?
[ similerly, I have another table with 50 Million records and the insert took 10 hrs without partition.
with partitioning the table, it took 18 hours... ]
SQL> select * from table(dbms_xplan.display);
PLAN_TABLE_OUTPUT
Plan hash value: 4195045590
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 643K| 34M| | 12917 (3)| 00:02:36 |
|* 1 | HASH JOIN | | 643K| 34M| 2112K| 12917 (3)| 00:02:36 |
| 2 | VIEW | index$_join$_001 | 69534 | 1290K| | 529 (3)| 00:00:07 |
|* 3 | HASH JOIN | | | | | | |
| 4 | INDEX FAST FULL SCAN| PK_ACCOUNT_MASTER_BASE | 69534 | 1290K| | 181 (3)| 00:00
| 5 | INDEX FAST FULL SCAN| ACCOUNT_MASTER_BASE_IDX2 | 69534 | 1290K| | 474 (2)| 00:00
PLAN_TABLE_OUTPUT
| 6 | TABLE ACCESS FULL | TB_SISADMIN_BALANCE | 2424K| 87M| | 6413 (4)| 00:01:17 |
Predicate Information (identified by operation id):
1 - access("A"."VENDOR_ACCT_NBR"=SUBSTR("B"."ACCOUNT_NO",1,8) AND
"A"."VENDOR_CD"="B"."COMPANY_NO")
3 - access(ROWID=ROWID)
Open C1;
Loop
Fetch C1 Bulk Collect Into C_Rectype Limit 10000;
Forall I In 1..C_Rectype.Count
Insert test
col1,col2,col3)
Values
val1, val2,val3);
V_Rec := V_Rec + Nvl(C_Rectype.Count,0);
Commit;
Exit When C_Rectype.Count = 0;
C_Rectype.delete;
End Loop;
End;
Total Rec Inserted :- 2424233
PL/SQL procedure successfully completed.
Elapsed: 00:51:01.22
Edited by: user520824 on Jul 16, 2010 9:16 AMI'm concerned about the view in step 2 and the index join in step 3. A composite index with both columns might eliminate the index join and result in fewer read operations.
If you know which partition the data is going into beforehand you can save a little bit of processing by specifying the partition (which may not be a scalable long-term solution) in the insert - I'm not 100% sure you can do this on inserts but I know you can on selects.
The APPEND hint won't help the way you are using it - the VALUES clause in an insert makes it be ignored. Where it is effective and should help you is if you can do the insert in one query - insert into/select from. If you are using the loop to avoid filling up undo/rollback you can use a bulk collect to batch the selects and commit accordingly - but don't commit more often than you have to because more frequent commits slow transactions down.
I don't think there is a nologging hint :)
So, try something like
insert /*+ hints */ into ...
Select
A.Ing_Acct_Nbr, currency_Symbol,
Balance_Date, Company_No,
Substr(Account_No,1,8) Account_No,
Substr(Account_No,9,1) Typ_Cd ,
Substr(Account_No,10,1) Chk_Cd,
Td_Balance, Sd_Balance,
Sysdate, 'Sisadmin'
From Ideaal_Cons.Tb_Account_Master_Base A,
Ideaal_Staging.Tb_Sisadmin_Balance B
Where A.Vendor_Acct_Nbr = Substr(B.Account_No,1,8)
And A.Vendor_Cd = b.company_no
;Edited by: riedelme on Jul 16, 2010 7:42 AM -
Poor performances on tables FAGL_011ZC, FAGL_011PC and FAGL_011VC
We note from some time on a very very poor performances of some programme accessing these tables.
The tables are very small, still standard.
We see these tables does not have particular indexes, and their buffering is "allowed but disabled".
In you opinion can we activate the buffering of these tables ? Are there controindications ?
Our system is ECC5 quite aligned as SP level.Client is looking upon TDMS (Test Data Migration Server) to replicate the PRD data into lower systems.
-
Unable to perform "insert into table select ...." using DBF_JDBC30
below i post my code :
import java.util.*;
import java.sql.*;
import java.io.*;
import com.hxtt.sql.*;
public class backofficeDbfVerification {
public backofficeDbfVerification(String path) throws SQLException {
Properties prop = new Properties();
prop.setProperty("user", "");
prop.setProperty("OtherExtensions","true");
prop.setProperty("Version Number", "03");
try {
Class.forName("com.hxtt.sql.dbf.DBFDriver").newInstance();
connection = DriverManager.getConnection("jdbc:DBF:/"+path, prop);
catch (ClassNotFoundException classnotfoundexception) {
System.out.println("could ot find HXTT class, make sure the classpath have been defined in your system !");
catch(Exception e) {
System.out.println(e.getMessage());
public void createBadRecord(String table, String newtable, int blth) throws SQLException {
Statement stmt = connection.createStatement();
String insert = "insert into \""+newtable+"\" select * from \""+table+"\"";
boolean bInsert = stmt.execute(insert);
stmt.close();
public static void main (String args[]) {
String path = "C:\\MASTER\\SOURCEDATA\\AREA";
try {
backofficeDbfVerification test = new backofficeDbfVerification(path);
test.createBadRecord("source\\area\\123.AA2","source\\area\\others\\89964568.AA1");
catch(SQLException sqx) {
System.out.println("Error "+sqx.getMessage());
i try to perform simple query : insert into tablea select from tableb which has same structure. the source table has only 9 rows, but the process ran very long time, no error message raised,it just run and never end (which make me upset for the whole day).
i have tried another simple query : insert into table1 values (1,2) using the same driver and execute successfully.
is it becoz of function limitation since i used evaluation copy ?Copy the answer from HXTT's support forum:
No function limitation. I just tested:
create table testa1 select * from test;
insert into testa1 select * from test;
Passed.
I also run your backofficeDbfVerification.java sample. Passed too. I'm using the same code as you. The difference is only my 89964568.AA1 and 123.AA2 is simulative tables files. If possible, please zip your 89964568.AA1 and 123.AA2 and email to [email protected] Thanks.
BTW, I pasted the little modified backofficeDbfVerification.java below:
import java.sql.*;
import java.util.*;
public class backofficeDbfVerification {
Connection connection;
public backofficeDbfVerification(String path) throws SQLException {
Properties prop = new Properties();
prop.setProperty("user", "");
prop.setProperty("OtherExtensions", "true");
prop.setProperty("Version Number", "03");
try {
Class.forName("com.hxtt.sql.dbf.DBFDriver").newInstance();
connection = DriverManager.getConnection("jdbc:DBF:/" + path, prop);
catch (ClassNotFoundException classnotfoundexception) {
System.out.println("could ot find HXTT class, make sure the classpath have been defined in your system !");
catch (Exception e) {
System.out.println(e.getMessage());
public void createBadRecord(String table, String newtable) throws
SQLException {
Statement stmt = connection.createStatement();
String insert = "insert into \"" + newtable + "\" select * from \"" +
table + "\"";
boolean bInsert = stmt.execute(insert);
stmt.close();
public static void main(String args[]) {
String path = "f:\\dbffiles";
try {
backofficeDbfVerification test = new backofficeDbfVerification(path);
test.createBadRecord("source\\area\\123.AA2",
"source\\area\\others\\89964568.AA1");
catch (SQLException sqx) {
System.out.println("Error " + sqx.getMessage());
} -
Insert in table with unique index
Hi
I Create a table save a factor for to calculate date, but other 2 columns are key table
CREATE TABLE TMP_FATOR
SETID VARCHAR2(5 BYTE) NOT NULL,
COMPANYID VARCHAR2(15 BYTE) NOT NULL,
FATOR NUMBER
CREATE UNIQUE INDEX IDX_TMP_FATOR ON TMP_FATOR
(SETID, COMPANYID)
NOLOGGINGI want to insert in table , but skip errors , I tried with
declare
i number;
begin
i:=1;
EXECUTE IMMEDIATE 'TRUNCATE TABLE SYSADM.TMP_FATOR';
BEGIN
INSERT INTO /*+ APPEND*/ SYSADM.TMP_FATOR
SELECT T1.SETID,
T1.COMPANYID,
SYSADM.pkg_ajusta_kenan.fnc_fator_dias_desconto(T1.SETID,T1.COMPANYID) fator
FROM SYSADM.PS_LOC_ITEM_SN T1;
EXCEPTION
WHEN DUP_VAL_ON_INDEX THEN
NULL;
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE(SQLERRM);
END;
COMMIT;
end;But did not work
Why ?The determinisic keyword is just part of the
declaration whether declaring a standalone function
or a packaged function.
SCOTT @ nx102 Local> create package test_pkg
2 as
3 function determin_foo( p_arg in number )
4 return number
5 deterministic;
6 end;
7 /
Package created.
Elapsed: 00:00:00.34
1 create or replace package body test_pkg
2 as
3 function determin_foo( p_arg in number )
4 return number
5 deterministic
6 is
7 begin
8 return p_arg - 1;
9 end;
0* end;
SCOTT @ nx102 Local> /
Package body created.
Elapsed: 00:00:00.14JustinCan I to have other procedures and functions inside pacckage ? -
Poor performance for the 1st select everyday for AFRU table
Hello everyone, I have performance problems with AFRU table. Every day, the first time I run a "Z" transaction, it takes around 100-120 seconds, but the second time and following it only takes four seconds. What could I do, in order to reduce the first execution time?
This is the select:
SELECT * FROM AFRU WHERE MANDT = :A0 AND CATSBELNR = :A1 AND BUDAT = :A2 AND PERNR = :A3 AND STOKZ <> :A4 AND STZHL = :A5
The execution plan for this select takes index AFRU~ZCA with an acceptable cost of 6.319. Index AFRU~ZCA is a nonunique index with these colums: MANDT + CATSBELNR + BUDAT + PERNR
I'll appreciate any ideas.
Thanks in advance,
Santi.What database system are you using (ASE, Oracle, etc?).
If ASE, for the general issue of the first exection of a query taking longer, the two most likely reasons would be
a) the table's data has aged out of cache so the query has to do a lot of physical i/o to read the data back into cache
or
b) the query plan for the query has aged out of statement cache and needs to be recompiled.
This query looks pretty simple, so the data cache seems much more likely.
To get a better feel, some morning run the query with
set statistics io on
set statistics time on
then run it again and look for differences in the physical vs logical i/o numbers and compile vs execution times.
You could use a scheduled event (Job Scheduler, cron job) to run the query or some query very like it a little earlier in the day to prime the data cache with the table data. -
Nologging direct-path insert into an indexed table
Hello,
Does anyone have an idea how I can suppress generation of undo logs for direct-path insert into an indexed table on 11.2.0.1.0:
CREATE TABLE TBL(ID NUMBER) NOLOGGING;
CREATE INDEX IDX ON TBL(ID) NOLOGGING;
INSERT /*+ APPEND */ INTO TBL SELECT /*+ APPEND */ ROWNUM FROM ...; -- Source table has 400,000,000+ rows
Regards,
Angel TsankovPl do not post duplicates - Why does Oracle not use direct-path insert when instructed to do so - pl continue the discussion in your original thread
-
On submit perform an insert on one table and an update on aother table
I am trying to perform and insert on the table one table (the wizard created my form the insert is going against the table that I created using the wizard) and on the form is on field that is also in another table. Therefore, I am trying to perform an update on one attribute of one table and a insert into another table. How do I do this in apex?
If you have used wizard to create form, then you may see a process of type 'Automatic Row Processing (DML)' in your page which will perform INSERT/UPDATE/DELETE on your form table. Here you can see APEX performs INSERT only when REQUEST is in 'INSERT, CREATE, CREATE_AGAIN, CREATEAGAIN'
So create one more PL/SQL page process which will execute at 'on Submit after validations' and write update process as follows
begin
-- pseudo table/columns
update tbl_second
set col1 = :p1_item
where pk_col = :p1_pk_item;
end;Make this process conditional so that it will perform UPDATE only when request value is in 'INSERT, CREATE, CREATE_AGAIN, CREATEAGAIN' ( i.e. only when you are inserting into your form table)
Cheers,
Hari
p.s. I think you may also need to update the second table when some-one updates your form table.
Edited by: Hari_639 on Oct 26, 2009 9:46 AM -
Different Performance for a view/table
Hi,
I have a view called "Myview" which has a poor performance on one database (DBTEST) but with good performance on another database (DBDEV)
I checked the indexes on both and all of them were in place on both databases.
DBTEST and DBDEV both installed on the same Unix machine (They share the same resources).
Since both databases are configured similarly I'm wondering why querying Myview view takes two times more to return records?
How can I identify where the problem is? The "consistent gets" and "physical reads" parameters are about 2 times more in DBTEST when I query the view. I believe this is why I have poor performance in DBTEST.
Could someone give me an advice on what db parameters I should verify to identify the problem?
DBTEST> select status from Myview where id =100;
elapse time: 40 seconds
DBDEV> select status from Myview where id =100;
elapse time: 22 seconds
DBTEST> select count(*) from Myview;
5123 rows selected
DBDEV> select count(*) from Myview;
4022 rows selected
Thanks,
AmirThere are 13 tables plus one view underlying the Myview
The tables which are not listed, are lookup tables and contain equal number of rows on both DBs:
DBDEV
TableName No of Rows
user_role 3023
project 2059
project_year 647
doc_tab 3091
user 3155
org 2639
region 125
application 3353
DBDEV
TableName No of Rows
user_role 6362
project 5058
project_year 1516
doc_tab 8659
user 6936
org 6320
region 176
application 7325
Since Myview is using UNION clause I picked part of the execution plan:
DBDEV:
11 rows selected.
Elapsed: 00:00:16.01
Execution Plan
SELECT STATEMENT Optimizer=CHOOSE (Cost=525 Card=3 Bytes=111)
VIEW OF 'Myview' (Cost=525 Card=3 Bytes=111)
SORT (UNIQUE) (Cost=560 Card=3 Bytes=1103)
UNION-ALL
HASH JOIN (ANTI) (Cost=138 Card=1 Bytes=369)
HASH JOIN (Cost=135 Card=1 Bytes=356)
NESTED LOOPS (Cost=132 Card=1 Bytes=348)
NESTED LOOPS (OUTER) (Cost=131 Card=1 Bytes=330)
NESTED LOOPS (OUTER) (Cost=130 Card=1 ytes=308)
NESTED LOOPS (OUTER) (Cost=129 Card=1 Bytes=295)
FILTER
NESTED LOOPS (OUTER)
HASH JOIN (Cost=128 Card=1 Bytes=175)
VIEW OF 'Myview_PROJ_ALL_YEAR'
(Cost=123 Card=15 Bytes=2295)
MERGE JOIN (Cost=123 Card=15 Bytes=1935)
SORT (JOIN) (Cost=119 Card=529 Bytes=61893)
HASH JOIN (Cost=107 Card=529 Bytes=61893)
VIEW OF 'Myview_PROJECT' (Cost=100 Card=529 Bytes=44436)
SORT (UNIQUE) (Cost=100 Card=529 Bytes=40998)
UNION-ALL
HASH JOIN (Cost=9 Card=51 Bytes=2703)
TABLE ACCESS (FULL) OF 'GRANT_PROGRAM' (Cost=2 Card=15 Bytes=135)
TABLE ACCESS (FULL) OF 'PROJECT' (Cost=6 Card=51 Bytes=2244)
HASH JOIN (Cost=48 Card=129 Bytes=11610)
HASH JOIN (Cost=41 Card=127 Bytes=9779)
HASH JOIN (Cost=29 Card=94 Bytes=5922)
HASH JOIN (Cost=9 Card=51 Bytes=2703)
TABLE ACCESS (FULL) OF 'GRANT_PROGRAM' (Cost=2 Card=15 Bytes=135)
TABLE ACCESS (FULL) OF 'PROJECT' (Cost=6 Card=51 Bytes=2244)
TABLE ACCESS (FULL) OF 'APPLICATION' (Cost=19 Card=3353 Bytes=33530)
INDEX (FAST FULL SCAN) OF 'UK_user_INVOLVE' (UNIQUE) (Cost=11 Card=4527 Bytes=63378)
TABLE ACCESS (FULL) OF 'user_role' (Cost=6 Card=3023 Bytes=39299)
HASH JOIN (Cost=12 Card=298 Bytes=22350)
HASH JOIN (Cost=9 Card=51 Bytes=2907)
TABLE ACCESS (FULL) OF 'GRANT_PROGRAM' (Cost=2 Card=15 Bytes=135)
TABLE ACCESS (FULL) OF 'PROJECT' (Cost=6 Card=51 Bytes=2448)
TABLE ACCESS (FULL) OF 'region' (Cost=2 Card=69 Bytes=1242)
HASH JOIN (Cost=19 Card=51 Bytes=4335)
HASH JOIN (Cost=12 Card=51 Bytes=3366)
HASH JOIN (Cost=9 Card=51 Bytes=2907)
DBINT:
9 rows selected.
Elapsed: 00:00:34.03
Execution Plan
SELECT STATEMENT Optimizer=CHOOSE (Cost=941 Card=3 Bytes=111)
VIEW OF 'Myview' (Cost=941 Card=3 Bytes=111)
SORT (UNIQUE) (Cost=976 Card=3 Bytes=1106)
UNION-ALL
HASH JOIN (ANTI) (Cost=253 Card=1 Bytes=370)
NESTED LOOPS (OUTER) (Cost=250 Card=1 Bytes=357)
NESTED LOOPS (OUTER) (Cost=250 Card=1 Bytes=341)
NESTED LOOPS (OUTER) (Cost=249 Card=1 Bytes=318)
HASH JOIN (Cost=248 Card=1 Bytes=304)
NESTED LOOPS (Cost=245 Card=1 Bytes=296)
HASH JOIN (Cost=243 Card=2 Bytes=556)
FILTER
HASH JOIN (OUTER)
VIEW OF 'Myview_PROJ_ALL_YEAR' (Cost=229 Card=35 Bytes=5355)
MERGE JOIN (Cost=229 Card=35 Bytes=4550)
SORT (JOIN) (Cost=226 Card=1262 Bytes=148916)
HASH JOIN (Cost=198 Card=1262 Bytes=148916)
VIEW OF 'Myview_PROJECT' (Cost=183 Card=1262 Bytes=106008)
SORT (UNIQUE) (Cost=183 (Card=1262 Bytes=100528)
UNION-ALL
HASH JOIN (Cost=15 Card=126 Bytes=6678)
TABLE ACCESS (FULL) OF 'GRANT_PROGRAM' (Cost=2 Card=28 Bytes=252)
TABLE ACCESS (FULL) OF 'PROJECT' (Cost=12 Card=126 Bytes=5544)
HASH JOIN (Cost=98 Card=454 Bytes=41314)
HASH JOIN (Cost=88 Card=448 Bytes=34496)
HASH JOIN (Cost=48 Card=206 Bytes=12978)
HASH JOIN (Cost=15 Card=126 Bytes=6678)
TABLE ACCESS (FULL) OF 'GRANT_PROGRAM' (Cost=2 Card=28 Bytes=252)
TABLE ACCESS (FULL) OF 'PROJECT' (Cost=12 Card=126 Bytes=5544)
TABLE ACCESS (FULL) OF 'APPLICATION' (Cost=32 Card=7325 Bytes=73250)
INDEX (FAST FULL SCAN) OF 'UK_user_INVOLVE' (UNIQUE) (Cost=39 Card=15889
Bytes=222446)
TABLE ACCESS (FULL) OF 'user_role' (Cost=9 Card=6362 Bytes=89068)
HASH JOIN (Cost=18 Card=556 Bytes=41700)
TABLE ACCESS (FULL) OF 'region' (Cost=2 Card=88 Bytes=1584)
HASH JOIN (Cost=15 Card=126 Bytes=7182)
TABLE ACCESS (FULL) OF 'GRANT_PROGRAM' (Cost=2 Card=28 Bytes=252)
TABLE ACCESS (FULL) OF 'PROJECT' (Cost=12 Card=126 Bytes=6048)
HASH JOIN (Cost=28 Card=126 Bytes=10836)
HASH JOIN (Cost=18 Card=126 Bytes=8316)
You see alapse time for querying DBTEST is sometimes 2 times more than DBDEV. BTW, I checked all indexes for both database are in place.
Based on the information provided can you tell me what the problem is?
Thanks,
Maybe you are looking for
-
Word 2007 hyperlinks not showing up in converted PDF
Using Acrobat Standard (Windows) 9.5.2: My Word 2007 hyperlinks not showing up in converted PDFs. This, even though I've got the Preferences set correctly (I believe -- see screenshot). Please help!
-
GRC AC 10: Emergency Access Management, Logon button is disabled (GRAC_SPM)
Hello Gurus, I have configured Emergency Access Management in GRC AC 10. GRC Box (SID) : GR1 client 100 Backend ERP system : D24 client 100 The FIREFIGHTER in GRC system : FFUSER1 Z_SAP_GRAC_SUPERUSER_MGMTUSER Z_SAP_GRC_FN_BASE Z_SAP_GRC_NWBC In the
-
BPA shows domain prepped for Exchange 2007 SP1 after SP3 install
Hello, I have a Client with a domain that is at 2003 domain functional level and 2000 forest level. The client had Exchange 2007 SP1 and I have just upgraded it to SP3. There is one CAS, one Hub Transport, and one Mailbox Server (All three had Exchan
-
I downloaded iOS 6 and now app store is a lot slower.. Why?
-
SQL 2012 cluster installation fails with an IP address error
I am building an eight node cluster and have successfully installed two SQL instances on it. Now, I cannot install any additional instances because of the following error: The resource with identifier 'Type='IP Address' NetworkName='Cluster Network 1