Any general tips on getting better performance out of multi table insert?

Question

I have been struggling with coding a multi table insert which is the first time I ever use one and my Oracle skills are pretty poor in general so now that the query is built and works fine I am sad to see its quite slow.
I have checked numerous articles on optimizing but the things I try dont seem to get me much better performance.
First let me describe my scenario to see if you agree that my performance is slow...
its an insert all command, which ends up inserting into 5 separate tables, conditionally (at least 4 inserts, sometimes 5 but the fifth is the smallest table). Some stats on these tables as follows:
Source table: 5.3M rows, ~150 columns wide. Parallel degree 4. everything else default.
Target table 1: 0 rows, 27 columns wide. Parallel 4. everything else default.
Target table 2: 0 rows, 63 columns wide. Parallel 4. default.
Target table 3: 0 rows, 33 columns wide. Parallel 4. default.
Target table 4: 0 rows, 9 columns wide. Parallel 4. default.
Target table 5: 0 rows, 13 columns wide. Parallel 4. default.
The parallelism is just about the only customization I myself have done. Why 4? I dont know it's pretty arbitrary to be honest.
Indexes?
Table 1 has 3 index + PK.
Table 2 has 0 index + FK + PK.
Table 3 has 4 index + FK + PK
Table 4 has 3 index + FK + PK
Table 5 has 4 index + FK + PK
None of the indexes are anything crazy, maybe 3 or 4 of all of them are on multiple columns, 2-3 max. The rest are on single columns.
The query itself looks something like this:
insert /*+ append */ all
when 1=1 then
into table1 (...) values (...)
into table2 (...) values (...)
when a=b then
into table3 (...) values (...)
when a=c then
into table3 (...) values (...)
when p=q then
into table4(...) values (...)
when x=y then
into table5(...) values (...)
select .... from source_table
Hints I tried are with append, without append, and parallel (though adding parallel seemed to make the query behave in serial, according to my session browser).
Now for the performance:
It does about 8,000 rows per minute on table1. So that means it should also have that much in table2, table3 and table4, and then a subset of that in table5.
Does that seem normal or am I expecting too much?
I find articles talking about millions of rows per minute... Obviously i dont think I can achieve that much... but maybe 30k or so on each table is a reasonable goal?
If it seems my performance is slow, what else do you think I should try? Is there any information I may try to get to see if maybe its a poorly configured database for this?
P.S. Is it possible I can run this so that it commits every x rows or something? I had the heartbreaking event of a network issue giving me this sudden "ora-25402: transaction must roll back" after it was running for 3.5 hours. So I lost all the progress it made... have to start over. plus i wonder if the sheer amount of data being queued for commit/rollback is causing some of the problem?
Edited by: trant on Jun 27, 2011 9:29 PM

Answer 1

Looks like there are about 54 sessions on my database, 7 of the sessions belong to me (2 taken by TOAD and 4 by my parallel slave sessions and 1 by the master of those 4)
In v$session_event there are 546 rows, if i filter it to the SIDs of my current session and order my micro_wait_time desc:
510     events in waitclass Other     30670     9161     329759     10.75     196     3297590639     1736664284     1893977003     0     Other
512     events in waitclass Other     32428     10920     329728     10.17     196     3297276553     1736664284     1893977003     0     Other
243     events in waitclass Other     21513     5     329594     15.32     196     3295935977     1736664284     1893977003     0     Other
223     events in waitclass Other     21570     52     329590     15.28     196     3295898897     1736664284     1893977003     0     Other
241     row cache lock     1273669     0     42137     0.03     267     421374408     1714089451     3875070507     4     Concurrency
241     events in waitclass Other     614793     0     34266     0.06     12     342660764     1736664284     1893977003     0     Other
241     db file sequential read     13323     0     3948     0.3     13     39475015     2652584166     1740759767     8     User I/O
241     SQL*Net message from client     7     0     1608     229.65     1566     16075283     1421975091     2723168908     6     Idle
241     log file switch completion     83     0     459     5.54     73     4594763     3834950329     3290255840     2     Configuration
241     gc current grant 2-way     5023     0     159     0.03     0     1591377     2685450749     3871361733     11     Cluster
241     os thread startup     4     0     55     13.82     26     552895     86156091     3875070507     4     Concurrency
241     enq: HW - contention     574     0     38     0.07     0     378395     1645217925     3290255840     2     Configuration
512     PX Deq: Execution Msg     3     0     28     9.45     28     283374     98582416     2723168908     6     Idle
243     PX Deq: Execution Msg     3     0     27     9.1     27     272983     98582416     2723168908     6     Idle
223     PX Deq: Execution Msg     3     0     25     8.26     24     247673     98582416     2723168908     6     Idle
510     PX Deq: Execution Msg     3     0     24     7.86     23     235777     98582416     2723168908     6     Idle
243     PX Deq Credit: need buffer     1     0     17     17.2     17     171964     2267953574     2723168908     6     Idle
223     PX Deq Credit: need buffer     1     0     16     15.92     16     159230     2267953574     2723168908     6     Idle
512     PX Deq Credit: need buffer     1     0     16     15.84     16     158420     2267953574     2723168908     6     Idle
510     direct path read     360     0     15     0.04     4     153411     3926164927     1740759767     8     User I/O
243     direct path read     352     0     13     0.04     6     134188     3926164927     1740759767     8     User I/O
223     direct path read     359     0     13     0.04     5     129859     3926164927     1740759767     8     User I/O
241     PX Deq: Execute Reply     6     0     13     2.12     10     127246     2599037852     2723168908     6     Idle
510     PX Deq Credit: need buffer     1     0     12     12.28     12     122777     2267953574     2723168908     6     Idle
512     direct path read     351     0     12     0.03     5     121579     3926164927     1740759767     8     User I/O
241     PX Deq: Parse Reply     7     0     9     1.28     6     89348     4255662421     2723168908     6     Idle
241     SQL*Net break/reset to client     2     0     6     2.91     6     58253     1963888671     4217450380     1     Application
241     log file sync     1     0     5     5.14     5     51417     1328744198     3386400367     5     Commit
510     cursor: pin S wait on X     3     2     2     0.83     1     24922     1729366244     3875070507     4     Concurrency
512     cursor: pin S wait on X     2     2     2     1.07     1     21407     1729366244     3875070507     4     Concurrency
243     cursor: pin S wait on X     2     2     2     1.06     1     21251     1729366244     3875070507     4     Concurrency
241     library cache lock     29     0     1     0.05     0     13228     916468430     3875070507     4     Concurrency
241     PX Deq: Join ACK     4     0     0     0.07     0     2789     4205438796     2723168908     6     Idle
241     SQL*Net more data from client     6     0     0     0.04     0     2474     3530226808     2000153315     7     Network
241     gc current block 2-way     5     0     0     0.04     0     2090     111015833     3871361733     11     Cluster
241     enq: KO - fast object checkpoint     4     0     0     0.04     0     1735     4205197519     4217450380     1     Application
241     gc current grant busy     4     0     0     0.03     0     1337     2277737081     3871361733     11     Cluster
241     gc cr block 2-way     1     0     0     0.06     0     586     737661873     3871361733     11     Cluster
223     db file sequential read     1     0     0     0.05     0     461     2652584166     1740759767     8     User I/O
223     gc current block 2-way     1     0     0     0.05     0     452     111015833     3871361733     11     Cluster
241     latch: row cache objects     2     0     0     0.02     0     434     1117386924     3875070507     4     Concurrency
241     enq: TM - contention     1     0     0     0.04     0     379     668627480     4217450380     1     Application
512     PX Deq: Msg Fragment     4     0     0     0.01     0     269     77145095     2723168908     6     Idle
241     latch: library cache     3     0     0     0.01     0     243     589947255     3875070507     4     Concurrency
510     PX Deq: Msg Fragment     3     0     0     0.01     0     215     77145095     2723168908     6     Idle
223     PX Deq: Msg Fragment     4     0     0     0     0     145     77145095     2723168908     6     Idle
241     buffer busy waits     1     0     0     0.01     0     142     2161531084     3875070507     4     Concurrency
243     PX Deq: Msg Fragment     2     0     0     0     0     84     77145095     2723168908     6     Idle
241     latch: cache buffers chains     4     0     0     0     0     73     2779959231     3875070507     4     Concurrency
241     SQL*Net message to client     7     0     0     0     0     51     2067390145     2000153315     7     Network
(yikes, is there a way to wrap that in equivalent of other forums' tag?)
v$session_wait;
223     835     PX Deq Credit: send blkd     sleeptime/senderid     268697599     000000001003FFFF     passes     1     0000000000000001     qref     0     00     1893977003     0     Other     0     10     WAITING
241     22819     row cache lock     cache id     13     000000000000000D     mode     0     00     request     5     0000000000000005     3875070507     4     Concurrency     -1     0     WAITED SHORT TIME
243     747     PX Deq Credit: send blkd     sleeptime/senderid     268697599     000000001003FFFF     passes     1     0000000000000001     qref     0     00     1893977003     0     Other     0     7     WAITING
510     10729     PX Deq Credit: send blkd     sleeptime/senderid     268697599     000000001003FFFF     passes     1     0000000000000001     qref     0     00     1893977003     0     Other     0     2     WAITING
512     12718     PX Deq Credit: send blkd     sleeptime/senderid     268697599     000000001003FFFF     passes     1     0000000000000001     qref     0     00     1893977003     0     Other     0     4     WAITING
v$sess_io:
223     0     5779     5741     0     0
241     38773810     2544298     15107     27274891     0
243     0     5702     5688     0     0
510     0     5729     5724     0     0
512     0     5682     5678     0     0