Slow record insertion when using millions of queries in one transaction

Question

For test purposes, we play a table creation scenario (no indexes) under multiple conditions : we insert records in bulk mode or one by one, with or without transactions, etc.
In general, the record insertion is ok, but not when we try to insert 1 million record one by one (sending 1 million INSERT commands) in a single transaction: in this case, the insertion is quick enough for the first 100000 records approximatively, but then, it becomes extremely slow, so that it would take several days to complete. This doe not happen whithout the transaction.
We were not able to find the database parameters to change to gain a better performance : rollback? transactions? undo? what else?
Does anybody have an idea of teh parameters to modify?
Thank-you in advance.

Answer 1

>
For test purposes, we play a table creation scenario (no indexes) under multiple conditions : we insert records in bulk mode or one by one, with or without transactions, etc.
In general, the record insertion is ok, but not when we try to insert 1 million record one by one (sending 1 million INSERT commands) in a single transaction: in this case, the insertion is quick enough for the first 100000 records approximatively, but then, it becomes extremely slow, so that it would take several days to complete. This doe not happen whithout the transaction.
>
Hi
How are you inserting the one million records when you do one at a time? If it's within a loop, you are probably doing a COMMIT as well within the loop. This will cause log file sync waits as LGWR will be too busy writing the redo entries. This will slow down the insert process as well as the performance of database. This is an expected bahaviour. Commit causes checkpoint to triiger which in turn will make log writer write the redo entries in the online redo logs. This is a serial process.
You can do the following methods to insert bulk records
a) INSERT /*+ APPEND */ into table select * from stage;
This will cause direct path load to happen bypassing buffer cache and reducing the redo to a great extent. However if you have foreign keys enabled in the table, it will silently ignore the direct path directive and do the conventional load.
b) Forall....SELECT ...BULK COLLECT..... This will be a good method when you do from PL/SQL
c) When you do within a loop
    Declare
    v_commit_cnt Number := 0;
   Begin
    For i in (select col1, col2, col3 from billion_record_table)
    Loop
     v_commit_cnt := v_commit_cnt + 1;
     Insert into target values (i.col1, i.col2, i.col3);
     If (v_commit_cnt >= 50000) Then
      commit;
      v_commit_cnt := 0;
     End If;
   End Loop;
   COMMIT;
End;
/4) If the target table is a staging table, you can do a CTAS (Create table as SELECT)
and many more options if you plan well in advance.