Low Performance from FORALL Insert

Question

Hi,
I have to insert some bulk records into a table so I create a index by table variable
type mytabletype mytable%rowtype index by binary_integer;
mytablevariable mytabletype;
Then I populate the data after processing into this variable by running in a loop:
loop
--process and generate column values here
mytablevariable(idx).column1:=value1;
mytablevariable(idx).column2:=value2;
idx:=idx+1;
end loop;
then i use a forall to do the bulk insert
forall newidx in mytablevariable.first .. mytablevariable.last
insert into mytable values mytablevariable(newidx);
I printed the time taken for various steps and found that forall is the bottle neck . For inserting some 50000 records it is taking almost 4 mins!!. Is there something wrong in the way I am populating data into table variable?
-Sumit

Answer 1

Sumit Chauhan wrote:
I did a kind of similar operation for testing whereby I did a BULK COLLECT into a nested table variable and inserted the same data by using FORALL into another table (with same schema). The insert took close to 10 secs.Using elapsed time as a baseline for measuring database performance is totally flawed. I often post the following example here. The very same SQL. Same execution plan. Same data. Executed within seconds of one another. But vastly different execution times. So if the very same SQL has variable execution times, how do you expect to measure different techniques and SQLs and execution plans and the like using time?
SQL> set timing on
SQL> select count(*) from all_objects;
  COUNT(*)
     65121
Elapsed: 00:00:23.31
SQL> select count(*) from all_objects;
  COUNT(*)
     65121
Elapsed: 00:00:01.53The major reason for the difference in this example is physical I/O (1st SQL) and logical I/O (2nd SQL). The second query benefited from the I/O done by the first query, loading data into the buffer cache.
If you want to measure, you need to measure work. The amount of work that needs to be done and not how long it took to do that work. As the above example clearly shows that a query doing the same amount of work, can do it in different elapsed execution times.
Moreover, I kind of need to pull data from one table do some time consuming processing and store the results into another table so as to save the processing time , when results of calculation are used.The best way to do this, is to only use the SQL engine. As soon as you use the PL language, you introduce another software layer and increase the number of moving parts.
Thus the mantra of Maximize SQL and Minimize PL/SQL+.
No other language is as fast and as capable as SQL when it comes to crunching database data. If you do need to add PL into the mix because SQL is simply not capable of performing the required processing, then you will use bulk collection.
Bulk collection does one and one thing only. It creates a bigger "+communication pipe+" between the SQL and PL engines. So instead of PL getting a single row at a time from the SQL engine, it gets a batch of a 100. Instead of it sending a single row at a time to the SQL engine, it sends a 100.
This reduces the number of context switches between the two - and increases performance. However, this is not a linear increase. It does not mean that the bigger the communication pipe, the faster the bulk processing.
Also, the bigger you make this pipe, the more expensive non-shared memory (PGA) is needed.
So there is a very real limit to the performance improvement you will see with bulk processing. And there is a direct cost, in terms of expensive PGA, associated with it.
The PL engine's default bulk processing limit is a 100 rows. You may found slight performance improvement when using a limit of a 1000 rows. Beyond that, unlikely to see any performance improvement. In fact, you may see the opposite as the amount of private memory consumed can negatively impact server memory, increase swapping, and slow the entire server down.