BULK COLLECT and FORALL with dynamic INSERT.

Question

Hello,
I want to apply BULK COLLECT and FORALL feature for a insert statement in my procedure for performance improvements as it has to insert a huge amount of data.
But the problem is that the insert statement gets generated dynamically and even the table name is found at the run-time ... so i am not able to apply the performance tuning concepts.
See below the code
PROCEDURE STP_MES_INSERT_GLOBAL_TO_MAIN
  (P_IN_SRC_TABLE_NAME                  VARCHAR2 ,
  P_IN_TRG_TABLE_NAME                  VARCHAR2 ,
  P_IN_ED_TRIG_ALARM_ID                 NUMBER ,
  P_IN_ED_CATG_ID                 NUMBER ,
  P_IN_IS_PIECEID_ALARM IN CHAR,
  P_IN_IS_LAST_RECORD IN CHAR
IS
      V_START_DATA_ID                 NUMBER;
      V_STOP_DATA_ID                   NUMBER;
      V_FROM_DATA_ID                   NUMBER;
      V_TO_DATA_ID                       NUMBER;
      V_MAX_REC_IN_LOOP              NUMBER := 30000;
      V_QRY1         VARCHAR2(32767);
BEGIN
            EXECUTE IMMEDIATE 'SELECT MIN(ED_DATA_ID), MAX(ED_DATA_ID) FROM '|| P_IN_SRC_TABLE_NAME INTO V_START_DATA_ID , V_STOP_DATA_ID;
            --DBMS_OUTPUT.PUT_LINE('ORIGINAL START ID := '||V_START_DATA_ID ||' ORIGINAL  STOP ID := ' || V_STOP_DATA_ID);
            V_FROM_DATA_ID := V_START_DATA_ID ;
            IF (V_STOP_DATA_ID - V_START_DATA_ID ) > V_MAX_REC_IN_LOOP THEN
                    V_TO_DATA_ID := V_START_DATA_ID + V_MAX_REC_IN_LOOP;
            ELSE
                   V_TO_DATA_ID := V_STOP_DATA_ID;
            END IF;
          LOOP
                BEGIN
             LOOP      
                        V_QRY1 := ' INSERT INTO '||P_IN_TRG_TABLE_NAME||
                        ' SELECT * FROM '||P_IN_SRC_TABLE_NAME ||
                        ' WHERE ED_DATA_ID BETWEEN ' || V_FROM_DATA_ID ||' AND ' || V_TO_DATA_ID;
                EXECUTE IMMEDIATE V_QRY1;
commit;
                                 V_FROM_DATA_ID :=  V_TO_DATA_ID + 1;
                        IF  ( V_STOP_DATA_ID - V_TO_DATA_ID > V_MAX_REC_IN_LOOP ) THEN
                              V_TO_DATA_ID := V_TO_DATA_ID + V_MAX_REC_IN_LOOP;
                        ELSE
                              V_TO_DATA_ID := V_TO_DATA_ID + (V_STOP_DATA_ID - V_TO_DATA_ID);
                        END IF;
                EXCEPTION
                         WHEN OTHERS THEN.............
....................so on Now you can observer here that P_IN_SRC_TABLE_NAME is the source table name which we get as a parameter at run-time. I have used 2 table in the insert statement P_IN_TRG_TABLE_NAME (in which i have to insert data) and P_IN_SRC_TABLE_NAME(from where i have to insert data)
  V_QRY1 := ' INSERT INTO '||P_IN_TRG_TABLE_NAME||
                        ' SELECT * FROM '||P_IN_SRC_TABLE_NAME ||
                        ' WHERE ED_DATA_ID BETWEEN ' || V_FROM_DATA_ID ||' AND ' || V_TO_DATA_ID;
                EXECUTE IMMEDIATE V_QRY1;now when i appy the bulk collect and forall feature i am facing the out of scope problem....see the code below ...
BEGIN
            EXECUTE IMMEDIATE 'SELECT MIN(ED_DATA_ID), MAX(ED_DATA_ID) FROM '|| P_IN_SRC_TABLE_NAME INTO V_START_DATA_ID , V_STOP_DATA_ID;
            --DBMS_OUTPUT.PUT_LINE('ORIGINAL START ID := '||V_START_DATA_ID ||' ORIGINAL  STOP ID := ' || V_STOP_DATA_ID);
            V_FROM_DATA_ID := V_START_DATA_ID ;
            IF (V_STOP_DATA_ID - V_START_DATA_ID ) > V_MAX_REC_IN_LOOP THEN
                    V_TO_DATA_ID := V_START_DATA_ID + V_MAX_REC_IN_LOOP;
            ELSE
                   V_TO_DATA_ID := V_STOP_DATA_ID;
            END IF;
          LOOP
                DECLARE
                 TYPE TRG_TABLE_TYPE IS TABLE OF P_IN_SRC_TABLE_NAME%ROWTYPE;
                 V_TRG_TABLE_TYPE TRG_TABLE_TYPE;
                 CURSOR TRG_TAB_CUR IS
                 SELECT * FROM P_IN_SRC_TABLE_NAME
                 WHERE ED_DATA_ID BETWEEN V_FROM_DATA_ID AND V_TO_DATA_ID;
                 V_QRY1 varchar2(32767);
                BEGIN
                OPEN TRG_TAB_CUR;
                LOOP
                FETCH TRG_TAB_CUR BULK COLLECT INTO V_TRG_TABLE_TYPE LIMIT 30000;
                FORALL I IN 1..V_TRG_TABLE_TYPE.COUNT
                V_QRY1 := ' INSERT INTO '||P_IN_TRG_TABLE_NAME||' VALUES V_TRG_TABLE_TYPE(I);'
                EXECUTE IMMEDIATE V_QRY1;
                EXIT WHEN TRG_TAB_CUR%NOTFOUND;
                END LOOP;
                CLOSE TRG_TAB_CUR;
                        V_FROM_DATA_ID :=  V_TO_DATA_ID + 1;
                        IF  ( V_STOP_DATA_ID - V_TO_DATA_ID > V_MAX_REC_IN_LOOP ) THEN
                              V_TO_DATA_ID := V_TO_DATA_ID + V_MAX_REC_IN_LOOP;
                        ELSE
                              V_TO_DATA_ID := V_TO_DATA_ID + (V_STOP_DATA_ID - V_TO_DATA_ID);
                        END IF;
                EXCEPTION
                         WHEN OTHERS THEN.........so on
But the above code is not helping me ,  what i am doing wrong ??? how can i tune this dynamically generated statement to use bulk collect for better performace ......
Thanks in Advance !!!!

Answer 1

Hello,
a table name cannot be bind as a parameter in SQL, this wont't compile:
EXECUTE IMMEDIATE ' INSERT INTO :1 VALUES ......
USING P_IN_TRG_TABLE_NAME ...but this should work:
EXECUTE IMMEDIATE ' INSERT INTO ' || P_IN_TRG_TABLE_NAME || ' VALUES ......You cannot declare a type that is based on a table which name is in a variable.
PL/SQL is stronly typed language, a type must be known at compile time, a code like this is not allowed:
PROCEDURE xx( src_table_name varchar2 )
DECLARE
   TYPE tab IS TABLE OF src_table_name%ROWTYPE;
  ...This can be done by creating one big dynamic SQL - see example below (tested on Oracle 10 XE - this is a slightly simplified version of your procedure):
CREATE OR REPLACE
PROCEDURE stp1(
  p_in_src_table_name                  VARCHAR2 ,
  p_in_trg_table_name                  VARCHAR2 ,
  v_from_data_id     NUMBER := 100,
  v_to_data_id       NUMBER := 100000
IS
BEGIN
  EXECUTE IMMEDIATE q'{
  DECLARE
     TYPE trg_table_type IS TABLE OF }' || p_in_src_table_name || q'{%ROWTYPE;
     V_TRG_TABLE_TYPE TRG_TABLE_TYPE;
     CURSOR TRG_TAB_CUR IS
     SELECT * FROM }' || p_in_src_table_name ||
     q'{ WHERE ED_DATA_ID BETWEEN :V_FROM_DATA_ID AND :V_TO_DATA_ID;
  BEGIN
      OPEN TRG_TAB_CUR;
      LOOP
            FETCH TRG_TAB_CUR BULK COLLECT INTO V_TRG_TABLE_TYPE LIMIT 30000;
            FORALL I IN 1 .. V_TRG_TABLE_TYPE.COUNT
            INSERT INTO }' || p_in_trg_table_name || q'{ VALUES V_TRG_TABLE_TYPE( I );
            EXIT WHEN TRG_TAB_CUR%NOTFOUND;
      END LOOP;
      CLOSE TRG_TAB_CUR;
  END; }'
  USING v_from_data_id, v_to_data_id;
  COMMIT;
END;But this probably won't give any performace improvements. Bulk collect and forall can give performance improvements when there is a DML operation inside a loop,
and this one single DML operates on only one record or relatively small number of records, and this DML is repeated many many times in the loop.
I guess that your code is opposite to this - it contains insert statement that operates on many records (one single insert ~ 30000 records),
and you are trying to replace it with bulk collect/forall - INSERT INTO ... SELECT FROM will almost alwayst be faster than bulk collect/forall.
Look at simple test - below is a procedure that uses INSERT ... SELECT :
CREATE OR REPLACE
PROCEDURE stp(
  p_in_src_table_name                  VARCHAR2 ,
  p_in_trg_table_name                  VARCHAR2 ,
  v_from_data_id     NUMBER := 100,
  v_to_data_id       NUMBER := 100000
IS
V_QRY1 VARCHAR2(32767);
BEGIN
      V_QRY1 := ' INSERT INTO '||   P_IN_TRG_TABLE_NAME ||
                ' SELECT * FROM '|| P_IN_SRC_TABLE_NAME ||
                ' WHERE ed_data_id BETWEEN :f AND :t ';
      EXECUTE IMMEDIATE V_QRY1
      USING V_FROM_DATA_ID, V_TO_DATA_ID;
      COMMIT;
END;
/and we can compare both procedures:
SQL> CREATE TABLE test333
  2  AS SELECT level ed_data_id ,
  3            'XXX ' || LEVEL x,
  4            'YYY ' || 2 * LEVEL y
  5  FROM dual
  6  CONNECT BY LEVEL <= 1000000;
Table created.
SQL> CREATE TABLE test333_dst AS
  2  SELECT * FROM test333 WHERE 1 = 0;
Table created.
SQL> set timing on
SQL> ed
Wrote file afiedt.buf
  1  BEGIN
  2     FOR i IN 1 .. 100 LOOP
  3        stp1( 'test333', 'test333_dst', 1000, 31000 );
  4     END LOOP;
  5* END;
SQL> /
PL/SQL procedure successfully completed.
Elapsed: 00:00:22.12
SQL> ed
Wrote file afiedt.buf
  1  BEGIN
  2     FOR i IN 1 .. 100 LOOP
  3        stp( 'test333', 'test333_dst', 1000, 31000 );
  4     END LOOP;
  5* END;
SQL> /
PL/SQL procedure successfully completed.
Elapsed: 00:00:14.86without bulk collect ~ 15 sec.
bulk collect version ~ 22 sec. .... 7 sec longer / 15 sec. = about 45% performance decrease.