How can I load my data faster? Is there a SQL solution instead of PL/SQL?

Question

11.2.0.2
Solaris 10 sparc
I need to backfill invoices from a customer. The raw data has 3.1 million records. I have used pl/sql to load these invoices into our system (dev), however, our issue is the amount of time it's taking to run the load - effectively running at approx 4 hours. (Raw data has been loaded into a staging table)
My research keeps coming back to one concept: sql is faster than pl/sql. Where I'm stuck is the need to programmatically load the data. The invoice table has a sequence on it (primary key = invoice_id)...the invoice_header and invoice_address tables use the invoice_id as a foreign key. So my script takes advantage of knowing the primary key and uses that on the subsequent inserts to the subordinate invoice_header and invoice_address tables, respectively.
My script is below. What I'm asking is if there are other ideas on the quickest way to load this data...what am I not considering? I have to load the data in dev, qa, then production so the sequences and such change between the environments. I've dummied down the code to protect the customer; syntax and correctness of the code posted here (on the forum) is moot...it's only posted to give the framework for what I currently have.
Any advice would be greatly appreciated; how can I load the data faster knowing that I need to know sequence values for inserts into other tables?
DECLARE
   v_inv_id        invoice.invoice_id%TYPE;
   v_inv_addr_id    invoice_address.invoice_address_id%TYPE;
   errString        invoice_errors.sqlerrmsg%TYPE;
   v_guid          VARCHAR2 (128);
   v_str           VARCHAR2 (256);
   v_err_loc       NUMBER;
   v_count         NUMBER := 0;
   l_start_time    NUMBER;
   TYPE rec IS RECORD
      BILLING_TYPE             VARCHAR2 (256),
      CURRENCY                 VARCHAR2 (256),
      BILLING_DOCUMENT         VARCHAR2 (256),
      DROP_SHIP_IND            VARCHAR2 (256),
      TO_PO_NUMBER        VARCHAR2 (256),
      TO_PURCHASE_ORDER   VARCHAR2 (256),
      DUE_DATE                 DATE,
      BILL_DATE                DATE,
      TAX_AMT                  VARCHAR2 (256),
      PAYER_CUSTOMER           VARCHAR2 (256),
      TO_ACCT_NO          VARCHAR2 (256),
      BILL_TO_ACCT_NO          VARCHAR2 (256),
      NET_AMOUNT               VARCHAR2 (256),
      NET_AMOUNT_CURRENCY      VARCHAR2 (256),
      ORDER_DT             DATE,
      TO_CUSTOMER         VARCHAR2 (256),
      TO_NAME             VARCHAR2 (256),
      FRANCHISES       VARCHAR2 (4000),
      UPDT_DT                  DATE
   TYPE tab IS TABLE OF rec
                  INDEX BY BINARY_INTEGER;
   pltab           tab;
   CURSOR c
   IS
      SELECT   billing_type,
               currency,
               billing_document,
               drop_ship_ind,
               to_po_number,
               to_purchase_order,
               due_date,
               bill_date,
               tax_amt,
               payer_customer,
               to_acct_no,
               bill_to_acct_no,
               net_amount,
               net_amount_currency,
               order_dt,
               to_customer,
               to_name,
               franchises,
               updt_dt
        FROM   BACKFILL_INVOICES;
BEGIN
   l_start_time := DBMS_UTILITY.get_time;
   OPEN c;
   LOOP
      FETCH c
      BULK COLLECT INTO pltab
      LIMIT 1000;
      v_err_loc := 1;
      FOR i IN 1 .. pltab.COUNT
      LOOP
         BEGIN
            v_inv_id :=  SEQ_INVOICE_ID.NEXTVAL;
            v_guid := 'import' || TO_CHAR (CURRENT_TIMESTAMP, 'hhmissff');
            v_str := str_parser (pltab (i).FRANCHISES); --function to string parse  - this could be done in advance, yes.
            v_err_loc := 2;
            v_count := v_count + 1;
            INSERT INTO    invoice nologging
                 VALUES   (v_inv_id,
                           pltab (i).BILL_DATE,
                           v_guid,
                           '111111',
                           'NONE',
                           TO_TIMESTAMP (pltab (i).BILL_DATE),
                           TO_TIMESTAMP (pltab (i).UPDT_DT),
                           'READ',
                           'PAPER',
                           pltab (i).payer_customer,
                           v_str,
                           '111111');
            v_err_loc := 3;
            INSERT INTO    invoice_header nologging
                 VALUES   (v_inv_id,
                           TRIM (LEADING 0 FROM pltab (i).billing_document), --invoice_num
                           NULL,
                           pltab (i).BILL_DATE,                 --invoice_date
                           pltab (i).TO_PO_NUMBER,
                           NULL,
                           pltab (i).net_amount,
                           NULL,
                           pltab (i).tax_amt,
                           NULL,
                           NULL,
                           pltab (i).due_date,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           TO_TIMESTAMP (SYSDATE),
                           TO_TIMESTAMP (SYSDATE),
                           PLTAB (I).NET_AMOUNT_CURRENCY,
                           (SELECT   i.bc_value
                              FROM   invsvc_owner.billing_codes i
                             WHERE   i.bc_name = PLTAB (I).BILLING_TYPE),
                           PLTAB (I).BILL_DATE);
            v_err_loc := 4;
            INSERT INTO    invoice_address nologging
                 VALUES   (invsvc_owner.SEQ_INVOICE_ADDRESS_ID.NEXTVAL,
                           v_inv_id,
                           'BLAH INITIAL',
                           pltab (i).BILL_DATE,
                           NULL,
                           pltab (i).to_acct_no,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           SYSTIMESTAMP,
                           NULL);
            v_err_loc := 5;
            INSERT INTO    invoice_address nologging
                 VALUES   ( SEQ_INVOICE_ADDRESS_ID.NEXTVAL,
                           v_inv_id,
                           'BLAH',
                           pltab (i).BILL_DATE,
                           NULL,
                           pltab (i).TO_ACCT_NO,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           SYSTIMESTAMP,
                           NULL);
            v_err_loc := 6;
            INSERT INTO    invoice_address nologging
                 VALUES   ( SEQ_INVOICE_ADDRESS_ID.NEXTVAL,
                           v_inv_id,
                           'BLAH2',
                           pltab (i).BILL_DATE,
                           NULL,
                           pltab (i).TO_CUSTOMER,
                           pltab (i).to_name,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           SYSTIMESTAMP,
                           NULL);
            v_err_loc := 7;
            INSERT INTO    invoice_address nologging
                 VALUES   ( SEQ_INVOICE_ADDRESS_ID.NEXTVAL,
                           v_inv_id,
                           'BLAH3',
                           pltab (i).BILL_DATE,
                           NULL,
                           'SOME PROPRIETARY DATA',
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           NULL,
                           SYSTIMESTAMP,
                           NULL);
            v_err_loc := 8;
            INSERT
              INTO    invoice_event nologging (id,
                                                         eid,
                                                         root_eid,
                                                         invoice_number,
                                                         event_type,
                                                         event_email_address,
                                                         event_ts)
            VALUES   ( SEQ_INVOICE_EVENT_ID.NEXTVAL,
                      '111111',
                      '222222',
                      TRIM (LEADING 0 FROM pltab (i).billing_document),
                      'READ',
                      'some_user@some_company.com',
                      SYSTIMESTAMP);
            v_err_loc := 9;
            INSERT INTO   backfill_invoice_mapping
                 VALUES   (v_inv_id,
                           v_guid,
                           pltab (i).billing_document,
                           pltab (i).payer_customer,
                           pltab (i).net_amount);
            IF v_count = 10000
            THEN
               COMMIT;              
            END IF;
         EXCEPTION
            WHEN OTHERS
            THEN
               errString := SQLERRM;
               INSERT INTO   backfill_invoice_errors
                    VALUES   (
                                pltab (i).billing_document,
                                pltab (i).payer_customer,
                                errString || ' ' || v_err_loc
               COMMIT;
         END;
      END LOOP;
      v_err_loc := 10;
      INSERT INTO   backfill_invoice_timing
           VALUES   (
                       ROUND ( (DBMS_UTILITY.get_time - l_start_time) / 100,
                              2)
                       || ' seconds.',
                       (SELECT   COUNT (1)
                          FROM   backfill_invoice_mapping),
                       (SELECT   COUNT (1)
                          FROM   backfill_invoice_errors),
                       SYSDATE
      COMMIT;
      EXIT WHEN c%NOTFOUND;
   END LOOP;
   COMMIT;
EXCEPTION
   WHEN OTHERS
   THEN
      errString := SQLERRM;
      INSERT INTO   backfill_invoice_errors
           VALUES   (NULL, NULL, errString || ' ' || v_err_loc);
      COMMIT;
END;

Answer 1

Hello
You could use insert all in your case and make use of sequence.NEXTVAL and sequence.CURRVAL like so (excuse any typos - I can't test without table definitions). I've done the first 2 tables, so it's just a matter of adding the rest in...
INSERT ALL
     INTO      invoice nologging
                VALUES   (     SEQ_INVOICE_ID.NEXTVAL,
                               BILL_DATE,
                                my_guid,
                                '111111',
                                'NONE',
                                CAST(BILL_DATE AS TIMESTAMP),
                                CAST(UPDT_DT AS TIMESTAMP),
                                'READ',
                                'PAPER',
                                payer_customer,
                                parsed_francises,
                                '111111'
     INTO      invoice_header
          VALUES   (      SEQ_INVOICE_ID.CURRVAL,
                    TRIM (LEADING 0 FROM billing_document), --invoice_num
                    NULL,
                    BILL_DATE,                 --invoice_date
                    TO_PO_NUMBER,
                    NULL,
                    net_amount,
                    NULL,
                    tax_amt,
                    NULL,
                    NULL,
                    due_date,
                    NULL,
                    NULL,
                    NULL,
                    NULL,
                    NULL,
                    SYSTIMESTAMP,
                    SYSTIMESTAMP,
                    NET_AMOUNT_CURRENCY,
                    bc_value,
                    BILL_DATE)
     SELECT 
     src.billing_type,
          src.currency,
          src.billing_document,
          src.drop_ship_ind,
          src.to_po_number,
          src.to_purchase_order,
          src.due_date,
          src.bill_date,
          src.tax_amt,
          src.payer_customer,
          src.to_acct_no,
          src.bill_to_acct_no,
          src.net_amount,
          src.net_amount_currency,
          src.order_dt,
          src.to_customer,
          src.to_name,
          src.franchises,
          src.updt_dt,
          str_parser (src.FRANCHISES) parsed_franchises,
          'import' || TO_CHAR (CURRENT_TIMESTAMP, 'hhmissff') my_guid,
          i.bc_value
        FROM        BACKFILL_INVOICES src,
             invsvc_owner.billing_codes i
     WHERE   i.bc_name = src.BILLING_TYPE;Some things to note
1. Don't commit in a loop - you only add to the run time and load on the box ultimately reducing scalability and removing transactional integrity. Commit once at the end of the job.
2. Make sure you specify the list of columns you are inserting into as well as the values or columns you are selecting. This is good practice as it protects your code from compilation issues in the event of new columns being added to tables. Also it makes it very clear what you are inserting where.
3. If you use WHEN OTHERS THEN... to log something, make sure you either rollback or raise the exception. What you have done in your code is say - I don't care what the problem is, just commit whatever has been done. This is not good practice.
HTH
David
Edited by: Bravid on Oct 13, 2011 4:35 PM