Nested Loops...looping through one month of data at a time year by year

Question

Hi all,
I'm trying to create an insert statement that loops through a table that has 10 years of data (2001 to 2010) month by month to minimize impact on server and commits more frequently to avoid filling up the redo logs and rollback tablespaces. The table is large, has about 40 millions records per year. Lets say the structure of the table is the following:
Customer_ID number(9),
Order_Item_1 number(6),
Order_Item_2 number(6),
Order_Item_3 number(6),
Order_date date
The table is in flat format but I want to normalize it so that it looks like the following:
Customer_ID Order_Seq Order_Item Order_date
999999999 1 555555 01-jan-2001
999999999 2 666666 01-jan-2001
999999999 3 444444 01-jan-2001
888888888 1 555555 03-jan-2001
888888888 2 666666 03-jan-2001
But because I want to loop through month by month....I need to set it up so that it loops through month by month, year by year (Using the Order Date Field) and Order_item by Order_item. Something like:
so my insert statements would be something like if I hardcoded instead of put the insert statement into a loop:
insert into orders_normalized
(Customer_id,Order_seq,Order_item,Order_date) select customer_id,1,Order_item,Order_date where Order_item_1 is not null and to_char(order_date,'yyyy') = '2001' and to_char(order_date,'mm')='01';
insert into orders_normalized
(Customer_id,Order_seq,Order_item,Order_date) select customer_id,2,Order_item,Order_date where Order_item_2 is not null and to_char(order_date,'yyyy') = '2001' and to_char(order_date,'mm')='01';
insert into orders_normalized
(Customer_id,Order_seq,Order_item,Order_date) select customer_id,3,Order_item,Order_date where Order_item_3 is not null and to_char(order_date,'yyyy') = '2001' and to_char(order_date,'mm')='01';
insert into orders_normalized
(Customer_id,Order_seq,Order_item,Order_date) select customer_id,1,Order_item,Order_date where Order_item_1 is not null and to_char(order_date,'yyyy') = '2001' and to_char(order_date,'mm')='02';
insert into orders_normalized
(Customer_id,Order_seq,Order_item,Order_date) select customer_id,2,Order_item,Order_date where Order_item_2 is not null and to_char(order_date,'yyyy') = '2001' and to_char(order_date,'mm')='02';
insert into orders_normalized
(Customer_id,Order_seq,Order_item,Order_date) select customer_id,3,Order_item,Order_date where Order_item_3 is not null and to_char(order_date,'yyyy') = '2001' and to_char(order_date,'mm')='03';
Hope this makes sense.
Thanks

Answer 1

Does the sequence of items in an order really matter? In other words, do we really need to preserve that an item was in position 2 versus position 1? I bet that the sequence or position of each item in an order is not meaningful. They were probably numbered 1, 2, and 3 just to make them uniquely named columns so there would be three slots to hold up to 3 items in the denormalized table.
You only have about 400 million rows to insert, so it could feasibly be done in a single transaction (depending on your database environment).
You can always do a create table as select (CTAS) to help with undo / redo issues and get better performance. You could run it in parallel, and spit it out to a new table partitioned by month. Single DDL statement running in parallel making your new table--sounds good to me.
How about something like this:
CREATE TABLE ORDERS_NORMALIZED
(CUSTOMER_ID, ORDER_ITEM, ORDER_DATE)
PARTITION BY RANGE(ORDER_DATE)
PARTITION p200901 VALUES LESS THAN (TO_DATE('200902','YYYYMM')),
PARTITION p200902 VALUES LESS THAN (TO_DATE('200903','YYYYMM')),
PARTITION p201012 VALUES LESS THAN (TO_DATE('201101','YYYYMM'))
as SELECT CUSTOMER_ID, ORDER_ITEM_1, ORDER_DATE
   FROM OTHER_TABLE
   WHERE ORDER_ITEM_1 IS NOT NULL
   UNION ALL
   SELECT CUSTOMER_ID, ORDER_ITEM_2, ORDER_DATE
   FROM OTHER_TABLE
   WHERE ORDER_ITEM_2 IS NOT NULL
   UNION ALL
   SELECT CUSTOMER_ID, ORDER_ITEM_3, ORDER_DATE
   FROM OTHER_TABLE
   WHERE ORDER_ITEM_3 IS NOT NULL.....................
Out of curiosity, why not normalize it further? You could have used two tables instead of one.
One (ORDER) with:
ORDER_ID
CUSTOMER_ID
DATE
Order_id would be a new surrogate key / primary key.
Another table (ORDER_ITEM) with:
ORDER_ID
ORDER_ITEM
It would be a table that links ORDERS to ITEMS. You get the idea.