Eliminate duplicate while fetching data from source

Question

Hi All,
CUSTOMER TRANSACTION
CUST_LOC     CUT_ID          TRANSACTION_DATE     TRANSACTION_TYPE
100          12345          01-jan-2009          CREDIT
100          23456          15-jan-2000          CREDIT
100          12345          01-jan-2010          DEBIT
100          12345          01-jan-2000          DEBITNow as per my requirement, i need to fetch data from CISTOMER_TRANSACTION table for those customer which has transaction in last 10 years. In my above data, customer 12345 has transaction in last 10 years, whereas for customer 23456, does not have transaction in last 10 years so will eliminate it.
Now, CUSTOMER_TRANSACTION table has approximately 100 million records. So, we are fectching data in batches. Batching is divided into months. Total 120 months. Below is my query.
select *
FROM CUSTOMER_TRANSACTION CT left outer join
(select distinct CUST_LOC, CUT_ID FROM CUSTOMER_TRANSACTION WHERE TRANSACTION_DATE >= ADD_MONTHS(SYSDATE, -120) and TRANSACTION_DATE < ADD_MONTHS(SYSDATE, -119) CUST
on CT.CUST_LOC = CUST.CUST_LOC and CT.CUT_ID = CUST.CUT_IDThru shell script, months number will change. -120:-119, -119:-118 ....., -1:-0.
Now the problem is duplication of records.
while fetching data for jan-2009, it will get cust_id 12345 and will fetch all 3 records and load it into target.
while fetching data for jan-2010, it will get cust_id 12345 and will fetch all 3 records and load in into target.
So instead of having only 3 records, for customer 12345 it will be having 6 records. Can someone help me on how can i eliminate duplicate records from getting in.
As of now i have 2 ways in mind.
1. Fetch all records at once. Which is impossible as it will give space issue.
2. After each batch, run a procedure which will delete duplicate records based on cust_loc, cut_id and transaction_date. But again it will have performance problem.
I want to eliminate it while fetching data from source.
Edited by: ace_friends22 on Apr 6, 2011 10:16 AM

Answer 1

You can do it this way....
SELECT DISTINCT cust_doc,
                cut_id
  FROM customer_transaction
WHERE transaction_date >= ADD_MONTHS(SYSDATE, -120)
   AND transaction_date < ADD_MONTHS(SYSDATE, -119)However please note that - if want to get the transaction in a month like what you said earlier jan-2009 and jan-2010 and so on... you might need to use TRUNC...
Your date comparison could be like this... In this example I am checking if the transaction date is in the month of jan-2009
AND transaction_date BETWEEN ADD_MONTHS(TRUNC(SYSDATE,'MONTH'), -27)  AND LAST_DAY(ADD_MONTHS(TRUNC(SYSDATE,'MONTH'), -27)) Your modified SQL...
SELECT *
  FROM customer_transaction 
WHERE transaction_date BETWEEN ADD_MONTHS(TRUNC(SYSDATE,'MONTH'), -27)  AND LAST_DAY(ADD_MONTHS(TRUNC(SYSDATE,'MONTH'), -27))Testing..
--Sample Data
CREATE TABLE customer_transaction (
cust_loc number,
cut_id number,
transaction_date date,
transaction_type varchar2(20)
INSERT INTO customer_transaction VALUES (100,12345,TO_DATE('01-JAN-2009','dd-MON-yyyy'),'CREDIT');
INSERT INTO customer_transaction VALUES (100,23456,TO_DATE('15-JAN-2000','dd-MON-yyyy'),'CREDIT');
INSERT INTO customer_transaction VALUES (100,12345,TO_DATE('01-JAN-2010','dd-MON-yyyy'),'DEBIT');
INSERT INTO customer_transaction VALUES (100,12345,TO_DATE('01-JAN-2000','dd-MON-yyyy'),'DEBIT');
--To have three records in the month of jan-2009
UPDATE customer_transaction
   SET transaction_date = TO_DATE('02-JAN-2009','dd-MON-yyyy')
WHERE cut_id = 12345
   AND transaction_date = TO_DATE('01-JAN-2010','dd-MON-yyyy');
UPDATE customer_transaction
   SET transaction_date = TO_DATE('03-JAN-2009','dd-MON-yyyy')
WHERE cut_id = 12345
   AND transaction_date = TO_DATE('01-JAN-2000','dd-MON-yyyy');
commit;
--End of sample data
SELECT *
  FROM customer_transaction 
WHERE transaction_date BETWEEN ADD_MONTHS(TRUNC(SYSDATE,'MONTH'), -27)  AND LAST_DAY(ADD_MONTHS(TRUNC(SYSDATE,'MONTH'), -27));Results....
CUST_LOC     CUT_ID TRANSACTI TRANSACTION_TYPE
      100      12345 01-JAN-09 CREDIT
      100      12345 02-JAN-09 DEBIT
      100      12345 03-JAN-09 DEBITAs you can see, there are only 3 records for 12345
Regards,
Rakesh
Edited by: Rakesh on Apr 6, 2011 11:48 AM