Avoid duplicates processing

Question

Hi All
I have a requirement in which I get a flat from source system everyday
I load this flat file into a staging table. This flat file can contain upto 6 million records
Next day when the flat file comes it will come with changed and unchanged records.
ie even if a record has not changed it will again come up in the file, the next day.
From the staging table, this data is processed and goes to other tables.
Since 75% of data will be duplicate, I do not want to process all the data everyday.
How do I do this? Getting only the unchanged records from source system is ruled out and also
I cant do a "diff" command on Unix between new and old files to load only changed records
I am thinking of 2 ways to do it
1) Create a duplicate set of table similar to staging table and load the previous days data in
this. While selecting the data use the MINUS operator on the table and do the processing
2) Retain the previous days data in the same table and while processing SELECT DISTINCT
from the table .
Which is the best way to do this, or any other alternate solutions to this
Performance is a major factor here and am using Oracle 8i.
Thanks
Ashwin N.

Answer 1

If you are going to pre-process the rows to eliminate the duplicates, I would suspect that the MINUS approach would be faster. I would do something like:
load into new
SELECT * FROM new
MINUS
SELECT * FROM old;
processing to real table
DROP TABLE old;
RENAME new TO old;
CREATE TABLE new AS
SELECT * FROM old
WHERE 1=0;However, I'm not entirely convinced that the processing into the real table would be enough faster to offset the time taken to eliminate the duplicates before processing. My first impulse, if you can do your processing in a SQL statement, would be to just load the records into an empty staging table every day and do an update and an insert statement something like:
UPDATE real_table r
SET (col1, co2, col2, ... , coln) =
    (SELECT col1, co2, col2, ... , coln
     FROM staging_table s
     WHERE r.pk1 = s.pk1 AND
           r.pk2 = s.pk2 AND
           (r.col1 <> s.col1 OR
            r.col2 <> s.col2 OR
            r.coln <> s.coln)
WHERE EXISTS (SELECT 1
              FROM staging_table s
              WHERE r.pk1 = s.pk1 AND
                    r.pk2 = s.pk2);
INSERT INTO real_table
SELECT *
FROM staging_table s
WHERE NOT EXISTS (SELECT 1
                  FROM real_table r
                  WHERE s.pk1 = r.pk1 AND
                        s.pk2 = r.pk2);You should rebuild the PK on staging_table after the load. If you do this paralell, nologging, it should be reasonably quick. Even better would be to get the data sorted by PK fro mthe source system, because you could then use the NOSORT option on the index.
TTFN
John