Using a date field to remove 'duplicates'

Question

Hi all,
Really struggling here, would appreciate some fresh ideas !!
I have a ton of rows in a table that are duplicates with the exception of a date field that differs very slightly.
e.g.
TO_CHAR(START_DATE,'
13-JUN-2007 12:39:03
13-JUN-2007 12:39:10 << dup
18-JUN-2007 05:49:15
18-JUN-2007 05:49:21 << dup
13-JUN-2007 09:02:25
13-JUN-2007 09:02:32 << dup
It would be great if I could identify these dups in SQL and remove them. I've sampled the data and the times are out by between 5 and 7 seconds so if I could weed out all rows that have another row with a time that differs by <10 seconds that would be ideal.
My other, but not as thorough, idea was to drop the seconds and I would be then able to drop them as if they were true duplicates ... run the risk of not catching some that way though.
Any ideas?
Many thanks in advance!
Adam

Answer 1

Assuming that you could legititmately have two records for the same day that are more then 10 seconds apart (and you would not consider these to be duplicates, then something along these lines should do it. In the initial select, I have bolded the "duplicates"
pre]SQL> SELECT * FROM rt;
SNO SDATE
1 13-jun-2007 12:39:03
1 13-jun-2007 12:39:10
1 13-jun-2007 12:39:12
1 13-jun-2007 12:39:45
2 18-jun-2007 05:49:15
2 18-jun-2007 05:49:21
3 13-jun-2007 09:02:25
3 13-jun-2007 09:02:32
SQL> DELETE FROM rt
2 WHERE rowid IN (SELECT rid
3 FROM (SELECT sno, sdate, rowid rid,
4 LAG(sdate) OVER (PARTITION BY sno
5 ORDER BY sdate) rn
6 FROM rt)
7 WHERE (sdate - NVL(rn, sdate))*24*60*60 BETWEEN 1 and 10);
4 rows deleted.
SQL> SELECT * FROM rt;
SNO SDATE
1 13-jun-2007 12:39:03
1 13-jun-2007 12:39:45
2 18-jun-2007 05:49:15
3 13-jun-2007 09:02:25
This keeps the oldest of the duplicates by sdate, if you want to keep the newest, then something like:
SQL> rollback;
Rollback complete.
SQL> DELETE FROM rt
  2  WHERE rowid IN (SELECT rid
  3                  FROM (SELECT sno, sdate, rowid rid,
  4                               LEAD(sdate) OVER (PARTITION BY sno
  5                                                ORDER BY sdate) rn
  6                        FROM rt)
  7                  WHERE (NVL(rn, sdate) - sdate)*24*60*60 BETWEEN 1 and 10);
4 rows deleted.
SQL> SELECT * FROM rt;
       SNO SDATE
         1 13-jun-2007 12:39:12
         1 13-jun-2007 12:39:45
         2 18-jun-2007 05:49:21
         3 13-jun-2007 09:02:32John