Query to eliminate duplicates

I have a table that lists some data. Every day should have one and only one record per organization. I am trying to weed out the duplicates in one query. It doesn't matter which of the duplicates gets deleted, so I could use a (where rownum = 2) clause, but I'm not sure how to get there.
This query exposes the duplicates:
select DATE_INDEX, ORG_ID
from STAFFING_TURNOVER
group by DATE_INDEX, ORG_ID
having COUNT(ID) > 1
order by DATE_INDEX, ORG_IDCan someone help me turn this into a delete query that deletes the second record? Because it is based on a combination of date + organization, the syntax for a subquery-based delete is escaping me.

Delete from STAFFING_TURNOVER Where RowID in
(Select Min(RowID) From STAFFING_TURNOVER Group by DATE_INDEX, ORG_ID Having Count(1) > 1)
Oops, someone posted the same thing.
If you need to you can execute it more than once if you have more than one. You can also perform an elegant minus query or use a Sub-query to ID the dups. The problem will be performance and depend on how much data you have.
Edited by: Dev on Dec 1, 2010 3:25 PM

Similar Messages

Select query that eliminate duplicates

I am struck at the below scenario. need your help in modify my query based on the below sample data. My objective is to create a EVENT AND EVENT_LOW using the data available from TEM_GT and TXN table.
TEM_GT (global temporary table)
est_id primary key, trans_id, trp_id, amount
1 111 2221 1.5
2 111 3332 2.0
3 112 4443 3.0
TXN table
trans_id, trans_type
111 type1
112 type1
EVENT table
event_id primary key, trans_id, trans_type, flag.
1000 111 type1 N
1001 112 type1 N
EVENT_LOW table
event_low_id primary key, event_id, est_id, amount.
9991 1000 1 1.5
9992 1000 1 2.0
9993 1001 2 3.0
insert into(event_low_id, event_id, est_id, amount)
select event_low_id_s.nextval e.event_id, tg.est_id, tg.amount from
from TEM_GT tg, EVENT e
WHERE
tg.trans_id = e.trans_id
AND e.flag = 'N'
Based on TEM_GT and TXN gt, populating data into EVENT table. Now when I try to populate data in the EVENT_LOW table, the above query returns 5 rows instead of 3 rows. This is due to trp_id in TEM_GT table. I do not want to add trp_id in EVENT table, and want to handle the duplicate elimination in the above select query.
I wrote the below sql. It almost resolve my problem. But sometimes it returns different value for the amount column. I wanted my event_low result as below for the est_id 1.
event_low_id primary key, event_id, est_id, amount.
9991 1000 1 1.5
9992 1000 1 2.0
But sometime it returns as below
event_low_id primary key, event_id, est_id, amount.
9991 1000 1 1.5
9992 1000 1 1.5
or
event_low_id primary key, event_id, est_id, amount.
9991 1000 1 2.0
9992 1000 1 2.0
select *
from (select x.*,
row_number() over (partition by event_id order by event_id) rn
from (seelct e.event_id, tg.est_id, tg.amount
from TEM_GT tg, EVENT e
WHERE
tg.trans_id = e.trans_id
AND e.flag = 'N'
) x
where rn = 1
I am using Oracle 11g. Any help. Thanks in advance.

Hi Suresh,
Please read SQL and PL/SQL FAQ
Whenever you create a thread always post sample data as CREATE TABLE and INSERT statements.
Additionally when you put some code or output please enclose it between two lines starting with {noformat}{noformat}
i.e.:
{noformat}{noformat}
SELECT ...
{noformat}{noformat}
Regarding your problem you said:
Based on TEM_GT and TXN gt, populating data into EVENT table. Now when I try to populate data in the EVENT_LOW table, the above query returns 5 rows instead of 3 rows.
The above query cannot return 5 rows with the data you have posted. You have 3 rows in TEM_GT and 2 rows in EVENT table. The table are joined ontg.trans_id = e.trans_id AND e.flag = 'N'
And the result of the query with your posted data is *3 rows*. So try to be clear.
If your intention is to insert into EVENT_LOW only one row from TEM_GT for the same trans_id then you have to tell us which is the criteria to choose the row between the duplicates (the first by est_id order??).
Also be careful to post valid statements. The statement below is not valid as it is missing the table name after INSERT INTO and there are 2 consecutive FROM keywords:insert into(event_low_id, event_id, est_id, amount)
select event_low_id_s.nextval e.event_id, tg.est_id, tg.amount from
from TEM_GT tg, EVENT e
WHERE
tg.trans_id = e.trans_id
AND e.flag = 'N'
I will make the assumption that you want to select the first row order by est_id during insertion. If this is not what you want, please change the analytic function:INSERT INTO event_low (event_low_id, event_id, est_id, amount)
WITH got_data AS
SELECT e.event_id, tg.est_id, tg.amount
, ROW_NUMBER() OVER(PARTITION BY tg.trans_id ORDER BY tg.est_id) AS rn
FROM tem_gt tg, event e
WHERE tg.trans_id = e.trans_id
AND e.flag = 'N'
SELECT event_low_id_s.NEXTVAL
, event_id, est_id, amount
FROM got_data
WHERE rn = 1;
If this is not what you want please post CREATE TABLE and INSERT statement, and post your expected output (for insertion into EVENT_LOW table) for the data you have posted.
Regards.
Al

How to Eliminate Duplicates in Hibernate Criteria Query

Hi all,
Anyone who know how to eliminate duplicates in Hibernate Criteria Query ?
Thanks,
Lanz

You should try asking this in a hibernate forum

Query help to eliminate duplicate inserts

Hi
I am inserting records using Insert and select statements. I am having composite primary key on client_id,begin_Dt,end_Dt and service_cd_desc on my target table.
In order to eliminate duplicate composite key being inserted into the target table I am using NOT IN clause but some how I am error as
ORA-00001: unique constraint violated (<schema>.<constraint>)
query is as below:
insert into stage_2_authorized_service
(CLIENT_ID
,BEGIN_DT
,END_DT
,DHS_SERVICE_CD
,SERVICE_CD_DESC
,DHS_SERVICE_GROUP_CD
,SERVICE_GROUP_DESC
,AUTHORIZED_DOLLARS
,AUTHORIZED_UNITS
,DHS_AUTHORIZED_UNITS_TYPE_CD
,AUTHORIZED_UNITS_TYPE_DESC
,CONTRACT_NUMBER
,SERVICE_CD
,SERVICE_GROUP_CD
SELECT DISTINCT
stage_1.CLIENT_ID,
stage_1.BEGIN_DT,
stage_1.END_DT,
stage_1.service_cd as dhs_service_cd,
dim.service_cd_desc as service_cd_desc,
stage_1.SERVICE_GROUP_CD as dhs_service_group_cd,
decode(stage_1.SERVICE_GROUP_cd, '1','NURSING FACILITY','2', 'CLASS','3','CBA','4','STATE OPERATED ICF/MR CAMPUS','5','STATE OPERATED ICF/MR COMMUNITY','6','NON-STATE-OPERATED ICF/MR','7','Community Care','8','Hospice','9','LTC SUPPORT SERVICES (PROVIDER ENROLLMENT)','10','SWING BEDS (WAS PART OF NURSING FACILITY)','11','PACE','12','HCS','13','HCS-O','14','MRLA (MHMR NEW WAIVER)','15','TEXAS HOME LIVING (TxHmL)','16','Deaf Blind Waiver','17','Consolidated Waiver','18','Medically Dependent Children Program (MDCP)','19','STAR+PLUS') as service_group_desc,
stage_1.AUTHORIZED_DOLLARS,
stage_1.AUTHORIZED_UNITS,
stage_1.UNIT_TYPE as DHS_AUTHORIZED_UNITS_TYPE_CD,
decode(stage_1.UNIT_TYPE,'1','WEEK','2','MONTH','4','PER AUTHORIZATION','5','DAILY') as AUTHORIZED_UNITS_TYPE_DESC ,
CONTRACT_NUMBER,
stage_1.service_cd,
stage_1.service_group_cd
FROM
STAGE_1_MG_SERVICE_REQUEST stage_1, stage_3_servcd_servgrp_dim dim
where
(stage_1.client_id,stage_1.begin_dt,stage_1.end_dt,dim.service_cd_desc) not in (select distinct stage_1.client_id,stage_1.begin_dt,stage_1.end_dt,dim.service_cd_desc
from stage_1_mg_service_request stage_1, stage_3_servcd_servgrp_dim dim
where stage_1.service_cd = dim.service_cd and
stage_1.service_group_cd = dim.service_group_cd and
stage_1.service_group_cd in ('2','3','16','17','18')
Can some one tell me whats wrong with above query
Any help is appreciated...!
Thanks
J

Yes my target table doesn't have data init at all.Well that is not how you use not in to avoid unique constraint violations
You need to select only the values that will not violate the constraint something like this untested example.
select
    client_id,
    begin_dt,
    end_dt,
    service_cd_desc,
    dhs_service_cd,
    dhs_service_group_cd,
    service_group_desc,
    authorized_dollars,
    authorized_units,
    dhs_authorized_units_type_cd,
    authorized_units_type_desc,
    contract_number,
    service_cd,
    service_group_cd
from
    SELECT
    count(*) over (partition by
stage_1.CLIENT_ID,
stage_1.BEGIN_DT,
stage_1.END_DT,
dim.service_cd_desc) the_count,
    stage_1.CLIENT_ID,
    stage_1.BEGIN_DT,
    stage_1.END_DT,
    stage_1.service_cd as dhs_service_cd,
    dim.service_cd_desc as service_cd_desc,
    stage_1.SERVICE_GROUP_CD as dhs_service_group_cd,
    decode(stage_1.SERVICE_GROUP_cd, '1','NURSING FACILITY','2', 'CLASS','3','CBA',
        '4','STATE OPERATED ICF/MR CAMPUS','5','STATE OPERATED ICF/MR COMMUNITY',
        '6','NON-STATE-OPERATED ICF/MR','7','Community Care','8','Hospice',
        '9','LTC SUPPORT SERVICES (PROVIDER ENROLLMENT)',
        '10','SWING BEDS (WAS PART OF NURSING FACILITY)','11','PACE','12','HCS',
        '13','HCS-O','14','MRLA (MHMR NEW WAIVER)','15','TEXAS HOME LIVING (TxHmL)',
        '16','Deaf Blind Waiver','17','Consolidated Waiver',
        '18','Medically Dependent Children Program (MDCP)','19','STAR+PLUS') as service_group_desc,
    stage_1.AUTHORIZED_DOLLARS,
    stage_1.AUTHORIZED_UNITS,
    stage_1.UNIT_TYPE as DHS_AUTHORIZED_UNITS_TYPE_CD,
    decode(stage_1.UNIT_TYPE,'1','WEEK','2','MONTH',
        '4','PER AUTHORIZATION','5','DAILY') as AUTHORIZED_UNITS_TYPE_DESC ,
    CONTRACT_NUMBER,
    stage_1.service_cd,
    stage_1.service_group_cd
    FROM
    STAGE_1_MG_SERVICE_REQUEST stage_1, stage_3_servcd_servgrp_dim dim
    where
    (stage_1.client_id,stage_1.begin_dt,stage_1.end_dt,dim.service_cd_desc) not in
    (select distinct stage_1.client_id,stage_1.begin_dt,stage_1.end_dt,dim.service_cd_desc
    from stage_1_mg_service_request stage_1, stage_3_servcd_servgrp_dim dim
    where stage_1.service_cd = dim.service_cd and
    stage_1.service_group_cd = dim.service_group_cd and
    stage_1.service_group_cd in ('2','3','16','17','18'))
where the_count = 1;Link to a working example here
Re: insert via insert into ....(select * from)....validtion of duplicate re

Eliminate duplicate while fetching data from source

Hi All,
CUSTOMER TRANSACTION
CUST_LOC     CUT_ID          TRANSACTION_DATE     TRANSACTION_TYPE
100          12345          01-jan-2009          CREDIT
100          23456          15-jan-2000          CREDIT
100          12345          01-jan-2010          DEBIT
100          12345          01-jan-2000          DEBITNow as per my requirement, i need to fetch data from CISTOMER_TRANSACTION table for those customer which has transaction in last 10 years. In my above data, customer 12345 has transaction in last 10 years, whereas for customer 23456, does not have transaction in last 10 years so will eliminate it.
Now, CUSTOMER_TRANSACTION table has approximately 100 million records. So, we are fectching data in batches. Batching is divided into months. Total 120 months. Below is my query.
select *
FROM CUSTOMER_TRANSACTION CT left outer join
(select distinct CUST_LOC, CUT_ID FROM CUSTOMER_TRANSACTION WHERE TRANSACTION_DATE >= ADD_MONTHS(SYSDATE, -120) and TRANSACTION_DATE < ADD_MONTHS(SYSDATE, -119) CUST
on CT.CUST_LOC = CUST.CUST_LOC and CT.CUT_ID = CUST.CUT_IDThru shell script, months number will change. -120:-119, -119:-118 ....., -1:-0.
Now the problem is duplication of records.
while fetching data for jan-2009, it will get cust_id 12345 and will fetch all 3 records and load it into target.
while fetching data for jan-2010, it will get cust_id 12345 and will fetch all 3 records and load in into target.
So instead of having only 3 records, for customer 12345 it will be having 6 records. Can someone help me on how can i eliminate duplicate records from getting in.
As of now i have 2 ways in mind.
1. Fetch all records at once. Which is impossible as it will give space issue.
2. After each batch, run a procedure which will delete duplicate records based on cust_loc, cut_id and transaction_date. But again it will have performance problem.
I want to eliminate it while fetching data from source.
Edited by: ace_friends22 on Apr 6, 2011 10:16 AM

You can do it this way....
SELECT DISTINCT cust_doc,
                cut_id
FROM customer_transaction
WHERE transaction_date >= ADD_MONTHS(SYSDATE, -120)
   AND transaction_date < ADD_MONTHS(SYSDATE, -119)However please note that - if want to get the transaction in a month like what you said earlier jan-2009 and jan-2010 and so on... you might need to use TRUNC...
Your date comparison could be like this... In this example I am checking if the transaction date is in the month of jan-2009
AND transaction_date BETWEEN ADD_MONTHS(TRUNC(SYSDATE,'MONTH'), -27) AND LAST_DAY(ADD_MONTHS(TRUNC(SYSDATE,'MONTH'), -27)) Your modified SQL...
SELECT *
FROM customer_transaction
WHERE transaction_date BETWEEN ADD_MONTHS(TRUNC(SYSDATE,'MONTH'), -27) AND LAST_DAY(ADD_MONTHS(TRUNC(SYSDATE,'MONTH'), -27))Testing..
--Sample Data
CREATE TABLE customer_transaction (
cust_loc number,
cut_id number,
transaction_date date,
transaction_type varchar2(20)
INSERT INTO customer_transaction VALUES (100,12345,TO_DATE('01-JAN-2009','dd-MON-yyyy'),'CREDIT');
INSERT INTO customer_transaction VALUES (100,23456,TO_DATE('15-JAN-2000','dd-MON-yyyy'),'CREDIT');
INSERT INTO customer_transaction VALUES (100,12345,TO_DATE('01-JAN-2010','dd-MON-yyyy'),'DEBIT');
INSERT INTO customer_transaction VALUES (100,12345,TO_DATE('01-JAN-2000','dd-MON-yyyy'),'DEBIT');
--To have three records in the month of jan-2009
UPDATE customer_transaction
   SET transaction_date = TO_DATE('02-JAN-2009','dd-MON-yyyy')
WHERE cut_id = 12345
   AND transaction_date = TO_DATE('01-JAN-2010','dd-MON-yyyy');
UPDATE customer_transaction
   SET transaction_date = TO_DATE('03-JAN-2009','dd-MON-yyyy')
WHERE cut_id = 12345
   AND transaction_date = TO_DATE('01-JAN-2000','dd-MON-yyyy');
commit;
--End of sample data
SELECT *
FROM customer_transaction
WHERE transaction_date BETWEEN ADD_MONTHS(TRUNC(SYSDATE,'MONTH'), -27) AND LAST_DAY(ADD_MONTHS(TRUNC(SYSDATE,'MONTH'), -27));Results....
CUST_LOC     CUT_ID TRANSACTI TRANSACTION_TYPE
      100      12345 01-JAN-09 CREDIT
      100      12345 02-JAN-09 DEBIT
      100      12345 03-JAN-09 DEBITAs you can see, there are only 3 records for 12345
Regards,
Rakesh
Edited by: Rakesh on Apr 6, 2011 11:48 AM

How do I eliminate duplicate email accounts on my Mac?

how do I eliminate duplicate email accounts on my Mac?

Go to Mail Preferences/Accounts. Select the Account you want to remove and hit the minus button.

How to eliminate duplicates from multiple subscriptions in iCal?

Is there a way to eliminate duplicate entries created when subscribing to multiple calendars? As an example - I have a subscription with each of my kids school calendars (one is in middle school the other elementary). I end up with several duplicates such as the following:
February 14, 2014
* Valentines Day (calendar)
* Valentines Day (elementary)
* Valentines Day (middle school)
Thanks for your help!
-Curt

Your specification can be interpreted more than one way.
In the examples you give, "duplicate records" not only have the same EMPNO, but all the data is the same. If this is what you are trying to avoid, use DISTINCT. Instead ofFROM EMP_TEST Ewrite FROM (select distinct * from EMP_TEST) EThis will eliminate duplicates if and only if every column has the same value.
Again, not sure if this is what you want but it would work for your sample data.
P.S. I agree with sb92075, you should make EMPNO a primary key and avoid the problem altogether.

How to eliminate duplicate copies after doing a sort

I did a filter/sort across 100's of folders then exported the results to a separate folder.
I later noticed way too many duplicates in the new folder.
Does Lightroom have a feature to automatically remove or eliminate duplicates when doing such a process??
If not, any other ideas??

Spaceman, you're basically correct that LR doesn't write your changes to the original file itself. LR is a non-destructive editor, which is really a good thing. It means that if you screw up an edit, you can always go back and start over. Or if you decided to convert the image to B&W, and now want to go back and re-edit it in color, you can do that. There are, however, a couple of ways to do what you want.
The easiest way is to use the Export dialog. This will create a copy of your image(s) with all the edits applied. Those copies can then be viewed in any image viewer program. You can export as jpeg or tiff files. Based on your question, I'm guessing you typically shoot and process jpeg files (since most image viewers won't display raw files), so LR will create new jpegs, with all your edits. You can give these files new names on export, or write them to a different folder. You can even automatically import the images into the LR catalog if you want. (I don't, as if I ever want to re-edit the images I'll start with the original file.)
LR can also write the changes to what's called a "sidecar" file, that you can copy along with the original image file. But I don't think very many apps know what to do with that file, either.

How do i eliminate duplicates in itunes 11?

how to eliminate duplicate songs in itunes?

Simbalover wrote:
You don't have to delete them. Go to Preferences >> Store >> Uncheck show itunes in icloud Purchases.
Didn't work (but I left it unchecked anyway).
All of my duplicates are from CD's that I own and imported myself, not purchased through iTunes.

Want to eliminate duplicates in iphoto???

How do I eliminate duplicates in iphoto??

For dealing with duplicates in iPhoto check out iPhoto Library Manager Duplicate Annihilator Decloner or iPhoto Duplicate Cleaner

I must have many duplicates over 10,000 photos-impossible! How can I find and eliminate duplicates

How can I find and eliminate duplicate photos? iPhoto says I have 10,000 or more images which cannot be!
I assume as I imported them from various discs, cameras etc there overlap?
Thank you

Some apps for dealing with duplicates in iPhoto
iPhoto Library Manager
Duplicate Annihilator
Decliner
iPhoto Duplicate Cleaner

How do you eliminate duplicate iphoto photos?

How do you eliminate duplicate iphoto photos?

There are two apps, actually one app and one Applescript script, for finding and removing duplicate photos from iPhoto. They are:
Duplicate Annihilator
iPhoto AppleScript to Remove Duplicates
OT

How do I eliminate duplicates from my nano

How do I eliminate duplicates from my nano?

Hey geor39,
To find and remove all the songs that show up more than once in your library, follow the steps in the article below:
How to find and remove duplicate items in your iTunes library
http://support.apple.com/kb/ht2905
If you're using iTunes 11, the feature is under View instead of File.
All the best,
David

Eliminate duplicates in merged lists

is there an esay way to eliminate duplicates in a merge?

Hi herbio,
More information will help to find a solution. How/what did you merge?
Hint 1: if you sort by that column, you will see duplicates come together.
Hint 2: Filter by other Columns may help.
The more you tell, the better we can help .
Regards,
Ian.

How do I eliminate duplicate songs ?, how do I eliminate duplicate songs ?

how do I eliminate duplicate songs ?

I've written a script called DeDuper which can help remove unwanted duplicates.
See this thread for background.
tt2

Query to eliminate duplicates

Similar Messages

Maybe you are looking for