Insert data from table1 with PK to remove duplicates from table2 without PK

Question

There will probably be a simple solution to my issue although it is currently evading me. I have a table that contains 14 million rows. There are many duplicates within this table that I need to remove. I was hoping an old trick I used in MS Access would
help, this solution was to create an identical table and add a PK on the filed I wanted to remove the duplicates from. Then just copy+paste append from one table to the other and Access would just drop the rows that were duplicated. Very simple yet worked
quickly and effectively. Is there a quick way to do this in MS SQL as everything i've tried so far has failed. Any help would be greatly appreciated.
Ian

Answer 1

try below
create table source_table (
column1 int,
column2 varchar(32)
create table destination_table
column1 int primary key,
column2 varchar(32)
insert into source_table (column1, column2) values (1, 'VAL2');
insert into source_table (column1, column2) values (2, 'VAL3');
insert into source_table (column1, column2) values (3, 'VAL4');
insert into source_table (column1, column2) values (4, 'VAL5');
insert into source_table (column1, column2) values (4, 'VAL5');
insert into source_table (column1, column2) values (4, 'VAL5');
GO
-- simple method
WITH CTE AS
select
ROW_NUMBER() OVER (partition by column1 ORDER BY column2) rno, column1, column2
from source_table )
insert into destination_table(column1, column2)
select column1, column2 from CTE where rno =1
-- inserting in while loop little chunks(500000), you can reduce if you want
Declare @startid int =0, @totalrecords int = 14000000, @count int = 500000
while(@startid <= @totalrecords)
begin
WITH CTE AS
select
ROW_NUMBER() OVER (partition by column1 ORDER BY column2) rno, column1, column2
from source_table where column1 between @startid and @startid+@count)
insert into destination_table(column1, column2)
select column1, column2 from CTE where rno =1
Print convert(varchar , @startid+@count)+ ' processed' -- indication while running
set @startid = @startid+@count
end
Thanks
Saravana Kumar C