Transfering 100+million records

Question

Hi Guys
i have source table (TAB1) having 100+ million records.. needs to copy to another database... to reduce the numbers... created the join with another table (TAB2) which reduced it to 10+ million records but data services taking long time (almost 40 minutes) to show the numbers (1000) in monitor.
See the attached pic for the dataflow created..
could you guys please help me to let me know what is the best practice to implement same in less time...
DS4.2 on windows server 2008
Data Services server has 32 GB RAM (out of which this job is using 6GB only)
SQL SERVER has 64 GB RAM (out of which this job is using 22GB only)
Regards

Answer 1

Does that mean you don't have access to your target db either? Because that's where you create your linked server.
Anyway, try and put your join and filter conditions directly in the query transform, under the FROM-tabs and WHERE-tabs, i.e. without using a global variable. The sql generated will look like:
     SELECT  tab1.* FROM TAB1
     INNER JOIN TAB2
     ON tab1.c1 = tab2.c2
     WHERE tab2.date >= <fromdate>
If you then set the "Transfer type" of the Data_Transfer transform to "Table" and make sure that the temp table is created in your source db (I hope you'll have the authorisation to do so), you will get:
     INSERT INTO temp
     SELECT  tab1.* FROM TAB1
     INNER JOIN TAB2
     ON tab1.c1 = tab2.c2
     WHERE tab2.date >= <fromdate>
This bypasses DS processing completely and most probably will be the best you can achieve if you cannot touch your db environment itself.
On another note, the native MS SQL Server client 2008 will outperform any ODBC driver.