Daily Incremental Load

Question

Hi All,
I got a situation where I want to load 9GB/50Million+ of data daily into my warehouse system from a 10g source database. This is just daily incremental volume. With PL/SQL what would be fastest and best way of doing this without giving heat to the source.
Thanks in Advance.

Answer 1

bi_man wrote:
With PL/SQL what would be fastest and best way of doing this without giving heat to the source. With PL/SQL? No fast and best way. As I can already see a clunky and somewhat flawed PL/SQL cursor loop selecting from the remote server over a db-link, inserting locally, doing incremental commits.
So forget PL/SQL. Data problems need to be addressed first and foremost using SQL. PL/SQL is used to manage the overall processing flow of that SQL, and deal with exceptions and so on.
The basic method for "moving rows" between source and destination tables is via an INSERT..SELECT statement.
Due to the nature and size of the move, it can be done as smaller discrete steps - allowing you to better manage the move of rows and even run it in parallel. You can do this via the DBMS_PARALLEL_EXECUTE package. You can also do that via custom criteria.
For example, every day you move yesterday's orders from the OLTP system into the warehouse. Orders are received 24x7. So you can create a procedure that takes an hour (date/time) as input, and copy all order the orders submitted during that hour. As part of this process, the procedure can log the hourly copy process in a table - listing the hour and number of rows copied. Likewise it can first check that log table to ensure that the hour has not already been successfully copied.
You now can manage the copy process at hourly level, have restart capabilities should a specific copy process fail, and the ability to start several copy processes at the same time for parallel processing.
One factor that will have an impact on the performance of this, is the size and speed of the network pipe between remote database and local database. If this is a major factor, then you could decide that a INSERT..SELECT across that network pipe is simply too expensive and slow. And instead look at options like unloading source data into a CSV file, compressing it, and then transferring it across the network. On the destination server, the file can then be uncompressed and loaded using SQL*Loader or an external table.
If the source and target tables are identical and partitioned, you can use DataPump to export the specific day's partition from the source table, and then import it into the empty partition for that day in the target table.