Best practice for sqlldr -- direct to core or to stage first?

Question

We want to begin using sql loader to load simple (but big) tables that have, up to this point, been loaded via perl and it's DBI connection to Oracle. The target tables typically receive 10-20 million rows per day (parsed log data from many thousands of machines) and at any one time can hold more than a billion total records PER TABLE. These tables are pretty simple (typically 5-10 columns, 2 or 3 part primary keys). They are partitioned BY MONTH (DAY is always one of the primary key columns) and set up on very large SAN disk arrays, stripped, etc. I can use sqlldr to load the core tables directly, OR, I could use sqlldr to load a staging table on a daily basis, then PL/SQL and SQL+ to move data from the staging table to the core. My instinct tells me that the second route is SAFER, that is there is less chance that something catastrophic could corrupt the core table, but obviously this would (a) take more time to develop and (b) reduce our over-all throughput.
If I go the first route, loading the core directly with sqlldr, what is the worst thing that could possibly happen? That is, in anyone's experience, can a sqlldr problem corrupt a very large table? Does the likelihood of a catastrophic problem increase in proportion to the number of rows already in the target table? Are there strategies that will mitigate potential catastrophies besides going to staging and then to core via pl/sql? For example, if my core is partitioned by month, might I limit potential damage only to the current month? Are there any known potential pitfalls to using sqlldr directly in this fashion?
Thanks
matthew rapaport
[email protected]

Answer 1

Wow, thanks everyone!
1. External tables... I'd thought of this, but in our development group we have no direct access to the DBMS server so we'd have to do some workflow to move the data files to the dbms server and then write the merge. If sql loader will do the job directly (to the core) without risk, then that seems to be the most straight-forward way to go.
2. The data in the raw files is very clean, this being done in the step that parses the raw logs (100-500mb each) to the "insert files" (~20mb each), and there would be no transformations in moving data from staging to core, so again that appears to argue for direct-to-core loading.
3. The data is collected by DAY, but reported on mostly by MONTH (e.g., select day, sum(col), count(col), from TABLE where day between A and B, group by day, order by day, etc where A and B are usually the first and last day of the month) and that is why the tables are partitioned by month, but perhaps this is not the best practice (???). I'm not the DBA, but I can make suggestions... What do you think?
4. Time to review my sqlldr docs! I haven't used it in a couple of years, and I'm keeping my fingers crossed that it can handle the particular delimiter used in these files (pipe-tab-pipe expressed in perl as "|\t|". If I recall it can, but I'm not sure how to express the tab...
Meanwhile, thank you very much, you have all been a BIG help... Strange no one asked me how it was that a Microsoft company was using Oracle :-) ... I work for DANGER INC (was www.danger.com if anyone interested) which is now owned (about 9 months now) by Microsoft, and this is the legacy reporting system... :-)
matthew rapaport
[email protected]
[email protected]