How to get the last run date.

Question

We intend to develop an incremental data load mapping using this strategy:
1) The mapping reads the date it was last run from an auxiliary table.
2) It selects from the source only those rows that were inserted or updated after said date.
3) Then, a post-mapping process updates the last run date in the auxiliary table, using SYSDATE.
The problem with this logic is that there is a gap: if the mapping starts running at 1:00 and ends at 2:00, the rows that are inserted in between will never be loaded.
Is there any way to get the value when the mapping started running? Is there a better way to do this?
Any help would be appreciated.
Juan Algaba

Answer 1

There is always the possibility of some record updates slipping through the crack if you are depending on dates unless you are very carefull. All of the audit tasks that the OWB-generate code performes take time. Any pre- or post- process that needs to run takes time. So which date is the best cuttoff point to equate to "when the last run of the merge (or insert or update) statement completed"?
Plus, how do you handle reloads if the previous load failed and your mapping had incremental commits?
Is your source on another server? If so, are the dates in perfect synch? The audit tables populate with sysdate of your runtime schema. Is that the same as the sysdate on your source remote database?
I would qualify my query to look for all updates since the start of the last run that finished successfully - adjusted if neccesary for sysdate differences if it is on a remote schema. And make sure that your code handles any reloads gracefully in the event that this brings back data that you have already loaded once. .
Because we use Oracle Streams to load a local staging area, we also have custom code to dump the primary keys of all data changes to utility staging tables while streams is updating the local copy. So, our Person table has an st_Person_delta table that just holds the primary keys that have been updated by Streams since the last ETL run.
During datamart load we disable the streams apply to stabilize our environment, and join these lists of pk's to their source tables to drive our ETL. So we only select data where Streams has performed an update to the row since our last run. When we are done our ETL, we truncate the primary key staging tables, and then turn streams back on to start loading up our new delta into our staging tables again..
The ETL gets pretty complex though when many tables join together in one mapping and you need to check all possible source table deltas to see if any of them got updated to determine the delta for a given dimension or fact record, but it works great once you get it all done.