Incremental loading - select taking a lot of time - urgent please

Question

Hi,
I have a huge source oracle table with millions of records. I am using a last_update_date field from this table to select the new records. I have the last_update_date indexed. Also, please note number of distinct values of this field is not high.
1.
I have the last_update_date of the previous load in another metadata table (in the target system though).
In my mapping, I have the source, I then have a metadata table from where I get the value that is the last_update_date in the prev load.
I have a expression which actually converts this value from the metadata table to date field using TO_DATE(...). I have a filter which has inputs from this expression and the fields that I require from the source table. I then use te filter condition as INOUTGRP1.LAST_UPDATE_DATE(from source) > INOUTGRP1.LAST_LOADED_DATE (from metedata). This does not use the index becasue it takes a lot of time to load (approx 30 mins for 10k new records).
2.
Just to test I hard coded the LAST_LOADED_DATE in the filter condition like
INOUTGRP1.LAST_UPDATE_DATE(from source) > TO_DATE('03/15/2007 05:30:00 PM', 'MM/DD/YYYY HF:MI:SS PM'). This uses the index and takes about 15 seconds to load the same 10k records.
I cant hard-code the date because this will keep changing as I load.
Any suggestions please.
Thanks a lot in advance!
Maruthi

Answer 1

Hi
I had exact performance issus as your before. Here is my solution.
1. create a function like this:
d_edw_run_date date;
-- main body
BEGIN
select edw_run_date
into d_edw_run_date
from estg_mrif.loading_date;
RETURN d_edw_run_date;
EXCEPTION
WHEN OTHERS THEN
NULL; -- enter any exception code here
RETURN NULL;
END;
you can add your to_date function in the select statement. Let's name this one as GET_EDW_RUN_DATE.
2. Call this function in each of your mapping as a Pre_Mapping operator.
3. Drag all of your source plus the VALUE of this Function (in Pre_Mapping) to a filter. Add the filter condition as INOUTGRP1.LAST_UPDATE_DATE(from source) > INOUTGRP1.VALUE.
You'll be satisfied with the performance now because the returned value from the function will be used as a Constant Value in your generated SELECT and surely your index will be used.
As you didn't mention what your DB version is, you can have an additional fix it if this one doesn't work very well. For example change the condition "> " to "Between <the start date> and <A constant date>" .
Hope this will work for you.
Regards,
Jenny