Ledger detail and associated account. URGENT

Question

Ahsan/Navin,
The question primarily revolves around dimension tables and incremental load. If we take out SCD1/2 out of the scene for now, could you please answer Yes or No for the below questions? Please it is kind of urgent as the client is really worried about these aspects of data transfer in incremental loads. As always, appreciate your quick response. Assume that first full load has been done.
1. If we change a key field (effective date) of an account that does not have a new transaction associated with it in the ledger – will the change come over after an incremental load?
2. If we change a non key field (title) on a prior effective date of an account that does not have a new transaction associated with it in the ledger – will the change come over after an incremental load?
3. If we change a non key field (title) on a prior effective date of an account that does have a new transaction associated with it in the ledger – will it reflect after an incremental load?
4. During a load process Full/Incremental, dimension tables get loaded first right? IF not, and facts are loaded first, wont they error out seeing that no associated dimension details exist?
5. We have a few custom tables which dont have any sort of a datetimestamp columns. Is bringing them all over everytime load happens, the only option? What is the best method to handle these kind of situations?
Thanks,
Dan

Answer 1

For your first 3 questions, it depends on the LAST UPDATE DATE in the key tables in the source system. This differs a little bit across the various source systems but in general, if the source system table has a "Update Date" which is used as a filter for the Incrmental load SQL Qualifier, then it will capture that record...and if you only have SCD Type 1, the whole row gets updated for all the fields. If you have SCD Type 2 enabled, then ONLY dimension columns that are part of TYPE2_DIFF will result in end dating the existing record and Iinserting a NEW record. So for SCD Type 2, you need to see if additional columns need to be added to capture history. OOTB only the records in the TYPE2DIFF are compared for capturing history.
For your other questions:
4. During a load process Full/Incremental, dimension tables get loaded first right? IF not, and facts are loaded first, wont they error out seeing that no associated dimension details exist?
A: Yes, Dims are usually loaded first for each star schema. However, note that there are sometimes Persisted staging, and other auxiliary tables for facts that may be loaded but in general yes, DIMS before FACTS.
5. We have a few custom tables which dont have any sort of a datetimestamp columns. Is bringing them all over everytime load happens, the only option? What is the best method to handle these kind of situations?
A: If you NEED to capture history on these custom tables, you can do this w custom changes on the DW side via Informatica. If you dont need history then I dont see the problem with reloading them every time..do you have a large data volume?? If not, then why is it an issue??