Multiple source databases

Question

Hello,
I need to load data into dimensions and cube from multiple source databases. We have separate transaction databases for each country, but data must be loaded into common target schema. At the moment we have created mappings which loads data from one source database.
What is the best way to populate, for example customers dimension from customers transaction tables (located in multiple source databases) if business identifiers can overlap?

Answer 1

I would break this up into two pieces: the load from the source systems into the staging area, and then the load from the staging area into the target table, which you mentioned was a dimension.
There is no reason you would have to use views or synonyms if you don't want to. You could approach this either with a SET operator, in this case a union, that brought all the sources together into the staging table. However, I would likely create multiple mappings for each source system. I would create a mapping for each table that you have to pull data from, and then create a separate process flow that controls all the tables being pulled from a particular source database. This allows you to schedule the extractions from the source systems independently of each other in case they need to run at different times. The end result of all of this will be a series of staging tables that are loaded from source tables from all the different locations. For each of the singular mappings, I would create a constant that defined the source of the data for this mapping, and load that constant into a column called SOURCE_SYSTEM, or perhaps COUNTRY, as you mentioned. Then, this SOURCE_SYSTEM/COUNTRY column + SOURCE_SYSTEM_ID would serve as the natural key.
You can choose to persist these tables (as an ODS of sorts) or not... that is up to you and your requirements.
Finally, I would create a single mapping that loads the final target table from the single staging table.
Let me know if this doesn't make sense.
Regards,
Stewart Bryson