Use of ODI in Data Import scenario

Question

Hi,
We are contemplating to use ODI for a Data Import scenario. The flow goes something like this:
1.A user specified flat file or xml with a mapping between the file columns and base table columns serves as the input.
(Note that the file column names could be anything and that is why we use mapping).
2. This file data is stored in a stager table and certain validations are run on this data.
3. This data is then imported into the base tables.
I assume we cannot use ODI for step 1 as the file columns are not fixed and can vary. We need to programmatically interpret the data from the mappings. (Is there a way to do this in ODI?).
If we use ODI for step 3 to import data from stager to base tables:
- If we have a million records to be imported, then how performant is ODI? Do we need to invoke ODI in batches of (a few thousands) to improve performance?
- Thanks in Advance,
Raghu

Answer 1

Hi Jont,
Thanks for your reply..
Here is an example of the mapping that we use:
Flat File columns:
AccName
AccLoc
Mapping (Specified by the user at run time):
AccName -->(Maps to) Account.Name
AccLoc --> (Maps to) Account.Location
The user would map the file columns to the final target entity fields (like Account.Name) as above.
Since, we have to store this data in a intermediate staging table, we would have a mapping internally (which is fixed), like
Account.Name -->(maps to) AccStager.Name
Account.Location -->(Maps to) AccStager.Location
where AccStager.Name is the staging table field name.
Thus, by using these two sets of mapping, we store the file data into staging table.
Hope this is clear...
Thanks,
Raghuveer