Implementing surrogate keys in dimensions

hello,
First thing, I'm new to ODI! I am using Oracle data integrator 10.1.3.
I have a dimension table 'Dim_Contracts' as target table. The structure is as follows:
PK_Dim_Contract Primary key (surrogate key - to be populated from an Oracle database sequence in the target)
Contract_ID (normal field in target - no constraints in target- to be populated from source - originally a primary key in source)
+ other dimension attributes.
from what i have googled out and read in the forum, i cannot define 'PK_Dim_Contract' as the primary key of my dimension (target) table, to be able to update it from the oracle sequence defined - rather the 'contract_ID', which is the natural key should be the primary key. Is that correct? If yes, isn't it against dimension modelling principle?
More to the point, my question is: How do I populate a sequence in my primary key field in the target table?
Thanks for your help.
Regards,
Anju

Hello Anju,
Welcome in the ODI community ;).
What I suggest you is to set the UNIQUE KEY on Contract_ID in your target. This way you will be able to use flow control and do Incremental Update Loading.
PK_Dim_Contract (surrogate key) can be your primary key in the dabatase.
To populate PK_Dim_Contract from an Oracle Sequence, create it first in your Oracle DB. Add a new sequence to your project (left pane), choose Natural Sequence, choose your schema and enter the name of your Oracle Sequence.
In your interface, define the mapping of PK_Dim_Contract as
:<ODI_SEQUENCE_NAME>_NEXTVALand execute this mapping on the target.
Note: :<ODI_SEQUENCE_NAME>_NEXTVAL works only for SQL Statements. If you want to use the sequence somewhere else, use the following syntax :
#<ODI_SEQUENCE_NAME>_NEXTVALHope it helps,
Jerome

Similar Messages

About Surrogate Key and Dimension Key on OWB 10.2

Hi, everyone.
I am using OWB 10.2 and I have a question about Surrogate key and Dimension Key.
I indicated the foreign key as VARCHAR2 type in Fact Table and Dimension Key as VARCHAR2 type is operated as Primary key in Dimension Table. I made Single Level in Dimension Table.
I know that Dimension Key stores the surrogate ID for dimension and is the primary key of the table. Also, Surrogate ID should be only NUMBER type.
So, in this case, Surrogate ID is NUMBER type
Dimension key should be NUMBER type to store the surrogate ID.
But, Dimension key also should operate the primary to relate Foreign key as VARCHAR2 type.
How I can solve this confusing condition?
Please let me know that.
JWS

Hi JWS,
From a SQL point of view it should not be a problem to join a NUMBER field to a VARCHAR2 field because during execution there will be an implicite cast for the NUMBER value to a VARCHAR2 value. See the example below.
SELECT * FROM DUAL
WHERE 1 = '1'From an OWB point of view it is not possible to have a Dimension with an NUMBER value Key that has a relation to a VARCHAR2 value Foreign key in a Fact table. This is caused due to the creation of a Fact table in OWB in which the Foreign keys in it are build from de Dimension tables that refer to them.
You will loose the reference to the Dimension when changing the type of the Foreign Key.
To resolve this issue I would advise you to use a Sequence that generates your Surrogate Key (NUMBER type) for the Dimension table and store it in the Primary Key Column (VARCHAR2 type).
When validating the mapping you will get a warning, but when executing this should give no problems.
Regards,
Ilona

How to create surrogate key in dimension without unique value

Hi, I have a dimension where there is no column with unique value. I want to add a surrogate key to replace the existing primary key which is derived from concatenating 3 columns(e.g. 'A'||'B'||'C'). I'm thinking of using sequence. But this won't allow me to link the dimension to fact table. How do I come up with surrogate key under this situation? Thanks. ~Tracy

I'm actually trying to accomplish something similar myself.
In my sources I've got two sorts of customers, ones that are directly reported, and ones whose information is provided with sales records (this is stored in module ODS).
Of course identification is different, but in the datamart (module DWH) I'm sort of forced to use an equivalent way of loading (due to the way it first used to work). To accelerate lookups on dimensions, I copy the ODS surrogate key to DWH dimensions, but this does not work for the 'inbuilt' customers because they do not have a surrogate key in the ODS.
They DO have means of unique identification, and at first I thought I could concatenate these (also 3) columns to use as identification code. Unfortunately this is VARCHAR2, where the surrogate key is (naturally) NUMBER.
So now it looks like I'm forced to first build a table in ODS especially for these 'inbuilt' customers and assign a surrogate key (by sequence) to it, this way it conforms to how 'normal' customers are loaded into DWH.
I guess you'll have to pull of the same trick, i.e. create a table with either only the 'translation' of D-code to a surrogate key or all information that is fed into the dimension, which then can be used as a lookup or as complete source when loading data into your datamart.
Good luck, Patrick

Maintainence of surrogate keys

Hi,
Does anyone suggest what ways of maintaining the
surrogate keys in dimension tables? I really want
to know if OWB have this kind of feature to
achieve this purpose?
Regards,
Raymond

Fair enough, I think I misunderstood your question. I will defer to someone able to better discuss the relational side of things. I know this is / can be done using insert triggers, etc. - but since this is a completely common thing to do in a DW, I would expect OWB to have special functionality to address this without resorting to code.
I don't know OWB though, so sorry I can't give a specific answer.
Thx,
Scott

Surrogate key for fact

Hi all,
i don't know that in ETL update of existing records in fact can happen or not. IF it happen then how can i handle it? is it recommendable to implement surrogate key for fact tables to update the existing records of fact? That work like primary key for fact table. This doesn't have any relation with OWB. This is a general concept about ETL.
Regards,
Sumanta

Yes, you can update a fact.
You generally do this by adding a unique constraint to the fact table which comprises the identifying primary key columns from the source tables, but exclusive of the surrogate key. The surrogate key will be the actual primary key of the table.
Now, when you make an insert/update mapping you include a sequence that generates your surrogate key, and then select the 'match by' on the fact table to match by the unique key (actual source table values which determine uniqueness) rather than the primary key. Then, on the surrogate key column on the fact table which is being populated by the sequence, you set its property to not update the surrogate key when an update is being applied to the row.
This way the update statement will not update the surrogate key when updating the fact based upon the source identifiers that identify this fact.
I'm sure I could explain this more clearly... but I'm very pressed for time right now.
Cheers,
Mike

How to Maintain Surrogate Key Mapping (cross-reference) for Dimension Tables

Hi,
What would be the best approach on ODI to implement the Surrogate Key Mapping Table on the STG layer according to Kimball's technique:
"Surrogate key mapping tables are designed to map natural keys from the disparate source systems to their master data warehouse surrogate key. Mapping tables are an efficient way to maintain surrogate keys in your data warehouse. These compact tables are designed for high-speed processing. Mapping tables contain only the most current value of a surrogate key— used to populate a dimension—and the natural key from the source system. Since the same dimension can have many sources, a mapping table contains a natural key column for each of its sources.
Mapping tables can be equally effective if they are stored in a database or on the file system. The advantage of using a database for mapping tables is that you can utilize the database sequence generator to create new surrogate keys. And also, when indexed properly, mapping tables in a database are very efficient during key value lookups."
We have a requirement to implement cross-reference mapping tables with Natural and Surrogate Keys for each dimension table. These mappings tables will be populated automatically (only inserts) during the E-LT execution, right after inserting into the dimension table.
Someone have any idea on how to implement this on ODI?
Thanks,
Danilo

Hi,
first of all please avoid bolding something. After this according Kimball (if i remember well) is a 1:1 mapping, so no-surrogate key.
After that personally you could use Lookup Table
http://www.odigurus.com/2012/02/lookup-transformation-using-odi.html
or make a simple outer join filtering by your "Active_Flag" column (remember that this filter need to be inside your outer join).
Let us know
Francesco

Best Practice loading Dimension Table with Surrogate Keys for Levels

Hi Experts,
how would you load an Oracle dimension table with a hierarchy of at least 5 levels with surrogate keys in each level and a unique dimension key for the dimension table.
With OWB it is an integrated feature to use surrogate keys in every level of a hierarchy. You don't have to care about
the parent child relation. The load process of the mapping generates the right keys and cares about the relation between the parent and child inside the dimension key.
I tried to use one interface per Level and created a surrogate key with a native Oracle sequence.
After that I put all the interfaces in to one big Interface with a union data set per level and added look ups for the right parent child relation.
I think it is a bit too complicated making the interface like that.
I will be more than happy for any suggestions? Thank you in advance!
negib
Edited by: nmarhoul on Jun 14, 2012 2:26 AM

Hi,
I do like the level keys feature of OWB - It makes aggregate tables very easy to implement if your sticking with a star schema.
Sadly there is nothing off the shelf with the built in knowledge modules with ODI , It doesnt support creating dimension objects in the database by default but there is nothing stopping you coding up your own knowledge module (use flex fields maybe on the datastore to tag column attributes as needed)
Your approach is what I would have done, possibly use a view (if you dont mind having it external to ODI) to make the interface simpler.

Building dimensions that are based on surrogate keys

Hi -
I am new to AWM and have a question.
I read the docs, but I still do not understand how to build a cube using AWM.
So we have a DW with dimensions and fact tables. The dimension tables use surrogate keys.
The fact table uses the surrogate keys of the dimensions.
I want to define the time dimension in AWM.
The unique key for the dimension is the surrogate key (time_key).
In AWM I defined levels: day, month and year.
Would the hierarchy for the day level be:
time_key -> day?
Thanks,
Frank

Thankyou so much.
now when I try that here is the error that I get for that
line:
1118: Implicit coercion of a value with static type
flash.display:DisplayObject to a possibly unrelated type
flash.display:MovieClip.
I've placed a zipfile with my FLA, .as and .xml here:
http://bigfins.com/temp/test5xml.zip
The actionscript alone is below.
Thankyou again for any assistance.

How we use Surrogate Keys for snowflake dimension

Hi All,
my question is - How we use Surrogate Keys for snowflake dimension
i heard from some body Surrogate Keys only work with star schema.
please correct me if i wrong.
Regards,
Manish

Hi manishcal16PPS,
According to your description, you can only create natural key in your dimension. But it's not working when using surrogate key. Right?
In Analysis Services, the snowflake schema of the dimensions are represented by more than one dimension table in other words its takes multiple dimension tables to define a dimension. Surrogate key are just some extra, redundant, unique key based on the
natural key. So there's no direct relationship or some limitations between surrogate keys and snowflake schema.
In this scenario, since there's relationship between the two dimensions, you should create natural key. For using natural key or surrogate key. Please refer to an article below:
Surrogate Key vs. Natural Key
For understanding star/snowflake schema, please see:
Understanding Star and Snowflake Schemas
Regards,
Simon Hou
TechNet Community Support

Surrogate Key and Map for Cube

Hi
I am new to Data Warehousing and am trying to use OWB 11g.
I am trying to create dimensions with multiple levels. When I create more than one level it need to have surrogate as well business key for each dimension level. But I can create only one surrogate in the dimension, there is no option to create multiple surrogate keys in the same dimension. so what am I missing?
My second question is regarding cube. Do I need to create a Mapping for a cube? if yes, should I move the data to the cube from the dimensions? and where will the measures come from? do i need to load the measures or they will be calculated automatically?
please reply...
regards
Arif

hi
Got it, Yes that was the reason,
The table was not properly deployed after the dimension was modified.
Anyway, the describe of the table is as follows
describe arif.QUESTION_DIM
Name Null Type
DIMENSION_KEY NOT NULL NUMBER
IGV_ID NUMBER
PER_ID NUMBER
DIM_ID NUMBER
IGO_ID NUMBER
INQ_ID NUMBER
ID NUMBER
DIM_ORDEM NUMBER
DIM_AMBITO VARCHAR2(3)
DIM_NOME VARCHAR2(150)
10 rows selected
Now, I am having another problem,
when, I deploy the Map to load the data from three different tables, it gives the following problem
Name               Action               Status          Log
QUESTION_MAP          Create               Warning          ORA-06550: line 297, column 25:
                                        PLS-00302: component 'ID' must be declared
QUESTION_MAP          Create               Warning          ORA-06550: line 1153, column 11:
                                        PL/SQL: SQL Statement ignored
QUESTION_MAP          Create               Warning          ORA-06550: line 1155, column 15:
                                        PL/SQL: ORA-00904: "QUESTION_DIM"."ID": invalid identifier
QUESTION_MAP          Create               Warning          ORA-06550: line 1155, column 31:
                                        PLS-00302: component 'ID' must be declared
QUESTION_MAP          Create               Warning          ORA-06550: line 233, column 1:
                                        PL/SQL: SQL Statement ignored
QUESTION_MAP          Create               Warning          ORA-06550: line 2539, column 11:
                                        PL/SQL: SQL Statement ignored
QUESTION_MAP          Create               Warning          ORA-06550: line 2541, column 15:
                                        PL/SQL: ORA-00904: "QUESTION_DIM"."ID": invalid identifier
QUESTION_MAP          Create               Warning          ORA-06550: line 2541, column 31:
                                        PLS-00302: component 'ID' must be declared
QUESTION_MAP          Create               Warning          ORA-06550: line 297, column 9:
                                        PL/SQL: ORA-00904: "QUESTION_DIM"."ID": invalid identifier
Edited by: user643560 on Oct 22, 2008 9:38 AM

Problems maintaning surrogate keys

Hi,
I am trying to load a dimension table. the initial load worked fine. now for the regular load if there is one a new record in the source table it does adds 1 new record in the dimension table. But the problem is the surrogate keys(I am using a oracle sequence for this) skips the number of record in the dimension table and assigs next value to the new record. The lode type is "insert/Update".Constraint used for maching is the primary key of source table.
e.g
--initial run.. table had 5 records before
-- Second run.. table has 6 records but the surrogate keys value for the 6th record will be '11' insted of being '6'
can you help me with this.....

Nawneet,
after I posted this problem on the forum i read an article that said i should do something like what you said. so i changed the seq to an function which returns only the next value when called. i imported it to the map and used it insted of the seq...but still no use....the map is doing the same thing during the second run. i also tried chaning the load type from insert/update to update/insert. also changed the default operating mode to row based....still no use...:((.....
what happens is during the first run 5 records get loaded perfectly and every thign is fine, but if I run it again with no new changes in the source records the map runs fine and the data is still good. but what happens is the nextval of the sequence jumps to 11...when i am expecting it to remain at 6 as no new records have been added to the dimension table. I think some how owb is seeing 5 records in the source table during the second run and uses the next val frmo the cache...but does not use them and then since it not used this values go waste.
Problem: I can not schedule the load of the dimension table for every day as the next value of this seq will keep on skeeping and when ever a new record is added to dimension table will have a large value .

Loading data into Fact/Cube with surrogate keys from SCD2

We have created 2 dimensions, CUSTOMER & PRODUCT with surrogate keys to reflect SCD Type 2.
We now have the transactional data that we need to load.
The data has a customer id that relates to the natural key of the customer dimension and a product id that relates to the natural key of the product dimension.
Can anyone point us in the direction of some documentation that explains the steps necessary to populate our fact table with the appropriate surrgoate key?
We assume that we need to have an lookup table between the current version of the customer and the incoming transaction data - but not sure how to go about this.
Thanks in advance for your help.
Laura

Hi Laura
There is another way to handling SCD and changing Facts. This is to use a different table for the history. Let me explain.
The standard approach has these three steps:
1. Determine if a change has occurred
2. End Date the existing record
3. Insert a new record into the same table with a new Start Date and dummy End Date, using a new surrogate key
The modified approach also has three steps:
1. Determine if a change has occurred
2. Copy the existing record to a history table, setting the appropriate End Date en route
3. Update the existing record with the changed information giving the record a new Start Date, but retaining the original surrogate key
What else do you need to do?
The current table can use the surrogate key as the primary key with the natural key being being a unique key. The history table has the surrogate key and the end date in the primary key, with a unique key on the natural key and the end date. For end user queries which in more than 90% of the time go against current data this method is much faster because only current records are in the main table and no filters are needed on dates. If a user wants to query history and current combined then a view which uses a union of the main and historical data can be used. One more thing to note is that if you adopt this approach for your dimension tables then they always keep the same surrogate key for the index. This means that if you follow a strict Kimball approach to make the primary key of the fact table be a composite key made up of the foreign keys from each dimension, you NEVER have to rework this primary key. It always points to the correct dimension, thereby eliminating the need for a surrogate key on the fact table!
I am using this technique to great effect in my current contract and performance is excellent. The splitter at the end of the map splits the data into three sets. Set one is for an insert into the main table when there is no match on the natural key. Set two is when there is a match on the natural key and the delta comparison has determined that a change occurred. In this case the current row needs to be copied into history, setting the End Date to the system date en route. Set three is also when there is a match on the natural key and the delta comparison has determined that a change occurred. In this case the main record is simply updated with the Start Date being reset to the system date.
By the way, I intend to put a white paper together on this approach if anyone is interested.
Hope this helps
Regards
Michael

How to load composite business key into Dimension

Hi,
Maybe what I'm asking is very basic but, well, here I go:
I have a dimension, CollectingAgent, and several tables to populate from:
Bank (bank_code, bank_name ) [PK: bank_code]
Agency (bank_code, agency_code, agency_name, address, etc) [PK: bank_code, agency_code]
Now, for my purposes, collecting agent is the bank+agency descriptions concatenated. I don't know hoy to load, or map, the combinated key (bank_code, agency_code) into the business field of the dimension. OWB won't let me define more than one attribute as business key.
I'm using OWB 10g Rel2.
Thanks in advance for your help,
--oswaldo.
[osantos]

Ok, let me try to give an example made up with completely bogus information. I'm going to make a dimension called clothing that has two hierarchies that look like this:
"Item" hierarchy:
All Clothing --> Item Type --> Color / Item
"Color" hierarchy:
All Clothing --> Color --> Color / Item
Item type would be values such as pants, shirts, dresses, etc. The low level "color / item" is just a combination of those two (i.e. "Blue Shirt") - and note in this example that it will have 2 composite natural keys. It will roll up over the item type hierarchy under shirts, and under the color hierarchy in blue. Here's a sample of what I'm talking about:
(Item hier)
All Products
...Pants
......Blue Pants
......Red Pants
......Green Pants
...Shirts
......Blue Shirts
......Red Shirts
etc.
(Color Hier)
All Products
...Blue
......Blue Pants
......Blue Shirts
...Red
......Red Pants
......Red Shirts
like I said, completely bogus, but this will let me illustrate how to set up the dimension. I'd create the following attributes (hard to format, sorry):
Key....................Surrogate Key
ID.......................Natural Key
Item_Type_ID......Natural Key
Color_ID..............Natural Key
Short_Desc.........short description
Long_Desc..........long description
Having created these attributes, I'd then create the following levels and assign the attributes as follows:
All_Products_Level:
...Key
...ID
...Short_Desc
...Long_Desc
Item_Type_Level:
...Key
...Item_Type_ID
...Short_Desc
...Long_Desc
Color_Level:
...Key
...Color_ID
...Short_Desc
...Long_Desc
Color_Item_Level:
...Key
...Color_ID
...Item_Type_ID
...Short_Desc
...Long_Desc
Should be pretty easy from there. The whole trick is to have a SERIES of different natural keys - a "generic" one (ID) for dimension levels you don't really care about, and "named" ones (color_id, item_type_id) you can use for given levels. And note that you CAN have more than one natural key at any level, and you can have some natural keys that are not used at all levels.
For your simple dimension, you can probably do the following (note that I'm assuming the lowest level has your composite two business keys, while the top level is completely separate):
TOP_LEVEL:
...Key (for the surrogate key)
...ID (for the business natural key)
...Short_Desc
...Long_Desc
BOTTOM_LEVEL:
...Key (for the surrogate key)
...Business_natural_key_1 (rename to be match your first business key)
...Business_natural_key_2 (rename to match your second business key)
...Short_Desc
...Long_Desc
Hope this helps! If you need to see something a little more concrete, feel free to email me at SPowell at columbus.rr.com, send me some details on exactly what your keys are, etc. and I'll whip up a quick example for you.
Thanks,
Scott

Hash partitioning v. list partitioning on surrogate key + partition pruning

Hi,
Have a large fact table with surrogate keys, therefore queries are of form
select dimension.attribute..
from fact, dima, dimb..
where facta.dima_surrogate_key = dima.dimension_key
and facta.dimb_surrogate_key = dimb.dimension_key
and dima.attribute = <value>
and dimb.attribute = <value>
Would ideally like partition pruning to happen but will this happen if hash partition on facta.surrogate_key
Likewise could list partition on facta.dima_surrogate_key and further sub-partition on hash of factb.dima_surrogate_key.
Any advice much appreciated.

user5716448 wrote:
Hi,
Version 11.2.0.1
fact table structure
PRODUCT_ID NUMBER
RETAILER_ID NUMBER
OUTLET_ID NUMBER
CALENDAR_ID NUMBER
BRANCH_ID NUMBER
PUBLISHER_ID NUMBER
DISTRIBUTOR_ID NUMBER
TRANS_TYPE_ID NUMBER
TRANS_QTY NUMBER (10)
TRANS_VALUE (10,4)
No date on fact table (just surrogate_id for calendar whihc links to calendar/date dimension.
Although queries can be by date of transaction, most aren't.
Potential to grow to 3 billion rows.
Considering hash partitioning on the product_id, simply to break data down and product_id is the largest dimension.About hash partitioning – in this case it is probably all about the ability to run in parallel. Do not have any info on that, so I cannot comment further.
>
sqls are varied, lots of different types some query all dimensions, sometimes a few. Not the straightforward date examples in the manual.You can pick a dimension that is frequently used by the SQLs. I understand that there is no perfect one, but even if you pick just a “good” one you might have a good deal of partition elimination.
>
Users run 3rd part ad-hoc reporting layer which has to allow them to report against the star in any way they want.
Star transformation hint enabled. Have heard in deciding number of hash partitions, partition size should geneerally be < 2gb.
e.g transactions for a given product for customers belonging to a given multiple in a given week
select trans_qty, trans_value, m.prod_name, m.prod_num, r.cust_name, w.branch_name, rtt.trans_date, rtt,trans_type
from retailer_transaction rt, media m, wholesaler w, calendar c, retailer r, trans_type rtt
where rt.issue_id = m.dimension_key
and m.prod_num = 600
and rt.branch_id = w.dimension_key
and rt.outlet_id = r.dimension_key
and r.multiple_num = 700
and rt.calendar_id = calendar.dimension_key
and m.issue_year_week = 201110
and rt.trans_type_id = rtt.dimension_keyLastly, you need to focus on weather and how to partition your indexes (I assume you have bunch of bitmaps). This decision is at least as important as partitioning the table.

What can I do to create natural keys for dimension (cwm2, dbms_aw)?

I create OLAP catalog and AW via cwm2 and dbms_aw packages. Unfortunately made AW uses surrogate keys (that is names of levels are stuck to values of the dimensions, e.g. LEVELNAME_value). Is there any way to change it?

You should be able use the API invocation:
dbms_awm.set_awdimload_spec_parameter(<loadspec>,
<cwm source dim owner>,
<cwm source dim name>,
'UNIQUE_RDBMS_KEY',
'NO')
Change the last parameter from NO to YES.
This requires that the key (dimension) values in your source RDBMS tables are unique across levels.
For example, if you have source RDBMS data like:
CITIES_TABLE
KEY PARENT
BOSTON MASSACHUSETTS
NEW_YORK NEW_YORK
HARTFORD CONNECTICUT
STATES_TABLE
MASSACHUSETTS
NEW_YORK
CONNECTICUT
Then you need to leave the parameter as NO to get 6 AW dimension values of:
CITY_BOSTON
CITY_NEW_YORK
CITY_HARTFORD
STATE_MASS
STATE_NEW_YORK
STATE_CONNECTICUT
If you set it to YES for source data like above, you will only incorrectly get 5 AW dimension values of:
BOSTON
NEW_YORK
HARTFORD
MASSACHUSETTS
CONNECTICUT
Alternatively you could set it to YES and change your source data for the offending row to something like:
CITIES_TABLE
KEY PARENT
BOSTON MASSACHUSETTS
NEW_YORK_CITY NEW_YORK
HARTFORD CONNECTICUT

Implementing surrogate keys in dimensions

Similar Messages

Maybe you are looking for