Ssis to load dimension table
i want to be able to generate a dimension out of fact table
the attribute is the name of a store which can be spelt anyway.
but in the dimension table, i want all the member values with similar spelling to correspond to a single store and mapped to a unique key.
how is this one using ssis
the fact table may have the followng for walmart
walmart
walmartab
walmart123
how to map all these walmarts to a single walmart and also associate a unique key
the dimension table should have
ID storename displayname
1 walmart walmart123
2 walmart walmart
3 walmart walmartab
OR, would it be a better idea to clean the source and then have a dimension like below?
1 walmart
please advice wahat is the general best practice for designing as well as loading the dimension table
thanks very much
do you mean, the fact table needs to be cleansed? or can it be left alone with diffent values within the fact tablebeing mapped to a single value, which requires cleaning the dimension?
which is the preferred approach and why?
if the dimension is cleaned this would be the resulting dimension
ID storename displayname
1 walmart walmart123
2 walmart walmart
3 walmart walmartab
OR, would it be a better idea to clean the source and then have a dimension like below?
1 walmart
Its dimension that needs cleansing. The fact will only have fk to dimension using key value isnt it? So you need to make sure you complete the cleansing in the dimension and based on which you map the key values correctly in fact to correspond to survived key
value for collapsed values
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs
Similar Messages
-
Best Practice loading Dimension Table with Surrogate Keys for Levels
Hi Experts,
how would you load an Oracle dimension table with a hierarchy of at least 5 levels with surrogate keys in each level and a unique dimension key for the dimension table.
With OWB it is an integrated feature to use surrogate keys in every level of a hierarchy. You don't have to care about
the parent child relation. The load process of the mapping generates the right keys and cares about the relation between the parent and child inside the dimension key.
I tried to use one interface per Level and created a surrogate key with a native Oracle sequence.
After that I put all the interfaces in to one big Interface with a union data set per level and added look ups for the right parent child relation.
I think it is a bit too complicated making the interface like that.
I will be more than happy for any suggestions? Thank you in advance!
negib
Edited by: nmarhoul on Jun 14, 2012 2:26 AMHi,
I do like the level keys feature of OWB - It makes aggregate tables very easy to implement if your sticking with a star schema.
Sadly there is nothing off the shelf with the built in knowledge modules with ODI , It doesnt support creating dimension objects in the database by default but there is nothing stopping you coding up your own knowledge module (use flex fields maybe on the datastore to tag column attributes as needed)
Your approach is what I would have done, possibly use a view (if you dont mind having it external to ODI) to make the interface simpler. -
Update/Load Dimension table attributes(non-key elements)
Hi,
Is there a configuration change in DAC or Informatica where I can opt to load all the dimension tables during an incremental load, irrespective of an associated transaction in the fact table? I observed that for all those accounts with no corresponding transactions are extracted but dropped by the time they reach the actual dimension table. So in the final target DWH, only those unique accounts are brought which have an associated transaction.
I know we can create an aux mapping to include some kind of a flag to trick DAC into thinking the corresponding base fact table also has changed and then bring over the dimension data. But is there a simple alternative?
For example, In GL we have an account dimension which has an associated line transaction details for each account in the fact table. I just change the Account name from A to B. No change or updates to the associated line entry in the FACT table. Now I run an incremental load. Will the change ( new account name) be recognized and brought over? OR will it be brought over only when the associated line entry also changes?
Usually when the dollar amounts dont change, nobody cares but in our case the client wants it. I have googled extensively to understand incremental loading but cud not get a clear response to the above question. Does SCD2 play any role in this?
Thanks,
DanAhsan,
The question primarily revolves around dimension tables and incremental load. If we take out SCD1/2 out of the scene for now, could you please answer Yes or No for the below questions? Please it is kind of urgent as the client is really worried about these aspects of data transfer in incremental loads. As always, appreciate your quick response. Assume that first full load has been done.
1. If we change a key field (effective date) of an account that does not have a new transaction associated with it in the ledger – will the change come over after an incremental load?
2. If we change a non key field (title) on a prior effective date of an account that does not have a new transaction associated with it in the ledger – will the change come over after an incremental load?
3. If we change a non key field (title) on a prior effective date of an account that does have a new transaction associated with it in the ledger – will it reflect after an incremental load?
4. During a load process Full/Incremental, dimension tables get loaded first right? IF not, and facts are loaded first, wont they error out seeing that no associated dimension details exist?
5. We have a few custom tables which dont have any sort of a datetimestamp columns. Is bringing them all over everytime load happens, the only option? What is the best method to handle these kind of situations?
Thanks,
Dan -
Load fact table with null dimension keys
Dear All,
We have OWB 10g R2 and ROLAP star schema. In our source system some rows don’t have all attributes populated with values (null value), and this empty attributes are dimension (business) keys in star schema. Is it possible to load fact table with such rows (some dimension keys are null) in the OWB mappings? We use cube operator in mappings.
Thanks And Regards
MiranThe dimension should have a row indicating UNKNOWN, this will have a business key outside of the normal range e.g. -999999.
In the mapping the missing business keys can then be NVL'd to -999999.
Cheers
Si -
ERROR while loading time dimension table
i need to load time dimension from csv to oracle table, while loading i got the error.
my source data type is date and target is date.
ODI-1226: Step sample day fails after 1 attempt(s).
ODI-1240: Flow sample day fails while performing a Loading operation. This flow loads target table W_SAMPLE_DATE.
ODI-1228: Task SrcSet0 (Loading) fails on the target ORACLE connection Target_Oracle.
Caused By: java.sql.SQLException: ORA-30088: datetime/interval precision is out of range
while creating c$ table
create table WORKSCHEMA.C$_0W_SAMPLE_DATE
C3_ROW_WID NUMBER(10) NULL,
C1_CALENDAR_DATE TIMESTAMP() NULL,
C2_DAY_DT TIMESTAMP() NULL
)check the source data and use the correct function eg TO_DATE(SRC.DATE, 'MM?DD/YYYY') use NVL if required.
-
Incremental load into the Dimension table
Hi,
I have the problem in doing the incremental load of the dimension table.Before loading into the dimension table,i would like to check the data in the dimnesion table.
In my dimension table i have one not null surrogate key and the other null dimension tables.The not null surrogate key, i am populating with the Sequence Generator.
To do the incremental load i have done the following.
I made lookup into the dimension table and looked for a key.The key from the lookup table i have passed to the expression operator.In the expression operator i have created one field and hard coded one flag based on the key from the lookup table.I passed this flag to the filter operator and rest of the fields from the source.
By doing this i am not able to pass the new records to the dimension table.
Can you please help me.
I have another question also.
How do i update one not null key in the fact table.
Thanks
VinayHi Mark,
Thanks for your help to solve my problem.I thought i share more information by giving the sql.
I am giving below the 2 sqls, i would like to achieve through OWB.
Both the following tasks need to be accomplished after loading the fact table.
task1:
UPDATE fact_table c
SET c.dimension_table_key =
(SELECT nvl(dimension_table.dimension_table_key,0)
FROM src_dimension_table t,
dimension_table dimension_table
WHERE c.ssn = t.ssn(+)
AND c.date_src_key = to_number(t.date_src(+), '99999999')
AND c.time_src_key = to_number(substr(t.time_src(+), 1, 4), '99999999')
AND c.wk_src = to_number(concat(t.wk_src_year(+), concat(t.wk_src_month(+), t.wk_src_day(+))), '99999999')
AND nvl(t.field1, 'Y') = nvl(dimension_table.field1, 'Y')
AND nvl(t.field2, 'Y') = nvl(dimension_table.field2, 'Y')
AND nvl(t.field3, 'Y') = nvl(dimension_table.field3, 'Y')
AND nvl(t.field4, 'Y') = nvl(dimension_table.field4, 'Y')
AND nvl(t.field5, 'Y') = nvl(dimension_table.field5, 'Y')
AND nvl(t.field6, 'Y') = nvl(dimension_table.field6, 'Y')
AND nvl(t.field7, 'Y') = nvl(dimension_table.field7, 'Y')
AND nvl(t.field8, 'Y') = nvl(dimension_table.field8, 'Y')
AND nvl(t.field9, 'Y') = nvl(dimension_table.field9, 'Y')
WHERE c.dimension_table_key = 0;
fact table in the above sql is fact_table
dimesion table in the above sql is dimension_table
source table for the dimension table is src_dimension_table
dimension_table_key is a not null key in the fact table
task2:
update fact_table cf
set cf.key_1 =
(select nvl(max(p.key_1),0) from dimension_table p
where p.field1 = cf.field1
and p.source='YY')
where cf.key_1 = 0;
fact table in the above sql is fact_table
dimesion table in the above sql is dimension_table
key_1 is a not null key in the fact table
Is it possible to achieve the above tasks through Oracle Warehouse builder(OWB).I created the mappings for loading the dimension table and fact table and they are working fine.But the above two queries i am not able to achieve through OWB.I would be thankful if you can help me out.
Thanks
Vinay -
Loading dimensions using interface table
Hello,
I am trying to load dimensions into EPMA 11.1.1.3 using interface tables and need some direction. I have an accounts dimension file with the following columns ( Parent, Child, Alias, Description, DataType,Aggregation). I configured a data source using configuration wizard (Interface tables).
I see in the database that some sample tables are created. Now, should I create a separate table with the columns listed above?
Can someone let me know what are the next steps to successfully import accounts dimension?
ThanksCan someone plz respond?
This is important for me. I have gone thru' the documentation and also the sample tables but don't understand why are there multiple tables for a particular dimension viz. hierarchy, property etc?
Please tell me the right way.
Thanks again!! -
Should my dimension table be this big
Im in the process of building my first product dimension for a star schema and not sure if im doing this correctly. A brief explanation of our setup
We have products (dresses) made by designers for specific collections each of which has a range of colors and each color can come in many sizes. For our UK market this equates to some 1.9 million
product variants. Flattening the structure out to Product+designer+collection gives about 33,000 records but when you add all the colors and then all the colors sizes it pumps that figure up to 1.9million. My “rolled own” incremental ETL load runs
ok just now, but we are expanding into the US market and our projections indicate that our products variants could multiple 10 fold. Im a bit worried about performance of the ETL just now and in the future.
Is 1.9m records not an awful lot of records to incrementally load (well analyse) for a dimension table nightly as it is never mind what that figure may grow to when we go to US?
I thought of somehow reducing this by using a snowflake but would this not just reduce the number of columns in the dimensions and not the row count?
I then thought of separating the colors and size into their own dimension, but this doesn’t seem right as they are attributes of products and also I would lose the relationship between products, size & color I.e. I would have to go through the
fact table (which ive read is not a good choice.) for any analysis.
Am I correct in thinking these are big numbers for a dimension table? Is it even possible to reduce the number somhow?
Still learning so welcome any help.
ThanksHi Plip71,
In my opinion, It is always good to reduce the Dimension Volume as much as possible for better performance.
Is there any Hierarchy in you product Dimension?.. Going for a snowflake for this problem is a bad idea..
Solution 1:
From the details given by, It is good to Split the Colour and Size as seperate dimension. This will reduce the vloume of dimension and increase the column count in the fact(seperate WID has to be maintained in the fact table). but, this will improve the
performance of the cube. before doint this please check the layout requirement from the business.
Solution 2:
Check the distinct count of Item varient used in fact table. If it is very less, they try creating a linear product dimension. i.e, create an view for the product dimension doing a inner join with the fact table. so that only the used Dimension member will
be loaded in the Cube Product Dimension. hence volume is reduced with improvement in performance and stability of the cube.
Thanks in advance, Sorry for the delayed reply ;)
Anand
Please vote as helpful or mark as answer, if it helps Regards, Anand -
Static and small dimension table should I use lookup
Hi Gurus,
I am creating interface to load my fact table.
Generally for different codes to replace with surrogate keys I use dimension table for look-up in interface.
but I have question if dimension table is very small and has only static values like
Sur_key, Code, Description.
1, C, Create
2 , R, Redeem
in such case if in interface for fact table, I just write expression "if source_table.column=C then 1 else 2".
Please advice if this is better way or not.
Thanks in advance.I agree - it is much better practice to use lookups to populate surrogate keys. Using hard-coded values in code requires the developer to always remember that dependency on dimensional data and can cause unexpected results down the line if the keys are updated or are inconsistent across environments.
-
What is '#Distinct values' in Index on dimension table
Gurus!
I have loaded my BW Quality system (master data and transaction data) with almost equivalent volume as in Production.
I am comparing the sizes of dimension and fact tables of one of the cubes in Quality and PROD.
I am taking one of the dimension tables into consideration here.
Quality:
/BIC/DCUBENAME2 Volume of records: 4,286,259
Index /BIC/ECUBENAME~050 on the E fact table /BIC/ECUBENAME for this dimension key KEY_CUBENAME2 shows #Distinct values as 4,286,259
Prod:
/BIC/DCUBENAME2 Volume of records: 5,817,463
Index /BIC/ECUBENAME~050 on the E fact table /BIC/ECUBENAME for this dimension key KEY_CUBENAME2 shows #Distinct values as 937,844
I would want to know why the distinct value is different from the dimension table count in PROD
I am getting this information from the SQL execution plan, if I click on the /BIC/ECUBENAME table in the code. This screen gives me all details about the fact table volumes, indexes etc..
The index and statistics on the cube is up to date.
Quality:
E fact table:
Table /BIC/ECUBENAME
Last statistics date 03.11.2008
Analyze Method 9,767,732 Rows
Number of rows 9,767,732
Number of blocks allocated 136,596
Number of empty blocks 0
Average space 0
Chain count 0
Average row length 95
Partitioned YES
NONUNIQUE Index /BIC/ECUBENAME~P:
Column Name #Distinct
KEY_CUBENAMEP 1
KEY_CUBENAMET 7
KEY_CUBENAMEU 1
KEY_CUBENAME1 148,647
KEY_CUBENAME2 4,286,259
KEY_CUBENAME3 6
KEY_CUBENAME4 322
KEY_CUBENAME5 1,891,706
KEY_CUBENAME6 254,668
KEY_CUBENAME7 5
KEY_CUBENAME8 9,430
KEY_CUBENAME9 122
KEY_CUBENAMEA 10
KEY_CUBENAMEB 6
KEY_CUBENAMEC 1,224
KEY_CUBENAMED 328
Prod:
Table /BIC/ECUBENAME
Last statistics date 13.11.2008
Analyze Method 1,379,086 Rows
Number of rows 13,790,860
Number of blocks allocated 187,880
Number of empty blocks 0
Average space 0
Chain count 0
Average row length 92
Partitioned YES
NONUNIQUE Index /BIC/ECUBENAME~P:
Column Name #Distinct
KEY_CUBENAMEP 1
KEY_CUBENAMET 10
KEY_CUBENAMEU 1
KEY_CUBENAME1 123,319
KEY_CUBENAME2 937,844
KEY_CUBENAME3 6
KEY_CUBENAME4 363
KEY_CUBENAME5 691,303
KEY_CUBENAME6 226,470
KEY_CUBENAME7 5
KEY_CUBENAME8 8,835
KEY_CUBENAME9 124
KEY_CUBENAMEA 14
KEY_CUBENAMEB 6
KEY_CUBENAMEC 295
KEY_CUBENAMED 381Arun,
The cube in QA and PROD are compressed. Index building and statistics are also up to date.
But I am not sure what other jobs are run by BASIS as far as this cube in production is concerned.
Is there any other Tcode/ Func Mod etc which can give information about the #distinct values of this Index or dimension table?
One basic question, As the DIM key is the primary key in the dimension table, there cant be duplicates.
So, how would the index on Ftable on this dimension table show #distinct values less than the entries in that dimension table?
Should the entries in dimension table not exactly match with the #Distinct entries shown in
Index /BIC/ECUBENAME~P on this DIM KEY? -
Dimension key 16 missing in dimension table /BIC/DZPP_CP1P
Hi all,
I have a problem with an infocube ZPP_CP1. I am not able to delete nor load any data. It was working fine till some time back.
Below is the outcome of running RSRV check on this cube. I tried to run the error correction in RSRV. But n o use
Please help.
Dimension key 16 missing in dimension table /BIC/DZPP_CP1P
Message no. RSRV018
Diagnosis
The dimension key 16 that appears as field KEY_ZPP_CP1P in the fact table, does not appear as a value of the DIMID field in the dimensions table /BIC/DZPP_CP1P.
There are 17580 fact records that use the dimension key 16.
The facts belonging to dimension key 16 are therefore no longer connected to the master data of the characteristic in dimension.
Note that errors that are reported for the package dimension are not serious (They are thus shown as warnings (yellow) and not errors (red). When deleting transaction data requests, it can arise that the associated entries in the package dimension have already been deleted. As a result, the system terminates when deleting what can be a very large number of fact records. At the moment, we are working on a correction which will delete such data which remains after deletion of the request. Under no circumstances must you do this manually. Also note that data for request 0 cannot generally be deleted.
The test investigates whether all the facts are zero. If this is the case, the system is able to remove the inconsistency by deleting these fact records. If the error cannot be removed, the only way to re-establish a consistent status is to reconstruct the InfoCube. It may be possible for SAP to correct the inconsistency, for which you should create an error message.
Procedure
This inconsistency can occur if you use methods other than those found in BW to delete data from the SAP BW tables (for example, maintaining tables manually, using your own coding or database tools).Hi Ansel,
There has been no changes in the cube. I am getting this problem in my QA server. So I retransported the cube again from Dev to QA. But did not help me..
Any other ideas??
Regards,
Adarsh -
Fact Tables without Dimension Tables.
Hi,
I am currently working on a project where a data warehouse is being developed using an application database as the source. Since the application database is already normalised, the tables are being loaded in as they are and the consultant involved is creating
data marts for them. I have asked about why he is not creating separate dimension and fact tables as data marts, and he has told me that he will create a fact table with the dimensions as column names without any dimension tables created as you might expect
as per star/snowflake schema.
Is this right, as I always thought you had to have dimension and fact tables, especially if you want to start using SSAS?
Regards,
W.create a fact table with the dimensions as column names without any dimension tables created as you might expect as per star/snowflake schema.
This is the trouble with denormalization: where is the stopping point?
If the data warehouse is the source for SSAS OLAP cubes, it is better to go with the usual star schema.
Kalman Toth Database & OLAP Architect
SQL Server 2014 Database Design
New Book / Kindle: Beginner Database Design & SQL Programming Using Microsoft SQL Server 2014 -
Missing most detailed table for dimension tables
Hi ,
I am getting this following error
Business Model Core:
[nQSError: 15003] Missing most detailed table for dimension tables: [Dim - Customer,Dim - Account Hierarchy,Dim - Account Region Hierarchy,Fact - Fins - Period Days Count].
[nQSError: 15001] Could not load navigation space for subject area Core.
I got this error when I tried to configure # of Elapsed Days and # of Cumulative Elapsed Days by following way-
1. Using the Administration Tool, open OracleBIAnalyticsApps.rpd.
Configuration Steps for Controlling Your Data Set
Configuring Oracle Financial Analytics 5-51
The OracleBIAnalyticsApps.rpd file is located at:
ORACLE_INSTANCE\bifoundation\OracleBIServerComponent\coreapplication_
obisn\repository
2. In the Business Model and Mapping layer, go the logical table Fact - Fins - Period
Days Count.
3. Under Sources, select the Fact_W_DAY_D_PSFT logical table source.
4. Clear the Disabled option in the General tab and click OK.
5. Open the other two logical table sources, Fact_W_DAY_D_ORA and Fact_W_
DAY_D_PSFT, and select the Disabled option.
6. Add the "Fact - Fins - Period Days Count" and "Dim - Company" logical tables to
the Business Model Diagram. To do so, right-click the objects and select Business
Model Diagram, Selected Tables Only.
7. In the Business Model Diagram, create a new logical join from "Dim - Company"
to "Fact - Fins - Period Days Count." The direction of the foreign key should be
from the "Dim - Company" logical table to the "Fact - Fins - Period Days Count"
table. For example, on a (0,1):N cardinality join, "Dim - Company" will be on the
(0/1) side and "Fact - Fins - Period Days Count" will be on the N side.
8. Under the Fact - Fins - Period Days Count logical table, open the "# of Elapsed
Days" and "# of Cumulative Elapsed Days" metrics, one at a time.
9. Go to the Levels tab. For the Company dimension, the Logical Level is set to All.
Click the X button to remove it. Repeat until the Company dimension does not
have a Logical Level setting.
10. Make sure to check Global Consistency to ensure there are no errors, and then
save the RPD file.
Please help me to resolve.
Thanks,
SoumitroCould you let me know how you resolved this. I am facing the same.
-
"Missing most detailed table for dimension tables" eror when I run the Global Consistency check
ERRORS:
Business Model DAC Measures:
[nQSError: 15003] Missing most detailed table for dimension tables: [D_DETAILS,D_EXECUTION_PLAN,D_TASK].
[nQSError: 15001] Could not load navigation space for subject area DAC Measures.
I am also attaching my Business Model layer for easy understanding. I have a fact table and several Dimension table. I got this error only after creating the following hierarchies:
Execution Plan -> Tasks -> Details
Start Date Time Hierarchy
End Date Time Hierarchy
Is there a solution for this problem? Thanks in advance!Yes ! My Task Hierarchy has 3 dimension tables that form a hierarchy :Execution Plan -> Tasks -> Detail
All the 3 levels in the hierarchy are 3 different dimension tables. -
How to update a fact table when a dimension table is reloaded
We have implemented BI Apps 796. Insertion into W_EMPLOYEE_D table which stores all the employee information had stopped one year back as some company security policy restricted the informatica worklfows to pick up the data. (PER_ALL_PEOPLE_F was a HRMS table and it contained sensitive information line SSN and salary, was inaccessible to the user which informatica uses and the SDE mapping used to return 0 rows).
Now we have the approval to see those rows and the dimension table is loaded with some 100 new employees who joined in last one year.
The ROW_WID of W_EMPLOYEE_D is referenced in lot of fact tables and for all those missing employees the WID in the fact table is 0.
Now that we have all employees, how to make the FACT table point to the correct WID and not store 0. Has anyone faced this problem before?? Writing an update statement will be a tedious task as there are so many fact tables that join to w_employee_d. Also our company uses Sales, Procurement, Finance modules of OB Apps (which constitutes atleast 20 fact tables)
Any guidance is appreciated. Thanks in advanceHello Kostis,
thank you for your answer. I don't fully understand you. Can you show me short example, please? I create alias table for time dimension on Physical Layer - original table is TimeDayDim and I create aliases TimeDayDim1, TimeDayDim2, TimeDayDim3, TimeDayDim4. Then I create foreign key Fact.Time1 -> TimeDayDim1, Fact.Time2 -> TimeDayDim2, Fact.Time3 -> TimeDayDim3, Fact.Time4 -> TimeDayDim4. And what now? Must I create these table api Bussines Model and create new time dimensions at bussiness model????
I need in Answers ONE Time dimension. I think I must split my fact table to four tables ... (time1, place1 ...) (time2, place2 ...) (time3 place3...) (time4 place4...) then link those tables to Time dimension (but I dont know where I can split those tables - on Physical Layer or on Bussines Layer).
I suppose that I will have in Answers one time dimension and four facts tables and I will be able to query them. (for example: Time.Days, Fact1.Place1, Fact3.Speed, Fact4.Count Criteria: Time.Year = 2008)
Best Regards Vlada
Maybe you are looking for
-
I recently upgrade from Iphotos 09 to 11 and my place assignments are gone. Please help me with this. Also, how can I assing places to pictures in iphotos 11? Thanks. Vishy <Email Edited by Host>
-
Load flat files from S3 into SQL Server 2008 R2
Hi, We have a few customers dropping files in Amazon S3. Could you please let me know how to load this data into SQL Server 2008 R2 database using SSIS? We are 2008 R2 BIDS environment. Thanks in advance.............. Ione
-
IPhoto '08 stuck in July, recent events missing
iPhoto 7.1.4 is behaving oddly. The last 12 Months folder is stuck on 08/07/2008 and all the imports since then are missing. The original images are there in the originals folder. I've tried rebuilding the library and using iPhoto Library Manager but
-
How can I embed Variant To Data in a subVI and pass in the Type input?
If I use Create Control on the Type input of Variant To Data, I get an undefined type error unless I insert a control of a specific type into the cluster control that was created. This seems to prevent passing a parameter into a subVI for this purpos
-
Hi there, I have a question regarding testing eCATT. I have Golden client which I have authorization to create/change eCATT. However there is no test data in this DEV client. In other DEV client, it has test data but I don't have authorization to