SSIS Fact table load

Hi Experts,
Need you valuable suggestion.
I am loading a fact table which is having 16 look ups to get the surrogate keys from dimensions.Volume is around 3million.
It is taking lot of time to load the table (around 3-4 hours). Is it because of large number of lookup in a single package?
can you please give some design tips to reduce my load time? Also I am having 6 date fields to lookup with Date dim. Is there any way to get connection once and use it ? I came to know about cache manager for that. Can anyone please help me to understand
that?
Thanks
mukejee

hi, I am now doing the same action with you,
loading a fact table which is having 14 look ups to get the surrogate keys from dimensions, while I encounter tough errors when executing fact table, errors are here:
[FactoryLookup [352]] Error: Row yielded no match during lookup.
[FactoryLookup [352]] Error: The "component "FactoryLookup" (352)" failed because error code 0xC020901E occurred, and the error row disposition on "output "Lookup Output" (354)" specifies failure on error. An error occurred on the specified object of the
specified component.
[DTS.Pipeline] Error: The ProcessInput method on component "FactoryLookup" (352) failed with error code 0xC0209029. The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and
will cause the Data Flow task to stop running.
[DTS.Pipeline] Error: Thread "WorkThread0" has exited with error code 0xC0209029.
I have no idea where goes wrong, Could you please help me?
Thanks a lot.

Similar Messages

Fact Table loading in parallel

Hi
I have 3 fact tables which loads daily 300,000 records and runs in sequence . now i want to run these 3 fact tables load in parallel .
if i run these 3 fact table load in parallel will there be any performance impact on running quries ?
regards
Srinivas

Hello,
I have had a case once where the parallel mechanism in SSIS was consuming so much memory that the pacakge failed every now and then. I have had to revert and make the tasks run sequentially. Luckily this is done simply by joining them in the BI designer.
After that the package no longer failed. In short the impact is not so much on the SQL server engine running the queries, but on the System that runs the SSIS package and consumes too much memory.
Jan D'Hondt - SQL server BI development

Using WHERE NOT EXISTS for a Fact Table Load

I'm trying to set up a fact table load using T SQL, and I need to use WHERE NOT EXISTS. All of the fields from the fact table are listed in the WHERE NOT EXISTS clause. What I expect is that if the value of any one of the fields is different, that the whole
record be treated as a new record, and inserted into the table. However, in my testing, when I 'force' a field value, new records are not inserted.
The following is my query:
declare
@Created_By nchar(50)
,@Created_Date datetime --do we need utc check?
,@Updated_By nchar(50)
,@Updated_Date datetime
select @Created_By = system_user
,@Created_Date = getdate()
,@Updated_By = system_user
,@Updated_Date = getdate()
insert fact.Appointment
Slot_ID
, Slot_DateTime
, Slot_StartDateTime
, Slot_EndDateTime
, Slot_Duration_min
, Slot_CreateDateTime
, Slot_CreateDate_DateKey
, Healthcare_System_ID
, Healthcare_Service_ID
, Healthcare_Supervising_Service_ID
, Healthcare_Site_ID
, Booked_Appt_ID
, Appt_Notification_Submission_DateKey
, Appt_Notification_Completion_DateKey
, Appt_Notification_Duration
, Appt_Notification_ID
, Patient_ID
, Physician_ID
, Referral_ID
, Specialty
, LanguageRequested
, Created_Date
, Created_By
, Updated_Date
, Updated_By
select distinct
Slot.Slot_ID
, Slot.Slot_Start_DateTime as Slot_DateTime --???
, Slot.Slot_Start_DateTime
, Slot.Slot_End_DateTime
, datediff(mi,slot.Slot_Start_DateTime,slot.Slot_End_Datetime) as Slot_Duration_Min
, Slot.Created_Date as Slot_CreateDateTime
, SlotCreateDate.Date_key as Slot_CreateDate_DateKey
, HSite.Healthcare_System_ID
, HSite.Healthcare_Service_ID
, HSite.Healthcare_Service_ID as Healthcare_Supervising_Service_ID
, HSite.Healthcare_Site_ID
, Ref.Booked_Appt_ID
, ApptSubmissionTime.Date_key as Appt_Notification_Submission_DateKey
, ApptCompletionTime.Date_key as Appt_Notification_Completion_DateKey
, datediff(mi,appt.SubmissionTime,appt.CompletionTime) as Appt_Notification_Duration
, Appt.Appt_Notification_ID
, pat.Patient_ID
, 0 as Physician_ID
, ref.Referral_ID
, Hsrv.Specialty
, appt.[Language] as LanguageRequested
,@Created_Date as Created_Date
,@Created_By as Created_By
,@Updated_Date as Updated_Date
,@Updated_By as Updated_By
from dim.Healthcare_System HSys
inner join dim.Healthcare_Service HSrv
on HSys.Healthcare_System_ID = HSrv.HealthCare_System_ID
inner join dim.Healthcare_Site HSite
on HSite.HealthCare_Service_ID = HSrv.Healthcare_Service_ID
and HSite.HealthCare_System_ID = HSrv.HealthCare_System_ID
inner join dim.Referral Ref
on Ref.ReferralSite_ID = HSite.Site_ID
and Ref.ReferralService_ID = HSite.Service_ID
and Ref.ReferralSystem_ID = HSite.System_ID
right join (select distinct Slot_ID, Source_Slot_ID, Slot_Start_DateTime, Slot_End_DateTime, Created_Date from dim.slot)slot
on ref.Source_Slot_ID = slot.Source_Slot_ID
inner join dim.Appointment_Notification appt
on appt.System_ID = HSys.System_ID
inner join dim.Patient pat
on pat.Source_Patient_ID = appt.Source_Patient_ID
inner join dim.SystemUser SysUser
on SysUser.Healthcare_System_ID = HSys.Healthcare_System_ID
left join dim.Calendar SlotCreateDate
on SlotCreateDate.Full_DateTime = cast(Slot.Created_Date as smalldatetime)
left join dim.Calendar ApptSubmissionTime
on ApptSubmissionTime.Full_DateTime = cast(appt.SubmissionTime as smalldatetime)
left join dim.Calendar ApptCompletionTime
on ApptCompletionTime.Full_DateTime = cast(appt.CompletionTime as smalldatetime)
where not exists
select
Slot_ID
, Slot_DateTime
, Slot_StartDateTime
, Slot_EndDateTime
, Slot_Duration_min
, Slot_CreateDateTime
, Slot_CreateDate_DateKey
, Healthcare_System_ID
, Healthcare_Service_ID
, Healthcare_Supervising_Service_ID
, Healthcare_Site_ID
, Booked_Appt_ID
, Appt_Notification_Submission_DateKey
, Appt_Notification_Completion_DateKey
, Appt_Notification_Duration
, Appt_Notification_ID
, Patient_ID
, Physician_ID
, Referral_ID
, Specialty
, LanguageRequested
, Created_Date
, Created_By
, Updated_Date
, Updated_By
from fact.Appointment
I don't have any issues with the initial insert, but records are not inserted on subsequent inserts when one of the WHERE NOT EXISTS field values changes.
What am I doing wrong?
Thank you for your help.
cdun2

so I set up a WHERE NOT EXIST condition as shown below. I ran the query, then updated Slot_Duration_Min to 5. Some of the Slot_Duration_Min values resolve to 15. What I expect is that when I run the query again, that the records where Slot_Duration_Min resolves
to 15 should be inserted again, but they are not. I am using or with the conditions in the WHERE clause because if any one of the values is different, then a new record needs to be inserted:
declare
@Created_By nchar(50)
,@Created_Date datetime
,@Updated_By nchar(50)
,@Updated_Date datetime
select
@Created_By = system_user
,@Created_Date = getdate()
,@Updated_By = system_user
,@Updated_Date = getdate()
insert fact.Appointment
Slot_ID
, Slot_DateTime
, Slot_StartDateTime
, Slot_EndDateTime
, Slot_Duration_min
, Slot_CreateDateTime
, Slot_CreateDate_DateKey
, Healthcare_System_ID
, Healthcare_Service_ID
, Healthcare_Supervising_Service_ID
, Healthcare_Site_ID
, Booked_Appt_ID
, Appt_Notification_Submission_DateKey
, Appt_Notification_Completion_DateKey
, Appt_Notification_Duration
, Appt_Notification_ID
, Patient_ID
, Physician_ID
, Referral_ID
, Specialty
, LanguageRequested
, Created_Date
, Created_By
, Updated_Date
, Updated_By
select distinct
Slot.Slot_ID
, Slot.Slot_Start_DateTime as Slot_DateTime --???
, Slot.Slot_Start_DateTime
, Slot.Slot_End_DateTime
, datediff(mi,slot.Slot_Start_DateTime,slot.Slot_End_Datetime) as Slot_Duration_Min
, Slot.Created_Date as Slot_CreateDateTime
, SlotCreateDate.Date_key as Slot_CreateDate_DateKey
, HSite.Healthcare_System_ID
, HSite.Healthcare_Service_ID
, HSite.Healthcare_Service_ID as Healthcare_Supervising_Service_ID
, HSite.Healthcare_Site_ID
, Ref.Booked_Appt_ID
, ApptSubmissionTime.Date_key as Appt_Notification_Submission_DateKey
, ApptCompletionTime.Date_key as Appt_Notification_Completion_DateKey
, datediff(mi,appt.SubmissionTime,appt.CompletionTime) as Appt_Notification_Duration
, Appt.Appt_Notification_ID
, pat.Patient_ID
, 0 as Physician_ID
, ref.Referral_ID
, Hsrv.Specialty
, appt.[Language] as LanguageRequested
,@Created_Date as Created_Date
,@Created_By as Created_By
,@Updated_Date as Updated_Date
,@Updated_By as Updated_By
from dim.Healthcare_System HSys
inner join dim.Healthcare_Service HSrv
on HSys.Healthcare_System_ID = HSrv.HealthCare_System_ID
inner join dim.Healthcare_Site HSite
on HSite.HealthCare_Service_ID = HSrv.Healthcare_Service_ID
and HSite.HealthCare_System_ID = HSrv.HealthCare_System_ID
inner join dim.Referral Ref
on Ref.ReferralSite_ID = HSite.Site_ID
and Ref.ReferralService_ID = HSite.Service_ID
and Ref.ReferralSystem_ID = HSite.System_ID
right join (select distinct Slot_ID, Source_Slot_ID, Slot_Start_DateTime, Slot_End_DateTime, Created_Date from dim.slot)slot
on ref.Source_Slot_ID = slot.Source_Slot_ID
inner join dim.Appointment_Notification appt
on appt.System_ID = HSys.System_ID
inner join dim.Patient pat
on pat.Source_Patient_ID = appt.Source_Patient_ID
inner join dim.SystemUser SysUser
on SysUser.Healthcare_System_ID = HSys.Healthcare_System_ID
left join dim.Calendar SlotCreateDate
on SlotCreateDate.Full_DateTime = cast(Slot.Created_Date as smalldatetime)
left join dim.Calendar ApptSubmissionTime
on ApptSubmissionTime.Full_DateTime = cast(appt.SubmissionTime as smalldatetime)
left join dim.Calendar ApptCompletionTime
on ApptCompletionTime.Full_DateTime = cast(appt.CompletionTime as smalldatetime)
where not exists
select
Slot_ID
, Slot_DateTime
, Slot_StartDateTime
, Slot_EndDateTime
, Slot_Duration_min
, Slot_CreateDateTime
, Slot_CreateDate_DateKey
, Healthcare_System_ID
, Healthcare_Service_ID
, Healthcare_Supervising_Service_ID
, Healthcare_Site_ID
, Booked_Appt_ID
, Appt_Notification_Submission_DateKey
, Appt_Notification_Completion_DateKey
, Appt_Notification_Duration
, Appt_Notification_ID
, Patient_ID
, Physician_ID
, Referral_ID
, Specialty
, LanguageRequested
, Created_Date
, Created_By
, Updated_Date
, Updated_By
from fact.Appointment fact
where
Slot.Slot_ID = fact.Slot_ID
or
Slot.Slot_Start_DateTime = fact.Slot_DateTime
or
Slot.Slot_Start_DateTime = fact.Slot_StartDateTime
or
Slot.Slot_End_DateTime = fact.Slot_EndDateTime
or
datediff(mi,slot.Slot_Start_DateTime,slot.Slot_End_Datetime) =
fact.Slot_Duration_min
or
Slot.Created_Date = fact.Slot_CreateDateTime
or
SlotCreateDate.Date_key = fact.Slot_CreateDate_DateKey
or
HSite.Healthcare_System_ID = fact.Healthcare_System_ID
or
HSite.Healthcare_Service_ID = fact.Healthcare_Service_ID
or
HSite.Healthcare_Service_ID =
fact.Healthcare_Service_ID
or
HSite.Healthcare_Site_ID = fact.Healthcare_Site_ID
or
Ref.Booked_Appt_ID = fact.Booked_Appt_ID
or
ApptSubmissionTime.Date_key =
fact.Appt_Notification_Submission_DateKey
or
ApptCompletionTime.Date_key =
fact.Appt_Notification_Completion_DateKey
or
datediff(mi,appt.SubmissionTime,appt.CompletionTime) = fact.Appt_Notification_Duration
or
Appt.Appt_Notification_ID = fact.Appt_Notification_ID
or
pat.Patient_ID =
fact.Patient_ID
or
0 = 0
or
ref.Referral_ID = fact.Referral_ID
or
Hsrv.Specialty = fact.Specialty
or
appt.[Language] = fact.LanguageRequested

Fact table loading

Hi all,
We have two dimensions and one fact table as
Dim 1 Geography Dim2 consumer
geo_id town cons_id cons_name geo_id
G1 ABC C1 A1 G1
G2 DEF C2 B1 G2
G3 GHI C3 C1 G2
G4 JKL C4 D1 G2
G5 MNO C5 E1 G1
C6 F1 G1
Fact FACT1
FACTI_ID GEO_ID CONS_ID
F1 G1 C1
F2 G2 C2
F3 G2 C3
F4 G2 C4
F5 G1 C5
F6 G1 C6
should we need to load G3,G4,G5 into fact table

actuallay when we are creating repository in obiee by linking fact and dimension tables and queiring the list of consumers in each town we are getting oUtput as
TOWN COUNT(CONS_ID) (from consumer table)
ABC 3
DEF 3
but we need the output as
TOWN COUNT(CONS_ID) (from consumer table)
ABC 3
DEF 3
GHI 0
JKL 0
MNO 0
we have tried with outer join between the fact and dimensional table but no use.
As the GEO_ID (G3,G4,G5,G6) are not in the fact table the matching between fact and dimension is not taking place
So Kindly reply....

Issue with non calculated column in a fact table

Hi All,
With 3 facts(Fact1,Fact2,Fact3) and 2 Confirmed Dimensions my joins work fine in Criteria when I include All calculated columns from facts. If I try to include a non calculated column from Fact1(Which is a number Data type) Columns from Fact2 and Fact3 show Null values. I know it is not recommended to include dimension columns in fact , does OBIEE not support Number type non calculated columns as well? Is there any work around that I can bring in my non calculated column from Fact and still get results for other fact columns.Iam at 11.1.1.7 of OBIEE
Let me know if Iam not clear.
Your help is much Appreciated.
Thanks,
Vineela.

i would like to add 2 fields into my fact tables - LOAD ID (populated by a sequence during each load) and LOAD DATE.
as these fields are not related to any reporting dimensions, it is still possible to add them in OWB ? the fact wizard always ask for a foreign key to a dimension ...
Duncan,
If you want to add non dimensional attributes to a fact by using OWB, you can create additional measures to it and use them as attributes.
Igor

Non dimensional attributes in a fact table

i would like to add 2 fields into my fact tables - LOAD ID (populated by a sequence during each load) and LOAD DATE.
as these fields are not related to any reporting dimensions, it is still possible to add them in OWB ? the fact wizard always ask for a foreign key to a dimension ...

i would like to add 2 fields into my fact tables - LOAD ID (populated by a sequence during each load) and LOAD DATE.
as these fields are not related to any reporting dimensions, it is still possible to add them in OWB ? the fact wizard always ask for a foreign key to a dimension ...
Duncan,
If you want to add non dimensional attributes to a fact by using OWB, you can create additional measures to it and use them as attributes.
Igor

Help Help : Error loading fact table

Hi
I am strugglinh since last 2 days .SSIS is giving me torrid time
I am getting error while loadding the fact table
[Destination Fact Table [1099]] Error: An OLE DB error has occurred. Error code: 0x80040E21. An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E21 Description: "Multiple-step OLE DB operation generated errors. Check each OLE DB status value, if available. No work was done.".
[Destination Fact Table [1099]] Error: The "input "OLE DB Destination Input" (1112)" failed because error code 0xC020907B occurred, and the error row disposition on "input "OLE DB Destination Input" (1112)" specifies failure on error. An error occurred on the specified object of the specified component.
[DTS.Pipeline] Error: The ProcessInput method on component "Destination Fact Table" (1099) failed with error code 0xC0209029. The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.
Please help
I have already googled for this error and appied whatever tips were given.

What other messages are you getting (warning, errors)?
Does the package fail with the first row inserted in the destination fact table?
One thing you could try is to change the error output of the destination component to re-direct rows in case of an error; then send those rows to place (file/table) where you can take a closer look.

Load fact table with null dimension keys

Dear All,
We have OWB 10g R2 and ROLAP star schema. In our source system some rows don’t have all attributes populated with values (null value), and this empty attributes are dimension (business) keys in star schema. Is it possible to load fact table with such rows (some dimension keys are null) in the OWB mappings? We use cube operator in mappings.
Thanks And Regards
Miran

The dimension should have a row indicating UNKNOWN, this will have a business key outside of the normal range e.g. -999999.
In the mapping the missing business keys can then be NVL'd to -999999.
Cheers
Si

Transaction data can be loaded into the Fact table without loading the

Transaction data can be loaded into the Fact table without loading the corresponding master data (Example : Sales analysis transaction data can be loaded without populating any of its dimensions master data)
a. True
b. False

Hi Kutti,
True - You need to select the option in the infopackage - alwyas load even if no master data exists.
Bye
Dinesh

Fact tables not loading

Hi guys,
We have installed ODI 11 with OBIA and so far the installation has been done with out any problems.But the problem here is none of the fact tables is getting loaded what would be the root cause for this.
Most of the dimension tables got loaded.By default only 1 fact table got loaded,but there are 50 fact tables in that module(HRMS & finance) but when we execute the process is running but the tables ar not loading.
Its an urgent requirement,i had no clue whats happening.Any ideas guys.
TIA,
Kranthi.
Edited by: Kranthi.K on Apr 28, 2010 3:34 AM

First of all you need to check if all the dimensions got loaded or not?
Because if a crucial dimension like customer or something else, which is being used in all fact loads is not getting loaded then obviously none of your facts will be loaded.
Hope it helps.

Load data in a fact table

Hello,
I have implemented SCD2 dimension and mapping executing works fine.
Now I have question about loading data in a fact table.
How do I need to use OWB (maybe JOINER operator - Join condition - between dimensions and source table) in case of:
- update on source table
- delete on source table
I think the most simple is insert on source table. It is probably to_char(source_transaction_date,'dd.mm.yyyy') = to_char(sysdate,'dd.mm.yyyy'), if I load once a day..
What is the procedure for fact table mapping to handle updates and deletes on source table?
Regards

Some discussions in previous forums should help you
http://forums.sdn.sap.com/thread.jspa?threadID=2019448
http://forums.sdn.sap.com/thread.jspa?threadID=1908902
In the SAP tutorial, you can see a sample example of making fact tables.
http://help.sap.com/businessobject/product_guides/boexir32SP1/en/xi321_ds_tutorial_en.pdf
Arun

Problem with loading fact table in OWB3i

Hi,
I have a problem while trying to load a fact table using OWB 3i. Let me explain the situation :
I have 2 dimensions: Customer, Product say.
I have loaded the dimension tables separately, successfully using different mappings from source tables.
Now I want to load the fact table with the corresponding warehouse keys from these dimension tables through lookup transformation.
When I try to bring to lookup transformations for the 2 dim. tables, it gives an error--"Source and target are bound to different data providers". Using 1 dim table and 1 corresponding lookup transformation , the fact table is successfully loaded.
How to load a fact table when there are more than 1 dim. tables using lookup ?Is there any other way of solving the same problem?
Any suggestion will be highly appreciated.
regards
Dipanjan

Simply, try joining source tables with the join operator. In wich way you can create a single source object into the mapping suitable to be connected with yours fact tables.
Greetings

OBIA How to truncate all fact tables automatically before running a full load

With every run data gets appended to the fact tables.How to configure so that with every run data gets deleted before we start the load.
We have OBIA 11g

If you are using DAC for scheduling the ETL jobs, you can list out the fact tables that are being used in your execution plan and set the Refresh dates for those tables as NULL. This action makes the dac execution plan to consider loading the data freshly into the fact tables.
Steps to make Refresh date as NULL in DAC:
To make the refresh dates as NULL for your fact tables in DAC, go to Setup tab -> Physical Data Sources
Now select the connection Datawarehouse(In my case) -> Refresh Dates
Query your fact tables and go to Refresh Date column and click on the calendar Icon
Click on NULL button to make the refresh date as NULL for that particular table and click on ok and save.
In the similar manner do it for all the fact tables you want.
After the above process, once you run the execution plan then data will be loaded freshly.
Regards,
Obul

Load data from dimensions to fact table

Hi,
I have 10 dimension tables and one fact table. now i want to load data in to fact table from dimensions table. how i can do that.
Regards,
joy

To elaborate on "Mallee"'s answer, it's not good practice to load dimensional data into a fact table.
I guess what you mean is how to refer to dimensions when loading a fact table.
Simply refer to dimensions using lookups on the necessary dimensions, the lookup needs to be on the natural key you have that links facts to dimensions. Don't forget to use the dimension's surrogate key though when populating the fact table's foreign key columns pointing at the dimensions.
Hope this helps.
Cheers, Patrick

Fact table mapping/loading

I am new to OWB and trying to use OWB 9.0.4 to create a star schema on a oracle 9.2 DB. Using OWB client, I successfuly created dims and a fact table on the target DB. But I got stuck with how to use OWB to map the fact table with dims and source tables, and using OWB to generate SQL loading scripts which contains PL/SQL statement shown below:
insert into "fact_table" ("fact_count") ( select count(*) from target_table where dim1='dime_value1' and dim2='dim2_value2');
here dim1 and dim2 are the dims assocated with the fact table and they are also mapped to the corresponding target table columns.

Frank,
source table -> aggregator -> target
As input into the aggregator, you specify dim1, dim2 and any numeric value. If you do not have one, use a numeric constant such as 1. In the aggregator, you specify dim1, dim2 as the group by clause (operator properties). You then add a numeric output attribute (edit function), and go to the attribute properties. As a function, specify count from the numeric input attribute.
That would load your measure.
Cube function is not currently supported in the mapping editor, but you could include that in a materialized view that is based on the table.
Mark.

SSIS Fact table load

Similar Messages

Maybe you are looking for