OWB Lookup

Hi
I am relatively new to OWB . so if my question is very basic ,please bear with me. I would like to know what is the difference between lookup option and join we have in the mapping editor. Can anybody share the knowledge with some example.
I appreciate your help on this regard
Regards
Balaji

The key lookup is similar to the join, but it has two important differences:
- it results in an outer join, not equi-join. If the output record for an input record is not found, a null record will be generated anyway.
- You can (from the GUI) set the output value in this case (null record) by setting a property named 'Default value'.
This is useful, for example, when you define a mapping that loads a cube and when you define surrogate keys on the dimension. In this example, you create a Key Lookup operator that looks up the surrogate key in the dimension table and returns the corresponding original record to form the foreign key relationship. You can also define the Default Value in this scenario to point to a key referring an 'Unknown' or 'Others' record in the dimension. So for example, if during the loading process you encounter an unknown product in the sources, you can still generate a record for the fact table (cube) that will point to the 'Unknown' record in the products table.
Regards:
Igor

Similar Messages

OWB Lookup/Join

We are using OWB repository 10.2.0.2.0 and OWB client 10.2.0.2.8. The Oracle version is 10 G (10.2.0.2.0). OWB is installed on Sun 64 bit server.
As we use lookup in OWB mapping, We can use the lookup tables/source table to create a View as Join with source and lookup table in Database. The created view could be a source for OWB mapping. This will help us to have less lookups in OWB mapping. As we Join the lookup in Database server, we could get better performance.
Did any one in this forum use above approach in large projects?
Any idea?
What is the best approach, when lookup returns multiple values for single lookup key?
Thanks in advance.
RI

Hi RI,
This will help us to have less lookups in OWB mapping. As we Join the lookup in Database server, we could get better performance.
Did any one in this forum use above approach in large projects?OWB uses Oracle database as ETL-engine and in case of Set based execution mode (for PL/SQL mapping) you will not get better performance with view (when used instead of lookup of join operators).
Additional negative impact of "view" approach is lost of complete info for Lineage/Impact analysis (I think that for large projects this is a very helpful features).
What is the best approach, when lookup returns multiple values for single lookup key?I think this is design mistake - you should define unique key on lookup table
Regards,
Oleg

Lookup Idea??

We are using OWB repository 10.2.0.2.0 and OWB client 10.2.0.2.8. The Oracle version is 10 G (10.2.0.2.0). OWB is installed on Sun 64 bit server.
As we use lookup in OWB mapping, We have a situation to create lookup from same table for different results in same OWB map. Here is the situation.
1) Table Ltab
Lookup key = sourcekey1
and lookupcode in ( 'A', 'M')
2) Table Ltab
Lookup key = sourcekey1
and lookupcode in ( 'K', 'V')
We can use ( lookupcode= 'A' OR lookupcode = 'M') instead lookupcode in ( 'A', 'M') as well.
I do not see a way to code as above in OWB lookup operator.
Is it doable in OWB via lookup operator?
Alternatevely, we could create multiple views to support above situation and attach the corresponding views to lookup.
Did any one in this forum use above approach in large projects?
Any idea?
Thanks in advance.
RI

Hi,
I suggest using a joiner operator instead of the lookup. The lookup operator generates a left outer join anyway and in the join condition you have much more flexibility.
I would not recommend using views, since this splits your etl logik into two different locations.
Regards,
Carsten.

Dynamic lookup in OWB

Hello All,
Is there any dynamic lookup in OWB, if so, please let me know how to implement the same OWB.
awaiting for your reply......

Hello All,
Is there any dynamic lookup in OWB, if so, please let me know how to implement the same OWB.
awaiting for your reply......

Dynamic Lookup in OWB 10.1g

Can we execute dynamic lookup in OWB 10.1g?
I want update the columns of the target table, based on the previous values of the columns.
Suppose there is a record in the target table with previous status and current status columns.
The source table consist of 10 records which need to be processed one at a time in a single batch. Now we need to compare the status of record with the current status of target table. If the source contains next higher status then the current status of target record need to go to previous status and the new status coming from source need to overwrite the current status of target record.
We have tried using row based option as well as setting commit frequency equal to 1 but we are not able to get the required result.
how can we implement this in OWB10.1g?

OK, now what I would do in an odd case like this is to look at the desired FINAL result of a run rather than worry so much about the intermediate steps.
Based on your statement of the status incrementing upward, and only upward, your logic can actually be distilled down to the following:
At the end of the load, the current status for a given primary key is the maximum status, and the previous status will be the second highest status. All the intermediate status values are transitional status values that have no real bearing on the desired final result.
So, let's try a simple prototype:
--drop table mb_tmp_src; /* SOURCE TABLE */
--drop table mb_tmp_tgt; /*TARGET TABLE */
create table mb_tmp_src (pk number, val number);
insert into mb_tmp_src (pk, val) values (1,1);
insert into mb_tmp_src (pk, val) values (1,2);
insert into mb_tmp_src (pk, val) values (1,3);
insert into mb_tmp_src (pk, val) values (2,2);
insert into mb_tmp_src (pk, val) values (2,3);
insert into mb_tmp_src (pk, val) values (3,1);
insert into mb_tmp_src (pk, val) values (4,1);
insert into mb_tmp_src (pk, val) values (4,3);
insert into mb_tmp_src (pk, val) values (4,4);
insert into mb_tmp_src (pk, val) values (4,5);
insert into mb_tmp_src (pk, val) values (4,6);
insert into mb_tmp_src (pk, val) values (5,5);
commit;
create table mb_tmp_tgt (pk number, val number, prv_val number);
insert into mb_tmp_tgt (pk, val, prv_val) values (2,1,null);
insert into mb_tmp_tgt (pk, val, prv_val) values (5,4,2);
commit;
-- for PK=1 we will want a current status of 3, prev =2
-- for PK=2 we will want a current status of 3, prev =2
-- for PK=3 we will want a current status of 1, prev = null
-- for PK=4 we will want a current status of 6, prev = 5
-- for PK=5 we will want a current status of 5, prev = 4
Now, lets's create a pure SQL query that gives us this result:
select pk, val, lastval
from
select pk,
val,
max(val) over (partition by pk) maxval,
lag(val) over (partition by pk order by val ) lastval
from (
select pk, val
from mb_tmp_src mts
union
select pk, val
from mb_tmp_tgt mtt
where val = maxval
(NOTE: UNION, not UNION ALL to avoid multiples where tgt = src, and would want a distinct in the union if multiple instances of same value can occur in source table too)
OK, now I'm not at my work right now, but you can see how unioning (SET operator) the target with the source, passing the union through an expression to get the analytics, and then through a filter to get the final rows before updating the target table will get you what you want. And the bonus is that you don't have to commit per row. If you can get OWB to generate this sort of statement, then it can go set-based.
EDIT: And if you can't figure out how to get OWB to generate thisentirely within the mapping editor, then use it to create a view from the main subquery with the analytics, and then use that as the source in your mapping.
If your problem was time-based where the code values could go up or down, then you would do pretty much the same thing except you want to grab the last change and have that become the current value in your dimension. The only time you would care about the intermediate values is if you were coding for a type 2 SCD, in which case you would need to track all the changes.
Hope this helps.
Mike
Edited by: zeppo on Oct 25, 2008 10:46 AM

How to implement Dynamic lookup in OWB mappings(10g)

Hi,
Iam using OWB 10g version for developing the mappings.
Now I need to implement the Dynamic lookup in the mapping.Is there any transformations available in OWB to achieve this.
Please give me some information about the same.
Thanks in advance...

Hi,
first i have created a procedure witht he following code in the code editor...
BEGIN
Declare
Cursor C_brand_col is
Select cat from TBL_CAT_EDESC_BRAND ;
Vbrand varchar2(240);
Cursor C_bredesc_col is
Select EDESC_BRAND from TBL_CAT_EDESC_BRAND;
Vbredesc varchar2(240);
V_Command varchar2(30000);
Begin
Open C_brand_col;
Fetch C_brand_col into vbrand;
Open C_bredesc_col;
Fetch C_bredesc_col into vbredesc;
loop
V_Command := 'update sav_fc_sa_pc c set c.'||vbrand||'=';
V_Command := V_Command||'(select d.fc_brands_sa from TEST_brand d' ;
V_Command := V_Command||' where d.brand_edesc = '||''''||vbredesc||''''||' and c.cardno=d.cardno)';
dbms_output.put_line('10 '||V_command);
Execute immediate v_command;
Fetch C_brand_col into vbrand;
Exit when c_brand_col%notfound;
Fetch C_bredesc_col into vbredesc;
Exit when c_bredesc_col%notfound;
end loop;
commit;
end;
END;
then i validate it and deply it..
after that i create a mapping and in that mapping i first import the table TBL_CAT_EDESC_BRAND and drag and drop it into the mapping and again the i put the procedure into a transformation operator and connect the inoutgrp of the table to the transformation operator ingrp deploy it and run it...this is taking a lot of time .... so i am not sure whether i am doing the right thing...for this dynamic sql i dont need to pass any parameters. can i juz execute this procedure or should i create a mapping ???? i am totally confused... could you please help me.....how to proceed........
if i juz execute the dynamic sql it takes only 5 min in sql but i am not sure how ti implement it in owb... can you please help...
Thanks a many

Subsequent Lookup Operators causes OWB to generate undeployable mappings

Hi,
I am using OWB 11gR2 .
I am trying to create a fact loading mapping based on datavault.
That gives me error during deployment.
Validating and Generate doesn't give me an error.
So in trying to load the fact i tie various datavault tables together with a joiner operator.
All tables except the driving table are set to outer join role.
The output fields are tied to various lookup operator objects.
The output from those is tied to the target fact table.
All of this goes well, this mapping is deployable and upon genrate one can see the statements.
The problem arises when i try to insert another lookup operator between the output of one lookup operator and the fact.
That mapping does not give a validation error and also generate intermediate doesn't error.
Deploying doesn't work however, it complains of a incorrect identifier.
Inspecting the generate intermediate does reveal the problem:
OWB appends all of the join clauses from the first joiner to the total used for loading the fact as a where statement.
When you look at the first joiner though it just displays nicely all of the left outer join statements.
There is no where statement to be found on this first joiner.
It is only added at the fact stage at exactly the same place where the left outerjoins are from the first joiner.
Questions:
Is there a limit to the number of subsequent lookup operators one can use ? 2 can not be it i hope ..
Is there a patch for this ?
Other remarks; i have noticed that when i use more than 8 lookup operators on my canvas that the lookup conditons get corrupt.
It becomes something like: lookup.fieldname = null instead of lookup.fieldname = input.fieldname .
When this happens i have to correct every lookup operator on the mapping.
Is this known error ?
Hope somebody has an answer fro my first problem.
rgrds Mike
Edited by: MichaelR64 on 16-jan-2011 23:39

Hi,
I did some further testing:
This happens when there is an unequal number of lookups "attached"to the driving table.
What i mean is that if there is one lookup attached to a port of the driving table, then the next port that has a lookup can not have two (serially connected) lookups.
Or put the other way : if a port has two lookups(sequentially connected) than the errror disappears when all the other ports with lookups also have two lookups(serially connected that is).
At first i thought it had something to do with the joiner used in the first stage.
Replacing that with a view didn't solve it
In fact using a lookup where multiple rows output is specified causes owb to create this with a outer join.
It is this outer join part that is being mangled by owb as specified before.
If anyone can comment on this..

How do you do Dynamic Lookup in OWB?

Hi,
Is there a way to lookup records that have been processed but not yet committed in the db?
I have a data set that contains records that need to be inserted and updated. Some records in the data set have entries for both insert & updated i.e. the same record is inserted and updated in the data set. How could I determine if the record has been processed (inserted but not yet committed in the db)?
Thanks.
-bzx

Hi Nawneet,
Here is an example:
Let's say, the external table has the structure as follow:
company_id, company_name, company_url, address_line, city, state, country, last_changed_on
I have 2 tables:
COMPANY_TABLE, which has the following structure
company_id, company_name, company_url, last_changed_on
and ADDRESS_TABLE, which has the following structure
address_line, city, state, country, last_changed_on
Now, let's say I have the following 2 records in a source dataset (external table):
20, XYZ company, www.xyzcompany.com, 4500 longpond road, victor, ny, usa, oct 10, 1990
22, ABC company, www.abccompany.com, 4500 longpond road, victor, ny, usa, jan 11, 2010
I would like to add the above records as follow:
COMPANY_TABLE:
20, XYZ company, www.xyzcompany.com, oct 10, 1990
22, ABC company, www.abccompany.com, jan 11, 2010
ADDRESS_TABLE:
4500 longpond road, victor, ny, usa, jan 11, 2010
That is, the ADDRESS_TABLE should not add duplicate entry. We use address_line, city, state, & country columns to determine if the record already exist.
So, what I am expecting from the mapping is that it should first "insert" the following record:
4500 longpond road, victor, ny, usa, oct 10, 1990
and then "update" it later when it gets the next record:
4500 longpond road, victor, ny, usa, jan 11, 2010
I am using lookup operator on the ADDRESS_TABLE to determine if the address already exist but it only works if the 1st record has already been committed (i.e. in the table).
Thank you.
-bzx

How to model in OWB if you have a subquery in your select statement? (11.2)

Which operator (is it the Table operator) to use if I have a subquery as part of my select statement? (The way I do it works, but is not elegant. What I do is create a table for the subquery and than join that with the other sources to point to my target.) But, would appreciate the best way with OWB's features.
Here is the entire SQL (including the subquery):
SELECT to_numbe (null), course_id
at.book, lp.NAME,
(select b.pricing from qp_pricing_attributes b
where b.list_id = ll.list_id and b.product = at.product) sign,
ll.operand AS price, lhb.comments
FROM
pricing_attribs at,
list_lines ll,
list_h lp,
list_b lhb
WHERE 1 = 1
and ll.list_line_id = atrib.list_line_id
AND ll.list_header_id = lp.list_header_id
AND lp.list_header_id = lhb.list_header_id
Thanks you.

Hi
Doesn't need to be a table in the target. Let's take a very simple example, let's say you wanted to do the following;
select ename,(select dname from dept d where d.deptno=e.deptno) from emp e
Then DEPT is your lookup table and EMP is your driving table.
1. Add EMP on to the canvas.
2. Add in lookup operator
3. Bind lookup to DEPT
4. Hit finish
5. Map EMP.DEPT to DEPT.INGRP1 (the lookup operator)
6. Edit lookup operator, on Lookup Conditions tab set DEPTNO for lookup column, and DEPTNO for input attribute
7. Lookup complete now
You can add in a target table operator and map from the source EMP table and the output attributes of the lookup operator. you can change step 4 to carry on through wizard and define lookup, but I have chosen to do 5, to automatically get the names and datatypes of the inputs. There are lot of other options now in 11gR2 in the lookup.
Hope this makes sense.
Cheers
David

How to add new key lookup in Mapping ?

Hi,
I am learning OWB 10g from oracle.com site. I am following the Oracle-By-Example.
[http://www.oracle.com/technology/obe/obe_bi/Lesson6_Designing_ETL_Data_Flow_Mappings/designing_etl_data_flow_mappings.htm]
My OWB version is:
OWB Client : 10.2.0.1.31
OWB Repository : 10.2.0.1.0
But the screenshots are different than what I am getting on my OWB. Am I using the correct version of Oracle-By-Example?
I have created Mapping Dimension and Mapping Table. Now i want to add a Mapping Key Lookup. But as the screenshots shown in Oracle-By-Example are different, I cannot go further.
Please help me.
Thanks!
Yogini

Hi Yogini
Here are some steps, it should be fairly straightforward, you can view the online help for more information.
1. Drag and drop the Key Lookup operator on to the Mapping Editor.
2. The Lookup wizard opens on the Welcome page. Select Next to move to page 2.
3. Provide a name and description for the operator, default is KEY_LOOKUP. Hit Next to move to page 3.
4. On the Groups page, hit Next.
5. Select attributes to use in key lookup. For example those from the WAREHOUSES table in the OBE you are using that you will lookup in the COUNTRIES table. Shuttle those attrbutes to the right hand side. Hit Next to move to page 4.
6. Select the COUNTRIES table from the combo box nder 'Select the object which has the lookup result. In the lookup conditions table ensure the matching criteria is set ie . LOCATION_ID from WAREHOUSE matches with LOCATION_ID from COUNTRIES. Hit Next to move to page 5.
7. Here you can define the strategy for matches, just hit Next, then Finish. You have walked through all pages and are complete.
Cheers
David

Is there a limit on the number of Key Lookups against a table in a mapping

I'm using OWB 11.1 and have a mapping with 15 Key Lookups against the one table. When I Validate the mapping it objects to a Key Lookup not being connected, even thuogh in the mapping all the Key Lookups have been renamed to their relevant fields?
Is there a limit to the number of key lookups against the one table?

Thanks for the replys.
I'm getting a validation error, so can't run the mapping. Error VLD-1108: Operator Key_LOOKUP is not properly connected.
The issue is solved, after checking each of the Key_Lookups, one was not connected to an output. The error occurs if an output field isn't used (connected to) a table or other operator.
Edited by: user616385 on Sep 30, 2009 7:58 PM
Edited by: user616385 on Sep 30, 2009 8:01 PM

Mapping in OWB with primary key and foreign key relationship

Hi all,
I am new to this datawarehousing field. I have just started my career. I have to now create a mapping in owb where a table has a field which is a primary key of another table in the same staging area. If you guys could help me out with the a method it can be created that would be very helpful to me.
I thought of 2 ideas,
1. If I can use a look up, but then I am not sure if i can use a lookup for primary key, foreign key relationship. If I can use also, I do not know how to use that.
2. What if I can directly take that the first table and link the primary key of that table to the second table which uses that primary key of the first table as one of its fields.
I do not know how feasible these methods are. Please guys help me out.
Thanks in advance.

I have a similar case where table a and table b having relation but table a got inserted with data and table b is empty so there no values for foriegn key column in table b to realte with table a.
Now i want to load table b foriegn key with primary key column values of table a.
how can we do this in owb
thanks
kumar

Post-upgrade tasks for OWB 10g Release 2

Hi,
we're planning to upgrade from OWB 10.1.0.2 to 10.2.x.
Now, I'm trying to get a clear view on what will be the "post-upgrade" tasks, in terms of manual adaptations, corrections, etc. to get everything back up and running after the upgrade ...
E.g. : by means of test al already exported a dimension, and imported it in the 10.2 repository ... Turned out that in the pre-10.2 version, there were no validation problems, but as from 10.2 (although the upgrade and import were successful), I got 19 warnings and 2 errors ... because each level needs to have a business or surrogate key ...
It' s things like that I'm especially interested in, in order to determine the workload after the upgrade ...
So, feel free to share your experiences ; I'm looking forward to read them!
Thanks!
Grtz.

Here are some from my list. I stopped maintaining it a while ago, we have lots more where this came from. Some of it also apply to older versions of OWB.
Key lookup     You are to choose input and output attributes BEFORE being able to choose loookup table, all in a time consuming wizard
Expressions     Copy and paste into expression editor, validation fails for expressions using lowercase (such as "upper" or "substr")
Renaming attributes     Not possible anymore, you need to open details
Adding a table component     You must browse in a tree structure (time consuming), not search functionality anymore
Import     Import converting from 9.2 hangs after 15% in about 10% of all imports
Import     Some mappings are not possible to import, a total rewrite is necessary
Deployment     You can only deploy about 5-10 times before you need to restart OWB
Memory leak     If something goes wrong in OWB, memory is never cleared - JAWAW.exe needs to be removed manually from memory
Import     Importing mappings one by one (as this is sometimes needed due to OWB), the last directory used is OF COURSE not default when importing the next time
OWB GUI / mapping editor     When working with components in the mapping editor, an object hidden behind the object you are actually working with is selected prior to the object in visual focus to the user
BUG!     Urgent: CASE - wrong generated Code
ANSI JOINS     Remove ANSI-SQL from mapping configuration, otherwise (+) outer joins will not work
Transformations     Not possible to enter a transformation such as "sysdate", you must find it in a huge tree structure list (no search available)

Some Thoughts On An OWB Performance/Testing Framework

Hi all,
I've been giving some thought recently to how we could build a performance tuning and testing framework around Oracle Warehouse Builder. Specifically, I'm looking at was in which we can use some of the performance tuning techniques described in Cary Millsap/Jeff Holt's book "Optimizing Oracle Performance" to profile and performance tune mappings and process flows, and to use some of the ideas put forward in Kent Graziano's Agile Methods in Data Warehousing paper http://www.rmoug.org/td2005pres/graziano.zip and Steven Feuernstein's utPLSQL project http://utplsql.sourceforge.net/ to provide an agile/test-driven way of developing mappings, process flows and modules. The aim of this is to ensure that the mappings we put together are as efficient as possible, work individually and together as expected, and are quick to develop and test.
At the moment, most people's experience of performance tuning OWB mappings is firstly to see if it runs set-based rather than row-based, then perhaps to extract the main SQL statement and run an explain plan on it, then check to make sure indexes etc are being used ok. This involves a lot of manual work, doesn't factor in the data available from the wait interface, doesn't store the execution plans anywhere, and doesn't really scale out to encompass entire batches of mapping (process flows).
For some background reading on Cary Millsap/Jeff Holt's approach to profiling and performance tuning, take a look at http://www.rittman.net/archives/000961.html and http://www.rittman.net/work_stuff/extended_sql_trace_and_tkprof.htm. Basically, this approach traces the SQL that is generated by a batch file (read: mapping) and generates a file that can be later used to replay the SQL commands used, the explain plans that relate to the SQL, details on what wait events occurred during execution, and provides at the end a profile listing that tells you where the majority of your time went during the batch. It's currently the "preferred" way of tuning applications as it focuses all the tuning effort on precisely the issues that are slowing your mappings down, rather than database-wide issues that might not be relevant to your mapping.
For some background information on agile methods, take a look at Kent Graziano's paper, this one on test-driven development http://c2.com/cgi/wiki?TestDrivenDevelopment , this one http://martinfowler.com/articles/evodb.html on agile database development, and the sourceforge project for utPLSQL http://utplsql.sourceforge.net/. What this is all about is having a development methodology that builds in quality but is flexible and responsive to changes in customer requirements. The benefit of using utPLSQL (or any unit testing framework) is that you can automatically check your altered mappings to see that they still return logically correct data, meaning that you can make changes to your data model and mappings whilst still being sure that it'll still compile and run.
Observations On The Current State of OWB Performance Tuning & Testing
At present, when you build OWB mappings, there is no way (within the OWB GUI) to determine how "efficient" the mapping is. Often, when building the mapping against development data, the mapping executes quickly and yet when run against the full dataset, problems then occur. The mapping is built "in isolation" from its effect on the database and there is no handy tool for determining how efficient the SQL is.
OWB doesn't come with any methodology or testing framework, and so apart from checking that the mapping has run, and that the number of rows inserted/updated/deleted looks correct, there is nothing really to tell you whether there are any "logical" errors. Also, there is no OWB methodology for integration testing, unit testing, or any other sort of testing, and we need to put one in place. Note - OWB does come with auditing, error reporting and so on, but there's no framework for guiding the user through a regime of unit testing, integration testing, system testing and so on, which I would imagine more complete developer GUIs come with. Certainly there's no built in ability to use testing frameworks such as utPLSQL, or a part of the application that let's you record whether a mapping has been tested, and changes the test status of mappings when you make changes to ones that they are dependent on.
OWB is effectively a code generator, and this code runs against the Oracle database just like any other SQL or PL/SQL code. There is a whole world of information and techniques out there for tuning SQL and PL/SQL, and one particular methodology that we quite like is the Cary Millsap/Jeff Holt "Extended SQL Trace" approach that uses Oracle diagnostic events to find out exactly what went on during the running of a batch of SQL commands. We've been pretty successful using this approach to tune customer applications and batch jobs, and we'd like to use this, together with the "Method R" performance profiling methodology detailed in the book "Optimising Oracle Performance", as a way of tuning our generated mapping code.
Whilst we want to build performance and quality into our code, we also don't want to overburden developers with an unwieldy development approach, because what we'll know will happen is that after a short amount of time, it won't get used. Given that we want this framework to be used for all mappings, it's got to be easy to use, cause minimal overhead, and have results that are easy to interpret. If at all possible, we'd like to use some of the ideas from agile methodologies such as eXtreme Programming, SCRUM and so on to build in quality but minimise paperwork.
We also recognise that there are quite a few settings that can be changed at a session and instance level, that can have an effect on the performance of a mapping. Some of these include initialisation parameters that can change the amount of memory assigned to the instance and the amount of memory subsequently assigned to caches, sort areas and the like, preferences that can be set so that indexes are preferred over table scans, and other such "tweaks" to the Oracle instance we're working with. For reference, the version of Oracle we're going to use to both run our code and store our data is Oracle 10g 10.1.0.3 Enterprise Edition, running on Sun Solaris 64-bit.
Some initial thoughts on how this could be accomplished
- Put in place some method for automatically / easily generating explain plans for OWB mappings (issue - this is only relevant for mappings that are set based, and what about pre- and post- mapping triggers)
- Put in place a method for starting and stopping an event 10046 extended SQL trace for a mapping
- Put in place a way of detecting whether the explain plan / cost / timing for a mapping changes significantly
- Put in place a way of tracing a collection of mappings, i.e. a process flow
- The way of enabling tracing should either be built in by default, or easily added by the OWB developer. Ideally it should be simple to switch it on or off (perhaps levels of event 10046 tracing?)
- Perhaps store trace results in a repository? reporting? exception reporting?
at an instance level, come up with some stock recommendations for instance settings
- identify the set of instance and session settings that are relevant for ETL jobs, and determine what effect changing them has on the ETL job
- put in place a regime that records key instance indicators (STATSPACK / ASH) and allows reports to be run / exceptions to be reported
- Incorporate any existing "performance best practices" for OWB development
- define a lightweight regime for unit testing (as per agile methodologies) and a way of automating it (utPLSQL?) and of recording the results so we can check the status of dependent mappings easily
other ideas around testing?
Suggested Approach
- For mapping tracing and generation of explain plans, a pre- and post-mapping trigger that turns extended SQL trace on and off, places the trace file in a predetermined spot, formats the trace file and dumps the output to repository tables.
- For process flows, something that does the same at the start and end of the process. Issue - how might this conflict with mapping level tracing controls?
- Within the mapping/process flow tracing repository, store the values of historic executions, have an exception report that tells you when a mapping execution time varies by a certain amount
- get the standard set of preferred initialisation parameters for a DW, use these as the start point for the stock recommendations. Identify which ones have an effect on an ETL job.
- identify the standard steps Oracle recommends for getting the best performance out of OWB (workstation RAM etc) - see OWB Performance Tips http://www.rittman.net/archives/001031.html and Optimizing Oracle Warehouse Builder Performance http://www.oracle.com/technology/products/warehouse/pdf/OWBPerformanceWP.pdf
- Investigate what additional tuning options and advisers are available with 10g
- Investigate the effect of system statistics & come up with recommendations.
Further reading / resources:
- Diagnosing Performance Problems Using Extended Trace" Cary Millsap
http://otn.oracle.com/oramag/oracle/04-jan/o14tech_perf.html
- "Performance Tuning With STATSPACK" Connie Dialeris and Graham Wood
http://www.oracle.com/oramag/oracle/00-sep/index.html?o50tun.html
- "Performance Tuning with Statspack, Part II" Connie Dialeris and Graham Wood
http://otn.oracle.com/deploy/performance/pdf/statspack_tuning_otn_new.pdf
- "Analyzing a Statspack Report: A Guide to the Detail Pages" Connie Dialeris and Graham Wood
http://www.oracle.com/oramag/oracle/00-nov/index.html?o60tun_ol.html
- "Why Isn't Oracle Using My Index?!" Jonathan Lewis
http://www.dbazine.com/jlewis12.shtml
- "Performance Tuning Enhancements in Oracle Database 10g" Oracle-Base.com
http://www.oracle-base.com/articles/10g/PerformanceTuningEnhancements10g.php
- Introduction to Method R and Hotsos Profiler (Cary Millsap, free reg. required)
http://www.hotsos.com/downloads/registered/00000029.pdf
- Exploring the Oracle Database 10g Wait Interface (Robin Schumacher)
http://otn.oracle.com/pub/articles/schumacher_10gwait.html
- Article referencing an OWB forum posting
http://www.rittman.net/archives/001031.html
- How do I inspect error logs in Warehouse Builder? - OWB Exchange tip
http://www.oracle.com/technology/products/warehouse/pdf/Cases/case10.pdf
- What is the fastest way to load data from files? - OWB exchange tip
http://www.oracle.com/technology/products/warehouse/pdf/Cases/case1.pdf
- Optimizing Oracle Warehouse Builder Performance - Oracle White Paper
http://www.oracle.com/technology/products/warehouse/pdf/OWBPerformanceWP.pdf
- OWB Advanced ETL topics - including sections on operating modes, partition exchange loading
http://www.oracle.com/technology/products/warehouse/selfserv_edu/advanced_ETL.html
- Niall Litchfield's Simple Profiler (a creative commons-licensed trace file profiler, based on Oracle Trace Analyzer, that displays the response time profile through HTMLDB. Perhaps could be used as the basis for the repository/reporting part of the project)
http://www.niall.litchfield.dial.pipex.com/SimpleProfiler/SimpleProfiler.html
- Welcome to the utPLSQL Project - a PL/SQL unit testing framework by Steven Feuernstein. Could be useful for automating the process of unit testing mappings.
http://utplsql.sourceforge.net/
Relevant postings from the OTN OWB Forum
- Bulk Insert - Configuration Settings in OWB
http://forums.oracle.com/forums/thread.jsp?forum=57&thread=291269&tstart=30&trange=15
- Default Performance Parameters
http://forums.oracle.com/forums/thread.jsp?forum=57&thread=213265&message=588419&q=706572666f726d616e6365#588419
- Performance Improvements
http://forums.oracle.com/forums/thread.jsp?forum=57&thread=270350&message=820365&q=706572666f726d616e6365#820365
- Map Operator performance
http://forums.oracle.com/forums/thread.jsp?forum=57&thread=238184&message=681817&q=706572666f726d616e6365#681817
- Performance of mapping with FILTER
http://forums.oracle.com/forums/thread.jsp?forum=57&thread=273221&message=830732&q=706572666f726d616e6365#830732
- Poor mapping performance
http://forums.oracle.com/forums/thread.jsp?forum=57&thread=275059&message=838812&q=706572666f726d616e6365#838812
- Optimizing Mapping Performance With OWB
http://forums.oracle.com/forums/thread.jsp?forum=57&thread=269552&message=815295&q=706572666f726d616e6365#815295
- Performance of mapping with FILTER
http://forums.oracle.com/forums/thread.jsp?forum=57&thread=273221&message=830732&q=706572666f726d616e6365#830732
- Performance of the OWB-Repository
http://forums.oracle.com/forums/thread.jsp?forum=57&thread=66271&message=66271&q=706572666f726d616e6365#66271
- One large JOIN or many small ones?
http://forums.oracle.com/forums/thread.jsp?forum=57&thread=202784&message=553503&q=706572666f726d616e6365#553503
- NATIVE PL SQL with OWB9i
http://forums.oracle.com/forums/thread.jsp?forum=57&thread=270273&message=818390&q=706572666f726d616e6365#818390
Next Steps
Although this is something that I'll be progressing with anyway, I'd appreciate any comment from existing OWB users as to how they currently perform performance tuning and testing. Whilst these are perhaps two distinct subject areas, they can be thought of as the core of an "OWB Best Practices" framework and I'd be prepared to write the results up as a freely downloadable whitepaper. With this in mind, does anyone have an existing best practices for tuning or testing, have they tried using SQL trace and TKPROF to profile mappings and process flows, or have you used a unit testing framework such as utPLSQL to automatically test the set of mappings that make up your project?
Any feedback, add it to this forum posting or send directly through to me at [email protected]. I'll report back on a proposed approach in due course.

Hi Mark,
interesting post, but I think you may be focusing on the trees, and losing sight of the forest.
Coincidentally, I've been giving quite a lot of thought lately to some aspects of your post. They relate to some new stuff I'm doing. Maybe I'll be able to answer in more detail later, but I do have a few preliminary thoughts.
1. 'How efficient is the generated code' is a perennial topic. There are still some people who believe that a code generator like OWB cannot be in the same league as hand-crafted SQL. I answered that question quite definitely: "We carefully timed execution of full-size runs of both the original code and the OWB versions. Take it from me, the code that OWB generates is every bit as fast as the very best hand-crafted and fully tuned code that an expert programmer can produce."
The link is http://www.donnapkelly.pwp.blueyonder.co.uk/generated_code.htm
That said, it still behooves the developer to have a solid understanding of what the generated code will actually do, such as how it will take advantage of indexes, and so on. If not, the developer can create such monstrosities as lookups into an un-indexed field (I've seen that).
2. The real issue is not how fast any particular generated mapping runs, but whether or not the system as a whole is fit for purpose. Most often, that means: does it fit within its batch update window? My technique is to dump the process flow into Microsoft Project, and then to add the timings for each process. That creates a Critical Path, and then I can visually inspect it for any bottleneck processes. I usually find that there are not more than one or two dogs. I'll concentrate on those, fix them, and re-do the flow timings. I would add this: the dogs I have seen, I have invariably replaced. They were just garbage, They did not need tuning at all - just scrapping.
Gee, but this whole thing is minimum effort and real fast! I generally figure that it takes maybe a day or two (max) to soup up system performance to the point where it whizzes.
Fact is, I don't really care whether there are a lot of sub-optimal processes. All I really care about is performance of the system as a whole. This technique seems to work for me. 'Course, it depends on architecting the thing properly in the first place. Otherwise, no amount of tuning of going to help worth a darn.
Conversely (re. my note about replacing dogs) I do not think I have ever tuned a piece of OWB-generated code. Never found a need to. Not once. Not ever.
That's not to say I do not recognise the value of playing with deployment configuration parameters. Obviously, I set auditing=none, and operating mode=set based, and sometimes, I play with a couple of different target environments to fool around with partitioning, for example. Nonetheless, if it is not a switch or a knob inside OWB, I do not touch it. This is in line with my dictat that you shall use no other tool than OWB to develop data warehouses. (And that includes all documentation!). (OK, I'll accept MS Project)
Finally, you raise the concept of a 'testing framework'. This is a major part of what I am working on at the moment. This is a tough one. Clearly, the developer must unit test each mapping in a design-model-deploy-execute cycle, paying attention to both functionality and performance. When the developer is satisifed, that mapping will be marked as 'done' in the project workbook. Mappings will form part of a stream, executed as a process flow. Each process flow will usually terminate in a dimension, a fact, or an aggregate. Each process flow will be tested as an integrated whole. There will be test strategies devised, and test cases constructed. There will finally be system tests, to verify the validity of the system as a production-grade whole. (stuff like recovery/restart, late-arriving data, and so on)
For me, I use EDM (TM). That's the methodology I created (and trademarked) twenty years ago: Evolutionary Development Methodology (TM). This is a spiral methodology based around prototyping cycles within Stage cycles within Release cycles. For OWB, a Stage would consist (say) of a Dimensional update. What I am trying to now is to graft this within a traditional waterfall methodology, and I am having the same difficulties I had when I tried to do it then.
All suggestions on how to do that grafting gratefully received!
To sum up, I 'm kinda at a loss as to why you want to go deep into OWB-generated code performance stuff. Jeepers, architect the thing right, and the code runs fast enough for anyone. I've worked on ultra-large OWB systems, including validating the largest data warehouse in the UK. I've never found any value in 'tuning' the code. What I'd like you to comment on is this: what will it buy you?
Cheers,
Donna
http://www.donnapkelly.pwp.blueyonder.co.uk

Lookup Table and Target Table are the same

Hi All,
I have a requirement in which I have to lookup the target table and based on the records in it, I need to load a new record into the target table.
Being very specific,
Suppose I have a key column which when changes I want to generate a new id and then insert this new value.
The target table record structure looks like this
list_id list_key list_name
1 'A' 'NAME1'
1 'A' 'NAME2'
1 'A' 'NAME3'
2 'B' 'NAME4'
2 'B' 'NAME5'
As shown the target table list_id changes only when the list key changes. I need to generate the list_id value from within OWB mapping.
Can anyone throw some light as to how this can be done in OWB???
regards
-AP

Hello, AP
You underestimate the power of single mapping :) If you could tolerate using additional stage table (with is definitly recomended in case your table from example will account a lot of rows).
You underestimate the power of single mapping :) It you could tolerate using additional stage table (witch is definitely recommended in case your table from example will account a lot of rows), you could accomplish all you need within one mapping and without using PLSQL function. This is true as far as you could have several targets within one mapping.
Source ----------------------------------------------------- >| Join2 | ---- > Target 2
|------------------------ >|Join 1| --> Lookup table -->|
Target Dedup >|
Here Target your target table. Join 1 operator covers operations needed to get existing key mapping (from dedup) and find new mappings. Results are stored within Lookup Table target (operation type TRUNCATE/INSERT).
Join 2 is used to perform final lookup and load it into the Target 2 the same as Target
The approach with lookup table is fast and reliable and could run on Set base mode. Also you could revisit lookup table to find what key mapping are loaded during last load operation.
Serhit

OWB Lookup

Similar Messages

Maybe you are looking for