ODI - Set Based Transformations

I'm a Business Intelligence SC and have a slide on ELT and ODI that highlights three reasons by performance is good.
1) Leverage Set-based transformations
2) Improved performance for loading, no network hop
3) Takes advantage of existing hardware
I do not know how to explain Set Based Transformations. Is there someone that can explain how I can explain thisfor a non-ELT expert/layperson?
Regards,
Beth Rodgers
[email protected]

Since most of your queries are done by the RDBMS, you can load directly a set (of rows) instead of doing that row by row.
Set-based is in opposition with row-based and is of usually faster.
Hope it helps.
Regards,
JeromeFr

Similar Messages

Creation of a Marketing Profile Set based on the standard CRM_MKTTG_BP_BIRTHDATE Infoset thru the WebUI Segment Builder

Objective: Creation of a Marketing Profile Set based on the standard CRM_MKTTG_BP_BIRTHDATE Infoset thru the WebUI Segment Builder.
Problems: Standard CRM_MKTTG_BP_BIRTHDATE Infoset DOES NEVER RETURN ANY RESULTS on the segmentation builder of the WebUI client (no matter the filter you specify).
Other cases: However if we create a new Data source with data "Extraction from Query" (transaction RS02) based on this standard CRM_MKTTG_BP_BIRTHDATE Infoset and we test the functionality using transaction RSA3, IT DOES RETURN RESULTS (using the same filters as in the segmentation builder)! Of course the segmentation builder works with other attribute lists.
NEW: IF WE DON’T ESPECIFY ANY SEGMENTATION BASE IN THE PROFILE GROUP, IT DOES RETURN RESULTS!!! (GO TO THE END OF THE DOCUMENT FOR DETAILS)
Steps followed:
Review Standard Infoset CRM_MKTTG_BP_BIRTHDATE (Transaction SQ02).Review Data Reading Program code: do we have to change the standard code? We have only remove the "BREAK-POINT" statement.
Creation of a Data source based on the standard CRM_MKTTG_BP_BIRTHDATE Infoset (Transaction CRMD_MKTDS).
Creation of an Attribute List based on that standard Data source and selection the Birth date as a filter in the Attribute list (Transaction CRMD_MKTDS).
On the segment builder of the WebUI set any filter (Example: “TODAY”).
Steps in detail:
Review Standard Infoset CRM_MKTTG_BP_BIRTHDATE
SQ02.
(Image deleted)
Review Data Reading Program code (Code attached)
(Image deleted)
Creation of a Data source based on the standard CRM_MKTTG_BP_BIRTHDATE Infoset
Data source: Transaction: SPRO – Customer Relationship Management - Marketing – Segmentation - Maintain Data Sources and Attribute Lists: CRMD_MKTDS
Create Data Source
(Image deleted)
Infoset: CRM_MKTTG_BP_BIRTHDATE.
Business Partner Field: CRMT_MKTTG_IS_BIRTHDATE-PARTNER_GUID
(Image deleted)
Result:
(Image deleted)
Attribute List
Creation of the Attribute List
(Image deleted)
Assign Data Source:
(Image deleted)
Selection of the Birth date as a filter in the Attribute list:
(Image deleted)
Result:
(Image deleted)
Segment builder of the WebUI:
Access to WebUI with marketing role.
Create Segment:
Creation of Profile Set
Graphical modeler
Create Filter: We have already tried many different filters (we already seen notes referring to this matter).
Example1: “today”
Count target group. Here I would like to stand out that once you click on the “Count” option, immediately it shows 0 results. In other words, it doesn’t process any information. With other attribute lists it takes some time to process, no matter the filter or the attribute list you use. However if we create a new Data source with data "Extraction from Query" (transaction RS02) based on this standard CRM_MKTTG_BP_BIRTHDATE Infoset and we test the functionality using transaction RSA3, IT DOES RETURN RESULTS (with “today” filter and other)! So it seems that the problem has to do with the segment builder, not the Infoset.
Result: it never return any results no matter the filter you specify.
Other cases:
Creation of a new Data source with data "Extraction from Query" where you enter the name of your new query
Creation of a Data source based on the standard CRM_MKTTG_BP_BIRTHDATE Infoset
Standard Infoset CRM_MKTTG_BP_BIRTHDATE:
We fill the form and select “Extraction from Query”:
Press SAVE and define which fields will be available for selection and which of them will be visible.
RSA3: Extractor checker in order to test the Data source:
We specify the Data source:
IT DOES RETURN RESULTS:
ALV Grid:
NEW: It seems that it has to do with the Segmentation Basis selected in the creation of a Profile Set, because if we do not select any, it does return results. But we need to create this Profile Set based on a particular Segmentation Basis called BS Usuarios Registrados.
Segment builder – Profile Set of the WebUISegmentation Basis: we leave it in blanc.
Graphical modeler
Create Filter: We have already tried many different filters (we already seen notes referring to this matter).
Example1: “today”
Count target group.
Result: IT DOES RETURN RESULTS!!! àConclusion: the problem is the Segmentation Basis
Define Segmentation Basis:
Transaction: SPRO – Customer Relationship Management - Marketing – Segmentation – Classic Segmentation - Define Segmentation Basis.BS Usuarios Registrados:
Define Reports for Creating Segmentation Bases
Transaction: SPRO – Customer Relationship Management - Marketing – Segmentation – Classic Segmentation - Define Reports for Creating Segmentation BasesZMK_UTL_MGR_SITES_UREG
Define Usage for Segments:
Transacción: SPRO – Customer Relationship Management - Marketing – Segmentation – General Settings - Define Usage for Segments.ZBS_UR à02 Segmentatión Basis
Attribute List:Category:
There is no Category 02 Segmentatión Basis!!! Why?? Why is not posible to select Category 02 Segmentation Basis in the creation of the Attribute List???
Usage ID
NEW: It seems that it has to do with the Segmentation Basis selected in the creation of a Profile Set, because if we do not select any, it does return results. But we need to create this Profile Set based on a particular Segmentation Basis called BS Usuarios Registrados. Why is not posible to select Category 02 Segmentation Basis in the creation of the Attribute List???

It seems that the standard Infoset doesn’t work by itself with Segmentation Basis. Two SAP notes have been applied and a new Field Group has been added to the Infoset (in order to distint the Segmentation Base).
CRMT_MKTTG_IS_BIRTHDATE-MGRGUID
1966298: MGR_GUID is not populated correctly to infoset read program
2007687:Segmentation InfoSet queries saved with incorrect Master Group
For that reason we had to create a new Infoset based on the standard CRMT_MKTTG_IS_BIRTHDATE, but adding this new Field MGRGUID to the Group Fields.
Issue solved

Can Data Links be established between Data sets based on View Objects?

Hi all,
In the BI Publisher Documentation it's given that Datasets based on view object queries do not support Data Links / Group Links. We found out that only way to establish relationship between view object Datasets is to create a view link and then upload it to create a Dataset.
1. Is there any other way to establish relationship between view objects Datasets in DataModel editor itself just as in the case of data sets based on (SQL queries e.t.c.)?
2. If so can View object Datasets be linked to Datasets based on other Datasources?
3. Will the Datalinks for View object Datasets be supported in any of the upcoming releases. Is there any ER logged for this case?
Any insight on the above issues will be really helpful.
Thanks

Enhance the data source with date and time and populate these fields in the user exit using the function module IB_CONVERT_FROM_TIMESTAMP .
OR
You can create Z function module IB_CONVERT_FROM_TIMESTAMP in BW side and write a routine in update rules/transfer rules to populate date and time.
hope this helps ...
Ravi

Need to connect to upgraded Oracle EBS R12 version from R11. The current ODI set up is pointing to R11, Can i use the same connection to point to R12? Please hele experts..

Need to connect to upgraded Oracle EBS R12 version from R11. The current ODI set up is pointing to R11, Can i use the same connection to point to R12? Please hele experts...
Rp

1. in physical connections part can i use the same work schema
2. Can i use same contexts created or do i need to create everything as new and then try?
Thanks,
Rp.
Hi,
As you mentioned that you just upgraded the database, so the data is same and schema is same you can connect with the same work schema.
Yes, you can use the same contexts , but need to do Reverse Engineering for your new database.
And about data server, i think you also have no need to create new data server, if hostname,sid and port etc are same as these were with R11(consult with your DBAs regarding it)

ITunes Automatically Change EQ Setting Based On Which Airport Express Streaming? I have two Airport Expresses running at my house. Both are connected to different types of speakers. Neither speaker set has its own equalizer, and both require a different

I have two Airport Expresses running at my house. Both are connected to different types of speakers. Neither speaker set has its own equalizer, and both require a different EQ setting to sound just right. Does anyone know of a way to have iTunes automatically change its EQ setting based on which Airport Express it is streaming to?

I'm in the same boat. I have a 6 zone amplifier running with 6 airport expresses to different rooms inside and outside the house. Each has it's own acoustic characteristics. I'd really love to be able to set each airport to equalize based on my SPL frequency sweep I did for each room.
Setting the eq in itunes won't do it as that is global plus I have many more sources for auiod other than itunes that use the AEs directly.

Character Set based on Logon Language

Hi expert,
We have an issue here after upgrade from ECC5 to ECC6.
- In ECC5 (non-unicode), if we login with language ZH (chinese), the character set automatic set to Simplified Chinese.
- In ECC6(unicode), if we login with language ZH (chinese), the character set remain be Western European.
How can we set the system to automatic set the correct character set based on user logon language?
Currently in ECC6, user have to go to application toolbar -> Customize Local Layout -> Character Set to select the correct character set.
Any advice will be very appreciate.
Thank you.

Hi,
Does someone has any idea on my issue?
Thank you.

View offending records in audit viewer for set-based mappings

i have mappings which is configured to load data from Oracle9i external tables into Fact tables.
i am using set-based loading with reject limit set to 0. Whenever there's an offending record from the external table, the whole mapping will fail with message 'Fatal error or maximum error exceeded'. This is exactly how i want it.
now, when i check the mapping process in audit viewer, i expect to see which record caused this process to abort. instead, i only see the same message stated above (which is not very helpful in identify source of the problem, right ?)
i believe this is due to the mapping being set-based. unfortunately i can't use row-based as i do not want ANY record loaded into the fact table even if there is 1 offending record. In row-based mode, it seems to auto-commit after a certain number of records (bfor any errorneous records are encountered). i have attempted to set the 'commit block size' to something big value (eg. 50,000) but it still seems to commit after say every 2000 records ...
any suggestions ?
thanks

Since you need set based code in order to avoid commits, it is difficult to trap record level errors - the set based procedure will run to the end or encounter an error and all you will get is a generic SQL error code and message. The commit frequency should work - I am not aware of any problems in this area. Can you give more details about this?
It might also be possible to filter correct records by using a splitter and set it in such a way that correct records get loaded into the production table, while incorrect are 'parked' into a temporary table for review. Another possibility, if you need complex data cleansing prior to loading, would be to use an intermediate staging table and then use the row based mode on it, before transferring the correct records from this staging table to the appropriate production table. This last solution will slow down your process because it introduces an additional step, but might give you the loading flexibility you need.
Regards:
Igor

ORA-00903: Invalid table name - running Set based mapping

Hello,
Using OWB 10.2.04.36 and have created a mapping which reads data from Non-Oracle, ODBC source table, actually a worksheet in an Excel workbook which has been defined/set up using the Heterogeneous Service components.
I can view the data in the worksheet using the Design Center, Data Object Editor, Data Viewer tab so I know the data is accessible.
The mapping is performing a Loading Type: INSERT/UPDATE into a View which has an INSTEAD OF INSERT OR UPDATE OR DELETE ON view.
The mapping validates okay and is deployed successfully.
Yet when it is run in "Set based" Operating Mode from Control Center Manager the Execution Results show an "ORA-00903: Invalid table name" error is raised.
You cannot run the mapping in any Row based operating mode as Row based running fails with "ORA:22816: Unsupported feature with RETURNING clause" due to the generated code for the INSERT/UPDATE of the view using a RETURNING clause which is actioned on an INSTEAD OF trigger.
Looking at the generated package code I can strip out the SELECT statement from the MERGE statement for the alias "MERGE_SUBQUERY" and it runs and displays the expected result, however when the complete MERGE statement is taken and run I get the same ORA-00903 error that was reported in Control Center Manager.
Any ideas what the problem could be? I have another mapping that reads from the same source Excel workbook/worksheet and INSERT/UPDATE into a table without an INSTEAD OF trigger, this mapping deploys, runs successfully so the issue seems to be with the INSERT/UPDATE into the view. The views are what we require to be populated.
Thanks.

Hi,
But changing V_EMP_DEPT to EMP is not INSERTING/UPDATING to the view V_EMP_DEPT, what you propose is INSERTING/UPDATING into the table EMP. The code was only an example showing that the MERGE does not work when INSERTING/UPDATING into a view based on joining tables. For example say you wanted to INSERT/UPDATE the DNAME of V_EMP_DEPT then the MERGE statement generated by OWB PL/SQL mapping would use the code structure/template:-
MERGE INTO "V_EMP_DEPT" "V_EMP_DEPT"
   USING (SELECT 5369 "EMPNO",
                 'SMITH' "ENAME",
                 'CLERK' "JOB",
                 7902 "MGR",
                 To_Date('17/12/1980','DD/MM/YYYY') "HIREDATE",
                 800 "SAL",
                 'New Dept Name" "DNAME"
          FROM   Dual,
                 "DEPT" "DEPT"
          WHERE ("DEPT"."DEPTNO" = 20)) "MERGE_SUBQUERY"
   ON (    "V_EMP_DEPT"."EMPNO" = "MERGE_SUBQUERY"."EMPNO")
   WHEN NOT MATCHED THEN
      INSERT("V_EMP_DEPT"."EMPNO",
             "V_EMP_DEPT"."ENAME",
             "V_EMP_DEPT"."JOB",
             "V_EMP_DEPT"."MGR",
             "V_EMP_DEPT"."HIREDATE",
             "V_EMP_DEPT"."SAL",
             "V_EMP_DEPT"."DNAME")
      VALUES("MERGE_SUBQUERY"."EMPNO",
             "MERGE_SUBQUERY"."ENAME",
             "MERGE_SUBQUERY"."JOB",
             "MERGE_SUBQUERY"."MGR",
             "MERGE_SUBQUERY"."HIREDATE",
             "MERGE_SUBQUERY"."SAL",
             "MERGE_SUBQUERY"."DNAME")
   WHEN MATCHED THEN
      UPDATE
         SET "ENAME" = "MERGE_SUBQUERY"."ENAME",
             "JOB" = "MERGE_SUBQUERY"."JOB",
             "MGR" = "MERGE_SUBQUERY"."MGR",
             "HIREDATE" = "MERGE_SUBQUERY"."HIREDATE",
             "SAL" = "MERGE_SUBQUERY"."SAL",
             "DNAME" = "MERGE_SUBQUERY"."DNAME";
{code}
This was only an example my target view has a lot more columns being MERGE'd into the view and joined tables.
Cheers,
Phil

Error create new Application Set based on ApShell (BPC 5.1)

On BPC 5.1, I'm trying to create a new Application Set based on Apshell, called "UFG".
At the step where the OLAP database is created, I receive an error message stating the following.
Error message:The ApShell Role role referenced by the CubePermission permission does not exist in the UFG database.
Any help would be greatly appreciated.

Hi Joost,
yes it's a fresh 5.1 installation. I'm trying the steps you suggested, however I get the following errors:
If I try to assign the installation user to the AdminTeam I receive the following error:
Cast from string "SAP" to type 'integer' not valid.
If I attempt to assign the installation user directly to the SysAdmin Task Profile and to the AdminMbrAccPrf Member Access Profile, the error message is:
Error converting data type nvarchar to numeric.

Getting Errors while setting the Transformation Function

Hello All,
I was trying to attach a Transformation function to existing capture process.
But I am getting ORA-30625 error while I was trying to set my transformation function to one of the
DBA rule_names.
Following are details:
SQL> select rule_name from dba_rules;
RULE_NAME
TEMP1
TEMP3
TEMP5
TEMP7
SQL> DECLARE
2 action_ctx SYS.RE$NV_LIST;
3 ac_name VARCHAR2(30) := 'STREAMS$_TRANSFORM_FUNCTION';
4 BEGIN
5 SELECT RULE_ACTION_CONTEXT
6 INTO action_ctx
7 FROM DBA_RULES R
8 WHERE RULE_OWNER='STRMADMIN' AND RULE_NAME='TEMP1';
9 action_ctx.ADD_PAIR(ac_name,
10 SYS.ANYDATA.CONVERTVARCHAR2('hr.transformation'));
11 DBMS_RULE_ADM.ALTER_RULE(
12 rule_name => 'strmadmin.temp1',
13 action_context => action_ctx);
14 END;
15 /
DECLARE
ERROR at line 1:
ORA-30625: method dispatch on NULL SELF argument is disallowed
ORA-06512: at line 9
Can any body suggest me what to do...
Thanks in advance.
Regards,
Srinivas Jaltaru
( [email protected] )

Hello
I added the line you suggested, Compiling is done successfully, But Capture process
is not invoking the Transformation function that I have assigned to it.
Transformation function I am using is :
create or replace function scott.transformation (in_any IN SYS.AnyData)
RETURN SYS.AnyData
IS
rc NUMBER;
BEGIN
insert into scott.log values ('abc');
return in_any;
end;
1) Any extra initialization parameters are required..?
2) My Streams environment is having only a Capture process for capturing changes. There are no other
components like propagation, apply.
Appreciating your suggestions.
Thanks in advance.
Srinivas Jaltaru
( [email protected] )
Put the following line
action_ctx := SYS.RE$NV_LIST(SYS.RE$NV_ARRAY());
on the line immediately preceding the call to ADD_PAIR.
This should allow the pl/sql function to compile successfully.

OWB 10.2.0.4 really bad set-based delete performance

Hi, we recently upgraded to OWB 10.2.0.4, with one of the reasons being the ability to do set-based deletes instead of row-based. However, upon testing this, we're seeing maps that in row based deletes go from 30 - 40 seconds, now taking literally 1.5 to 2 HOURS to run.
I expected the SQL from the set based to take the form of:
delete from my_table
where (col_a, col_b, col_c) in (select a, b, c from ....)
but instead the format is different:
delete from my_table
where exists (select 1 from ....)
I don't quite understand what the SQL is trying to accomplish - and truthfully, it performs horribly compared to the hand-written version (explain plan shows estimated cost of 14,000 for my query, and over 5 million for the OWB query).
Has anyone else seen this - and is there a solution? Part of me wants to say I'm doing something wrong, but the other part says "sure, but it works fine in row-based mode(target only)" - exact same map.
Any ideas?
Thanks!
Scott

Hi everyone, we'll I've figured out what is causing the problem and how to fix it...but still don't understand why it causes the problem.
Here's a high level overview of the ETL - we find deleted record by selecting business key columns from our existing dimension table and doing a MINUS on the matching columns from the source table. If any records come out of this, it means the record was deleted on the source, and we go ahead and do a matching delete on the dimension table.
Here's where the odd thing happens though - there's a column called "source system name" that is part of the dimension business key. This column does NOT exist on the source system - it's just a hard coded constant (put in just in case we ever add an additional system in the future).
Basically, if we do the minus logic on all the columns EXCEPT for this one, and then connect a constant to the delete operator that has this hard coded value in - the delete takes FOREVER... On the other hand, if we actually put this field into the minus operator by simply repointing the existing constant there instead of directly to the delete table...the deletes magically start taking 30 seconds instead of 10 minutes to run.
No idea (at all) why this makes a diff, but it seems to - and it's a day and night different.
Hopefully this can help someone else out who runs into the same issue.
Thanks!
Scott

Correlation set based on correlation from raised ADF event payload

Hi,
In a BPEL process, how can we include a Correlation set based on correlation from raised ADF event payload ?
Thanks

Hi Charles,
Thanks for the response.
Actually, my requirement is : I have a BPEL Process which has a pick activity. The OnMessage branch of the Pick activity is mapped to a service which is invoked by a Mediator. This Mediator is Subscribed to a Create Event.
I am able to run my BPEL and Mediator instances separately but the problem is even though the Mediator is invoking the webservice, after the Create Event is raised, BPEL 's pick activity is not getting the response from the webservice due to which BPEL continues to be in running state.
This might be a problem with a Correlation Id seting .
How can i make by BPEL process know about the Mediator instance ? Any idea ?
Thanks
Nutan

Rule-based transformations perfomance

Hello.
Oracle 9.2.0.8.
We were tuned a replication on Oracle Streams.
There are 1 capture process and more than one apply processes.
We need to hide some columns in apply databases, for example, replace it with *.
Now we use triggers on apply schemas, which replace column value with * before insert. But with update and delete operations there are problems, because apply process generate errors - 'no data found' and put transactions in error queue.
Optimal variant is to use rule-based transformations on the capture process on a table, where are columns, which need to hide.
We wrote pl/sql transformation function and create rule with action_context associated with this function, but see, that processor loading of capture process increase doubly.
Please tell me, who work with transformations. Are really transformation hardly load capture machine or it is mistake? And how I can do aforesaid task with other methods?

Hello Kinney,
From what I've seen in your code, your code appears to be incorrectly written.
1. The transformaiton function should not execute the LCR, it just have to do the necessary modifications for the LCR. The apply process would execute the LCR and commits the transaction when all the LCRs executed.
2. You should not perform COMMIT in the transformation function. You need to eliminate the COMMIT from the code. The transformation function should not execute and COMMIT the LCR. The apply process would execute and COMMIT the transaction when it receives a commit lcr.
Please eliminate the following from the function and re-create the function:
lcr.EXECUTE(true);
commit;
Please test and see if the function is working as expected.
Thanks,
Rijesh

Correlation set based on Human task - receive activity

Hello everybody,
is it possible to define a correlation set based on a receive activity of an human task?
Thanks, Chris

Are you trying to set "Create with parameters" as default task flow activity ? in that case open the the task flow diagram , right click "Create with parameters" data control -> Mark Activity -> Default Activity.

Can the iPhone have settings profiles? I want the settings of what apps are using the "call home" automatically set based on my location or the time of day. Is this possible?

Can the iPhone have settings profiles? I want the settings of what apps are using the "call home" automatically set based on my location or the time of day. Is this possible? Of course the GPS location feature will need to be on in order for this to work. But the idea here would be to optimize battery life by disabling the 'call home' requests of apps. The settings profile should also support downloads as I would like to download and allow it when I am on WiFi but not so much when I am using my cellular data.

No, this is not possible as the sandboxed nature of iOS would not allow any app to modify another app like that. Such a feature would have to be enabled by Apple as a part of iOS itself. You can submit feedback to Apple about the idea if you wish to.
https://www.apple.com/feedback/

ODI - Set Based Transformations

Similar Messages

Maybe you are looking for