Work Schema Vs Staging Area

Can anybody explain what's the difference between Work Schema and Staging Area in ODI.
Work schema we specify while creating Physical schema in Topology manager.
Is Staging area same as work schema?
Thanks

Hi,
you have 3 step
1-source
2-staging area
3-target
in any mapping (designer9 you may choose in which step execute the operation (according to step technology). The work schema is the place in which you execute operation of this step. So, if you say to
LOGICAL : MY_SRC_DB
schema: ora_schema
work_schema: ora_w_schema
LOGICAL : MY_TARGET_DB
schema: ora_t_schema
work_schema: ora_wt_schema
you could say to odi having staging area different from target (for example SUNOPSIS MEMORY ENGINE)
the operation on source would be executed on ora_w_schema
the operation on staging would be executed on SUNOPSIS_MEMORY ENGINE's work
the operation on target would be executed on ora_wt_schema
At Oracle University teacher said me to put work=schema. You may fill in them different only for security issue (or performance)
i hope to be usefull, my english is not so good;P
Decaxd

Similar Messages

Staging area different from target - which KM???

Hi,
I need to transfer data from CSV to DB (only new inserts operation)
I am working the following KM's
LKM File to SQL
IKM SQL Control Append
It's working fine. However now i need to keep the staging area (where C$, I$ etc table are created) in a separate schema.
I have created a diff schema for staging area and selected in the interface "Staging area different from target"
However not sure of which KM's to use.
Please let me know how to achieve this.
Thanks,
Rosh

Hi
1st thing you have to give the workschema for temp tables when creating physical schema in dataserver for target.
Then you have to select the workschema in overview of interface as "staging different from target".
After doing this when you will use predefined KM for the interface it will create temp tables in workschema.
Now suppose you are not giving the workschma at the time of creating physical schema and you have selected "staging different than target" in interface.Here your C$ table will be created in your workschema but I$ table that is used by IKM is going to be created on target schema.So for this again you have change the IKM KM i.e. where to create I$ table (wokrschema) by selecting the corresponding logical schema.
So its better you give the workschema at the time of creating physical schema.
Here is the query to give privilege by the sys_dba
Grant create any table to ODI_TEMP.
Hope you got it
(Please mark the answer as correct or helpful and close the thread)
Thanks

Staging Area Schema

My source and target table is same and I am trying to update certain column in table XXX, I have set staging area different then target and specify the staging schema where I do have the create table prevelige but when I run this interface its trying to create temp table in target schema where I do not have preveilege to create table.Any clue ...?
In the interface I have checked all my mapping process is in staging area not source or target.
Any clue ...?
Thanks

HI,
It depends on the KM that you are using... If you are using a IKM Incremental Update then the process will create a temp table at target because it needs it to make the PK comparation.
My suggestions to work around it:
1) if your ETL allows, work with the IKM Control Append
2) If you need of a Incremental Update ELT just point the "Work Area" at the target Physical Schema to the "Staging Area" Schema, change the Database connection user to the user of Staging Area and give him the necessary grants at the tables of the target schema (insert and update i imagine),
Just to be sure, are you using Oracle DB as source, staging and target?

Staging area and base are in different schema/tablespace

Please can someone give me some advantages and disadvantages to keep staging tables and base tables in same schema. For eg we have staging area where we daily truncate staging table, load in the stg table and do transformation process. Once done we will move the staging table data to base tables. Base tables sizes are huge volume like 6 to 8 million rows as there is no purging done.
I want to suggest to my team that we should keep them in separate schemas as I understand it will be good from I/O point.
Is there any other reason to keep staging and base tables in separate schema/tablespace.

Hi,
Definitely I agree with previous answers. You wrote that staging area, transformation etc.. So I think it's a datawarehouse. Staging and base tables should be stored in different data files also in different harddrives. During the ETL task there can be high load on the Disk subsystem. Storing them in different schemas is another subject which is not related with the disk performance.
When it comes to blocksize of the datafile, there is a approach like this:
If the database is used for OLTP systems, it would be good idea to setup smaller size blocks. But when we're talking about a datawarehouse, it's highly recommended to use larger block sizes like 16KB or 32KB. This setting has an advantage related to reading, accessing data blocks.
Regards,
Cuneyt

Lookup issue when staging area diff from target

Hi,
I am using staging area different from target (csv file to database)
Everything is working fine the source data is moved to target. The interface runs suiceessfuly.
Now i want to add a lookup during transformation using a lookup table.
When i do that using lookups in quick edit section, i am getting an error.
When i run the interface, the C$ gets created in staging area - that is correct.
However when inserting from staging to target, now the ODI searches for the C$ table in target area instead of the staging..
Please help.
Thanks,
Rosh

Hi
1st thing you have to give the workschema for temp tables when creating physical schema in dataserver for target.
Then you have to select the workschema in overview of interface as "staging different from target".
After doing this when you will use predefined KM for the interface it will create temp tables in workschema.
Now suppose you are not giving the workschma at the time of creating physical schema and you have selected "staging different than target" in interface.Here your C$ table will be created in your workschema but I$ table that is used by IKM is going to be created on target schema.So for this again you have change the IKM KM i.e. where to create I$ table (wokrschema) by selecting the corresponding logical schema.
So its better you give the workschema at the time of creating physical schema.
Here is the query to give privilege by the sys_dba
Grant create any table to ODI_TEMP.
Hope you got it
(Please mark the answer as correct or helpful and close the thread)
Thanks

Staging area on source in Oracle

Hi,
My source and target, both are Oracle DBs.
a) I created one interface with STAGING AREA ON TARGET. For this I used LKM SQL to SQL and IKM SQL Incremental Update.
Its working fine and all the work tables, flow table etc are gettting created in the
Work Schema (ie. ODI_TRG) I have given while defining physical schema of my target.
I want to try with staging area on source....
b) I checked the "Staging area different from Target" option and selected the logical schema of the source in the Definitions tab of my interface.
In the corresponding physical schema, Work Schema I gave is ODI_SRC.
Now I expected all the work tables, flow tables etc to be created ODI_SRC. But still they are getting created in ODI_TRG!!! Why???
But finally to insert/update the target table, its trying to read the flow table from ODI_SRC, even though it created the flow table in ODI_TRG!!!!
And as a result its failing with this error code...
942 : 42000 : java.sql.SQLException: ORA-00942: table or view does not exist
Please clarify.

I think you didn't change your KM.
Step1: import the "IKM SQL to SQL Append" into your project
Step2: In your interface, on the definition table, select the "Staging Area Different From Target" checkbox, and select the schema of your source data in the drop-down list box.
Step3: go to the flow tab, and select the target, and select the IKM SQL to SQL Append.
Step4: Execute.
Refer this thread what KM to change.
Optimizing the KM

E$ table to be created in STAGING Area

Hello All,
I have an interface that has Staging Area different from TARGET. I have created an ODI Constraint in the target table in my model. Currently, when I run the scenario, the E$ table is created in the target database.
Because we are planning to use E$ for the tracking purpose for which the INtegration team would have access.
1) How can I force the E$ table be created in the ODI STAGING AREA (Database) rather than creating at the target database ?
2) What is the use of RECYCLING ERROR in IKM ?
Kindly clarify.
Thanks

Hi Nag,
Unfortunately a huge KM modification is necessary to accomplish this or divide in more than one interface (process). Let me try to explain:
An IKM Incremental Update create the I$ at target even when the stage area is another Logical Schema because it needs the target table to "decide" what records are to update and what are to insert.
Because of that, the internal the ODI algorithmic set the connections to the target once the I$ will be created there anyway.
To accomplish what you need it will be necessary to change the way that this kind of ETL is done and force the connections to the new staging area.
That is possible but will take some time.
An alternative solution, by process, could be;
1) make a copy of the target at staging area (if possible, I mean, if the amount of records allows it)
2) add 2 columns to this copy to flag the kind of dml (flg_ins, flg_upd for instance)
3) copy all desired constraints to the temporary table
4) use the same original interface but change the target to the temporary table.
5) at mapping check just "insert" to column flg_ins (value "Y" for instance)
6) at mapping check just "update" to column flg_upd (value "Y" for instance)
7) execute the interface
8) create a procedure with 2 steps:
9) step 1 - tab source: select all records from temporary table where flag_ins = 'Y'
- tab target; insert into original target
10)step 2 - tab source: select all records from temporary table where flag_upd = 'Y'
- tab target; update into original target
for the itens 9 and 10 will be necessary to choose the technology and respective Logical Schema.
If the amount of records at target doesn't allow this technique than:
Or will be necessary change the IKM
Or work with DBLINK from target to staging if both technologies are Oracle and to create a dblink will be possible. Any way IKM customization will be necessary, but a small one.
Does it make any sense to you?

XML as staging area

Hi All,
I am using SQL server as my source and XML as my target. I am using target area same as staging area
Now I want to look up fior all my work table i.e. E$.C$ and I$...where can I see these tables...i mean where are these tables stored when my staging area is XML?
Please help
Thanks

Hi,
ODI creates these temporary tables (E$, I$, C$) in the work schema of your staging area. So check your Staging area (on the definition tab), this will be the name of one of your logical schemas for SQL Server, then go to Topology Manager, find the logical schema and the physical schema which is associated through the Context in which you are running your process. Review the Physical Schema. There are two sections with two drop-down lists each: one section is for the for the data catalog/schema, the other one for the work catalog/schema. The temporary tables are created in the work schema (this might be "dbo" or what ever user you defined) of the work catalog.
Hope this helps,
Matt

Using a separate physical Work Schema

Hi,
There is a new Oracle data server that we are setting up and it will be used as a target.
The ODI documentation recommends to create a user dedicated for ODI (typically named ODI_TEMP)
We create a separate new user/schema called ODI_TEMP on the new Oracle database and grant it the necessary privileges.
So, now we have a physical target schema and a separate phyisical work schema (ODI_TEMP) on this new target database.
In ODI, we configure the new data server by going to Topology-> Physical Architecture-> Oracle- and select the New dataserver option. We enter the details for the physical target schema, ie the user connection details, etc. and save the new dataserver.
We then add a new physical schema to the newly created data server.
There are options to enter the details of the schema and also you can define here the separate workschema.
For the schema we select the target schema in the drop down. In the drop down for the workschema we can see ODI_TEMP nicely listed and select it.
But how or where should we enter the connection details for ODI_TEMP user/schema? It's connection details are different than the target schema.
Cheers.

Hi.
You will always need a work schema on each DB.
This is where ODI creates temp views, load tables etc.
You can have a separate staging database which you use as a staging area i.e. ODI_TEMP on a server of it's own. but you may also need a work schema on each target or source you connect to depending on technology used and where you tell ODI to do the transformations.
e.g. when you need to pull data from source to staging via a DBlink KM, ODI creates a view on the source (a work schema on the source is needed for this), and a view on the staging/target depending on how the interface is set up - again this will be created in the work schema of your staging area or on the work schema of the target.
You cannot really get away from this - I don’t think you should attempt to since there are many advantages of this approach, e.g. Executing filters on source means less data pulled from source to staging/target.
Hope this clears things up.

Maintaining the structure of a SQL Server staging area and dwh aligned with the Oracle data source

Hi,
I'm working in a context where the data source system, in Oracle, is a continuos work in progress. Each 1-2-3 weeks the data source system in the prod environment is updated with new tables or updated table (with new columns or altered columns). So, it is
important to apply in a rapid manner the related changes in the data structure of the SQL Server staging area and dwh.
The issues to solve are:
a. maintaining SQL Server data structure of the staging area aligned with the data structure of the Oracle data source;
b. maintaing SQL Server data structure of the staging area aligned with the data structure of the SQL Server staging area.
In order to solve these issues it could be useful to think to an authomatic manner to alert when a data structure change occurs and a simple manner to apply the changes on SQL Server data structure.
Any suggests, please? Many thanks

We use Oracle CDC service in SQLServer. It has a flag to indicate the schema changes happening at the Oracle end. We track the schema changes using it and then apply changes to sqlserver side. And regarding automation you can use Biml as suggested or .NET
scripts inside script task.
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Load file into Oracle without staging area

Is it possible to change LKM to do this?
Thanks

Yes it is.
You can set the 'Work Schema' (staging) of your topology object to the same as your target (Data Schema), then all objects will be created in the target (C$ I$ etc), therefore no staging area is required - although this is not best practice.
If you want to remove the staging tables altogether, then you can alter the knowledge module to load directly into the target in your LKM. It depends on the method you are using (external tables, sqlloader for example). You IKM will also need amending to basically do nothing, but has to be present for the interface to work.
Cheers
Bos
Edited by: Bos on Apr 7, 2011 8:54 AM

Oracle Data Integrator 11.1.1.5 Work Schema - List of Privileges

Hi All,
Oracle Data Integrator 11.1.1.5.
Extracting data from Oracle DB for Oracle EBS 12.1.3.
Customer created read-only schema (XXAPPS) to extract the data from EBS.
For ODI Work schema we now created one schema 'XBOL_ODI_TEMP' on the source DB. We are now looking for appropriate privileges that needs to be granted to XXAPPS and 'XBOL_ODI_TEMP' so that we won’t face the any error messageS related to permissions when we run ODI scenario?
We are now facing the error message : ODI-1227: Task SrcSet0 (Loading) fails on the source ORACLE connection VTB_ORACLE_EBS_1213.
Caused By: java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist.
Similar previliges can be granted to the work schema on target.
Venkat

i think it would be fine with only one schema(user) created at the source system which has got read access on the tables of the EBS DB. Now to resolve this error, assuming XXAPPS user is the one used,
in the topology --> data server(for EBS) --> physical schema the EBS schema name could be selected for Schema and XXAPPS as the work schema(for all ODI work related objects e.g. CDC)
Also, in the Data server the user XXAPPS needs to be used which has read access to EBS tables.
Now everytime ODI generates a query it will access a table lets say DUMMY as ,<EBS Schema>.DUMMY thus the reference is made.
Alternatively, you can create synonyms for EBS tables in XXAPPS schema.

Staging area for Management Agent files...?

Sorry for another question so quickly - I am reading the documentation first!
Management agents upload files to the OMS server, and these are parsed and loaded as data into the MR database, right?
Where is the staging area for these files (upload from all the management agents)? Is it possible to change it? From what I can gather it's
https://oms_server:1159/em/upload - but where is this in the real world? I've searched the OMS $ORACLE_HOME, but can't find any uploaded files
The reason for these questions is because we have a requirement to load these files to a shared area so that both our OMSes can access them - has anyone come across a configuration like this before? Does it work?
Many thanks (again!)
D

The upload area on the OMS would normally be <OMS_HOME>/sysman/recv.
You still might find no data files over here, as they may be uploaded to the OMS on that moment.
Yes, making this directory a shared location ,will work when running multiple OMS.
Regards
Rob

Staging area and target area

hi..friends.
should staging area and target area be same or different???
if same what are the advantages and dis-advantages???
and if not same what are the advantages and dis-advantages????
I want your help..
Thanks.

Staging area is the place where transformation takes place.
Basically c$,I$ tables gets created at this place and once processing is done , the same gets dropped .
By defalut staging area and traget area are the same . But most of the DBA does not like to have the temp tables created at the target as this cause dbms performance, maintance issue in long term.
Instead you can have a dedicated schema created just for staging and use the same in your interface.
Plus when your target is non dbms then you need to go for a dedicated staging area which can be a separate schema in your database or it could be in memory schema. You will find performance issue if you are using in memory schema as your staging area in case the data volume increase.
Thanks,
Sutirtha

How to change work schema ?

Hi,
Source : MS SQL database
Target : Oracle
I am loading the data from source table 'src_table' to target table 'trg_table'. I have created an interface for that. Currently ODI creating temporary tables like
$E_xyz(error)
$J_abc(Journalisam)
$C_abcd(check)
etc..
in my source schema (In MS SQL).
Can anybody let me know how I can store these temporary table in other schema ? What are the steps for that ?
How can I handle this things in production ?
Thanks,
Shrinivas

In topology ---> MS SQL database ----> physical schema ----> data schema ---> point it to your source schema .
work schema ---> point it to some other schema which resides in your MS SQL database
Thanks,
Sutirtha

Work Schema Vs Staging Area

Similar Messages

Maybe you are looking for