ODI Staging Area Confusion

Hi,
I am following an ODI Tutorial from http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/odi/odi_11g/odi_project_ff-to-ff/odi_project_flatfile-to-flatfile.htm. Basically it is writing a flat file to another flat file. My question in this case where will be the staging area?
in the article it is using knowledge modules IKM SQL to File Append and LKM File to SQL; based on the knowledge modules used it means staging area is in Oracle btu Idon't know why it is so.
Please help me in this regard and I will be very grateful.
Best Regards,

Your staging area has to be a technology with transformation capabilities i.e a relational DB. This could be a dedicated database such as SQL Server or Oracle or an in memory database such as SUNOPSIS Memory Engine. As both your source and target are files and don't have these capabilities you will have to choose the staging different from target option in your interface and then select the appropriate relational technology available in your ODI architecture.
The 2 inerfaces you have mentioned are not technology specific which means that you can choose any relational technolgy with SQL capabilities as your Satging Area, it does not have to be Oracle

Similar Messages

E$ table to be created in STAGING Area

Hello All,
I have an interface that has Staging Area different from TARGET. I have created an ODI Constraint in the target table in my model. Currently, when I run the scenario, the E$ table is created in the target database.
Because we are planning to use E$ for the tracking purpose for which the INtegration team would have access.
1) How can I force the E$ table be created in the ODI STAGING AREA (Database) rather than creating at the target database ?
2) What is the use of RECYCLING ERROR in IKM ?
Kindly clarify.
Thanks

Hi Nag,
Unfortunately a huge KM modification is necessary to accomplish this or divide in more than one interface (process). Let me try to explain:
An IKM Incremental Update create the I$ at target even when the stage area is another Logical Schema because it needs the target table to "decide" what records are to update and what are to insert.
Because of that, the internal the ODI algorithmic set the connections to the target once the I$ will be created there anyway.
To accomplish what you need it will be necessary to change the way that this kind of ETL is done and force the connections to the new staging area.
That is possible but will take some time.
An alternative solution, by process, could be;
1) make a copy of the target at staging area (if possible, I mean, if the amount of records allows it)
2) add 2 columns to this copy to flag the kind of dml (flg_ins, flg_upd for instance)
3) copy all desired constraints to the temporary table
4) use the same original interface but change the target to the temporary table.
5) at mapping check just "insert" to column flg_ins (value "Y" for instance)
6) at mapping check just "update" to column flg_upd (value "Y" for instance)
7) execute the interface
8) create a procedure with 2 steps:
9) step 1 - tab source: select all records from temporary table where flag_ins = 'Y'
- tab target; insert into original target
10)step 2 - tab source: select all records from temporary table where flag_upd = 'Y'
- tab target; update into original target
for the itens 9 and 10 will be necessary to choose the technology and respective Logical Schema.
If the amount of records at target doesn't allow this technique than:
Or will be necessary change the IKM
Or work with DBLINK from target to staging if both technologies are Oracle and to create a dblink will be possible. Any way IKM customization will be necessary, but a small one.
Does it make any sense to you?

Staging area and Target area in ODI

Hii...experts.
How Staging area is different from target area and same as target area?? what are their benefits and drawbacks in both??
Please suggest me with some pictorial representation..
Thanks

Hi SRK
I think you'll find your answer in this old white paper from oracle. The new version of the best practices are much smaller :/.
http://www.oracle.com/technetwork/middleware/data-integrator/overview/odi-bestpractices-datawarehouse-whi-129686.pdf
Page 87-95.
Hope it helps.
Regards,
Jerome
ps : p79-84 are interesting too

Unable to load staging area table in teradata from ODI

Hi
i'm unable to load data in Staging area in teradata. while loading i'm geeting following Error
org.apache.bsf.BSFException: exception from Jython:
Traceback (innermost last):
File "<string>", line 9, in ?
Load Error: See D:\CRM_MOD\/CRM_PRODUCT_SS_0.log for details
     at org.apache.bsf.engines.jython.JythonEngine.exec(JythonEngine.java:146)
     at com.sunopsis.dwg.codeinterpretor.k.a(k.java)
     at com.sunopsis.dwg.dbobj.SnpSessTaskSql.scripting(SnpSessTaskSql.java)
     at com.sunopsis.dwg.dbobj.SnpSessTaskSql.execScriptingOrders(SnpSessTaskSql.java)
     at com.sunopsis.dwg.dbobj.SnpSessTaskSql.execScriptingOrders(SnpSessTaskSql.java)
     at com.sunopsis.dwg.dbobj.SnpSessTaskSql.treatTaskTrt(SnpSessTaskSql.java)
     at com.sunopsis.dwg.dbobj.SnpSessTaskSqlC.treatTaskTrt(SnpSessTaskSqlC.java)
     at com.sunopsis.dwg.dbobj.SnpSessTaskSql.treatTask(SnpSessTaskSql.java)
     at com.sunopsis.dwg.dbobj.SnpSessStep.treatSessStep(SnpSessStep.java)
     at com.sunopsis.dwg.dbobj.SnpSession.treatSession(SnpSession.java)
     at com.sunopsis.dwg.cmd.DwgCommandSession.treatCommand(DwgCommandSession.java)
     at com.sunopsis.dwg.cmd.DwgCommandBase.execute(DwgCommandBase.java)
     at com.sunopsis.dwg.cmd.e.i(e.java)
     at com.sunopsis.dwg.cmd.g.y(g.java)
     at com.sunopsis.dwg.cmd.e.run(e.java)
     at java.lang.Thread.run(Unknown Source)
please any one help me
Thanks
Edited by: user12079238 on Oct 19, 2009 1:17 AM

Hi,
I believe you made some mistake in the Phisycal Schema, quit the last "\" from there....
Cezar Santos
[www.odiexperts.com]

Staging Area with OWB 10.2 - necessary or not?

Hi to all,
I have read so much about Staging Area and OWB 10.2 that I am totally confused: Some documents and powerpoints in the web say you do not need one, others say you need one. The thing is I am planning a DWH and now I am not sure if a staging area is necessary or not, because the mappings do the ETL jobs internal so I am not sure about the staging area. Most of my data sources are Tables/Views/MViews in a database.
Thank you very much for any help concerning this question!
Regards
Thomas

Would you prefer the answer that you MAY need one? Then again, you may just WANT one!
For example, if you are building against a high transaction volume, busy 24/7 OLTP system then you may find that you need a local snapshot in order to do a complete build with a consistent set of source data for all your numbers to be consistent.
Then again, you also may just find that bringing over just delta data into a local snapshot makes for much more efficient load rather than running against huge full remote tables if they are not well partitioned and/or indexed.
Then again, complex joins run against a remote system may run more efficiently if you bring the data across with simple table dumps into a staging area that you can index to optimize your queries rather than have to deal with poor performance of complex joins over a dblink. Especially if you need to perform complex joins accross more than one db link to multiple source source systems. How big a cartesion product do you want bouncing around the network to perform that sort of scenario? Sure, maybe you can do it - but how much are you going to impact performance across the boards doing things like that?
Is the source system already stresed to the max and sitting on a vintage piece of equipment, but your shiny new DW environment is blessed with tons of resources that will make the ETL run faster by several factors if you first copy the data over locally?
So, do you need a staging area?
Fact is that there is no generic correct answer to this question.
You have to look at the specifics of your data requirements and your environment to answer that question. There are costs and benefits to having a staging area, and you have to determine which way the cost/benefit analysis comes out for your specific project.
Mike

Lookup issue when staging area diff from target

Hi,
I am using staging area different from target (csv file to database)
Everything is working fine the source data is moved to target. The interface runs suiceessfuly.
Now i want to add a lookup during transformation using a lookup table.
When i do that using lookups in quick edit section, i am getting an error.
When i run the interface, the C$ gets created in staging area - that is correct.
However when inserting from staging to target, now the ODI searches for the C$ table in target area instead of the staging..
Please help.
Thanks,
Rosh

Hi
1st thing you have to give the workschema for temp tables when creating physical schema in dataserver for target.
Then you have to select the workschema in overview of interface as "staging different from target".
After doing this when you will use predefined KM for the interface it will create temp tables in workschema.
Now suppose you are not giving the workschma at the time of creating physical schema and you have selected "staging different than target" in interface.Here your C$ table will be created in your workschema but I$ table that is used by IKM is going to be created on target schema.So for this again you have change the IKM KM i.e. where to create I$ table (wokrschema) by selecting the corresponding logical schema.
So its better you give the workschema at the time of creating physical schema.
Here is the query to give privilege by the sys_dba
Grant create any table to ODI_TEMP.
Hope you got it
(Please mark the answer as correct or helpful and close the thread)
Thanks

XML as staging area

Hi All,
I am using SQL server as my source and XML as my target. I am using target area same as staging area
Now I want to look up fior all my work table i.e. E$.C$ and I$...where can I see these tables...i mean where are these tables stored when my staging area is XML?
Please help
Thanks

Hi,
ODI creates these temporary tables (E$, I$, C$) in the work schema of your staging area. So check your Staging area (on the definition tab), this will be the name of one of your logical schemas for SQL Server, then go to Topology Manager, find the logical schema and the physical schema which is associated through the Context in which you are running your process. Review the Physical Schema. There are two sections with two drop-down lists each: one section is for the for the data catalog/schema, the other one for the work catalog/schema. The temporary tables are created in the work schema (this might be "dbo" or what ever user you defined) of the work catalog.
Hope this helps,
Matt

Persistent Staging Area

We have a concept of a Persistent Staging Area (PSA) which functions very similar to a Type II SCD.
There is a couple of minor exceptions.
1. No Surrogate Key (the key for the table is the Natural Key and a SCD Starting Time Stamp)
2. We keep a Previous Begin Date to allow a developer to travers history
The lack of a SK does not seem to be a problem, using the existing SCD KM, but the Previous Begin Date is. There is no type to select when defining the column. Is there a way to create my own SCD column types? If so where is this done?
Thank you,
Troy

I'm quoting the ODI doc there.
When accessing an object properties through Oracle Data Integrator' substitution methods, if you specify the Flexfield Code, Oracle Data Integrator will substitute the Code by the flexfield value for the object instance.
For instance:
<%=odiRef.getTable("L", "MY_DATASTORE_FIELD", "W")%> will return the value of the flexfield MY_DATASTORE_FIELD for the current table.
<%=odiRef.getSrcTableList("", "[MY_DATASTORE_FIELD] ", ", ", "")%> will return the flexfield value for each of the source tables of the interface.
It is also possible to get the value of a flexfield through the getFlexFieldValue() method.
You find sometimes good things in the doc :-).
-FX

Work Schema Vs Staging Area

Can anybody explain what's the difference between Work Schema and Staging Area in ODI.
Work schema we specify while creating Physical schema in Topology manager.
Is Staging area same as work schema?
Thanks

Hi,
you have 3 step
1-source
2-staging area
3-target
in any mapping (designer9 you may choose in which step execute the operation (according to step technology). The work schema is the place in which you execute operation of this step. So, if you say to
LOGICAL : MY_SRC_DB
schema: ora_schema
work_schema: ora_w_schema
LOGICAL : MY_TARGET_DB
schema: ora_t_schema
work_schema: ora_wt_schema
you could say to odi having staging area different from target (for example SUNOPSIS MEMORY ENGINE)
the operation on source would be executed on ora_w_schema
the operation on staging would be executed on SUNOPSIS_MEMORY ENGINE's work
the operation on target would be executed on ora_wt_schema
At Oracle University teacher said me to put work=schema. You may fill in them different only for security issue (or performance)
i hope to be usefull, my english is not so good;P
Decaxd

Oracle 8.1.5 Invalid Staging Area Error

I get an error message: Invalid staging area. There is no top level components for Windows NT available for installing
from this staging area. I installed this version successfully running win98, but i get this error with XP. I tried renaming the symcjit.dll
as suggested in some of the earlier posting with no success. Any help is appreciated.
Thanks,
Will

copy disk to hardrive in a temporary directory. go to properties for setup.exe (right click on file used to install oracle). Select the compatiblity tab. make sure the check box "run this program in compatibility mode" is checked. In combo box select windows 98.

Lookup in staging area

We have a column on the source table which contain a person's country information. A single country name could be represented in different formats (for e.g. United States of America, USA, U.S.A). The target database has a table which stores the equivalence mapping for all these different possibilities (i.e. United States of America, USA, U.S.A are mapped to "USA")
We plan to have a design like this
Step 1. Load the country column to the staging area
Step 2. In the staging area, lookup the equivalences table, identify the country name and populate it on the staging area
Step 3. Transfer the data into target table.
The point where we have discussion is the Step-2. What would be an efficient way of resolving the equivalences and how we proceed in this step.
Any thoughts/suggestions on what would be a good or bad design here ?

Hi,
Let me try to help...
If I understood right you can achieve that in just one interface by check the join between the column "country" and the table where the all possible match are at staging area, it means:
1) create an interface
2) drag and drop all souce tables plus the table where are the countries are, from target.
3) drag and drop the "final" target table at target position
4) create a join between the "target countries tables" and "person table" by "country" column and check it to be executed at Staging Area.
Does it make any sense to you?
Cezar

How to print staging areas and doors on the pick doc. ?

Dear experts!
Thank you for your attention!
In our system, we use Lean WM.
We know staging areas and doors can aslo be printed on the pick doc. How to do that?
Best regard!
Tangdark

I'm not a WM expert, but am pretty sure nothing special is required for that. If the fields, where this information is stored, are used by the output processing program and the data is passed to the form and then printed by the form, there should be no problem. If data is there, but it's not printed, then you might need to adjust the output program and/or form.
You might want at least to try it first and use Search whenever possible, then come back with more specific questions, if necessary.

Staging area creation problem

Hi,
We have got media DVDs for R12.1.1 for Linux X86 from Oracle. We copied in the server one by one using cp command.
Ans now when we are starting staging creation using these copied DVD's from server, it's giving
Invalid Disk is inserted.
Please insert DVD labeled Oracle Applications Rapid Install - of Rdbms Disk 1
We removed STARTCD part initially which was staged properly using OS rm command.but when it was asking for second DVD it failed due to unavailability of 2nd DVD path
We copied all DVD's into /do1/R1211dump folder
Please advice on how to create staging area
Regards,

Hi,
Yes.
Also this is being executed by root user .
And when using oracle user, even though DVD dump folder is owned by oracle user
we are getting following error:
cannot unlink file for /tmp/staging/adautostg.pl: Permission denied at adautostg.pl line 365
cannot restore permissions to 0100777 for /tmp/staging/adautostg.pl: Permission denied at adautostg.pl line 365
cannot unlink file for /tmp/staging/RapidWiz.ini: Permission denied at adautostg.pl line 365
cannot restore permissions to 0100744 for /tmp/staging/RapidWiz.ini: Permission denied at adautostg.pl line 365
cannot unlink file for /tmp/staging/File/NCopy.pm: Permission denied at adautostg.pl line 365
cannot restore permissions to 0100744 for /tmp/staging/File/NCopy.pm: Permission denied at adautostg.pl line 365
cannot remove directory for /tmp/staging/File: Permission denied at adautostg.pl line 365
cannot remove directory for /tmp/staging: Operation not permitted at adautostg.pl line 365
Failed to copy RapidWiz.ini to /tmp/staging
The log file is located at /tmp/10311250/adautostaging.log
and
$ cd /tmp/staging/
$ ls -l
total 100
-rwxrwxrwx 1 root root 71841 Oct 31 12:50 adautostg.pl
drwxr-xr-x 2 root root 4096 Oct 31 12:26 File
-rwxr--r-- 1 root root 24329 Oct 31 12:26 RapidWiz.ini
$
Not sure how to proceed now..

Staging Area - Sans Transformations?

I am struggling to come up with a proper term for a database that will contain untransformed application data. This would be an EL process what will pull application data from the source, and insert it into the "staging" area. A
subsequent job will perform the ETL process into the data warehouse. The goal for this area is to have unaltered application data to use as the source for our ETL processes, where we will not have to deal with the retention policies associated with the
actual application data.

Often Staging in a data warehouse is permanent and long-term. If you purge the Staging area you have to make sure that all needed information is making it into the Data Warehouse. If you leave the data in Staging it's available for validation
and to support design changes in the Data Warehouse.
Also I'm seeing more direct access to this raw area of the DW, as "Big Data" workflows are gaining popularity and complementing more traditional approaches to analytics. Broadly, in "Big Data" the analyst/end-users get access
to raw source data quickly, and then do the transforms and reporting together as part of an ad-hoc analysis.
David
David http://blogs.msdn.com/b/dbrowne/

Staging area for Management Agent files...?

Sorry for another question so quickly - I am reading the documentation first!
Management agents upload files to the OMS server, and these are parsed and loaded as data into the MR database, right?
Where is the staging area for these files (upload from all the management agents)? Is it possible to change it? From what I can gather it's
https://oms_server:1159/em/upload - but where is this in the real world? I've searched the OMS $ORACLE_HOME, but can't find any uploaded files
The reason for these questions is because we have a requirement to load these files to a shared area so that both our OMSes can access them - has anyone come across a configuration like this before? Does it work?
Many thanks (again!)
D

The upload area on the OMS would normally be <OMS_HOME>/sysman/recv.
You still might find no data files over here, as they may be uploaded to the OMS on that moment.
Yes, making this directory a shared location ,will work when running multiple OMS.
Regards
Rob

ODI Staging Area Confusion

Similar Messages

Maybe you are looking for