Building staging area on DWH

Usually i use DTS or SSIS (microsoft) to import data to my Staging area.
Now all my data is in Oracle 11g DB and I would like to replace my
data transformation step to Streams Replication.
I would like to ask, do you thing it is a good chose ?
I'm Newbi on replication subject, i read this article
http://www.oracle.com/technology/products/dataint/pdf/twp_streams_replication_11gr1.pdf
But is still didn't helped me a lot how to begin
Is there any place i can learn more for Newbi ?

Oracle 11g Streams and Replication
http://www.oracle.com/pls/db111/portal.portal_db?selected=14&frame=#oracle_streams
http://www.oracle.com/pls/db111/portal.portal_db?selected=14&frame=#advanced_replication

Similar Messages

Maintaining the structure of a SQL Server staging area and dwh aligned with the Oracle data source

Hi,
I'm working in a context where the data source system, in Oracle, is a continuos work in progress. Each 1-2-3 weeks the data source system in the prod environment is updated with new tables or updated table (with new columns or altered columns). So, it is
important to apply in a rapid manner the related changes in the data structure of the SQL Server staging area and dwh.
The issues to solve are:
a. maintaining SQL Server data structure of the staging area aligned with the data structure of the Oracle data source;
b. maintaing SQL Server data structure of the staging area aligned with the data structure of the SQL Server staging area.
In order to solve these issues it could be useful to think to an authomatic manner to alert when a data structure change occurs and a simple manner to apply the changes on SQL Server data structure.
Any suggests, please? Many thanks

We use Oracle CDC service in SQLServer. It has a flag to indicate the schema changes happening at the Oracle end. We track the schema changes using it and then apply changes to sqlserver side. And regarding automation you can use Biml as suggested or .NET
scripts inside script task.
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Staging Area with OWB 10.2 - necessary or not?

Hi to all,
I have read so much about Staging Area and OWB 10.2 that I am totally confused: Some documents and powerpoints in the web say you do not need one, others say you need one. The thing is I am planning a DWH and now I am not sure if a staging area is necessary or not, because the mappings do the ETL jobs internal so I am not sure about the staging area. Most of my data sources are Tables/Views/MViews in a database.
Thank you very much for any help concerning this question!
Regards
Thomas

Would you prefer the answer that you MAY need one? Then again, you may just WANT one!
For example, if you are building against a high transaction volume, busy 24/7 OLTP system then you may find that you need a local snapshot in order to do a complete build with a consistent set of source data for all your numbers to be consistent.
Then again, you also may just find that bringing over just delta data into a local snapshot makes for much more efficient load rather than running against huge full remote tables if they are not well partitioned and/or indexed.
Then again, complex joins run against a remote system may run more efficiently if you bring the data across with simple table dumps into a staging area that you can index to optimize your queries rather than have to deal with poor performance of complex joins over a dblink. Especially if you need to perform complex joins accross more than one db link to multiple source source systems. How big a cartesion product do you want bouncing around the network to perform that sort of scenario? Sure, maybe you can do it - but how much are you going to impact performance across the boards doing things like that?
Is the source system already stresed to the max and sitting on a vintage piece of equipment, but your shiny new DW environment is blessed with tons of resources that will make the ETL run faster by several factors if you first copy the data over locally?
So, do you need a staging area?
Fact is that there is no generic correct answer to this question.
You have to look at the specifics of your data requirements and your environment to answer that question. There are costs and benefits to having a staging area, and you have to determine which way the cost/benefit analysis comes out for your specific project.
Mike

Staging Area Indexing

We currently planing to build our staging area from source database (Oracle 11g database)
So we decided to do 1 to 1 mapping for each table but there are some tables that contains huge data (millions of records) and for these tables we decide to do indexing to fasting the extraction using data services to our staging database but at the same time I think will affect the loading to staging area
So my question is the indexes is recommended to be created in staging area?

Hi Mohamed Anas,
Usually Creating indexes on the columns will give better performace the index does not take much space in the memory, there by resulting in faster SELECTs. I am not much expert to say that whether it is suggestable in staging area or not but while loading the data into any target table which is having indexed columns will effect the performace of the job ,
In general while loading the data into target we will drop the index in Preload SQL and will recreate it with post load SQL in target properties, because it will give better performace.
so as per the above even though if you created the indexes on columns, its better to drop it and re create it before loading into target that will give you better performance.
Hope you have got some idea,
Thanks,
subbu CH.

HU routing to Staging area based on destination Storage type

Hi Experts,
I have a requirement where I need to route the HUs coming out from same VAS work center to the different staging areas according to the destination storage types.
Say for example,
VAS Workcenter - VAS1
HUs completed out of the VAS1 workcenter should be put away into the destination storage types (RS11, RS12, RS13, RS14) through the staging areas (ST11, ST12, ST13, ST14 respectively)
Any ideas how to achieve this through customizing to route the HUs through relevant staging areas ( VAS1 -> ST11 -> RS11, VAS1 -> ST12 -> RS12, etc..,)
Rgds
-Shravan

Thanks to everyone for your response.
Regarding Juergen's questions, here I am talking about the inbound process
RSXX are final storage types, where the materials will be finally put away
Staging area(work center) is used for inbound where we consolidate the materials going to same storage type (building), to the one trolley
The product put away WT is created when we complete the HU in the staging area workcenter (Final step IBS1 in the process), so we are not able to know to which storage type the product will be moved, until the HU is completed in the staging work center.
Rgds
Shravan

Oracle 8.1.5 Invalid Staging Area Error

I get an error message: Invalid staging area. There is no top level components for Windows NT available for installing
from this staging area. I installed this version successfully running win98, but i get this error with XP. I tried renaming the symcjit.dll
as suggested in some of the earlier posting with no success. Any help is appreciated.
Thanks,
Will

copy disk to hardrive in a temporary directory. go to properties for setup.exe (right click on file used to install oracle). Select the compatiblity tab. make sure the check box "run this program in compatibility mode" is checked. In combo box select windows 98.

Lookup in staging area

We have a column on the source table which contain a person's country information. A single country name could be represented in different formats (for e.g. United States of America, USA, U.S.A). The target database has a table which stores the equivalence mapping for all these different possibilities (i.e. United States of America, USA, U.S.A are mapped to "USA")
We plan to have a design like this
Step 1. Load the country column to the staging area
Step 2. In the staging area, lookup the equivalences table, identify the country name and populate it on the staging area
Step 3. Transfer the data into target table.
The point where we have discussion is the Step-2. What would be an efficient way of resolving the equivalences and how we proceed in this step.
Any thoughts/suggestions on what would be a good or bad design here ?

Hi,
Let me try to help...
If I understood right you can achieve that in just one interface by check the join between the column "country" and the table where the all possible match are at staging area, it means:
1) create an interface
2) drag and drop all souce tables plus the table where are the countries are, from target.
3) drag and drop the "final" target table at target position
4) create a join between the "target countries tables" and "person table" by "country" column and check it to be executed at Staging Area.
Does it make any sense to you?
Cezar

How to print staging areas and doors on the pick doc. ?

Dear experts!
Thank you for your attention!
In our system, we use Lean WM.
We know staging areas and doors can aslo be printed on the pick doc. How to do that?
Best regard!
Tangdark

I'm not a WM expert, but am pretty sure nothing special is required for that. If the fields, where this information is stored, are used by the output processing program and the data is passed to the form and then printed by the form, there should be no problem. If data is there, but it's not printed, then you might need to adjust the output program and/or form.
You might want at least to try it first and use Search whenever possible, then come back with more specific questions, if necessary.

Staging area creation problem

Hi,
We have got media DVDs for R12.1.1 for Linux X86 from Oracle. We copied in the server one by one using cp command.
Ans now when we are starting staging creation using these copied DVD's from server, it's giving
Invalid Disk is inserted.
Please insert DVD labeled Oracle Applications Rapid Install - of Rdbms Disk 1
We removed STARTCD part initially which was staged properly using OS rm command.but when it was asking for second DVD it failed due to unavailability of 2nd DVD path
We copied all DVD's into /do1/R1211dump folder
Please advice on how to create staging area
Regards,

Hi,
Yes.
Also this is being executed by root user .
And when using oracle user, even though DVD dump folder is owned by oracle user
we are getting following error:
cannot unlink file for /tmp/staging/adautostg.pl: Permission denied at adautostg.pl line 365
cannot restore permissions to 0100777 for /tmp/staging/adautostg.pl: Permission denied at adautostg.pl line 365
cannot unlink file for /tmp/staging/RapidWiz.ini: Permission denied at adautostg.pl line 365
cannot restore permissions to 0100744 for /tmp/staging/RapidWiz.ini: Permission denied at adautostg.pl line 365
cannot unlink file for /tmp/staging/File/NCopy.pm: Permission denied at adautostg.pl line 365
cannot restore permissions to 0100744 for /tmp/staging/File/NCopy.pm: Permission denied at adautostg.pl line 365
cannot remove directory for /tmp/staging/File: Permission denied at adautostg.pl line 365
cannot remove directory for /tmp/staging: Operation not permitted at adautostg.pl line 365
Failed to copy RapidWiz.ini to /tmp/staging
The log file is located at /tmp/10311250/adautostaging.log
and
$ cd /tmp/staging/
$ ls -l
total 100
-rwxrwxrwx 1 root root 71841 Oct 31 12:50 adautostg.pl
drwxr-xr-x 2 root root 4096 Oct 31 12:26 File
-rwxr--r-- 1 root root 24329 Oct 31 12:26 RapidWiz.ini
$
Not sure how to proceed now..

Staging Area - Sans Transformations?

I am struggling to come up with a proper term for a database that will contain untransformed application data. This would be an EL process what will pull application data from the source, and insert it into the "staging" area. A
subsequent job will perform the ETL process into the data warehouse. The goal for this area is to have unaltered application data to use as the source for our ETL processes, where we will not have to deal with the retention policies associated with the
actual application data.

Often Staging in a data warehouse is permanent and long-term. If you purge the Staging area you have to make sure that all needed information is making it into the Data Warehouse. If you leave the data in Staging it's available for validation
and to support design changes in the Data Warehouse.
Also I'm seeing more direct access to this raw area of the DW, as "Big Data" workflows are gaining popularity and complementing more traditional approaches to analytics. Broadly, in "Big Data" the analyst/end-users get access
to raw source data quickly, and then do the transforms and reporting together as part of an ad-hoc analysis.
David
David http://blogs.msdn.com/b/dbrowne/

Staging area for Management Agent files...?

Sorry for another question so quickly - I am reading the documentation first!
Management agents upload files to the OMS server, and these are parsed and loaded as data into the MR database, right?
Where is the staging area for these files (upload from all the management agents)? Is it possible to change it? From what I can gather it's
https://oms_server:1159/em/upload - but where is this in the real world? I've searched the OMS $ORACLE_HOME, but can't find any uploaded files
The reason for these questions is because we have a requirement to load these files to a shared area so that both our OMSes can access them - has anyone come across a configuration like this before? Does it work?
Many thanks (again!)
D

The upload area on the OMS would normally be <OMS_HOME>/sysman/recv.
You still might find no data files over here, as they may be uploaded to the OMS on that moment.
Yes, making this directory a shared location ,will work when running multiple OMS.
Regards
Rob

Staging area needs Oracle universal Installer version 10.2.0.0.0 or later

Hi,
I am trying to install OEM agents thru run installer. but it says "staging area needs Oracle universal Installer version 10.2.0.0.0 or later", we are on we are on 10.1.0.5.0. How to download 10.2.0.0.0 or later.
Regards

# From Metalink click on "Patches & Updates” tab
# Select "Advanced Search”
# Enter "Universal Installer” under Product or Product Family
# Select "Oracle 10.2.0.0.0” release (Or higher)
# Select your Platform
# Change "Patch Type” to Any
# Change Classification to Any
# Click on Go

Staging area for inserts only?

If our DW will only over contain inserts, do we require a staging area?

Hi DarrenOD,
I personally like to put data in staging area before loading it to the Data Warehouse.It will give flexibility of debugging and very much needed when you have restricted time slot for data extraction from OLTP.
Following URL explains "Why do we need Staging Database"
http://www.dwbiconcepts.com/etl/27-basic-etl-concepts/137-why-do-we-need-staging-area-during-etl-load.html
Prathy
Prathy K

Staging area error and universal installer does not run

Everything was ok. I installed Oracle 9i r 2, AS 9.0.2, iFS 9.0.2.
Then I uninstall AS and iFS. But AS folder remains there (with inventory,contentsXML.. folder inside), even if I deleted it, when I tried to install AS again I get the error staging area error, and the folder appears again.
Another thing is that in oracle universal installer the installed and deinstall products options do not do anything. I only can select the install option
Did I do something wrong during the AS uninstall?
What Can I do now?
Thanks for your help,

Try looking here:
http://www.thomasfly.com/Oranux/#Step-by-step
follow these steps/configs and the install will go quite nice!

Staging area different from target

I am using oracle 10g as target database. I want to upload a flat data file into oracle table.
I uploaded sucessfully using LKM Oracle (SQL Loader) in Interface. In this case i am using oracle database as a staging area. So all load transferred to my database at the timing of processing.
I want to create staging area at my file server. So please help me out to resolve this problem.

A staging area must accept SQL syntax.
A File server won't accept SQL syntax.
So if you speak about Physical Server where you got your file and if there is a RDBMS installed in maybe but else I don't really know how you can...
You can specify another data server for your staging area or use the SUNOPSIS_MEMORY_ENGINE which use the cache I think but it's not optimized I think...
For this in the Definition Tab of your Interface choose Staging Area different from the Target and then select in the list box SUNOPSIS_MEMORY_ENGINE.

Building staging area on DWH

Similar Messages

Maybe you are looking for