Staging Area - Sans Transformations?

I am struggling to come up with a proper term for a database that will contain untransformed application data.  This would be an EL process what will pull application data from the source, and insert it into the "staging" area.  A
subsequent job will perform the ETL process into the data warehouse.  The goal for this area is to have unaltered application data to use as the source for our ETL processes, where we will not have to deal with the retention policies associated with the
actual application data.   

Often Staging in a data warehouse is permanent and long-term.  If you purge the Staging area you have to make sure that all needed information is making it into the Data Warehouse.  If you leave the data in Staging it's available for validation
and to support design changes in the Data Warehouse.
Also I'm seeing more direct access to this raw area of the DW, as "Big Data" workflows are gaining popularity and complementing more traditional approaches to analytics.  Broadly, in "Big Data" the analyst/end-users get access
to raw source data quickly, and then do the transforms and reporting together as part of an ad-hoc analysis.
David
David http://blogs.msdn.com/b/dbrowne/

Similar Messages

  • Staging area and target area

    hi..friends.
    should staging area and target area be same or different???
    if same what are the advantages and dis-advantages???
    and if not same what are the advantages and dis-advantages????
    I want your help..
    Thanks.

    Staging area is the place where transformation takes place.
    Basically c$,I$ tables gets created at this place and once processing is done , the same gets dropped .
    By defalut staging area and traget area are the same . But most of the DBA does not like to have the temp tables created at the target as this cause dbms performance, maintance issue in long term.
    Instead you can have a dedicated schema created just for staging and use the same in your interface.
    Plus when your target is non dbms then you need to go for a dedicated staging area which can be a separate schema in your database or it could be in memory schema. You will find performance issue if you are using in memory schema as your staging area in case the data volume increase.
    Thanks,
    Sutirtha

  • Lookup issue when staging area diff from target

    Hi,
    I am using staging area different from target (csv file to database)
    Everything is working fine the source data is moved to target. The interface runs suiceessfuly.
    Now i want to add a lookup during transformation using a lookup table.
    When i do that using lookups in quick edit section, i am getting an error.
    When i run the interface, the C$ gets created in staging area - that is correct.
    However when inserting from staging to target, now the ODI searches for the C$ table in target area instead of the staging..
    Please help.
    Thanks,
    Rosh

    Hi
    1st thing you have to give the workschema for temp tables when creating physical schema in dataserver for target.
    Then you have to select the workschema in overview of interface as "staging different from target".
    After doing this when you will use predefined KM for the interface it will create temp tables in workschema.
    Now suppose you are not giving the workschma at the time of creating physical schema and you have selected "staging different than target" in interface.Here your C$ table will be created in your workschema but I$ table that is used by IKM is going to be created on target schema.So for this again you have change the IKM KM i.e. where to create I$ table (wokrschema) by selecting the corresponding logical schema.
    So its better you give the workschema at the time of creating physical schema.
    Here is the query to give privilege by the sys_dba
    Grant create any table to ODI_TEMP.
    Hope you got it
    (Please mark the answer as correct or helpful and close the thread)
    Thanks

  • Staging area and base are in different schema/tablespace

    Please can someone give me some advantages and disadvantages to keep staging tables and base tables in same schema. For eg we have staging area where we daily truncate staging table, load in the stg table and do transformation process. Once done we will move the staging table data to base tables. Base tables sizes are huge volume like 6 to 8 million rows as there is no purging done.
    I want to suggest to my team that we should keep them in separate schemas as I understand it will be good from I/O point.
    Is there any other reason to keep staging and base tables in separate schema/tablespace.

    Hi,
    Definitely I agree with previous answers. You wrote that staging area, transformation etc.. So I think it's a datawarehouse. Staging and base tables should be stored in different data files also in different harddrives. During the ETL task there can be high load on the Disk subsystem. Storing them in different schemas is another subject which is not related with the disk performance.
    When it comes to blocksize of the datafile, there is a approach like this:
    If the database is used for OLTP systems, it would be good idea to setup smaller size blocks. But when we're talking about a datawarehouse, it's highly recommended to use larger block sizes like 16KB or 32KB. This setting has an advantage related to reading, accessing data blocks.
    Regards,
    Cuneyt

  • ODI Staging Area Confusion

    Hi,
    I am following an ODI Tutorial from http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/odi/odi_11g/odi_project_ff-to-ff/odi_project_flatfile-to-flatfile.htm. Basically it is writing a flat file to another flat file. My question in this case where will be the staging area?
    in the article it is using knowledge modules IKM SQL to File Append and LKM File to SQL; based on the knowledge modules used it means staging area is in Oracle btu Idon't know why it is so.
    Please help me in this regard and I will be very grateful.
    Best Regards,

    Your staging area has to be a technology with transformation capabilities i.e a relational DB. This could be a dedicated database such as SQL Server or Oracle or an in memory database such as SUNOPSIS Memory Engine. As both your source and target are files and don't have these capabilities you will have to choose the staging different from target option in your interface and then select the appropriate relational technology available in your ODI architecture.
    The 2 inerfaces you have mentioned are not technology specific which means that you can choose any relational technolgy with SQL capabilities as your Satging Area, it does not have to be Oracle

  • Building staging  area on DWH

    Usually i use DTS or SSIS (microsoft) to import data to my Staging area.
    Now all my data is in Oracle 11g DB and I would like to replace my
    data transformation step to Streams Replication.
    I would like to ask, do you thing it is a good chose ?
    I'm Newbi on replication subject, i read this article
    http://www.oracle.com/technology/products/dataint/pdf/twp_streams_replication_11gr1.pdf
    But is still didn't helped me a lot how to begin
    Is there any place i can learn more for Newbi ?

    Oracle 11g Streams and Replication
    http://www.oracle.com/pls/db111/portal.portal_db?selected=14&frame=#oracle_streams
    http://www.oracle.com/pls/db111/portal.portal_db?selected=14&frame=#advanced_replication

  • Oracle 8.1.5 Invalid Staging Area Error

    I get an error message: Invalid staging area. There is no top level components for Windows NT available for installing
    from this staging area. I installed this version successfully running win98, but i get this error with XP. I tried renaming the symcjit.dll
    as suggested in some of the earlier posting with no success. Any help is appreciated.
    Thanks,
    Will

    copy disk to hardrive in a temporary directory. go to properties for setup.exe (right click on file used to install oracle). Select the compatiblity tab. make sure the check box "run this program in compatibility mode" is checked. In combo box select windows 98.

  • Lookup in staging area

    We have a column on the source table which contain a person's country information. A single country name could be represented in different formats (for e.g. United States of America, USA, U.S.A). The target database has a table which stores the equivalence mapping for all these different possibilities (i.e. United States of America, USA, U.S.A are mapped to "USA")
    We plan to have a design like this
    Step 1. Load the country column to the staging area
    Step 2. In the staging area, lookup the equivalences table, identify the country name and populate it on the staging area
    Step 3. Transfer the data into target table.
    The point where we have discussion is the Step-2. What would be an efficient way of resolving the equivalences and how we proceed in this step.
    Any thoughts/suggestions on what would be a good or bad design here ?

    Hi,
    Let me try to help...
    If I understood right you can achieve that in just one interface by check the join between the column "country" and the table where the all possible match are at staging area, it means:
    1) create an interface
    2) drag and drop all souce tables plus the table where are the countries are, from target.
    3) drag and drop the "final" target table at target position
    4) create a join between the "target countries tables" and "person table" by "country" column and check it to be executed at Staging Area.
    Does it make any sense to you?
    Cezar

  • All my PDF forms are automatically transformed in Word forms,how can I change that and only choose the PDF I want to transform

    All my PDF forms are automatically transformed in word forms,how can I change that and only decide which PDF to be transformed in word

    Hi Ulfsch,
    Thank you for posting on the Adobe Forums, kindly try the step mentioned below.
    1) Right click on any of the PDF file and choose the option open with
    2) Select choose default program
    3) Select Adobe Reader/Acrobat from the list
    5) Check the box "always open this type of file with selected program" (not the exact same wording)
    6) Click OK
    If you are on Mac, CTRL+Click on the file.
    Go to Get info
    Go to Open with
    Change it to Adobe Reader/Acrobat and click change all.
    Thanks,
    Vikrantt Singh

  • How to print staging areas and doors on the pick doc. ?

    Dear experts!
    Thank you for your attention!
    In our system, we use Lean WM.
    We know staging areas and doors can aslo be printed on the pick doc.  How to do that?
    Best regard!
    Tangdark

    I'm not a WM expert, but am pretty sure nothing special is required for that. If the fields, where this information is stored, are used by the output processing program and the data is passed to the form and then printed by the form, there should be no problem. If data is there, but it's not printed, then you might need to adjust the output program and/or form.
    You might want at least to try it first and use Search whenever possible, then come back with more specific questions, if necessary.

  • Staging area creation problem

    Hi,
    We have got media DVDs for R12.1.1 for Linux X86 from Oracle. We copied in the server one by one using cp command.
    Ans now when we are starting staging creation using these copied DVD's from server, it's giving
    Invalid Disk is inserted.
    Please insert DVD labeled Oracle Applications Rapid Install - of Rdbms Disk 1
    We removed STARTCD part initially which was staged properly using OS rm command.but when it was asking for second DVD it failed due to unavailability of 2nd DVD path
    We copied all DVD's into /do1/R1211dump folder
    Please advice on how to create staging area
    Regards,

    Hi,
    Yes.
    Also this is being executed by root user .
    And when using oracle user, even though DVD dump folder is owned by oracle user
    we are getting following error:
    cannot unlink file for /tmp/staging/adautostg.pl: Permission denied at adautostg.pl line 365
    cannot restore permissions to 0100777 for /tmp/staging/adautostg.pl: Permission denied at adautostg.pl line 365
    cannot unlink file for /tmp/staging/RapidWiz.ini: Permission denied at adautostg.pl line 365
    cannot restore permissions to 0100744 for /tmp/staging/RapidWiz.ini: Permission denied at adautostg.pl line 365
    cannot unlink file for /tmp/staging/File/NCopy.pm: Permission denied at adautostg.pl line 365
    cannot restore permissions to 0100744 for /tmp/staging/File/NCopy.pm: Permission denied at adautostg.pl line 365
    cannot remove directory for /tmp/staging/File: Permission denied at adautostg.pl line 365
    cannot remove directory for /tmp/staging: Operation not permitted at adautostg.pl line 365
    Failed to copy RapidWiz.ini to /tmp/staging
    The log file is located at /tmp/10311250/adautostaging.log
    and
    $ cd /tmp/staging/
    $ ls -l
    total 100
    -rwxrwxrwx 1 root root 71841 Oct 31 12:50 adautostg.pl
    drwxr-xr-x 2 root root 4096 Oct 31 12:26 File
    -rwxr--r-- 1 root root 24329 Oct 31 12:26 RapidWiz.ini
    $
    Not sure how to proceed now..

  • Staging area for Management Agent files...?

    Sorry for another question so quickly - I am reading the documentation first!
    Management agents upload files to the OMS server, and these are parsed and loaded as data into the MR database, right?
    Where is the staging area for these files (upload from all the management agents)? Is it possible to change it? From what I can gather it's
    https://oms_server:1159/em/upload - but where is this in the real world? I've searched the OMS $ORACLE_HOME, but can't find any uploaded files
    The reason for these questions is because we have a requirement to load these files to a shared area so that both our OMSes can access them - has anyone come across a configuration like this before? Does it work?
    Many thanks (again!)
    D

    The upload area on the OMS would normally be <OMS_HOME>/sysman/recv.
    You still might find no data files over here, as they may be uploaded to the OMS on that moment.
    Yes, making this directory a shared location ,will work when running multiple OMS.
    Regards
    Rob

  • Staging area needs Oracle universal Installer version 10.2.0.0.0 or later

    Hi,
    I am trying to install OEM agents thru run installer. but it says "staging area needs Oracle universal Installer version 10.2.0.0.0 or later", we are on we are on 10.1.0.5.0. How to download 10.2.0.0.0 or later.
    Regards

    # From Metalink click on "Patches & Updates&rdquo; tab
    # Select "Advanced Search&rdquo;
    # Enter "Universal Installer&rdquo; under Product or Product Family
    # Select "Oracle 10.2.0.0.0&rdquo; release (Or higher)
    # Select your Platform
    # Change "Patch Type&rdquo; to Any
    # Change Classification to Any
    # Click on Go

  • Staging area for inserts only?

    If our DW will only over contain inserts, do we require a staging area?

    Hi DarrenOD,
    I personally like to put data in staging area before loading it to the Data Warehouse.It will give flexibility of debugging and very much needed when you have restricted time slot for data extraction from OLTP.
    Following URL explains "Why do we need Staging Database"
    http://www.dwbiconcepts.com/etl/27-basic-etl-concepts/137-why-do-we-need-staging-area-during-etl-load.html
    Prathy
    Prathy K

  • Staging area error and universal installer does not run

    Everything was ok. I installed Oracle 9i r 2, AS 9.0.2, iFS 9.0.2.
    Then I uninstall AS and iFS. But AS folder remains there (with inventory,contentsXML.. folder inside), even if I deleted it, when I tried to install AS again I get the error staging area error, and the folder appears again.
    Another thing is that in oracle universal installer the installed and deinstall products options do not do anything. I only can select the install option
    Did I do something wrong during the AS uninstall?
    What Can I do now?
    Thanks for your help,

    Try looking here:
    http://www.thomasfly.com/Oranux/#Step-by-step
    follow these steps/configs and the install will go quite nice!

Maybe you are looking for

  • How do I make one i-movie out of 10 different i-movies?

    I have 10 small one minute long i-movies. I want to make them into one single i-movie with 10 chapters. How can I do that? When I try to import (or drag in) my other i-movies I get this error: +The file could not be imported: The file "xxx" can't be

  • Unable to connect to MySQL Database using MX

    I am unable to connect to MySQL database hosted on a remote server. I get the following error message: "HTTP Error Code 404 File Not Found. Here are some possible reason for the problem: 1) There is no testing server running on the server machine. 2)

  • How do I remove STARTXXL when opening Firefox?

    When I open Firefox 5 I am greeted by a page inviting me to 'customise' my start-up. I live in the UK and this product is very much aimed at the US market. i just don't want it but re-installing firefox has no effect.

  • How to create hierarchy map?

    Can somebody help me? Just some suggestions can be helpful. Thanks. Oliver.

  • #include "Types.r" where can i find Types.r

    Its appears I have a missing dependency - just wondering where I might be able to find it looking through the old SDK cant seem to locate it...