ETL process

hi all,
I am new to OWB10g. I need to know what is exact ETL process to follow in OWB. And how to do the incremental loading. please tell me the Steps to follow??

If you are referring to incremental in terms of time/date increments, you can accomplish this in a mapping as follows:
1. SOURCE TABLE --> FILTER (filters out data so only the last 5 days, or last 1 week, or last 2 weeks of data will be accepted).
2. FILTER --> TARGET TABLE 1(This can be a simple load type of truncate/insert)
3. TARGET TABLE 1 --> Historical TARGET TABLE (This can be a load type of 'upsert' known as a merge or insert/update.)
What happens here is a source table has only the last few days or weeks (you specify it) of data extracted. It then truncates the target table, and loads only the data that the filter extracted. This will be a small subset of the original large source data. Finally, you can merge this smaller subset into the historical table, which will merge based on certain matching criteria. This Historical table will never truncate, therefore it will hold all history. The initial TARGET TABLE 1 will only hold a small subset of data, and can only be queried when you need to look at only the most recent extraction quickly.
-Greg

Similar Messages

  • Unique Index Error while running the ETL process

    Hi,
    I have Installed Oracle BI Applications 7.9.4 and Informatica PowerCenter 7.1.4. I have done all the configuration steps as specified in the Oracle BI Applications Installation and Configuration Guide. While running the ETL process from DAC for Execution Plan 'Human Resources Oracle 11.5.10' some tasks going to status Failed.
    When I checked the log files for these tasks, I found the following error
    ANOMALY INFO::: Error while executing : CREATE INDEX:W_PAYROLL_F_ASSG_TMP:W_PRL_F_ASG_TMP_U1
    MESSAGE:::java.lang.Exception: Error while execution : CREATE UNIQUE INDEX
    W_PRL_F_ASG_TMP_U1
    ON
    W_PAYROLL_F_ASSG_TMP
    INTEGRATION_ID ASC
    ,DATASOURCE_NUM_ID ASC
    ,EFFECTIVE_FROM_DT ASC
    NOLOGGING PARALLEL
    with error java.sql.SQLException: ORA-12801: error signaled in parallel query server P000
    ORA-01452: cannot CREATE UNIQUE INDEX; duplicate keys found
    EXCEPTION CLASS::: java.lang.Exception
    I found some duplicate rows in the table W_PAYROLL_F_ASSG_TMP with the combination of the columns on which it is trying to create INDEX. Can anyone give me information for the following.
    1. Why it is trying to create the unique index on the combination of columns which may not be unique.
    2. Is it a problem with the data in the source database (means becoz of duplicate rows in the source system).
    How we need to fix this error. Do we need to delete the duplicate rows from the table in the data warehouse manually and re-run the ETL process or is there any other way to fix the problem.

    This query will identify the duplicate in the Warehouse table preventing the Index from being built:
    select count(*), integration_id, src_eff_from_dt from w_employee_ds group by integration_id, src_eff_from_dt having count(*)>1;
    To get the ETL to finish issue this delete to the W_EMPLOYEE_DS table:
    delete from w_employee_ds where integration_id = '2' and src_eff_from_dt ='04-JAN-91';
    To fix it so this does not happen again on another load you need to find the record in the Vision DB, it is in the PER_ALL_PEOPLE_F table. I have a Vision source and this worked:
    select rowid, person_id , LAST_NAME FROM PER_ALL_PEOPLE_F
    where EFFECTIVE_START_DATE = '04-JAN-91';
    ROWID PERSON_ID
    LAST_NAME
    AAAWXJAAMAAAwl/AAL 6272
    Kang
    AAAWXJAAMAAAwmAAAI 6272
    Kang
    AAAWXJAAMAAAwmAAA4 6307
    Lee
    delete from PER_ALL_PEOPLE_F
    where ROWID = 'AAAWXJAAMAAAwl/AAL';

  • SQL server agent jobs throws random errors while the ETL process works fine.

    Hi,
    I have this problem with SQL agent jobs.
    We went from sql2008 to sql2012 and migrated SSIS without problems. The ETL process runs fine and the OLAP cubes are processed.
    I have a job which calls the master execution dtsx for a particulair customer. When the ETL load and OLAP is processed it should go to the next customer. The problem i have is that the agent logs some errors for random customers. I tried to do only two clients
    in one job this works then i add the third client and then it fails (log wise) for a customer which did run successfully before when there were only two customers.
    Despite the error message the ETL did run, there were no duplicate keys and OLAP was processed???
    Im very close to pull all my hair, because some combinations like two customers work, and placing these two customers with a third one it then fails again. (again cubes are processed and data is integer yet i keep getting these annoying errors in the log)
    Perhaps someone could help me further. 
    -Miracles are easy, the impossible takes a bit longer-

    Just double-click on the Agent job, then click on Steps property page (on your left), you must be able to see a list of steps with the action "On Failure" which you should examine.
    Arthur My Blog

  • Bi Content in 7.0 - Did SAP Convert all content to use new ETL process?

    Hello.
    I am working with BI 7.0 for the first time.  I have 8 years of BW 3.x experience.  We are starting a new BI project and we are in the process of activating Business Content.  We have BI_CONT level 7 (the latest as of today I believe).  I appears that SAP has not converted over its Business Content to use the new ETL process (Transformations, DTP's, etc).  For example, I am looking at basic areas such as AP, AR, SD, MM, GL, etc and the BI content is still using the 3.x ETL (transfer rules, update rules). Is something not installed right on my system or has SAP not converted it's content to the new ETL yet?
    Thanks in advance for your help.

    Jose,
    Some new content is released using the new DTP.  Most content remains in its delivered 3.x format.  If you want to use DTPs for this content you have to manually create the DTPs after installing the 3.x objects.  If you right-click on an InfoCube, you will see some new options in the context menu, including "Create Transformation...," and "Create Data Transfer Process...."  Context menu for DataSouce now contains a "Migrate" option that will allow you to migrate to the new DTP from 3.x.  Also, other objects such as Transfer Rules and Update Rules contain context menu options for "Create Transformation."
    Hope this helps.

  • Monitor ETL process using SNMP

    Hi,
    I have an ETL process that runs continuously and add lines to a table under Oracle 11g on Linux. I would like to monitor the proper operation of the ETL using an external SNMP manager (eg Zabbix). To do this the algorithm would be like this:
    - List and count lines that have been to the table durig the last n minutes;
    - Update an SNMP counter
    - Repeat every n minutes.
    I was thinking about creating a small Perl subagent for net-snmp that would execute an SQL query using JDBC, but is there a better way to do this? For instance by using an Ora

    I assume you're talking about scripting this? Probably the easiest thing to do would be to use Curl. I've attached an example. You'll need to change a couple values for it to work.
    The output is different than what you'd get if you did a "df" using the service account. You'll have to decide which output you require.

  • Data/database availability between ETL process

    Hi
    I am not sure whether this is right forum to ask this question. But still I am requesting for help.
    We have a DSS database of size 1Tb. The batch process runs between 12:00 hours till 6:00 am. the presentation/reporting schema is of size 400 GB. Through the nightly batch job, most of the tables in the presentation layer get truncated/modified. Due to business nature and requirement, this presentation layer needs to be available 24*7. As the ETL process modify/changes database, hence the system is not available between 12:00 till 6 am.
    The business requirement is that: Before ETL process starts, take a backup of the presentation layer, transfer the application to this backed-up area and then let the ETL process proceed. Once the ETL process finished, move the application to this latest area.
    Based on the size of the database and schema size, does any one , how to take backup/restore the presentation layer in the same database in a efficient way.

    There are a couple of possibilities here. You certainly need two sets of tables, one for loading and the other for presentation. You could use synonyms to control which one is the active reporting set and which is the ETL set, and switch them over at a particular time. Another approach would be to use partition exchange to exchange data and index segments between the the two table sets.
    I would go for the former method myself.

  • FIM 2010 R2 SP1 Reporting ETL Process for SCSM 2012 R2?

    Hi,
    First question: is FIM 2012 R2 SP1 Reporting supported on System Centre 2012 R2 or only System Centre 2012? I have followed the MS FIM Reporting deployment guide, and everything seems to work, except the ETL process (2nd question below)
    Second question: if it is supported, than how do we get the ETL process defined here (http://technet.microsoft.com/en-us/library/jj133844%28v=ws.10%29.aspx)  to work with these versions? The powershell script provided doesn't work on SC 2012 R2.
    Third question: how do we force the whole process so we can view data in the FIM Reports? as at present there is no data in any of the reports even after I manually ran these SCDW jobs: Extract_dw_SCSMServer, transform.common, load.common
    Thank you,
    DW

    Although it could work, if not officially announced as supported - it is not "officially" supported, so you're deploying it on your own risk and MS won't help you if any problem occurs. Please be aware of that.
    Keep trying

  • Document or link about ETL process troubleshooting

    Hello Gurus,
              will you please provide some document or link about ETL process troubleshooting.
    Many thanks,
    Frank Zhang

    Hi,
    Check in the following links you get good idea about ETL.
    http://www.sap-press.de/download/dateien/939/sappress_sap_bw_data_retrieval.pdf
    http://www.thespot4sap.com/articles/SAP_BW_ETL_Fundamentals.asp
    Thanks
    reddy

  • Custom ETL processes creation in OWB

    Hi, we are working in a Oracle Utilities BI implementation project.
    Some of the KPI identified, need a custom development - creation of complete ETL processes:
    - Extractor in CC&B (in COBOL)
    - Workflows in OWB
    - Configuration in BI
    We were able to create 4 custom ETL processes (2 fact and 2 related dimensions) including the COBOL extract program, the OWB entities (Files, Staging tables, Tables, Mappings and Workflows) and the BI tables.
    We already have the data in the BI database and in some cases we are able to configure zones to show the data in BI.
    We are facing some problems when we try to use a custom Fact and a custom Dimension together, for instance:
    Case 1. When we create a zone to show : Number of quotes - Measure 1 : tf=CM_CF_QUOTE.FACT_CNT func=SUM dt=END_DATE_KEY the graph is displayed.
    Case 2. When we create a zone to show : Number of accepted quotes - Measure 1 : tf=CM_CF_QUOTE.FACT_CNT func=SUM dt=END_DATE_KEY the and Fixed Dimensional Filter 1 : tf=CM_CD_QUOTE_TYPE.QUOTE_STATUS_CD oper==(10), the graph is displayed.
    Case 3. When we create a zone to show : Number of ongoing quotes - Measure 1 : tf=CM_CF_QUOTE.FACT_CNT func=SUM dt=END_DATE_KEY the and Fixed Dimensional Filter 1 : tf=CM_CD_QUOTE_TYPE.QUOTE_STATUS_CD oper==(20), the graph is displayed.
    Case 4. When we create a zone to show : Number of ongoing quotes - Measure 1 : tf=CM_CF_QUOTE.FACT_CNT func=SUM dt=END_DATE_KEY the and Fixed Dimensional Filter 1 : tf=CM_CD_QUOTE_TYPE.QUOTE_STATUS_CD oper==(20), the graph is displayed.
    Case 5. But when we create a zone to show : Number of quotes sliced by quote status - Measure 1 : tf=CM_CF_QUOTE.FACT_CNT func=SUM dt=END_DATE_KEY the and Dimensional Filter 1 : tf=CM_CD_QUOTE_TYPE.QUOTE_STATUS_CD, no graph is displayed.
    Case 6. A different problem occurs in the single fact. We try to show the number of processes of a given type. The type of the process is a UDDGEN. When we load the zone to show the graph the following error appears: Measure1 : tf=CM_F_CF_CSW.FACT_CNT func=SUM dt=ACTV_DATE_KEY and Fixed Dimensional Filter 1 : tf=CM_F_CF_CSW.UDDGEN1 oper==(ENTRADA)
    An error is displayed: No join defined between fact table CM_F_CF_CSW and dimension table CM_F_CF_CSW. The dimension tabelentered in the zone parameter could be invalid for this fact table.
    Does anyone had the same problem??????
    Edited by: user11256032 on 10/Jun/2009 11:51
    Edited by: user11256032 on 10/Jun/2009 11:54

    Hi user11256032, I just stumbled upon this by accident. The reason no-one has answered yet, is because it is in the wrong forum. (I can understand that you thought it belonged here.) Please post the question to the Oracle Utilities forum, which is here Utilities If that link doesn't work, go to Forum Home, then choose Industries, then Utilities. You may have to select "More ..." on Industries.
    Actually, I suspect there was an SR created for these, so your question may have been answered already.
    If you don't mind me asking, which customer is this for?
    Jeremy

  • ETL processing Performance and best practices

    I have been tasked with enhancing an existing ETL process. The process includes dumping data from a flat file to staging tables and process records from the initial tables to the permanent table. The first step, extracting data from flat file to staging
    tables is done by Biztalk, no problems here. The second part, processing records from staging tables and updating/inserting permanent tables is done in .Net. I find this process inefficient and prone to deadlocks because the code loads the data from the initial
    tables(using stored procs) and loops through each record in .net and makes several subsequent calls to stored procedures to process data and then updates the record. I see a variety of problems here, the process is very chatty with the database which is a
    big red flag. I need some opinions from ETL experts, so that I can convince my co-workers that this is not the best solution.
    Anonymous

    I'm not going to call myself an ETL expert, but you are right on the money that this is not an efficient way to work with the data. Indeed very chatty. Once you have the data in SQL Server - keep it there. (Well, if you are interacting with other data
    source, it's a different game.)
    Erland Sommarskog, SQL Server MVP, [email protected]

  • ETL process steps in OWB

    Hi friends,
    <li> can i know the etl steps involved in owb. if so can you provide the etl steps now.
    <li> And also there is any diagram for owb, Covering owb etl process from source to target(including BI components) too. (like available for infomatica in a diagrammatical manner) like how it is taking data from source and how it is performing etl process and how it is transforming it to data warehouse and how BI components utilizing it.
    Thanks
    Regards,
    Saro.
    Your Answers will be marked.

    Hi Saro,
    OWB does not enforce some specific steps on you. You define your datawarehouse architecure according to your needs. OWB perfectly support you to
    + extract the data from the source systems or files
    + load into the database (staging)
    + process the data within the database, e.g. loading it from the staging area into the core (normalized or star schema)
    + load it into OLAP cubes
    + design and monitor the overall process using process flows
    Regards,
    Carsten.

  • ETL Process - snapshot too old

    Hi friends:
    I would like to get some comments from you about the following situation:
    In an ETL process, usually we have gotten the Oracle error "Snapshot too old". We are doing various modifications in our ETL process (developed via PL/SQL or OWB), however, we have not had success. Could you give me some advise in order to avoid this problem?
    Thanks in advance.

    It's sth with your Rollback segment settings.Pls set the MINEXTENTS more larger and set OPTIMAL,if the sutiation still occurs,get the rollback segment offline first and then re-online the rollback segment.

  • ETL process but no use RSA1

    Hi All,
    How to ETL process but no use RSA1.
    Thanks...
    Tom Pemalang

    Hi Eddy,
       Is the file on the presentation serever? If so get the file loaded to the application server and gat a process chain created for the data load ... then you need to give authorization for RSPC and get the data loaded via a Process Chain ...
      Once the file is on the application server .. you even have the option of setting up a background job..
    Hope this helps.
    bestr egards,
    Kazmi

  • Assign value to one column when etl process is run

    hi,
    i have a table with column etl_id. my requirement is when i ran one etl process this column is updated by 1. when i run etl process second time the value is 2. if i run n times its value should be n.
    How do i implement this logic?
    please find my table.
    table name is WHOUSE_DS
    CREATE TABLE OLAPS.WHOUSE_DS
    (     ROW_WID NUMBER(10,0),
         INTEGRATION_ID NUMBER(10,0),
         DATASOURCE_NUM_ID NUMBER(10,0),
         ETL_ID NUMBER(10,0),
         WAREHOUSE_CODE VARCHAR2(30),
         WAREHOUSE_LOC_NAME VARCHAR2(30),
         WAREHOUSE_LOC_ADDRESS VARCHAR2(30),
         WAREHOUSE_LOC_CITY VARCHAR2(30),
         WAREHOUSE_LOC_COUNTY VARCHAR2(30),
         WAREHOUSE_LOC_STATE VARCHAR2(30),
         WAREHOUSE_LOC_COUNTRY VARCHAR2(30),
         ZIP_POSTAL VARCHAR2(30)
    The bold column should be updated by 1 each time when i run the etl process.
    Regards,
    Jyothy

    Hi Jyothy,
    1.Define a Sequence Generator
    2. Define a fuction ETL_ID_FN.
    The function definition should be as follows.
    Get the "Next Val" of Sequence generator and pass that value as a return value.
    3. Import the funciton into OWB.
    4. Open your mapping.
    5. Use Constant Operator
    6. Define a variable
    7. In the defintion of varaible, call the imported function.
    8. Validate it.
    9. Assign it to the field whichever you want to.
    Regards,
    Gowtham Sen.

  • In memory Lookups in ETL processing

    Hi All,
    I am writing a function in PL/SQL to convert unit of measurement like:
    convert(from_unit, to_unit) returns float
    I have couple of lookup tables which will be referenced in the function to convert units table_1 has 100 rows and and table_2 has 1000 rows.
    This function will be called during ETL processing for about half a million rows which can have significant I/O operations. I am considering two options
    1) PL/SQL table - pre-load both lookups in PL/SQL tables and use in function.
    2) CACH lookup table.
    However, I would like to have some opinion on my approach, please could you suggest which one will be better approach or if there are any other method to achieve good performance?
    Thanks
    techTD

    910150 wrote:
    Ok, I thought initially to write function with using couple of methods (PL/SQL table etc). There is no such thing as a PL/SQL "+table+". The correct term is array (and there are a couple of favours that can be used).
    Yes, these reside in memory. But in the "wrong" memory ito of scalability. And also in the "wrong" memory for optimal access and use by the SQL engine.
    Which makes this a problematic approach for SQL use. If PL/SQL functions are to be used, consider defining these as deterministic in order to reduce the number of context switches to the PL/SQL engine for executing these PL/SQL functions.
    Better still is doing that PL/SQL function code, inside the SQL statement.
    However, the logic inside is complex and would take some good amount of effort.SQL is vastly underestimated ito power and flexibility.
    I thought will check in the forum if some one has already done similar experiment.No need for "experimenting". The facts are simple. Calling PL/SQL functions from the SQL engine requires a context switch from the SQL engine to the PL/SQL engine.
    Context switching in any form (from inside a CPU between CPU rings, to between SQL and PL/SQL in an application s/w layer) is an overhead that needs to be carefully considered and implement so as to reduce the overhead to a minimum.
    If you have such a context switch from the SQL engine to the PL/SQL engine in a SQL projection, resulting in switch per row, for 1/2 a million rows - that overhead will add significant run-time to the SQL process.

Maybe you are looking for

  • SUBSTITUTION FOR DOCUMENT TYPE

    Dear Friends, I had an requirement of changing the Document type while posting the document. The requirement is as follows we have 2 users say user-1,user-2 both are posting a document with doc type SA (number range for doc type SA is 1001 - 2000). G

  • How do I make Pages the default program?

    Hi, This may be quite basic, but I can't seem figure it out. I've just migrated from a PC to a Mac and I have a lot of .doc's. I want to make Pages the default program to open all of my MS Word documents; however, when I select >open with>other>iWork

  • Switching between page layout and word processing

    I can't seem to switch between page layout and word processing view.  I've read all the posts on this and still can't figure out why I can't just type a document into Pages.  The only option I have in page layout view is to type into text boxes, whic

  • Lion Recovery Disk erased mounted volume

    Has anyone had this problem: I created a Lion Recovery Disk and it erased mounted volume? Can the data be recovered?

  • UWL custom attributes are not retreived

    Hi all, We are trying to add some custom attributes to our UWL. We are able to see the new columns but they remain null. It seems that the values are not retrieved from the backend.   We have tried defining custom attributes from workflow container,