Migration of huge data from norm tables to denorm tables for performance

We are planning to move the NORM tables to DENORM tables in Oracle DB for a client for performance issue. Any Idea on the design/approach we can use to migrate this HUGE data (2 billion records/ 5TB of data) in a window of 5 to 10 hrs (Or minimum than that also).
We have developed SQL that is one single query which contains multiple instance of same table and lots of join. Will that be helpful.

Jonathan Lewis wrote:
Lother wrote:
We are planning to move the NORM tables to DENORM tables in Oracle DB for a client for performance issue. Any Idea on the design/approach we can use to migrate this HUGE data (2 billion records/ 5TB of data) in a window of 5 to 10 hrs (Or minimum than that also).
We have developed SQL that is one single query which contains multiple instance of same table and lots of join. Will that be helpful.Unfortunately, the fact that you have to ask these questions of the forum tells us that you don't have the skill to determine whether or not the exercise is needed at all. How have you proved that denormalisation is necessary (or even sufficient) to solve the client's performance problems if you have no idea about how to develop a mechanism to restructure the data efficiently ?
Regards
Jonathan LewisYour brutal honesty is certainly correct. Another thing that is concerning to me is that it's possible that he's planning on denormalizing tables that are normalized for a reason. What good is a system that responds like a data warehouse but has questionable data integrity? I didn't even know where to begin with asking that question though.

Similar Messages

  • Data from 2 tables for jdbc sender adapter

    how to pickup data from 2 tables at a time when using a jdbc sender adapter?

    select <fields> from table1 where <condition>
    union
    select <fields> fromt table2 where <condition>
    also u can combine this with joins as pointed

  • Delete Variance data from COSB table for version 1

    Hi,
    We are trying to change the Valuation of version 1 from Legal to Profit center in OKEQ.
    While doing so, we are trying to remove variance indicator due to error message KT337.
    When trying to uncheck variance indicator for this version there is error message KV853.
    SAP per SAP note 337183, if there are any dependent entries in COSB table (CO Object: Total Variances/Results Analyses) we get this message.
    Now the question is how can i delete the variance data with version 1 from the table COSB.
    Message no. KT337 :
    Valuation variance from version "000"
    Diagnosis
    You are maintaining version 1 in controlling area 1000. The actual indicator is not active and at least one of the two indicators WIP and Variance are active. In this case, the valuation in version 1 must match the valuation of version '000'. This is not the case: 2 <> 0.
    Procedure
    Activating the indicators WIP and Variance with the actual indicator being inactive at the same time requires valuation 0.
    Message no. KV853
    Variance calculation cannot be reset - there are dependent entries
    Regards
    Raghu
    Edited by: Raghu Ram Thatavarthy on Nov 16, 2011 3:35 PM
    Edited by: Raghu Ram Thatavarthy on Nov 16, 2011 3:36 PM

    You may need to contact the SAP SLO (System landscape optimization) team

  • How to extract data from monthly tables for annual balance chart?

    Hello Again!
    I am wanting to decide whether to enter all of my transactions into one table for the whole year or separate tables for each month. If I was to use one table for the whole year it would probably have in the region of 1000 entries, and as I understand it, Numbers doesn't work so well with such large tables. Also, as I will be mostly using Numbers on my iPad, where it runs slower, this may well slow things down considerably. Another reason for using Monthly Tables is that it would be easier to search.
    Having said this, I am using the following method to construct balance charts for my accounts which seems to depend on having just one table for all transactions for the year. Here is a sample of the transactions table:
    which updates the balance for each account after each new transaction is entered.
    Then I use the following table for creating the balance charts:
    It uses the following formula to return the balance at the end of each week for each account (this is the formula for Account 1) which then is displayed in the chart.
    =LOOKUP(A2,$Week Number,All Transactions :: Account 1)
    If I was to use the Monthly Tables method it is not obvious to me how I would create the Weekly Balance Table, given that I would need to draw data from 12 different tables, and that some of these Tables would contain the same Week Number (for weeks that overlap months).
    I would be grateful for any advice on achieving the result I am looking for by the most efficient method.
    Thanks
    Nick
    P.S. Hopefully this is the last question for a while!

    I wasn't urging you to keep an annual table but to spit it on an easier to rule basis.
    Split it in chuncks of exactly 35 days starting from the first one of the 'year' in operation
    Doing that everything will be easier
    the index of a day will be calculated by
    =DATEDIF(B,D,"D")+1
    The index of the 'custom_week' will be calculated by :
    =1INT((DATEDIF(B,D,"D")1)/7)
    So, it would be easy to gather datas from a given 'custom_week'.
    Yvan KOENIG (VALLAURIS, France) jeudi 5 août 2010 10:22:25

  • Procedure to Purge data from WF tables for Item Type POERROR

    Hi,
    I have executed concurrent program "Purge Obsolete Workflow Runtime Data" to purge old data for Item Type "REQAPPRV". After execution of the concurrent program i re-query to WF_ITEMS table to get the count of the records for "REQAPPRV" but all records did not get purged as "REQAPPRV" is present as PARENT_ITEM_KEY for Item Type "POERROR" in WF_ITEMS table.
    Now to purge records of "REQAPPRV" i need to first Purge records of "POERROR". Now the problem is END_DATE of those POERROR records is NULL at WF_ITEMS table that is why concurrent program "Purge Obsolete Workflow Runtime Data" does not able to purge them. How can i resolve this issue?? Please guide..
    Regards,
    Priyanka

    Hi,
    I have already gone thorugh MOS doc, cross checked the queries provided in document and found that status of PARENT_ITEM_TYPE [REQAPPRV] is "COMPLETE" and the status of Chilld ITEM_TYPE [POERROR] is "ACTIVE". I want to close or Purge the child ITEM_TYPE. Can you guide how can we achive that ??
    Regards,
    Priyanka

  • Performance Issue - Fetching latest date from a507 table

    Hi All,
    I am fetching data from A507 table for material and batch combination. I want to fetch the latest record based on the value of field DATBI. I have written the code as follows. But in the select query its taking more time. I dont want to write any condition in where claue for DATBI field because I have already tried with that option.
    SELECT kschl
               matnr
               charg
               datbi
               knumh
        FROM a507
        INTO TABLE it_a507
        FOR ALL ENTRIES IN lit_mch1
        WHERE kschl = 'ZMRP'
        AND   matnr = lit_mch1-matnr
        AND   charg = lit_mch1-charg.
    SORT it_a507 BY kschl matnr charg datbi DESCENDING.
      DELETE ADJACENT DUPLICATES FROM it_a507 COMPARING kschl matnr charg.

    Hi,
    These kind of tables will be storing large volumes of data. Thus while making a select on it, its important to use as many primary key fields as possible in the where condition. Here you can try mentioning KAPPL since its specific to a requirement. If its for purchasing use 'M' and try.
    if not lit_mch1[] is initial.
    SELECT kschl
    matnr
    charg
    datbi
    knumh
    FROM a507
    INTO TABLE it_a507
    FOR ALL ENTRIES IN lit_mch1
    WHERE kappl = 'M'
    AND kschl = 'ZMRP'
    AND matnr = lit_mch1-matnr
    AND charg = lit_mch1-charg.
    endif.
    SORT it_a507 BY kschl matnr charg datbi DESCENDING.
    DELETE ADJACENT DUPLICATES FROM it_a507 COMPARING kschl matnr charg.
    This should considerably increase the performance
    Regards,
    Vik

  • How to retrieve data from plsql table in BI publisher Data template

    Hi All,
    I have created a data template for XML publisher report. In data template i m getting data from plsql table. for that i have created one package with pipelined function. I am able to run that sql from sql developer .But if i run the concurrent program then i got error like "java.sql.SQLSyntaxErrorException: ORA-00904: "XXXXX": invalid identifier".
    I have used the same parameters in Data template and concurrent program....
    please clarify me what needs to be done....
    thanks in advance....
    Regards,
    Doss

    Hi Alex ,
    i am using pipelined function and get the data from cursor and load it into plsql table (nested table). and i use the below in my data template to fetch the data:
    <sqlStatement name="Q1">
    <![CDATA[select * from  table(PO_SPEND_RPT_PKG.generate_report(P_ORG_ID,P_SOB_ID,P_ORG_NAME,P_PERIOD_NAME,P_CLOSE_STATUS,P_E_PCARD_NEED,P_REPORT_TYPE))]]>
    </sqlStatement>
    if i run the above in sql developer i can get the result....from apps if i run i got the error "java.sql.SQLSyntaxErrorException: ORA-00904: "P_ORG_ID": invalid identifier"
    Edited by: kalidoss on Sep 14, 2012 4:32 AM

  • Help needed in  extracting data from PCD tables

    Hi Friends
    I Have a requiremnt for creating custom portal activity report ,even though
    we have  standard report, the extraced data will be used to create bw reports later.
    my part is to find a way to extract the data from PCD tables for creating
    custom portal activity reports
    i have selected the following  tables for the data extraction
    WCR_USERSTAT,WCR_WEBCONTENTSTAT,WCR_USERFIRSTLOGON,
    WCR_USERPAGEUSAGE.
    My questions are
    1.Did i select the Exact PCD tables?
    2.Can i use UME api  for  accessing the data from those tables?
    3.can i use  the data extracted  from PCD tables in JSPdynpage  or
    webdynpro apps?
    4.can i Querry  the  PCD tables from  JSPDynpage or Webdynpro
    Please help me in finding a solution for this
    Thanks
    Ashok Battula

    Hi daniel
    Can u tell  me weather i can develop the following  custom reports from those WCR tables
         Report Type
    1     Logins
          - Unique Count
          - Total Count
          - Most Active Users (by Partner Name)
          - Most Active Users (by Contact Name)
          - Entry Point (by page name)
          - Session Time
          - Hourly Traffic Analysis
    2     Login Failures
          - Total Count
          - Count by error message
          - Credentials Entered (by user name and password)
    3     Content Views (by File Name)
          - Unique Count
          - Total Count
          - Most requested Files
          - Most requested Pages
          - File Not Found
    4     Downloads (by File Name)
          - Unique Count
          - Total Count
          - Most requested Files
          - File Not Found
    5     Portal Administration
          - Site Content (by file name)
          - Site Content (by page name)
          - Latest Content (by file name)
          - Expired Content (by file name)
          - Subscriptions Count (by file name)
    6     Login History (by Partner, Contact Name)
          - No Login
          - First Login
          - Duration between registration and first login
          - Most Recent Login
          - Average Number of Logins
    plz  help me in find ing a way
    thanks
    ashok

  • Retrive data from HR tables

    Hi Experts,
    How to retrive data from HR tables for custom screen & store in custom table? I need Employee number, employee name, department, their location & mail id. When employee number is given, all other details must be fetched. Can you give a solution for this problem?
    Thanks,
    Kavitha

    Hi Kavitha,
    in HR you could get data from different tables based on infotypes. lets say if any data need to get from infotype 0001 (Org assignment), add "PA" before the infotype number so it would be PA0001.
    you could get the Employee no, name, department, location(i guess personal area or personal subarea) from PA0001 table and email id from PA105 infotype.
    Regards
    Raju

  • To extract Data from Pool Table Data Sources

    hi
    I want to extract data from Pool table, for that i want to create infoset for that pool table. can anyone please let me know the
    procedure to create info set on pool tables.
    Regards
    Atul
    Moderator message: please (re)search yourself first.
    Edited by: Thomas Zloch on Nov 8, 2010 12:54 PM

    Hi Atul
    You have a couple of options here:
    1) Create Infoset SQ02 on those tables and RSO2 - create generic ds
    2) Create functional Module and create generic ds using FM
    Replicate DS to BW and Build objects and map them in transformations and create dtp and IP.
    Refer to this [link|How to extract data from a pool table?; for more details.
    Regards
    Harsh

  • Migrating data from LookUp Table to Main Table without taking extract

    Dear Experts,
    Is there any way if we want to migrate full data from LookUp Table to Main Table in the same repository without taking the extract from MDM DM and not importing it using Import Manager?
    Thanks
    Gaurav Kumar Sharma

    Hi Gaurav,
    Yes there are ways,like you can create port for syndication and use MDIS to import it back but Extraction from DM would be more convenient though.You can also use APIs to syndicate the lookup data and import it back in MDM.
    Can you please detail what you want to achieve?
    Thanks,
    Ravi

  • Reading data from a table which can have millions of record

    Hi,
    I need to write a query in my report that can be able to fetch records from a z table that have arount 20 lac (= 2 million) records. As far i knoew we can not handle such a huge amount of data in our internal tables. because the size of internal table is fixed and that can be increased only by basis persons.
    So can anybody tell me the approach that i should follow such that i can split the data from this table.
    Or any other approach is also welcome.
    Please reply ASAP when you have the time.
    Thanks And Regards,
    Mayank
    Edited by: Thomas Zloch on Mar 6, 2010 9:25 PM

    Hi Mayank,
    reduce the data selected to the fields you really need, avoid select *. See online help on SELECT, addition PACKAGE SIZE.
    select field1 field2 field3
      into corresponding fields of table lt_table package size nnn WHERE ...
    * process  lt_table
    endselect.
    Regards,
    Clemens
    Edited by: Rob Burbank on Mar 9, 2010 12:29 PM

  • Creating XML file using data from database table

    I have to create an xml file using the data from multiple table. The problem That i am facing is the data is huge it is in millions so I was wondering that is there any efective way of creating such an xml file.
    It would be great if you can suggest some approach to achieve my requirement.
    Thanks,
    -Vinod

    An example from the forum: Re: How to generate xml file from database table
    Edited by: Marco Gralike on Oct 18, 2012 9:41 PM

  • How to export some data from the tables of an owner with integrity?

    Hi to all,
    How to export some data from the tables of an owner with integrity?
    I want to bring some data from all tables in a single owner of the production database for development environment.
    My initial requirements are: seeking information on company code (emp), contract status (status) and / or effective date of contract settlement (dt_liq_efetiva) - a small amount of data to developers.
    These three fields are present in the main system table (the table of contracts). Then I thought about ...
    - create a temporary table from the query results table to contract;
    - and then use this temporary table as a reference to fetch the data in other tables of the owner while maintaining integrity. But how? I have not found the answer, because: what to do when not there is the possibility of a join between the contract and any other table?
    I am considering the possibility of consulting the names of tables, foreign keys and columns above, and create dynamic SQL. Conceptually, something like:
    select r.constraint_name "FK name",
    r.table_name "FK table",
    r.column_name "FK column",
    up.constraint_name "Referencing name",
    up.table_name "Referencing table",
    up.column_name "Referencing column"
    from all_cons_columns up
    join all_cons_columns r
    using (owner, position), (select r.owner,
    r.constraint_name fk,
    r.table_name table_fk,
    r.r_constraint_name r,
    up.table_name table_r
    from all_constraints up, all_constraints r
    where r.r_owner = up.owner
    and r.r_constraint_name = up.constraint_name
    and up.constraint_type in ('P', 'U')
    and r.constraint_type = 'R'
    and r.owner = 'OWNERNAME') aux
    where r.constraint_name = aux.fk
    and r.table_name = aux.table_fk
    and up.constraint_name = aux.r
    and up.table_name = aux.table_r;
    -- + Dynamic SQL
    If anyone has any suggestions and / or reuse code to me thank you very much!
    After resolving this standoff intend to mount the inserts in utl_file by a table and create another program to read and play in the development environment.
    Thinking...
    Let's Share!
    My thanks in advance,
    Philips

    Thanks, Peter.
    Well, I am working with release 9.2.0.8.0. But the planning is migrate to 10g this year. So my questions are:
    With Data Pump can export data just from tables owned for me (SCHEMAS = MYOWNER) parameterizing the volume of data (SAMPLE) and filters to table (QUERY), right? But parameterizing a contract table QUERY = "WHERE status NOT IN (2,6) ORDER BY contract ":
    1º- the Data Pump automatically searches for related data in other tables in the owner? ex. parcel table has X records related (fk) with Y contracts not in (2,6): X * SAMPLE records will be randomly exported?
    2º- for the tables without relation (fk) and which are within the owner (MYOWNER) the data is exported only based on the parameter SAMPLE?
    Once again, thank you,
    Philips
    Reading Oracle Docs...

  • Method for Downloading Huge Data from SAP system

    Hi All,
    we need to download the huge data from one SAP system  & then, need to migrate into another SAP system.
    is there any better method, except downloading data through SE11/SE16 ? please advice.
    Thanks
    pabi

    I have already done several system mergers, and we usually do not have the need to download data.
    SAP can talk to SAP. with RFC and by using ALE/IDOC communication.
    so it is possible to send e.g. material master with BD10 per IDOC from system A to system B.
    you can define that the IDOC is collected, which means it is saved to a file on the application server of the receiving system.
    you then can use LSMW and read this file with several hundred thousand of IDOCs as source.
    Field mapping is easy if you use IDOC method for import, because then you have a 1:1 field mapping.
    So you need only to focus on the few fields where the values changes from old to new system..

Maybe you are looking for