Migration of huge data from norm tables to denorm tables for performance
We are planning to move the NORM tables to DENORM tables in Oracle DB for a client for performance issue. Any Idea on the design/approach we can use to migrate this HUGE data (2 billion records/ 5TB of data) in a window of 5 to 10 hrs (Or minimum than that also).
We have developed SQL that is one single query which contains multiple instance of same table and lots of join. Will that be helpful.
Jonathan Lewis wrote:
Lother wrote:
We are planning to move the NORM tables to DENORM tables in Oracle DB for a client for performance issue. Any Idea on the design/approach we can use to migrate this HUGE data (2 billion records/ 5TB of data) in a window of 5 to 10 hrs (Or minimum than that also).
We have developed SQL that is one single query which contains multiple instance of same table and lots of join. Will that be helpful.Unfortunately, the fact that you have to ask these questions of the forum tells us that you don't have the skill to determine whether or not the exercise is needed at all. How have you proved that denormalisation is necessary (or even sufficient) to solve the client's performance problems if you have no idea about how to develop a mechanism to restructure the data efficiently ?
Regards
Jonathan LewisYour brutal honesty is certainly correct. Another thing that is concerning to me is that it's possible that he's planning on denormalizing tables that are normalized for a reason. What good is a system that responds like a data warehouse but has questionable data integrity? I didn't even know where to begin with asking that question though.
Similar Messages
-
Data from 2 tables for jdbc sender adapter
how to pickup data from 2 tables at a time when using a jdbc sender adapter?
select <fields> from table1 where <condition>
union
select <fields> fromt table2 where <condition>
also u can combine this with joins as pointed -
Delete Variance data from COSB table for version 1
Hi,
We are trying to change the Valuation of version 1 from Legal to Profit center in OKEQ.
While doing so, we are trying to remove variance indicator due to error message KT337.
When trying to uncheck variance indicator for this version there is error message KV853.
SAP per SAP note 337183, if there are any dependent entries in COSB table (CO Object: Total Variances/Results Analyses) we get this message.
Now the question is how can i delete the variance data with version 1 from the table COSB.
Message no. KT337 :
Valuation variance from version "000"
Diagnosis
You are maintaining version 1 in controlling area 1000. The actual indicator is not active and at least one of the two indicators WIP and Variance are active. In this case, the valuation in version 1 must match the valuation of version '000'. This is not the case: 2 <> 0.
Procedure
Activating the indicators WIP and Variance with the actual indicator being inactive at the same time requires valuation 0.
Message no. KV853
Variance calculation cannot be reset - there are dependent entries
Regards
Raghu
Edited by: Raghu Ram Thatavarthy on Nov 16, 2011 3:35 PM
Edited by: Raghu Ram Thatavarthy on Nov 16, 2011 3:36 PMYou may need to contact the SAP SLO (System landscape optimization) team
-
How to extract data from monthly tables for annual balance chart?
Hello Again!
I am wanting to decide whether to enter all of my transactions into one table for the whole year or separate tables for each month. If I was to use one table for the whole year it would probably have in the region of 1000 entries, and as I understand it, Numbers doesn't work so well with such large tables. Also, as I will be mostly using Numbers on my iPad, where it runs slower, this may well slow things down considerably. Another reason for using Monthly Tables is that it would be easier to search.
Having said this, I am using the following method to construct balance charts for my accounts which seems to depend on having just one table for all transactions for the year. Here is a sample of the transactions table:
which updates the balance for each account after each new transaction is entered.
Then I use the following table for creating the balance charts:
It uses the following formula to return the balance at the end of each week for each account (this is the formula for Account 1) which then is displayed in the chart.
=LOOKUP(A2,$Week Number,All Transactions :: Account 1)
If I was to use the Monthly Tables method it is not obvious to me how I would create the Weekly Balance Table, given that I would need to draw data from 12 different tables, and that some of these Tables would contain the same Week Number (for weeks that overlap months).
I would be grateful for any advice on achieving the result I am looking for by the most efficient method.
Thanks
Nick
P.S. Hopefully this is the last question for a while!I wasn't urging you to keep an annual table but to spit it on an easier to rule basis.
Split it in chuncks of exactly 35 days starting from the first one of the 'year' in operation
Doing that everything will be easier
the index of a day will be calculated by
=DATEDIF(B,D,"D")+1
The index of the 'custom_week' will be calculated by :
=1INT((DATEDIF(B,D,"D")1)/7)
So, it would be easy to gather datas from a given 'custom_week'.
Yvan KOENIG (VALLAURIS, France) jeudi 5 août 2010 10:22:25 -
Procedure to Purge data from WF tables for Item Type POERROR
Hi,
I have executed concurrent program "Purge Obsolete Workflow Runtime Data" to purge old data for Item Type "REQAPPRV". After execution of the concurrent program i re-query to WF_ITEMS table to get the count of the records for "REQAPPRV" but all records did not get purged as "REQAPPRV" is present as PARENT_ITEM_KEY for Item Type "POERROR" in WF_ITEMS table.
Now to purge records of "REQAPPRV" i need to first Purge records of "POERROR". Now the problem is END_DATE of those POERROR records is NULL at WF_ITEMS table that is why concurrent program "Purge Obsolete Workflow Runtime Data" does not able to purge them. How can i resolve this issue?? Please guide..
Regards,
PriyankaHi,
I have already gone thorugh MOS doc, cross checked the queries provided in document and found that status of PARENT_ITEM_TYPE [REQAPPRV] is "COMPLETE" and the status of Chilld ITEM_TYPE [POERROR] is "ACTIVE". I want to close or Purge the child ITEM_TYPE. Can you guide how can we achive that ??
Regards,
Priyanka -
Performance Issue - Fetching latest date from a507 table
Hi All,
I am fetching data from A507 table for material and batch combination. I want to fetch the latest record based on the value of field DATBI. I have written the code as follows. But in the select query its taking more time. I dont want to write any condition in where claue for DATBI field because I have already tried with that option.
SELECT kschl
matnr
charg
datbi
knumh
FROM a507
INTO TABLE it_a507
FOR ALL ENTRIES IN lit_mch1
WHERE kschl = 'ZMRP'
AND matnr = lit_mch1-matnr
AND charg = lit_mch1-charg.
SORT it_a507 BY kschl matnr charg datbi DESCENDING.
DELETE ADJACENT DUPLICATES FROM it_a507 COMPARING kschl matnr charg.Hi,
These kind of tables will be storing large volumes of data. Thus while making a select on it, its important to use as many primary key fields as possible in the where condition. Here you can try mentioning KAPPL since its specific to a requirement. If its for purchasing use 'M' and try.
if not lit_mch1[] is initial.
SELECT kschl
matnr
charg
datbi
knumh
FROM a507
INTO TABLE it_a507
FOR ALL ENTRIES IN lit_mch1
WHERE kappl = 'M'
AND kschl = 'ZMRP'
AND matnr = lit_mch1-matnr
AND charg = lit_mch1-charg.
endif.
SORT it_a507 BY kschl matnr charg datbi DESCENDING.
DELETE ADJACENT DUPLICATES FROM it_a507 COMPARING kschl matnr charg.
This should considerably increase the performance
Regards,
Vik -
How to retrieve data from plsql table in BI publisher Data template
Hi All,
I have created a data template for XML publisher report. In data template i m getting data from plsql table. for that i have created one package with pipelined function. I am able to run that sql from sql developer .But if i run the concurrent program then i got error like "java.sql.SQLSyntaxErrorException: ORA-00904: "XXXXX": invalid identifier".
I have used the same parameters in Data template and concurrent program....
please clarify me what needs to be done....
thanks in advance....
Regards,
DossHi Alex ,
i am using pipelined function and get the data from cursor and load it into plsql table (nested table). and i use the below in my data template to fetch the data:
<sqlStatement name="Q1">
<![CDATA[select * from table(PO_SPEND_RPT_PKG.generate_report(P_ORG_ID,P_SOB_ID,P_ORG_NAME,P_PERIOD_NAME,P_CLOSE_STATUS,P_E_PCARD_NEED,P_REPORT_TYPE))]]>
</sqlStatement>
if i run the above in sql developer i can get the result....from apps if i run i got the error "java.sql.SQLSyntaxErrorException: ORA-00904: "P_ORG_ID": invalid identifier"
Edited by: kalidoss on Sep 14, 2012 4:32 AM -
Help needed in extracting data from PCD tables
Hi Friends
I Have a requiremnt for creating custom portal activity report ,even though
we have standard report, the extraced data will be used to create bw reports later.
my part is to find a way to extract the data from PCD tables for creating
custom portal activity reports
i have selected the following tables for the data extraction
WCR_USERSTAT,WCR_WEBCONTENTSTAT,WCR_USERFIRSTLOGON,
WCR_USERPAGEUSAGE.
My questions are
1.Did i select the Exact PCD tables?
2.Can i use UME api for accessing the data from those tables?
3.can i use the data extracted from PCD tables in JSPdynpage or
webdynpro apps?
4.can i Querry the PCD tables from JSPDynpage or Webdynpro
Please help me in finding a solution for this
Thanks
Ashok BattulaHi daniel
Can u tell me weather i can develop the following custom reports from those WCR tables
Report Type
1 Logins
- Unique Count
- Total Count
- Most Active Users (by Partner Name)
- Most Active Users (by Contact Name)
- Entry Point (by page name)
- Session Time
- Hourly Traffic Analysis
2 Login Failures
- Total Count
- Count by error message
- Credentials Entered (by user name and password)
3 Content Views (by File Name)
- Unique Count
- Total Count
- Most requested Files
- Most requested Pages
- File Not Found
4 Downloads (by File Name)
- Unique Count
- Total Count
- Most requested Files
- File Not Found
5 Portal Administration
- Site Content (by file name)
- Site Content (by page name)
- Latest Content (by file name)
- Expired Content (by file name)
- Subscriptions Count (by file name)
6 Login History (by Partner, Contact Name)
- No Login
- First Login
- Duration between registration and first login
- Most Recent Login
- Average Number of Logins
plz help me in find ing a way
thanks
ashok -
Hi Experts,
How to retrive data from HR tables for custom screen & store in custom table? I need Employee number, employee name, department, their location & mail id. When employee number is given, all other details must be fetched. Can you give a solution for this problem?
Thanks,
KavithaHi Kavitha,
in HR you could get data from different tables based on infotypes. lets say if any data need to get from infotype 0001 (Org assignment), add "PA" before the infotype number so it would be PA0001.
you could get the Employee no, name, department, location(i guess personal area or personal subarea) from PA0001 table and email id from PA105 infotype.
Regards
Raju -
To extract Data from Pool Table Data Sources
hi
I want to extract data from Pool table, for that i want to create infoset for that pool table. can anyone please let me know the
procedure to create info set on pool tables.
Regards
Atul
Moderator message: please (re)search yourself first.
Edited by: Thomas Zloch on Nov 8, 2010 12:54 PMHi Atul
You have a couple of options here:
1) Create Infoset SQ02 on those tables and RSO2 - create generic ds
2) Create functional Module and create generic ds using FM
Replicate DS to BW and Build objects and map them in transformations and create dtp and IP.
Refer to this [link|How to extract data from a pool table?; for more details.
Regards
Harsh -
Migrating data from LookUp Table to Main Table without taking extract
Dear Experts,
Is there any way if we want to migrate full data from LookUp Table to Main Table in the same repository without taking the extract from MDM DM and not importing it using Import Manager?
Thanks
Gaurav Kumar SharmaHi Gaurav,
Yes there are ways,like you can create port for syndication and use MDIS to import it back but Extraction from DM would be more convenient though.You can also use APIs to syndicate the lookup data and import it back in MDM.
Can you please detail what you want to achieve?
Thanks,
Ravi -
Reading data from a table which can have millions of record
Hi,
I need to write a query in my report that can be able to fetch records from a z table that have arount 20 lac (= 2 million) records. As far i knoew we can not handle such a huge amount of data in our internal tables. because the size of internal table is fixed and that can be increased only by basis persons.
So can anybody tell me the approach that i should follow such that i can split the data from this table.
Or any other approach is also welcome.
Please reply ASAP when you have the time.
Thanks And Regards,
Mayank
Edited by: Thomas Zloch on Mar 6, 2010 9:25 PMHi Mayank,
reduce the data selected to the fields you really need, avoid select *. See online help on SELECT, addition PACKAGE SIZE.
select field1 field2 field3
into corresponding fields of table lt_table package size nnn WHERE ...
* process lt_table
endselect.
Regards,
Clemens
Edited by: Rob Burbank on Mar 9, 2010 12:29 PM -
Creating XML file using data from database table
I have to create an xml file using the data from multiple table. The problem That i am facing is the data is huge it is in millions so I was wondering that is there any efective way of creating such an xml file.
It would be great if you can suggest some approach to achieve my requirement.
Thanks,
-VinodAn example from the forum: Re: How to generate xml file from database table
Edited by: Marco Gralike on Oct 18, 2012 9:41 PM -
How to export some data from the tables of an owner with integrity?
Hi to all,
How to export some data from the tables of an owner with integrity?
I want to bring some data from all tables in a single owner of the production database for development environment.
My initial requirements are: seeking information on company code (emp), contract status (status) and / or effective date of contract settlement (dt_liq_efetiva) - a small amount of data to developers.
These three fields are present in the main system table (the table of contracts). Then I thought about ...
- create a temporary table from the query results table to contract;
- and then use this temporary table as a reference to fetch the data in other tables of the owner while maintaining integrity. But how? I have not found the answer, because: what to do when not there is the possibility of a join between the contract and any other table?
I am considering the possibility of consulting the names of tables, foreign keys and columns above, and create dynamic SQL. Conceptually, something like:
select r.constraint_name "FK name",
r.table_name "FK table",
r.column_name "FK column",
up.constraint_name "Referencing name",
up.table_name "Referencing table",
up.column_name "Referencing column"
from all_cons_columns up
join all_cons_columns r
using (owner, position), (select r.owner,
r.constraint_name fk,
r.table_name table_fk,
r.r_constraint_name r,
up.table_name table_r
from all_constraints up, all_constraints r
where r.r_owner = up.owner
and r.r_constraint_name = up.constraint_name
and up.constraint_type in ('P', 'U')
and r.constraint_type = 'R'
and r.owner = 'OWNERNAME') aux
where r.constraint_name = aux.fk
and r.table_name = aux.table_fk
and up.constraint_name = aux.r
and up.table_name = aux.table_r;
-- + Dynamic SQL
If anyone has any suggestions and / or reuse code to me thank you very much!
After resolving this standoff intend to mount the inserts in utl_file by a table and create another program to read and play in the development environment.
Thinking...
Let's Share!
My thanks in advance,
PhilipsThanks, Peter.
Well, I am working with release 9.2.0.8.0. But the planning is migrate to 10g this year. So my questions are:
With Data Pump can export data just from tables owned for me (SCHEMAS = MYOWNER) parameterizing the volume of data (SAMPLE) and filters to table (QUERY), right? But parameterizing a contract table QUERY = "WHERE status NOT IN (2,6) ORDER BY contract ":
1º- the Data Pump automatically searches for related data in other tables in the owner? ex. parcel table has X records related (fk) with Y contracts not in (2,6): X * SAMPLE records will be randomly exported?
2º- for the tables without relation (fk) and which are within the owner (MYOWNER) the data is exported only based on the parameter SAMPLE?
Once again, thank you,
Philips
Reading Oracle Docs... -
Method for Downloading Huge Data from SAP system
Hi All,
we need to download the huge data from one SAP system & then, need to migrate into another SAP system.
is there any better method, except downloading data through SE11/SE16 ? please advice.
Thanks
pabiI have already done several system mergers, and we usually do not have the need to download data.
SAP can talk to SAP. with RFC and by using ALE/IDOC communication.
so it is possible to send e.g. material master with BD10 per IDOC from system A to system B.
you can define that the IDOC is collected, which means it is saved to a file on the application server of the receiving system.
you then can use LSMW and read this file with several hundred thousand of IDOCs as source.
Field mapping is easy if you use IDOC method for import, because then you have a 1:1 field mapping.
So you need only to focus on the few fields where the values changes from old to new system..
Maybe you are looking for
-
Need help to add the words in a text file to an arraylist
I am new to java and I really need some help for my project.I want to add the words of a text file to an arraylist. The text file consist of words and the following set of punctuation marks {, . ; : } and spaces. thanks in advance :-)
-
Printer HP LASERJET 1020 not working in WINDOWS 7 RDP but WORKING IN XP
Our clients connect to server 2008r2 through rdp when they try to print with hp laserjet 1020 printer Its shows failed to print would you like to view the print troubleshooter for assistance. This error only to windows 7 clients,. Windows xp client
-
How to create an image with a transparent shape
Hi This should be easy - The end result I want is this: a png which is mostly black (or any colour) but with a defined shape in it which is transparent. I have the shape. I select it. I seem to be able to make a mask from this (a selection which I ca
-
How do i update my itunes to the 11.1 software so that i can sync my iphone 4s?
How do i update my itunes to the 11.1 software so that i can sync my iphone 4s to my itunes account?
-
Condition types related to planned delivery costs (Freight) from a PO.
Hi Friends, I want to know which condition type is related to Freight in a purchase order. I have a PO which has different conditions like PBXX, FRB1, FRC1, etc. under the conditions tab. I want to get only freight related conditions through a bapi.