Most efficient method to process 2 million plus records from & to a Ztable

Hi All,
My requirement is as follows:
There is a table which has 20 and odd columns, and close to 2 million records.
Initially only 5 or 6 columns will have data. Now the requirement is to fetch them and populate the remaining columns of the table by looking into other tables.
Looking for the most efficient method to handle this as the data count is huge.
There should be an optimum balance between memory usage and time consumption.
Kindly share your expertise in this regard.
Thanks
Mani

Write   a Program to Download the data for that table column to be filled into Local file   .XLS Format.
Then   Write a report for Uploading the data from the local file   .XLS to    database table through internal table itab.
Loop at itab .
UPDATA database table   where   condition of the primary fields.
endloop.
first try this in the development and testing server , then go for the Production.
But   take backup of full exsisting Production data into   local file and also take neccesary approvals for   doing this task .
Reward Points if it is usefull..
Girish

Similar Messages

IFRAME into iMOVIE - most efficient method for importing?

What would be the most efficient method for importing iFrame movies from a camera into iMovie?
iFrame i suppose to save time and work more efficiently in lue of quality but I don't seem to find I way to import the movies faster than in other formats.
On a second note, inporting in iMovie from DV (tape) cameras dramaticly reduced the image quality. Do we still have the same issue when importing an iFrame movie?
Thank you for your help!

Im completly new myself to importing IFRAME into Imovie 11 as i only got my new Panasonic X920 Camcorder 2 days ago.Can you please tell me is there a big drop in quality from 1080 60p to IFRAME Quality.

Most efficient method of online replication!

Hello Guys,
I want to know the most efficient way of synchoronous (real time) replication among 2 oracle databases, that are on 2 different geographyical locations and connectivity among them is over the internet.
Both systems are linux based and oracle 11gR1 is installed.
The constraint is performance.
Kindly help.
Regards, Imran

1) Do you really, really need synchronous replication? Or just "near real time" replication? Synchronous replication requires that neither database commit until both databases have the change, which implies that you're using two-phase commit which implies that every single transaction requires multiple messages to be exchanged between the servers. Two servers that are widely separated are likely going to add a substantial overhead to the transaction. There are a small handful of cases where that might be reasonable, but for 99.9% of the applications out there, synchronous replication is something to be avoided at all costs.
2) What is the business problem you are trying to solve? Are you trying to create a failover database? Are you trying to replicate a small subset of data from one database to another? Something else?
3) What edition of Oracle are you using? Enterprise or standard?
Justin

Post processing of defect records from automatic Goods movement

During the run of COGI transaction, we got the List of all goods movements with errors.The list of errors can be saved at once and the other option is save for background (when we choose option List-> Save-> Save at once or Save for Background).In save at once option it saves the error & save for background shows the message "goods movements were entered as a background job".
Now my question is 1)What is the use of Save for BAckground? 2)If there are are multiple errors for a single raw material (say 7 errors)associated with different Process orders in COGI output.When we choose two or more error(say five errors we selected check boxes) the exercised option save for background the displayed message is self explainatory & it should have created a background job.But when I checked the SM37 (Simple Job Selection) it did not show us the Background Job details.
3)When I exercised the option of Save for Background & processed five error records, how can I find which I processed & where they gone?

This cannot happen that u dotn see in SM37
May be u r using wrong username there.Just recheck it with your username
or check for BACKFLUSH* Jobs
Confirm if it resolves
Reward points if useful
Regards
Manish

What's the best way to delete 2.4 million of records from table?

We are having two tables one is production one and another is temp table which data we want to insert into production table. temp table having 2.5 million of records and on the other side production table is having billions of records. the thing which we want to do just simple delete already existed records from production table and then insert the remaining records from temp to production table.
Can anyone guide what's the best way to do this?
Thanks,
Waheed.

Waheed Azhar wrote:
production table is live and data is appending in this table on random basis. if i go insert data from temp to prod table a pk voilation exception occured bcoz already a record is exist in prod table which we are going to insert from temp to prod
If you really just want to insert the records and don't want to update the matching ones and you're already on 10g you could use the "DML error logging" facility of the INSERT command, which would log all failed records but succeeds for the remaining ones.
You can create a suitable exception table using the DBMS_ERRLOG.CREATE_ERROR_LOG procedure and then use the "LOG ERRORS INTO" clause of the INSERT command. Note that you can't use the "direct-path" insert mode (APPEND hint) if you expect to encounter UNIQUE CONSTRAINT violations, because this can't be logged and cause the direct-path insert to fail. Since this is a "live" table you probably don't want to use the direct-path insert anyway.
See the manuals for more information: http://download.oracle.com/docs/cd/B19306_01/server.102/b14200/statements_9014.htm#BGBEIACB
Sample taken from 10g manuals:
CREATE TABLE raises (emp_id NUMBER, sal NUMBER
 CONSTRAINT check_sal CHECK(sal > 8000));
EXECUTE DBMS_ERRLOG.CREATE_ERROR_LOG('raises', 'errlog');
INSERT INTO raises
 SELECT employee_id, salary*1.1 FROM employees
 WHERE commission_pct > .2
 LOG ERRORS INTO errlog ('my_bad') REJECT LIMIT 10;
SELECT ORA_ERR_MESG$, ORA_ERR_TAG$, emp_id, sal FROM errlog;
ORA_ERR_MESG$ ORA_ERR_TAG$ EMP_ID SAL
ORA-02290: check constraint my_bad 161 7700
(HR.SYS_C004266) violatedIf the number of rows in the temp table is not too large and you have a suitable index on the large table for the lookup you could also try to use a NOT EXISTS clause in the insert command:
INSERT INTO <large_table>
SELECT ...
FROM TEMP A
WHERE NOT EXISTS (
SELECT NULL
FROM <large_table> B
WHERE B.<lookup> = A.<key>
);But you need to check the execution plan, because a hash join using a full table scan on the <large_table> is probably something you want to avoid.
Regards,
Randolf
Oracle related stuff blog:
http://oracle-randolf.blogspot.com/
SQLTools++ for Oracle (Open source Oracle GUI for Windows):
http://www.sqltools-plusplus.org:7676/
http://sourceforge.net/projects/sqlt-pp/

What would be best approach to migrate millions of records from on premise SQL server to Azure SQL DB?

Team,
In our project, we have a requirement of data migration. We have following scenario and I really appreciate any suggestion from you all on implementation part of it.
Scenario:
We have millions of records to be migrated to destination SQL database after some transformation.
The source SQL server is on premise in partners domain and destination server is in Azure.
Can you please suggest what would be best approach to do so.
thanks,
Bishnu
Bishnupriya Pradhan

You can use SSIS itself for this
Have a batch logic which will identify data batches within source and then include data flow tasks to do the data transfer to Azure. The batch size chosen should be as per buffer meory availability + parallel tasks executing etc.
You can use ODBC or ADO .NET connection to connect to azure.
http://visakhm.blogspot.in/2013/09/connecting-to-azure-instance-using-ssis.html
Please Mark This As Answer if it solved your issue
Please Vote This As Helpful if it helps to solve your issue
Visakh
My Wiki User Page
My MSDN Page
My Personal Blog
My Facebook Page

Most efficient method of storing configuration data for huge volume of data

The scenario in which i'm boggled up is as follows:
I have a huge volume of raw data (as CSV files).
This data needs to be rated based on the configuration tables.
The output is again CSV data with some new fields appended to the original records.
These new fields are derived from original data based on the configuration tables.
There are around 15 configuration tables.
Out of these 15 tables 4 tables have huge configurations.
1 table has 15 million configuration data of 10 colums.
Other three tables have around 1-1.5 million configuration data of 10-20 columns.
Now in order to carry forward my rating process, i'm left with the following methods:
1) Leave the configurations in database table. Query the table for each configuration required.
Disadvantage: Even if the indexes are created on the table, it takes a lot of time to query 15 configuration tables for each record in the file.
2) Load the configurations as key value pairs in RAM using a suitable collection (Eg HashMap)
Advantage: Processing is fast
Disadvantage: Takes around 2 GB of RAM per instance.
Also when the CPU context swithes (as i'm using a 8 CPU server), the process gets hanged up for 10 secs.
This happens very frequently, so the net-net speed which i get is again less
3) Store the configurations as CSV sorted files and then perform a binary search on it.
Advantages: No RAM usage, Same configuration shared by multiple instances
Disadvantages: Only 1 configuration table has an integer key, so cant use this concept for other tables
(If i'm wrong in that please correct)
4) Store the configurations as an XML file
Dont know the advantages/disadvantages for it.
Please suggest with the methodology which should be carried out....
Edited by: Vishal_Vinayak on Jul 6, 2009 11:56 PM

Vishal_Vinayak wrote:
2) Load the configurations as key value pairs in RAM using a suitable collection (Eg HashMap)
Advantage: Processing is fast
Disadvantage: Takes around 2 GB of RAM per instance.
Also when the CPU context swithes (as i'm using a 8 CPU server), the process gets hanged up for 10 secs.
This happens very frequently, so the net-net speed which i get is again lessSounds like you don't have enough physical memory. Your application shouldn't be hanging at all.
How much memory is attached to each CPU? e.g. numactl --show

How to make this faster?? read millions of record from txt file

Hi there,
I got an issue. There is a txt file contains 2 million records, I also got another file contains over 10000 numbers. Now, I need to compare this 10000 numbers with that 2 million records if any records contains a number which belongs to 10000 number set, i retrieve this record and keep it. later on, when i finish the comparison i'll write all the result records into a txt file.
What kind of data structure shall i use to keep the records and numbers? how to make the comparison quicker? Any idea will do!
Thanks!

if i were to do it, i will insert bout the records into the db. then do an sql statement on the two tables to get the results. Then get the rs and output it to another text file.
just my opinion. not sure if this is faster.
Message was edited by:
clarenceloh

What is the best way to load 14 million COPA records from BW into HANA?

I have been managing a project in which we are attempting to load COPA data from BW into HANA using Open Hub and we continue to run into memory allocation errors in BW. We have been able to load 350,000 records.
Any suggestions on what the best approach would be along with BW memory parameters.
Your replies are appreciated.
Rob

Hello,
this seems to be issue in BW caused by big volume of migrated data. I do not think that this is HANA related problem. I would suggest to post this message into BW area - you might get much better support there.
But to help as much as I can - I found this (see point 7):
http://help.sap.com/saphelp_nw04/helpdata/en/66/76473c3502e640e10000000a114084/frameset.htm
7. Specify the number of rows per data package for the data records to be extracted. You can use this parameter to control the maximum size of a data package, and hence also how many main memories need to be made available to structure the data package.
Hope it helps.
Tomas

Most efficient coding method to get from WFM Digital to Timestamp Array and Channel Number Array - Earn Kudos Here !

I'm Fetching data from a digitizer card using the niHSDIO Fetch Waveform VI. After Fetching the data I want to keep the Timestamp and Digital Pattern which I'll stream to TMDS file.
What is the most efficient method of striping out the Arrays I'm interested in keeping?
The attached VI shows the input format and desired output. The Record Length is always 1. I'll be streaming 100,000+ records to file using a producer-consumer architecture with the consumer performing a TDMS write.
I'm assuming only the WDT Fetch gives you the time from t0.
Attachments:
Digital Waveform to Array Coding.vi ‏11 KB

Hi bmann2000,
I'm not sure about efficiency but this method definitely works. I've just used a 'Get Digital Waveform Component' function and the 'Digital to Boolean Array' function.
Hope this helps.
Chris
National Instruments - Tech Support
Attachments:
Digital Waveform to Array Coding 2.vi ‏15 KB

Efficient method to insert large number of data into table

Hi,
I have a procedure that accepts an input parameter, that contains, a comma seperated values as input.
Something like G12-UHG,THA-90HJ,NS-98039,........There can be more than 90,000 values in that comma seperated input paramter.
What is the most efficient way to do an insert in this case?.
3 methods I have in mind are :
1) Get individual tokens from CSV and use a plain old loop and do an insert.
2) Use BULK COLLECT & FOR ALL. However I don't know how to do this, since input is not from cursor, rather a parameter.
3) Use Table collections. Again this involves plain old looping through the collection. Same as 1st method.
Please do suggest the most efficient method.
Thanks

90,000 values?? Whats the data type of the input parameter?
you can use the string to row conversion trick if you want and do a single insert
SQL> with t as (select 'ABC,DEF GHI,JKL' str from dual)
2 select regexp_substr(str,'[^,]+', 1, level) list
3 from t connect by level <= NVL( LENGTH( REGEXP_REPLACE( str, '[^,]+', NULL ) ), 0 ) + 1
4 /
LIST
ABC
DEF GHI
JKL Edited by: Karthick_Arp on Feb 13, 2009 2:18 AM

How to Cool Down a MacBook Pro most efficient?

Hello,
I bought a MacBook Pro 13' a few days ago, and I'm very happy about the computer! Very good quality! I just have one little problem, when I'm gaming Flight Simulator, X-Plane, then my MacBook gets really hot. I know that there's methods to cool down a computer, but what's the best and most efficient methods to cool down my MacBook Pro 13' during gaming?
Hope some one is able to help ! Thank you !
AMLaursen

No I just found it on Amazon, I have a Belkin one I picked up at a supermarket near home. I don't game but I find Skype get the computer hot so I use the cooler then.
The one I have brings the temp down and the fans slow down as well.
I would have thought that they are all very much similar, just blowing air under the computer case, the Belkin ones leave a gap under the computer for the hot air to escape, not sure if that helps or not.
http://www.amazon.co.uk/Belkin-Cooling-Stand-Laptops-17-inch/dp/B001HNOLBI/ref=s r_1_3?ie=UTF8&qid=1400791130&sr=8-3&keywords=laptop+cooler
It is a case I think of getting one and trying it out.

Need help / Advice ; manage daily millions of records;;plz help me:)

Hi all,
I've only 2 years of experience in Oracle DBA. I need advice from Experts:)
To begin, the company I work for, decide to daily save in our Oracle database about 40 millions of records in our only table (User tables). These records should be daily imported from csv or xml feeds into one table.
This 's a project that need :
- Study the performance
- Study What is required in terms of hardware
As a leader in the market, Oracle 's the only DBMS that could support this size of data, but what's the limit of Oracle in this case? can Oracle support and manage perfectly daily 40 millions of records and for many years, ie We need all data of this table, we can't consider after a period that we don't need history: we need to save all data and without purge the history and this for many years i suppose!!! you can imagine 40 daily millions of records and for many years!!!
Then we need to consolidate from this table different views (or maybe materalized view) for each department and business inside the company, one other project that need study!
My questions 're :Using Oracle Database 10g Enterprise Edition Release 10.2.0.1.0:
1-Can Oracle support and perfectly manage daily 40 millions of records and for many years?
2-Study the performance ; which solutions, technics could I use to improve the performance of :
- Daily Loading 40 millions of records from csv or xml file/files?
- Daily Consolidate / managing different views/ materalized view from this big table?
3- What is required in terms of hardware? features / Technologies( maybe clusters...)
Hope that experts help me and advice me! thank you very much for your atention :)

1-Can Oracle support and perfectly manage daily 40 millions of records and for many years?Yes
2-Study the performance ; which solutions, technics could I use to improve >>>the performance of :Send me your email, and I can send you a Performance tuning metodology pdf.
You can see my email on my profile.
Daily Loading 40 millions of records from csv or xml file/files?DIrect Load
- Daily Consolidate / managing different views/ materalized view from this big table?You can use table partitions, one partition for each day.
Regards,
Francisco Munoz Alvarez

How to DELETE millions of records. How to make it fast.

Hi
I need to delete near abt 134 millions of records from tables.
How to make it faster? any trick , any settings.
I am using Oracle 9i on Linux box.
If suppose i use TRUNCATE . does it deletes all objects defined over tables like constraint, indexes etc.
Thanks,
Kuldeep

hi
SQL> create table te as select * from all_objects;
Table created.
SQL> create index te_ind on te ( owner);
Index created.
SQL> truncate table te;
Table truncated.
SQL> select index_name , status from user_indexes where table_name = 'TE';
INDEX_NAME STATUS
TE_IND VALID
SQL> create table ti as select * from all_objects;
Table created.
SQL> create index ti_ind on ti ( owner);
Index created.
SQL> drop table ti;
Table dropped.
SQL> select index_name , status from user_indexes where table_name = 'TI';
no rows selected
SQL>regards
Taj

Deleting 3 millon duplicated records from 4 million records

Hi,
I want to delete 3 million duplicate records from 4 million records in production. We dont have the partitions. I cann't recreate table using CTAS Option. So I have to delete the data in batch because of shared environment and people acceessing the data.
Is there any fastest way to delete the data in stead of using bulk delete by using max rowid.
please help me out.
Regards,
Venkat.
Edited by: ramanamadhav on Aug 16, 2011 8:41 AM

After deletion of data( with suggestion given by Justin), make sure that latest statistics are also taken.
After such a heavy delete there is always a mismatch between num_rows and actual count.
the best way is to shrink space and then gather table statistics, other wise optimizer will get confused and it will not take into consideration correct statistics.
For e.g.
16:41:55 SQL> create table test_tab_sm
as
select level col1, rpad('*888',200,'###') col2
from dual connect by level <= 100000;16:42:05 2 16:42:05 3 16:42:05 4
Table created.
Elapsed: 00:00:00.49
16:42:08 SQL> exec dbms_stats.gather_table_stats(ownname => 'KDM', tabname => 'TEST_TAB_SM', cascade => true);
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.51
16:42:17 SQL> select table_name,num_rows,blocks,avg_row_len,chain_cnt from user_tables where table_name = 'TEST_TAB_SM';
TABLE_NAME NUM_ROWS BLOCKS AVG_ROW_LEN CHAIN_CNT
TEST_TAB_SM 100000 2942 205 0
1 row selected.
Elapsed: 00:00:00.01
16:42:28 SQL> delete from TEST_TAB_SM where mod(col1,2) =1;
50000 rows deleted.
Elapsed: 00:00:01.09
16:42:39 SQL> select table_name,num_rows,blocks,avg_row_len,chain_cnt from user_tables where table_name = 'TEST_TAB_SM';
TABLE_NAME NUM_ROWS BLOCKS AVG_ROW_LEN CHAIN_CNT
TEST_TAB_SM 100000 2942 205 0
1 row selected.
Elapsed: 00:00:00.01
16:42:47 SQL> exec dbms_stats.gather_table_stats(ownname => 'KDM', tabname => 'TEST_TAB_SM', cascade => true);
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.26
16:42:55 SQL> select table_name,num_rows,blocks,avg_row_len,chain_cnt from user_tables where table_name = 'TEST_TAB_SM';
TABLE_NAME NUM_ROWS BLOCKS AVG_ROW_LEN CHAIN_CNT
TEST_TAB_SM 50000 2942 205 0
1 row selected.
Elapsed: 00:00:00.01
16:43:27 SQL> alter table TEST_TAB_SM move;
Table altered.
Elapsed: 00:00:00.46
16:43:59 SQL> select table_name,num_rows,blocks,avg_row_len,chain_cnt from user_tables where table_name = 'TEST_TAB_SM';
TABLE_NAME NUM_ROWS BLOCKS AVG_ROW_LEN CHAIN_CNT
TEST_TAB_SM 50000 2942 205 0
1 row selected.
Elapsed: 00:00:00.03
16:44:06 SQL> exec dbms_stats.gather_table_stats(ownname => 'KDM', tabname => 'TEST_TAB_SM', cascade => true);
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.24
16:44:17 SQL> select table_name,num_rows,blocks,avg_row_len,chain_cnt from user_tables where table_name = 'TEST_TAB_SM';
TABLE_NAME NUM_ROWS BLOCKS AVG_ROW_LEN CHAIN_CNT
TEST_TAB_SM 50000 1471 205 0
1 row selected.
Elapsed: 00:00:00.01
16:44:24 SQL> We can see how the no. of blocks changes. It changed from 2942 to 1471 which is half and in your case it will be 1/4th times of what it was using.
There are other options for shrinking space as well other than MOVE.
alter table <table_name> shrink space;Try it out and you can see the difference yourself.

Most efficient method to process 2 million plus records from & to a Ztable

Similar Messages

Maybe you are looking for