Help: Loading of 500 million rows
Hi all,
I have to load 500 million rows to 5 tables based on some criteria.in case of failure i have to start from the row which caused the error.i should not roll back the inseted rows. could you please help me with best logic?
thanks
thiru
Hi,
Depending on your non-mentioned DB-version, you could use DML Error Logging here.
http://tkyte.blogspot.com/2005/07/how-cool-is-this.html
http://www.oracle.com/technology/oramag/oracle/06-mar/o26performance.html
http://download.oracle.com/docs/cd/B19306_01/server.102/b14231/tables.htm#ADMIN10261
Similar Messages
-
Handling million rows using jdbc
approximately how much time would it take to use jdbc and do a simple data import from an oracle database having 500 million rows?
Without a lot more information my answer would be too long.
It is likely that something is wrong with the design of a project if it involves bring 500 million records into a running java application.
Using a 1000 byte record size it would take 19 hours just to move the data over a 100mbit ethernet connection. -
Performance issue while loading 20 million rows
Hi all,
Loading done with 20 million rows into a table (which contains 173columns) using sql loader.
And direct=true has been used.
database is : 9i
OS : Sun OS 5.10 Sun V890, 16 cpu 32GB RAM
Elapsedtime is : 4 hours
But same volume is tried with the following details, into the same table (but columns are increased from 173 to 500)
Database : oracle 10.2.0.4.0 64 bit
OS : Sun Os 5.10 SUN-FIRE V6800, 24 cpu and 54GB RAM
Elapsed time : 6:06 hours
please tell me what could be the problem and how can I minimize the loading time?
Thanks in Advance
AnjiHi burleson,
AWR snap shot as follows.
DB Name DB Id Instance Inst Num Release RAC Host
REVACC 1015743016 REVACC 1 10.2.0.4.0 NO P4061AFMAP
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 342 16-Sep-09 19:30:53 38 2.7
End Snap: 343 16-Sep-09 20:30:07 36 2.6
Elapsed: 59.24 (mins)
DB Time: 195.22 (mins)
Cache Sizes
~~~~~~~~~~~ Begin End
Buffer Cache: 9,184M 9,184M Std Block Size: 16K
Shared Pool Size: 992M 992M Log Buffer: 10,560K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 1,097,030.72 354,485,330.18
Logical reads: 23,870.31 7,713,251.27
Block changes: 8,894.16 2,873,984.09
Physical reads: 740.82 239,382.82
Physical writes: 1,003.32 324,203.27
User calls: 28.54 9,223.18
Parses: 242.99 78,517.09
Hard parses: 0.03 8.55
Sorts: 0.60 193.91
Logons: 0.01 3.45
Executes: 215.63 69,676.00
Transactions: 0.00
% Blocks changed per Read: 37.26 Recursive Call %: 99.65
Rollback per transaction %: 0.00 Rows per Sort: 7669.57
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.99 Redo NoWait %: 100.00
Buffer Hit %: 96.93 In-memory Sort %: 100.00
Library Hit %: 99.97 Soft Parse %: 99.99
Execute to Parse %: -12.69 Latch Hit %: 99.52
Parse CPU to Parse Elapsd %: 91.82 % Non-Parse CPU: 99.64
Shared Pool Statistics Begin End
Memory Usage %: 44.50 44.46
% SQL with executions>1: 85.83 84.78
% Memory for SQL w/exec>1: 87.15 86.65
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
CPU time 11,632 99.3
db file scattered read 320,585 210 1 1.8 User I/O
SQL*Net more data from client 99,234 164 2 1.4 Network
log file parallel write 5,750 149 26 1.3 System I/O
db file parallel write 144,502 142 1 1.2 System I/O
Time Model Statistics DB/Inst: REVACC/REVACC Snaps: 342-343
-> Total time in database user-calls (DB Time): 11713.1s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
DB CPU 11,631.9 99.3
sql execute elapsed time 3,131.4 26.7
parse time elapsed 53.7 .5
hard parse elapsed time 1.2 .0
connection management call elapsed time 0.3 .0
hard parse (sharing criteria) elapsed time 0.1 .0
sequence load elapsed time 0.1 .0
repeated bind elapsed time 0.0 .0
PL/SQL execution elapsed time 0.0 .0
DB time 11,713.1 N/A
background elapsed time 613.1 N/A
background cpu time 454.5 N/A
Wait Class DB/Inst: REVACC/REVACC Snaps: 342-343
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc
Avg
%Time Total Wait wait Waits
Wait Class Waits -outs Time (s) (ms) /txn
User I/O 562,302 .0 304 1 51,118.4
System I/O 166,468 .0 295 2 15,133.5
Network 201,009 .0 165 1 18,273.5
Application 60 .0 5 82 5.5
Configuration 313 .0 4 12 28.5
Other 1,266 .0 3 2 115.1
Concurrency 9,305 .0 2 0 845.9
Commit 60 .0 1 21 5.5
Wait Events DB/Inst: REVACC/REVACC Snaps: 342-343
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
db file scattered read 320,585 .0 210 1 29,144.1
SQL*Net more data from clien 99,234 .0 164 2 9,021.3
log file parallel write 5,750 .0 149 26 522.7
db file parallel write 144,502 .0 142 1 13,136.5
db file sequential read 207,780 .0 93 0 18,889.1
enq: RO - fast object reuse 60 .0 5 82 5.5
write complete waits 135 .0 3 23 12.3
control file parallel write 2,501 .0 3 1 227.4
rdbms ipc reply 189 .0 2 12 17.2
control file sequential read 13,694 .0 2 0 1,244.9
buffer busy waits 8,499 .0 1 0 772.6
log file sync 60 .0 1 21 5.5
direct path write 8,290 .0 1 0 753.6
SQL*Net message to client 100,882 .0 1 0 9,171.1
log file switch completion 13 .0 0 38 1.2
os thread startup 2 .0 0 174 0.2
direct path read 25,646 .0 0 0 2,331.5
log buffer space 161 .0 0 1 14.6
latch free 7 .0 0 24 0.6
latch: object queue header o 180 .0 0 1 16.4
log file single write 11 .0 0 9 1.0
SQL*Net more data to client 893 .0 0 0 81.2
latch: cache buffers chains 767 .0 0 0 69.7
row cache lock 36 .0 0 2 3.3
LGWR wait for redo copy 793 .0 0 0 72.1
reliable message 60 .0 0 1 5.5
latch: cache buffers lru cha 11 .0 0 1 1.0
db file single write 1 .0 0 10 0.1
log file sequential read 10 .0 0 1 0.9
latch: session allocation 18 .0 0 1 1.6
latch: redo writing 4 .0 0 0 0.4
latch: messages 7 .0 0 0 0.6
latch: row cache objects 1 .0 0 0 0.1
latch: checkpoint queue latc 1 .0 0 0 0.1
PX Idle Wait 13,996 100.5 27,482 1964 1,272.4
SQL*Net message from client 100,881 .0 23,912 237 9,171.0
Streams AQ: qmn slave idle w 126 .0 3,442 27316 11.5
Streams AQ: qmn coordinator 255 50.6 3,442 13497 23.2
Streams AQ: waiting for time 1 100.0 545 544885 0.1
class slave wait 2 .0 0 2 0.2
Background Wait Events DB/Inst: REVACC/REVACC Snaps: 342-343
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
log file parallel write 5,750 .0 149 26 522.7
db file parallel write 144,502 .0 142 1 13,136.5
control file parallel write 2,501 .0 3 1 227.4
direct path write 8,048 .0 1 0 731.6
control file sequential read 3,983 .0 0 0 362.1
os thread startup 2 .0 0 174 0.2
direct path read 25,646 .0 0 0 2,331.5
log buffer space 161 .0 0 1 14.6
events in waitclass Other 924 .0 0 0 84.0
log file single write 11 .0 0 9 1.0
db file single write 1 .0 0 10 0.1
log file sequential read 10 .0 0 1 0.9
db file sequential read 2 .0 0 5 0.2
latch: cache buffers chains 42 .0 0 0 3.8
latch: redo writing 4 .0 0 0 0.4
buffer busy waits 2 .0 0 0 0.2
rdbms ipc message 77,540 24.8 54,985 709 7,049.1
pmon timer 1,185 100.0 3,456 2916 107.7
Streams AQ: qmn slave idle w 126 .0 3,442 27316 11.5
Streams AQ: qmn coordinator 255 50.6 3,442 13497 23.2
smon timer 112 .0 3,374 30125 10.2
Streams AQ: waiting for time 1 100.0 545 544885 0.1
Operating System Statistics DB/Inst: REVACC/REVACC Snaps: 342-343
Statistic Total
AVG_BUSY_TIME 161,850
AVG_IDLE_TIME 187,011
AVG_IOWAIT_TIME 0
AVG_SYS_TIME 9,653
AVG_USER_TIME 152,083
BUSY_TIME 3,887,080
IDLE_TIME 4,491,132
IOWAIT_TIME 0
SYS_TIME 234,325
USER_TIME 3,652,755
LOAD 11
OS_CPU_WAIT_TIME 9,700
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 57,204,736
VM_OUT_BYTES 0
PHYSICAL_MEMORY_BYTES 56,895,045,632
NUM_CPUS 24
Service Statistics DB/Inst: REVACC/REVACC Snaps: 342-343
-> ordered by DB Time
Physical Logical
Service Name DB Time (s) DB CPU (s) Reads Reads
SYS$USERS 11,931.9 11,848.9 2,608,446 ##########
REVACC 0.0 0.0 0 0
SYS$BACKGROUND 0.0 0.0 25,685 34,096
Service Wait Class Stats DB/Inst: REVACC/REVACC Snaps: 342-343
-> Wait Class info for services in the Service Statistics section.
-> Total Waits and Time Waited displayed for the following wait
classes: User I/O, Concurrency, Administrative, Network
-> Time Waited (Wt Time) in centisecond (100th of a second)
Service Name
User I/O User I/O Concurcy Concurcy Admin Admin Network Network
Total Wts Wt Time Total Wts Wt Time Total Wts Wt Time Total Wts Wt Time
SYS$USERS
525903 24088 9259 161 0 0 201012 16511
REVACC
0 0 0 0 0 0 0 0
SYS$BACKGROUND
36410 6292 46 35 0 0 0 0
I will provide entire report.. if it is not sufficient
Thanks
Anji
Edited by: user11907415 on Sep 17, 2009 6:39 AM -
Help on table update - millions of rows
Hi,
I am trying to do the following however the process is taking lot of time. Can someone help me the best way to do it.
qtemp1 - 500,000 rows
qtemp2 - 50 Million rows
UPDATE qtemp2 qt
SET product =
SELECT product_cd
FROM qtemp1 qtemp
WHERE qt.quote_id = qtemp.quote_id
processed_ind = 'P'
I have created indexes on product, product_cd and quote_id on both the tables.
Thank youThere are two basic I/O read operations that need to be done to find the required rows.
1. In QTEMP1 find row for a specific QUOTE_ID.
2. In QTEMP2 find all rows where PROCESSED_IND is equal to 'P'.
For every row in (2), the I/O in (1) is executed. So, if there are 10 million rows result for (2), then (1) will be executed 10 million times.
So you want QTEMP1 to be optimised for access via QUOTE_ID - at best it should be using a unique index.
Access on QTEMP2 is more complex. I assume that the process indicator is a column with low cardinality. In addition, being an process status indicator it is likely a candidate for being changed via UPDATE statements - in which case it is a very poor candidate for either a B+ tree index or a Bitmap index.
Even if indexed, a large number of rows may be of process type 'P' - in which case the CBO will rightly decide not to waste I/O on the index, but instead spend all the I/O instead on a faster full table scan.
In this case, (2) will result in all 50 million rows to be read - and for each row that has a 'P' process indicator, calling (1).
Anyway you look at this.. it is a major processing request for the database to perform. It involves a lot of I/O, can involve a huge number of nested SQL calls to QTEMP1... so this is obviously going to be a slow performing process. The majority of elapsed processing time will be spend waiting for I/O from disks. -
How to Load 100 Million Rows in Partioned Table
Dear all,
I a workling in VLDB application.
I have a Table with 5 columns
For ex- A,B,C,D,DATE_TIME
I CREATED THE RANGE (DAILY) PARTIONED TABLE ON COLUMN (DATE_TIME).
AS WELL CREATED NUMBER OF INDEX FOR EX,
INDEX ON A
COMPOSITE INDEX ON DATE_TIME,B,C
REQUIREMENT
NEED TO LOAD APPROX 100 MILLION RECORDS IN THIS TABLE EVERYDAY ( IT WILL LOAD VIA SQL LOADER OR FROM TEMP TABLE (INSERT INTO ORIG SELECT * FROM TEMP)...
QUESTION
TABLE IS INDEXED SO I AM NOT ABLE TO USE SQLLDR FEATURE DIRECT=TRUE.
SO LET ME KNOW WHAT THE BEST AVILABLE WAY TO LOAD THE DATA IN THIS TABLE ????
Note--> PLEASE REMEMBER I CAN'T DROP AND CREATE INDEX DAILY DUE TO HUGE DATA QUANTITY.Actually a simpler issue then what you seem to think it is.
Q. What is the most expensive and slow operation on a database server?
A. I/O. The more I/O, the more latency there is, the longer the wait times are, the bigger the response times are, etc.
So how do you deal with VLT's? By minimizing I/O. For example, using direct loads/inserts (see SQL APPEND hint) means less I/O as we are only using empty data blocks. Doing one pass through the data (e.g. apply transformations as part of the INSERT and not afterwards via UPDATEs) means less I/O. Applying proper filter criteria. Etc.
Okay, what do you do when you cannot minimize I/O anymore? In that case, you need to look at processing that I/O volume in parallel. Instead of serially reading and writing a 100 million rows, you (for example) use 10 processes that each reads and writes 10 million rows. I/O bandwidth is there to be used. It is extremely unlikely that a single process can fully utilised the available I/O bandwidth. So use more processes, each processing a chunk of data, to use more of that available I/O bandwidth.
Lastly, think DDL before DML when dealing with VLT's. For example, a CTAS to create a new data set and then doing a partition exchange to make that new data set part of the destination table, is a lot faster than deleting that partition's data directly, and then running a INSERT to refresh that partition's data.
That in a nutshell is about it - think I/O and think of ways to use it as effectively as possible. With VLT's and VLDB's one cannot afford to waste I/O. -
OC4J - How to get data from large table (more than 9 million rows) by EJB?
SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS
O I use Jdeveloper to create EJB that has finder methods to get data from a big table (more S than 9 million rows). Deploy is OK but when run client program I always get timeout
O error or not enough memory error,
S Can any one help me?
O urgent
SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS SOS SOSYour problem may be that you are simply trying to load so many objects (found by your finder) that you are exceeding available memory. For example if each object is 100 bytes and you try to load 1,000,000 objects thats 100Mb of memory gone.
You could try increasing the amount of memory available to OC4J with the appropriate argument on the command line (or in the 10gAS console). For example to make 1Gb available to OC4J you would add the argument:
-Xmx1000m
Of course you need have this available as hard memory on your server or you will incur serious swapping.
Chris -
Update Statement against 1.4 million rows
Hi,
I am trying to execute and update statement against a table with over a million rows in it.
NAME Null? Type
PR_ID NOT NULL NUMBER(12,0)
PR_PROP_CODE NOT NULL VARCHAR2(180)
VALUE CLOB(4000)
SHRT_DESC VARCHAR2(250)
VAL_CHAR VARCHAR2(500)
VAL_NUM NUMBER(12,0)
VAL_CLOB CLOB(4000)
UNIQUE_ID NUMBER(12,0)
The update i am trying to do is to take teh column VALUE and based on some parameters update one of three columns. When
I run the sql it just sits there with no error. I gave it 24 hours before killing the process.
UPDATE PR.PR_PROP_VAL PV
SET PV.VAL_CHAR = (
SELECT a.value_char FROM
select
ppv.unique_id,
CASE ppv.pr_prop_code
WHEN 'BLMBRG_COUNTRY' THEN to_char(ppv.value)
WHEN 'BLMBRG_INDUSTRY' THEN to_char(ppv.value)
WHEN 'BLMBRG_TICKER' THEN to_char(ppv.value)
WHEN 'BLMBRG_TITLE' THEN to_char(ppv.value)
WHEN 'BLMBRG_UID' THEN to_char(ppv.value)
WHEN 'BUSINESSWIRE_TITLE' THEN to_char(ppv.value)
WHEN 'DJ_EUROASIA_TITLE' THEN to_char(ppv.value)
WHEN 'DJ_US_TITLE' THEN to_char(ppv.value)
WHEN 'FITCH_MRKT_SCTR' THEN to_char(ppv.value)
WHEN 'ORIGINAL_TITLE' THEN to_char(ppv.value)
WHEN 'RD_CNTRY' THEN to_char(ppv.value)
WHEN 'RD_MRKT_SCTR' THEN to_char(ppv.value)
WHEN 'REPORT_EXCEP_FLAG' THEN to_char(ppv.value)
WHEN 'REPORT_LANGUAGE' THEN to_char(ppv.value)
WHEN 'REUTERS_RIC' THEN to_char(ppv.value)
WHEN 'REUTERS_TITLE' THEN to_char(ppv.value)
WHEN 'REUTERS_TOPIC' THEN to_char(ppv.value)
WHEN 'REUTERS_USN' THEN to_char(ppv.value)
WHEN 'RSRCHDIRECT_TITLE' THEN to_char(ppv.value)
WHEN 'SUMMIT_FAX_BODY_FONT_SIZE' THEN to_char(ppv.value)
WHEN 'SUMMIT_FAX_TITLE' THEN to_char(ppv.value)
WHEN 'SUMMIT_FAX_TITLE_FONT_SIZE' THEN to_char(ppv.value)
WHEN 'SUMMIT_TOPIC' THEN to_char(ppv.value)
WHEN 'SUMNET_EMAIL_TITLE' THEN to_char(ppv.value)
WHEN 'XPEDITE_EMAIL_TITLE' THEN to_char(ppv.value)
WHEN 'XPEDITE_FAX_BODY_FONT_SIZE' THEN to_char(ppv.value)
WHEN 'XPEDITE_FAX_TITLE' THEN to_char(ppv.value)
WHEN 'XPEDITE_FAX_TITLE_FONT_SIZE' THEN to_char(ppv.value)
WHEN 'XPEDITE_TOPIC' THEN to_char(ppv.value)
END value_char
from pr.pr_prop_val ppv
where ppv.pr_prop_code not in
('BLMBRG_BODY','ORIGINAL_BODY','REUTERS_BODY','SUMMIT_FAX_BODY',
'XPEDITE_EMAIL_BODY','XPEDITE_FAX_BODY','PR_DISCLOSURE_STATEMENT', 'PR_DISCLAIMER')
) a
WHERE
a.unique_id = pv.unique_id
AND a.value_char is not null
Thanks for any help you can provide.
GrahamWhat about this:
UPDATE pr.pr_prop_val pv
SET pv.val_char = TO_CHAR(pv.value)
WHERE pv.pr_prop_code IN ('BLMBRG_COUNTRY', 'BLMBRG_INDUSTRY', 'BLMBRG_TICKER', 'BLMBRG_TITLE', 'BLMBRG_UID', 'BUSINESSWIRE_TITLE',
'DJ_EUROASIA_TITLE', 'DJ_US_TITLE', 'FITCH_MRKT_SCTR', 'ORIGINAL_TITLE', 'RD_CNTRY', 'RD_MRKT_SCTR',
'REPORT_EXCEP_FLAG', 'REPORT_LANGUAGE', 'REUTERS_RIC', 'REUTERS_TITLE', 'REUTERS_TOPIC', 'REUTERS_USN',
'RSRCHDIRECT_TITLE', 'SUMMIT_FAX_BODY_FONT_SIZE', 'SUMMIT_FAX_TITLE', 'SUMMIT_FAX_TITLE_FONT_SIZE',
'SUMMIT_TOPIC', 'SUMNET_EMAIL_TITLE', 'XPEDITE_EMAIL_TITLE', 'XPEDITE_FAX_BODY_FONT_SIZE', 'XPEDITE_FAX_TITLE',
'XPEDITE_FAX_TITLE_FONT_SIZE', 'XPEDITE_TOPIC')
AND pv.value IS NOT NULL -
Copy 500 million records between 2 tables
Hi,
I have 2 tables T1 and T2 in the same database and schema as well.
But the Size of data in T1 is around 500 million. Which is the best way to Copy data from T1 to T2.
How fast is the SQL PLUS Copy Command?
Will BULK COLLECT into collection and FOR ALL statements help in doing the job at high speeds?
Are there any other options?(I only want to use SQL PLUS or PL/SQL)
please let me know if there is a way out.
Thanks,
VijayHi ,
Thank you very much for the response.
This is a good option too but afraid about the
logging it does at the back end. What kind of logging. Maybe you can switch it of (Triggers?).
And more over it works row by row.No it does not. It is not the same as
INSERT INTO T2 SELECT FROM T1 where ID = 1;
INSERT INTO T2 SELECT FROM T1 where ID = 2;
INSERT INTO T2 SELECT FROM T1 where ID = 3;
INSERT INTO T2 SELECT FROM T1 where ID = 4;
INSERT INTO T2 SELECT FROM T1 where ID = 5;
INSERT INTO T2 SELECT FROM T1 where ID = 6;
It does all at once.
INSERT INTO T2 SELECT FROM T1;
Also the time is very limited for my job to complete
copying data.
So I am skeptical to use this Option.
Everything else will be slow compared to that.
**Also the 2 tables have Same Data Definition.
Incase there is a better way to do this. Please
share.
Thank you again for the help.
VijayGood luck
Message was edited by:
Sven Weller -
1 million row full table scan in less than one minute?
All,
a customer has a table which is around 1 million rows.
The table size is 300 Mo.
The customer wants to calculate regular arithmetic operations,
and synthetic data on the whole table. He will use 'group by'
statements and 'sort by'.
He wants to know if it would be possible to get the results in
less than one minute, and what would be the necessary
architecture in order to achieve this?
He currently has a 4 Gb Ram machine, 4 processors (I think...).
It's a 7.3 database.
Would an upgrade to 8i or 9i help? Would partitioning help?
Would parallel query help? What else?
There will be no other concurrent requests on the system.
Thanks!Hi
If you have four processors parallel query will increase speed
of executing your statement. It will work better with
partitions. In version 7.3 is not very easy to manage partitons
because you have not partition tables - you have only partition
views. Maybe the speed is the same but management of partition
views is not so easy. So in version 8i and 9i it will be better.
If you have index for group by columns and they are 'NOT NULL'
it will be additional way of increasing speed beacause you will
eleminate sort operation.
Regards -
Update all rows in a table which has 8-10 million rows take for ever
Hi All,
Greetings!
I have to update 8million rows on a table. Basically have to reset the batch_id with the current batch number. it contains 8-10 million rows and i have tried with bulk update and then also it takes long time. below is the table structure
sales_stg (it has composite key of product,upc and market)
=======
product_id
upc
market_id
batch_id
process_status
I have to update batch_id,process_status to current batch_id (a number) and process_status as zero. I have to update all the rows with these values for batch_id = 0.
I tried bulk update an it takes more than 2hrs to do. (I limit the update to 1000).
Any help in this regard.
Naveen.The fastest way will probably be to not use a select loop but a direct update like in William's example. The main downside is if you do too many rows you risk filling up your rollback/undo; to keep things as simple as possible I wouldn't do batching except for this. Also, we did some insert timings a few years ago on 9iR1 and found that the performance curve on frequent commits started to level off after 4K rows (fewer commits were still better) so you could see how performance improves by performing fewer commits if that's an issue.
The other thing you could consider if you have the license is using the parallel query option. -
How to get data from large table (more than 9 million rows) by EJB?
I have a giant table, it has more than 9 million rows.
I want to use ejb finders method to get data from this table but always get not enough memory error or time out error,
Can anyone give me solutions?
ThxYour problem may be that you are simply trying to load so many objects (found by your finder) that you are exceeding available memory. For example if each object is 100 bytes and you try to load 1,000,000 objects thats 100Mb of memory gone.
You could try increasing the amount of memory available to OC4J with the appropriate argument on the command line (or in the 10gAS console). For example to make 1Gb available to OC4J you would add the argument:
-Xmx1000m
Of course you need have this available as hard memory on your server or you will incur serious swapping.
Chris -
Best method to update database table for 3 to 4 million rows
Hi All,
I have 3 to 4 million rows are there in my excel file and we have to load to Z-Table.
The intent is to load and keep 18 months of history in this table.
so what should be best way for huge volume of data to Z-Table from excel file.
If is from the program, is that the best way use the FM 'GUI_DOWNLOAD' and down load those entries into the internal table and directly do as below
INSERT Z_TABLE from IT_DOWNLOAD.
I think for the huge amount of data it goes to dump.
please suggest me the best possible way or any psudo code to insert those huge entries into that Z_TABLE.
Thanks in advance..Hi,
You get the dump because of uploading that much records into itnernal table from excel file...
in this case, do the follwowing.
data : w_int type i,
w_int1 type i value 1.
data itab type standard table of ALSMEX_TABLINE with header line.
do.
refresh itab.
w_int = w_int1..
w_int1 = w_int + 25000.
CALL FUNCTION 'ALSM_EXCEL_TO_INTERNAL_TABLE'
EXPORTING
FILENAME = <filename>
I_BEGIN_COL = 1
I_BEGIN_ROW = w_int
I_END_COL = 10
I_END_ROW = w_int1
TABLES
INTERN = itab
* EXCEPTIONS
* INCONSISTENT_PARAMETERS = 1
* UPLOAD_OLE = 2
* OTHERS = 3
IF SY-SUBRC <> 0.
* MESSAGE ID SY-MSGID TYPE SY-MSGTY NUMBER SY-MSGNO
* WITH SY-MSGV1 SY-MSGV2 SY-MSGV3 SY-MSGV4.
ENDIF.
if itab is not initial.
write logic to segregate the data from itab to the main internal table and then
insert records from the main internal table to database table.
else.
exit.
endif.
enddo.
Regards,
Siddarth -
Best way to refresh 5 million row table
Hello,
I have a table with 5 million rows that needs to be refreshed every 2 weeks.
Currently I am dropping and creating the table which takes a very long time and gives a warning related to table space at the end of execution. It does create the table with the actual number of rows but I am not sure why I get the table space warning at the end.
Any help is greatly appreciated.
Thanks.Can you please post your query.
# What is the size of temporary tablespace
# Is you query performing any sorts ?
Monitor the TEMP tablespace usage from below after executing your SQL query
SELECT TABLESPACE_NAME, BYTES_USED, BYTES_FREE
FROM V$TEMP_SPACE_HEADER;
{code} -
Problem with BULK COLLECT with million rows - Oracle 9.0.1.4
We have a requirement where are supposed to load 58 millions of rows into a FACT Table in our DATA WAREHOUSE. We initially planned to use Oracle Warehouse Builder but due to performance reasons, decided to write custom code. We wrote a custome procedure which opens a simple cursor and reads all the 58 million rows from the SOURCE Table and in a loop processes the rows and inserts the records into a TARGET Table. The logic works fine but it took 20hrs to complete the load.
We then tried to leverage the BULK COLLECT and FORALL and PARALLEL options and modified our PL/SQL code completely to reflect these. Our code looks very simple.
1. We declared PL/SQL BINARY_INDEXed Tables to store the data in memory.
2. We used BULK COLLECT into FETCH the data.
3. We used FORALL statement while inserting the data.
We did not introduce any of our transformation logic yet.
We tried with the 600,000 records first and it completed in 1 min and 29 sec with no problems. We then doubled the no. of rows to 1.2 million and the program crashed with the following error:
ERROR at line 1:
ORA-04030: out of process memory when trying to allocate 16408 bytes (koh-kghu
call ,pmucalm coll)
ORA-06512: at "VVA.BULKLOAD", line 66
ORA-06512: at line 1
We got the same error even with 1 million rows.
We do have the following configuration:
SGA - 8.2 GB
PGA
- Aggregate Target - 3GB
- Current Allocated - 439444KB (439 MB)
- Maximum allocated - 2695753 KB (2.6 GB)
Temp Table Space - 60.9 GB (Total)
- 20 GB (Available approximately)
I think we do have more than enough memory to process the 1 million rows!!
Also, some times the same program results in the following error:
SQL> exec bulkload
BEGIN bulkload; END;
ERROR at line 1:
ORA-03113: end-of-file on communication channel
We did not even attempt the full load. Also, we are not using the PARALLEL option yet.
Are we hitting any bug here? Or PL/SQL is not capable of mass loads? I would appreciate any thoughts on this?
Thanks,
Haranadh
Following is the code:
set echo off
set timing on
create or replace procedure bulkload as
-- SOURCE --
TYPE src_cpd_dt IS TABLE OF ima_ama_acct.cpd_dt%TYPE;
TYPE src_acqr_ctry_cd IS TABLE OF ima_ama_acct.acqr_ctry_cd%TYPE;
TYPE src_acqr_pcr_ctry_cd IS TABLE OF ima_ama_acct.acqr_pcr_ctry_cd%TYPE;
TYPE src_issr_bin IS TABLE OF ima_ama_acct.issr_bin%TYPE;
TYPE src_mrch_locn_ref_id IS TABLE OF ima_ama_acct.mrch_locn_ref_id%TYPE;
TYPE src_ntwrk_id IS TABLE OF ima_ama_acct.ntwrk_id%TYPE;
TYPE src_stip_advc_cd IS TABLE OF ima_ama_acct.stip_advc_cd%TYPE;
TYPE src_authn_resp_cd IS TABLE OF ima_ama_acct.authn_resp_cd%TYPE;
TYPE src_authn_actvy_cd IS TABLE OF ima_ama_acct.authn_actvy_cd%TYPE;
TYPE src_resp_tm_id IS TABLE OF ima_ama_acct.resp_tm_id%TYPE;
TYPE src_mrch_ref_id IS TABLE OF ima_ama_acct.mrch_ref_id%TYPE;
TYPE src_issr_pcr IS TABLE OF ima_ama_acct.issr_pcr%TYPE;
TYPE src_issr_ctry_cd IS TABLE OF ima_ama_acct.issr_ctry_cd%TYPE;
TYPE src_acct_num IS TABLE OF ima_ama_acct.acct_num%TYPE;
TYPE src_tran_cnt IS TABLE OF ima_ama_acct.tran_cnt%TYPE;
TYPE src_usd_tran_amt IS TABLE OF ima_ama_acct.usd_tran_amt%TYPE;
src_cpd_dt_array src_cpd_dt;
src_acqr_ctry_cd_array src_acqr_ctry_cd;
src_acqr_pcr_ctry_cd_array src_acqr_pcr_ctry_cd;
src_issr_bin_array src_issr_bin;
src_mrch_locn_ref_id_array src_mrch_locn_ref_id;
src_ntwrk_id_array src_ntwrk_id;
src_stip_advc_cd_array src_stip_advc_cd;
src_authn_resp_cd_array src_authn_resp_cd;
src_authn_actvy_cd_array src_authn_actvy_cd;
src_resp_tm_id_array src_resp_tm_id;
src_mrch_ref_id_array src_mrch_ref_id;
src_issr_pcr_array src_issr_pcr;
src_issr_ctry_cd_array src_issr_ctry_cd;
src_acct_num_array src_acct_num;
src_tran_cnt_array src_tran_cnt;
src_usd_tran_amt_array src_usd_tran_amt;
j number := 1;
CURSOR c1 IS
SELECT
cpd_dt,
acqr_ctry_cd ,
acqr_pcr_ctry_cd,
issr_bin,
mrch_locn_ref_id,
ntwrk_id,
stip_advc_cd,
authn_resp_cd,
authn_actvy_cd,
resp_tm_id,
mrch_ref_id,
issr_pcr,
issr_ctry_cd,
acct_num,
tran_cnt,
usd_tran_amt
FROM ima_ama_acct ima_ama_acct
ORDER BY issr_bin;
BEGIN
OPEN c1;
FETCH c1 bulk collect into
src_cpd_dt_array ,
src_acqr_ctry_cd_array ,
src_acqr_pcr_ctry_cd_array,
src_issr_bin_array ,
src_mrch_locn_ref_id_array,
src_ntwrk_id_array ,
src_stip_advc_cd_array ,
src_authn_resp_cd_array ,
src_authn_actvy_cd_array ,
src_resp_tm_id_array ,
src_mrch_ref_id_array ,
src_issr_pcr_array ,
src_issr_ctry_cd_array ,
src_acct_num_array ,
src_tran_cnt_array ,
src_usd_tran_amt_array ;
CLOSE C1;
FORALL j in 1 .. src_cpd_dt_array.count
INSERT INTO ima_dly_acct (
CPD_DT,
ACQR_CTRY_CD,
ACQR_TIER_CD,
ACQR_PCR_CTRY_CD,
ACQR_PCR_TIER_CD,
ISSR_BIN,
OWNR_BUS_ID,
USER_BUS_ID,
MRCH_LOCN_REF_ID,
NTWRK_ID,
STIP_ADVC_CD,
AUTHN_RESP_CD,
AUTHN_ACTVY_CD,
RESP_TM_ID,
PROD_REF_ID,
MRCH_REF_ID,
ISSR_PCR,
ISSR_CTRY_CD,
ACCT_NUM,
TRAN_CNT,
USD_TRAN_AMT)
VALUES (
src_cpd_dt_array(j),
src_acqr_ctry_cd_array(j),
null,
src_acqr_pcr_ctry_cd_array(j),
null,
src_issr_bin_array(j),
null,
null,
src_mrch_locn_ref_id_array(j),
src_ntwrk_id_array(j),
src_stip_advc_cd_array(j),
src_authn_resp_cd_array(j),
src_authn_actvy_cd_array(j),
src_resp_tm_id_array(j),
null,
src_mrch_ref_id_array(j),
src_issr_pcr_array(j),
src_issr_ctry_cd_array(j),
src_acct_num_array(j),
src_tran_cnt_array(j),
src_usd_tran_amt_array(j));
COMMIT;
END bulkload;
SHOW ERRORS
-----------------------------------------------------------------------------do you have a unique key available in the rows you are fetching?
It seems a cursor with 20 million rows that is as wide as all the columnsyou want to work with is a lot of memory for the server to use at once. You may be able to do this with parallel processing (dop over 8) and a lot of memory for the warehouse box (and the box you are extracting data from)...but is this the most efficient (and thereby fastest) way to do it?
What if you used a cursor to select a unique key only, and then during the cursor loop fetch each record, transform it, and insert it into the target?
Its a different way to do a lot at once, but it cuts down on the overall memory overhead for the process.
I know this isnt as elegant as a single insert to do it all at once, but sometimes trimming a process down so it takes less resources at any given moment is much faster than trying to do the whole thing at once.
My solution is probably biased by transaction systems, so I would be interested in what the data warehouse community thinks of this.
For example:
source table my_transactions (tx_seq_id number, tx_fact1 varchar2(10), tx_fact2 varchar2(20), tx_fact3 number, ...)
select a cursor of tx_seq_id only (even at 20 million rows this is not much)
you could then either use a for loop or even bulk collect into a plsql collection or table
then process individually like this:
procedure process_a_tx(p_tx_seq_id in number)
is
rTX my_transactions%rowtype;
begin
select * into rTX from my_transactions where tx_seq_id = p_tx_seq_id;
--modify values as needed
insert into my_target(a, b, c) values (rtx.fact_1, rtx.fact2, rtx.fact3);
commit;
exception
when others
rollback;
--write to a log or raise and exception
end process_a_tx;
procedure collect_tx
is
cursor tx is
select tx_seq_id from my_transactions;
begin
for rTx in cTx loop
process_a_tx(rtx.tx_seq_id);
end loop;
end collect_tx; -
Impdp imported several MB but still 0 out of Nn million rows?!?
Thank you in advance for looking into this. I look forward to any help and insight you may be able to provide.
Has anyone run into this issue when data pump shows "nMB but 0 out of millions of rows" NOT imported?!? There are two tables in my import that do this. I have also encountered this error as well ORA-39166: Object was not found. Which seemed to be circumvented ONLY after setting the characterset. But now it cannot import data for two tables.
Here is the background.
The import was sent to me in two files. The ddl in one dump and the data in another. So I had to first create the DDL then import the data.
Create the ddl parameterfile and import the metadata.
Found the tablespaces from the source db.
I created a ddl file via impdp. remapped the tablespaces, created the users.
Import the data.
I set the character set env variable and used the following settings in the parameter file.
CONTENT=all
DUMPFILE=DATA_PUMP_DIR:FILE.%U.dmp
LOGFILE=DATA_FILE_DIR:FILE_20131126.log
ENCRYPTION_PASSWORD='PASSWORD'
REMAP_SCHEMA=SCHEMA1:SCHEMA
JOB_NAME=SCHEMA_NAME
table_exists_action=append
FULL=N
PARALLEL=9
So I begin the import and this is the result : (more than one table is not importing I showed two importing in my example however.)
. . imported "SCHEMA"."TABLE_01":"SYS_P597103" 663.4 MB 0 out of 32084160 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P434333" 589.0 MB 0 out of 29672535 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P635480" 656.6 MB 0 out of 32553771 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P651574" 658.8 MB 0 out of 32960374 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P460604" 626.5 MB 0 out of 30213168 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P567240" 620.5 MB 0 out of 31071412 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P577162" 633.1 MB 0 out of 31488125 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P387361" 583.8 MB 0 out of 28632631 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P402882" 592.3 MB 0 out of 28999710 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P542284" 606.3 MB 0 out of 30659587 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P366463" 567.9 MB 0 out of 27642766 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P377498" 571.3 MB 0 out of 28178251 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P339562" 608.0 MB 0 out of 26744122 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P357772" 622.4 MB 0 out of 27225225 rows
. . imported "SCHEMA"."TABLE_02":"SYS_P578321" 234.1 MB 24498143 rows
. . imported "SCHEMA"."TABLE_02":"SYS_P578324" 240.2 MB 24498143 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P192761" 449.9 MB 0 out of 21476694 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P287781" 545.0 MB 0 out of 25607178 rows
. . imported "SCHEMA"."TABLE_01":"SYS_P336096" 558.2 MB 0 out of 26321351 rows
. . imported "SCHEMA"."TABLE_02":"SYS_P593408" 235.8 MB 24863112 rows
. . imported "SCHEMA"."TABLE_02":"SYS_P593423" 241.7 MB 24863112 rows
. . imported "SCHEMA"."TABLE_02":"SYS_P623103" 241.4 MB 25219732 rows
. . imported "SCHEMA"."TABLE_02":"SYS_P623104" 247.3 MB 25219732 rows
. . imported "SCHEMA"."TABLE_02":"SYS_P647262" 245.2 MB 25629915 rows
. . imported "SCHEMA"."TABLE_02":"SYS_P647342" 250.4 MB 25629915 rows
. . imported "SCHEMA"."TABLE_02":"SYS_P519883" 233.9 MB 23413438 rowsI responded to this post after I posted it because I found a solution. I'm not sure why I do not see it. Here it is:
1. Re: impdp imported several MB but still 0 out of Nn million rows?!?
OracleBarista Nov 30, 2013 9:43 PM (in response to OracleBarista)
interestingly enough i was able to apply the rows to both of these two tables if I did not set the NLS_LANG in the environment and import both of them together. That only worked for these two tables however.
I do see one thing I find interesting one table shows 73K rows but not data and zero count when I select rows. So obviously there is nothing there but I'm not sure why it's reporting rows. Any ideas?
Correct Answer
Helpful Answer
Here is my response to you (DK2010):
Sorry for the oversight should have been more thourough. The version the datapump export was sent to me in was 11.2.0.2 the version database I'm using is 11.2.0.1.
-- Futher response to my own post (pasted here). Further research proved my suspicion which was the table was not part of any DUMP file I was provided. I do not know why TOAD was showing it with 73K rows because SQL queries did not support that.
I believe this question can be closed.
Maybe you are looking for
-
How do I change the phone number where sms is sent to activate keychain?
How do I change the phone number where sms message is sent to activate Keychain? It keeps sendingit to an old phone #, and is most frustrating!
-
How do I install adobe flash 8?
How do I install adobe flash 8?
-
Hi All, in the JTree, is there a way to change the root handles icons, not through the UIManager? I need the Changes to apply only for the one instance. is there a key for the putClientProperty method? a work around through the TreeCellRenderer is po
-
Windows 10 Upgrade fails on HP ENVY x360 - 15-u110dx
I'm unable to upgrade to Windows 10, keep getting the following error 80240031, please help
-
Any possibility to assign a user to maintain his own data in SU01
Hi All, For all the developers we have assigned only display view to SU01. Can we give them access to su01 to change their own data (Only tabs Address, Logon Data and Tabs should be changeable. Tabs Roles, Profiles ahould not be changeable becoz. the