SGA: allocation forcing component growth
Oracle 11g - 11.1.0.7
Redht 5.3 4GB RAM
SGA MAX Size 1GB
SGA_target 800M
pga_aggregate_target = 600MB
========================
SQL> SHOW SGA:
Total system Global Area 1043886080
Fixed size: 2166288
Variable Size: 1002439152
Database Buffers: 33554432
redo Buffers 5726208
==========================
I was notified by 11g DB control that I need to increase my SGA size to 1000M (1GB) from 800M. I went into the server option/Intitialization file (I am using SPFILE) and only increased it by 50mb, making the SGA 850M total. I did not use the "impliment" option to increase the SGA to 1GB like 11g suggested - because normally, 800M suffices.
However, after I went into my initialization parameters, edited the "CURRENT' intit paramaters to reflect the new sga_target to be: 850MB vs. 800M, hit apply, and it just sits there waiting. I clicked on the 'Active Sessions' link on the main DB control page showing a wait activity, and I found the session that states: SGA: allocating forcing component growth." No error message, just sitting there trying to add the additonal 50MB to the current 800MB. I normally have no issues doing this, it recently started. Have I consumed all my availabe memory that Oracle can use?
Like I said, I only have 4GB in the box. Can I kill the session that is trying to execute the SQL to increase the SGA size?
Thanks,
Rob
Edited by: ronthe11gdba on May 7, 2009 11:27 AM
Edited by: ronthe11gdba on May 7, 2009 12:21 PM
Oracle 11g - 11.1.0.7
Redht 5.3 4GB RAM
SGA MAX Size 1GB
SGA_target 800M
pga_aggregate_target = 600MB
========================
SQL> SHOW SGA:
Total system Global Area 1043886080
Fixed size: 2166288
Variable Size: 1002439152
Database Buffers: 33554432
redo Buffers 5726208
==========================
I was notified by 11g DB control that I need to increase my SGA size to 1000M (1GB) from 800M. I went into the server option/Intitialization file (I am using SPFILE) and only increased it by 50mb, making the SGA 850M total. I did not use the "impliment" option to increase the SGA to 1GB like 11g suggested - because normally, 800M suffices.
However, after I went into my initialization parameters, edited the "CURRENT' intit paramaters to reflect the new sga_target to be: 850MB vs. 800M, hit apply, and it just sits there waiting. I clicked on the 'Active Sessions' link on the main DB control page showing a wait activity, and I found the session that states: SGA: allocating forcing component growth." No error message, just sitting there trying to add the additonal 50MB to the current 800MB. I normally have no issues doing this, it recently started. Have I consumed all my availabe memory that Oracle can use?
Like I said, I only have 4GB in the box. Can I kill the session that is trying to execute the SQL to increase the SGA size?
Thanks,
Rob
Edited by: ronthe11gdba on May 7, 2009 11:27 AM
Edited by: ronthe11gdba on May 7, 2009 12:21 PM
Similar Messages
-
Hi,
Is it possible to check component allocation at characteristic level other than 'plant'. e.g. sales org or customer
If yes, please let me know how to do it.
As per my understanding if allocation at the component is configured using other characteristic values, multi-level availability check ignores the components.
Regards,
RSHi,
I am trying to carry out MATP with allocations maintained at component level. Allocations are maintained in planning area at material - location combination.
The issue is, allocations maintained at parent material are getting considered but not at component level.
For your information, multilevel check functionality in APO availability check is getting triggered when I create sales order in ECC for parent materail i.e. I can see the navigation structure and missing parts list.
But the purpose is to restrict confirmations at parent material based on allocation restrictions at component level.
Let me know if you need further inputs.
Regards,
RS -
When SGA is initialized and when it will be allocated ? Whether in nomount or mount stage?
For example,
If the SHARED_POOL size is greater than the SGA_MAX_SIZE , When the error will be thrown WHEN WE STARTUP THE DATABASE , whether in NOMOUNT or MOUNT STAGE ?.Sorry mate I don't understand your question?
OP? Operating from operating system? Sorry if I missed something.
The SGA may not get all of the memory it requests initially for example AIX says yep the memory is available but logically allocates the memory and even steals pages back afterwards if they are unused to reallocate to other processes.
There are options in AIX to stop stealing of memory from the SGA but AIX is very good at allocating pages and its a memory efficiency/performance feature.
Windows on the other hand I know less about. The parameter PRE_PAGE_SGA is worth reviewing depending on your operating system Windows, HP-UX, VMS, Solaris, Linux and IBM AIX. -
Hello folks,
We have a dataguard setup with primary database on 10.2.0.3 on AIX. I have kept all the parameters same in parameter file but querying SHOW SGA gives different results on primary database and DR database which causes ORA-04030 - Out of memory error. Can you please suggest some guidance?
Parameters on Primary:
SQL> show parameter db_cache
NAME TYPE VALUE
db_cache_advice string OFF
db_cache_size big integer 368M
SQL> show parameter shared_
NAME TYPE VALUE
hi_shared_memory_address integer 0
max_shared_servers integer
shared_memory_address integer 0
shared_pool_reserved_size big integer 5M
shared_pool_size big integer 224M
shared_server_sessions integer
shared_servers integer 0
SQL> show parameter sga_
NAME TYPE VALUE
sga_max_size big integer 2512M
sga_target big integer 2000M
SQL> show parameter pga_
NAME TYPE VALUE
pga_aggregate_target big integer 250M
SQL>
===========================
Parameters on DR:
SQL> show parameter db_cache
NAME TYPE VALUE
db_cache_advice string OFF
db_cache_size big integer 368M
SQL> show parameter shared_
NAME TYPE VALUE
hi_shared_memory_address integer 0
max_shared_servers integer
shared_memory_address integer 0
shared_pool_reserved_size big integer 5M
shared_pool_size big integer 224M
shared_server_sessions integer
shared_servers integer 0
SQL> show parameter sga_
NAME TYPE VALUE
sga_max_size big integer 2512M
sga_target big integer 2000M
SQL> show parameter pga_
NAME TYPE VALUE
pga_aggregate_target big integer 250M
SQL>
==================================
Primary SGA --
SQL> show sga
Total System Global Area 2634022912 bytes
Fixed Size 2075184 bytes
Variable Size 1761609168 bytes
Database Buffers 754974720 bytes
Redo Buffers 115363840 bytes
SQL>
DR SGA --
SQL> show sga
Total System Global Area 2634022912 bytes
Fixed Size 2075184 bytes
Variable Size 805307856 bytes
Database Buffers 1711276032 bytes
Redo Buffers 115363840 bytes
SQL>
Why primary and DR Variable size & Database buffers is different?Though my database is small and max 50 users connect at a time on primary. I got this error on DR and never received the same error or any memory error on primary site. that wonders me..!!I am not sure how good you explore below threads, but if these do not helps more then may be a bug and Oracle support is the solution :
ora-04030 error in dataguard environment
11g and ORA-04030 problems
Regards
Girish Sharma -
ORA 04030 Out of process memory error
Dear experts,
I know there are multiple discussions around this error and I have been reading through most of them in the past one week or so, but looks like we are running out of options or are missing the color altogether. Ok, we are getting ORA-04030 - out of process memory while allocating....while one of our batch process runs in the night. It simply tries to insert/update to a table. Our installation is 11.2.0.1.0 with no RAC configuration and on 64-bit AIX having 6 cores, 12 CPUs and 16 GB memory.
We have checked the Workarea_Size_Policy is set to be as Auto so Oracle decides how much memory to allocate to PGA automatically on run-time based on the demand. And based on the AWR report it doesnt look like we are anywhere near the country having a PGA-deficit!! I am attaching the AWR report in a word document here for your reference.
Also attached below are the configurations and the ulimit values.
IKBTRN1> show parameter workarea;
NAME TYPE VALUE
workarea_size_policy string AUTO
oraipeikbtrn1:/home/oracle-> ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 4194304
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited
threads(per process) unlimited
processes(per user) unlimited
Now, nothing seems to have contributed to the out of process memory issue from Oracle standpoint. I would be happy to be proved wrong here, if I am wrong.
So, whats going wrong here? A possible memory leak which we cannot zero down to, a OS memory limit or something else?
Seeking expert's advise on this, and also sincerely appreciate your time in looking at this.
Thanks.
P.S - I am pasting the whole AWR report since there is no 'upload file' option here that I can see.
WORKLOAD REPOSITORY report for
DB Name DB Id Instance Inst num Startup Time Release RAC
IKBTRN1 54659199 IKBTRN1 1 06-Jun-11 02:06 11.2.0.1.0 NO
Host Name Platform CPUs Cores Sockets Memory (GB)
oraipeikbtrn1.******.com AIX-Based Systems (64-bit) 12 6 16.00
Snap Id Snap Time Sessions Cursors/Session
Begin Snap: 5952 26-Aug-11 03:00:48 34 2.0
End Snap: 5953 26-Aug-11 04:00:52 32 1.9
Elapsed: 60.07 (mins)
DB Time: 1.93 (mins)
Report Summary
Cache Sizes
Begin End
Buffer Cache: 1,056M 704M Std Block Size: 8K
Shared Pool Size: 3,456M 3,456M Log Buffer: 7,184K
Load Profile
Load Profile
Per Second Per Transaction Per Exec Per Call
DB Time(s): 0.0 2.0 0.02 0.02
DB CPU(s): 0.0 0.5 0.00 0.00
Redo size: 556.1 34,554.8
Logical reads: 151.4 9,407.6
Block changes: 1.9 119.8
Physical reads: 14.2 882.6
Physical writes: 9.5 590.4
User calls: 1.8 112.8
Parses: 1.5 93.7
Hard parses: 0.1 8.9
W/A MB processed: -0.1 -6.9
Logons: 0.0 1.6
Executes: 1.9 115.4
Rollbacks: 0.0 0.0
Transactions: 0.0
Instance Efficiency Percentages (Target 100%)
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 96.63 In-memory Sort %: 99.97
Library Hit %: 95.68 Soft Parse %: 90.49
Execute to Parse %: 18.74 Latch Hit %: 100.00
Parse CPU to Parse Elapsd %: 57.23 % Non-Parse CPU: 86.28
Shared Pool Statistics
Begin End
Memory Usage %: 85.72 85.76
% SQL with executions>1: 93.91 96.66
% Memory for SQL w/exec>1: 89.07 87.04
Top 5 Timed Foreground Events
Event Waits Time(s) Avg wait (ms) % DB time Wait Class
DB CPU 29 24.66
db file scattered read 3,456 17 5 14.92 User I/O
db file sequential read 4,304 17 4 14.77 User I/O
direct path read temp 764 17 22 14.31 User I/O
direct path write temp 259 5 21 4.70 User I/O
Host CPU (CPUs: 12 Cores: 6 Sockets: )
Load Average Begin Load Average End %User %System %WIO %Idle
1.39 1.37 0.2 0.2 0.2 99.6
Instance CPU
%Total CPU %Busy CPU %DB time waiting for CPU (Resource Manager)
0.1 20.5 0.0
Memory Statistics
Begin End
Host Mem (MB): 16,384.0 16,384.0
SGA use (MB): 4,704.0 4,352.0
PGA use (MB): 196.1 188.4
% Host Mem used for SGA+PGA: 29.91 27.71
Main Report
• Report Summary
• Wait Events Statistics
• SQL Statistics
• Instance Activity Statistics
• IO Stats
• Buffer Pool Statistics
• Advisory Statistics
• Wait Statistics
• Undo Statistics
• Latch Statistics
• Segment Statistics
• Dictionary Cache Statistics
• Library Cache Statistics
• Memory Statistics
• Streams Statistics
• Resource Limit Statistics
• Shared Server Statistics
• init.ora Parameters
Back to Top
Wait Events Statistics
• Time Model Statistics
• Operating System Statistics
• Operating System Statistics - Detail
• Foreground Wait Class
• Foreground Wait Events
• Background Wait Events
• Wait Event Histogram
• Wait Event Histogram Detail (64 msec to 2 sec)
• Wait Event Histogram Detail (4 sec to 2 min)
• Wait Event Histogram Detail (4 min to 1 hr)
• Service Statistics
• Service Wait Class Stats
Back to Top
Time Model Statistics
• Total time in database user-calls (DB Time): 115.9s
• Statistics including the word "background" measure background process time, and so do not contribute to the DB time statistic
• Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 101.69 87.75
DB CPU 28.58 24.66
parse time elapsed 10.14 8.75
hard parse elapsed time 9.92 8.56
failed parse elapsed time 4.92 4.25
hard parse (sharing criteria) elapsed time 4.27 3.68
connection management call elapsed time 0.42 0.36
PL/SQL compilation elapsed time 0.34 0.30
PL/SQL execution elapsed time 0.18 0.15
sequence load elapsed time 0.00 0.00
repeated bind elapsed time 0.00 0.00
DB time 115.88
background elapsed time 86.01
background cpu time 5.06
Back to Wait Events Statistics
Back to Top
Operating System Statistics
• *TIME statistic values are diffed. All others display actual values. End Value is displayed if different
• ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name
Statistic Value End Value
NUM_LCPUS 0
NUM_VCPUS 0
AVG_BUSY_TIME 1,260
AVG_IDLE_TIME 360,705
AVG_IOWAIT_TIME 534
AVG_SYS_TIME 483
AVG_USER_TIME 679
BUSY_TIME 16,405
IDLE_TIME 4,329,811
IOWAIT_TIME 7,284
SYS_TIME 7,092
USER_TIME 9,313
LOAD 1 1
OS_CPU_WAIT_TIME 503,900
PHYSICAL_MEMORY_BYTES 17,179,869,184
NUM_CPUS 12
NUM_CPU_CORES 6
GLOBAL_RECEIVE_SIZE_MAX 1,310,720
GLOBAL_SEND_SIZE_MAX 1,310,720
TCP_RECEIVE_SIZE_DEFAULT 16,384
TCP_RECEIVE_SIZE_MAX 9,223,372,036,854,775,807
TCP_RECEIVE_SIZE_MIN 4,096
TCP_SEND_SIZE_DEFAULT 16,384
TCP_SEND_SIZE_MAX 9,223,372,036,854,775,807
TCP_SEND_SIZE_MIN 4,096
Back to Wait Events Statistics
Back to Top
Operating System Statistics - Detail
Snap Time Load %busy %user %sys %idle %iowait
26-Aug 03:00:48 1.39
26-Aug 04:00:52 1.37 0.38 0.21 0.16 99.62 0.17
Back to Wait Events Statistics
Back to Top
Foreground Wait Class
• s - second, ms - millisecond - 1000th of a second
• ordered by wait time desc, waits desc
• %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
• Captured Time accounts for 78.2% of Total DB time 115.88 (s)
• Total FG Wait Time: 62.08 (s) DB CPU time: 28.58 (s)
Wait Class Waits %Time -outs Total Wait Time (s) Avg wait (ms) %DB time
User I/O 8,949 0 56 6 48.74
DB CPU 29 24.66
System I/O 1,916 0 3 1 2.18
Other 506 88 1 2 0.92
Configuration 2 50 1 500 0.86
Commit 37 0 1 18 0.56
Application 20 0 0 17 0.29
Network 4,792 0 0 0 0.01
Concurrency 1 0 0 0 0.00
Back to Wait Events Statistics
Back to Top
Foreground Wait Events
• s - second, ms - millisecond - 1000th of a second
• Only events with Total Wait Time (s) >= .001 are shown
• ordered by wait time desc, waits desc (idle events last)
• %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn % DB time
db file scattered read 3,456 0 17 5 59.59 14.92
db file sequential read 4,304 0 17 4 74.21 14.77
direct path read temp 764 0 17 22 13.17 14.31
direct path write temp 259 0 5 21 4.47 4.70
control file sequential read 1,916 0 3 1 33.03 2.18
ADR block file read 38 0 1 28 0.66 0.92
log buffer space 2 50 1 500 0.03 0.86
log file sync 37 0 1 18 0.64 0.56
enq: RO - fast object reuse 14 0 0 24 0.24 0.29
local write wait 44 0 0 1 0.76 0.03
SQL*Net message to client 4,772 0 0 0 82.28 0.01
Disk file operations I/O 110 0 0 0 1.90 0.00
ADR block file write 7 0 0 0 0.12 0.00
SQL*Net message from client 4,773 0 15,396 3226 82.29
Streams AQ: waiting for messages in the queue 720 100 3,600 5000 12.41
Back to Wait Events Statistics
Back to Top
Background Wait Events
• ordered by wait time desc, waits desc (idle events last)
• Only events with Total Wait Time (s) >= .001 are shown
• %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn % bg time
control file sequential read 4,950 0 35 7 85.34 40.74
control file parallel write 1,262 0 31 25 21.76 36.46
log file parallel write 383 0 4 10 6.60 4.37
db file parallel write 627 0 2 3 10.81 2.36
change tracking file synchronous read 56 0 2 34 0.97 2.21
os thread startup 17 0 1 88 0.29 1.74
ADR block file read 135 0 1 7 2.33 1.04
change tracking file synchronous write 56 0 1 15 0.97 0.98
SGA: allocation forcing component growth 8 100 1 100 0.14 0.93
db file sequential read 112 0 1 6 1.93 0.75
process diagnostic dump 94 0 0 1 1.62 0.09
ADR block file write 92 0 0 1 1.59 0.07
LGWR wait for redo copy 11 0 0 1 0.19 0.01
log file sync 2 0 0 3 0.03 0.01
ADR file lock 92 22 0 0 1.59 0.01
Parameter File I/O 24 0 0 0 0.41 0.01
direct path write 6 0 0 1 0.10 0.00
Disk file operations I/O 54 0 0 0 0.93 0.00
rdbms ipc message 17,637 97 61,836 3506 304.09
Streams AQ: waiting for time management or cleanup tasks 5 60 11,053 2210602 0.09
DIAG idle wait 7,203 100 7,203 1000 124.19
PX Idle Wait 1,802 100 3,604 2000 31.07
pmon timer 1,212 99 3,603 2973 20.90
Space Manager: slave idle wait 726 99 3,603 4963 12.52
smon timer 12 100 3,600 300004 0.21
Streams AQ: qmn slave idle wait 128 0 3,583 27993 2.21
Streams AQ: qmn coordinator idle wait 256 50 3,583 13996 4.41
SQL*Net message from client 293 0 2 5 5.05
Back to Wait Events Statistics
Back to Top
Wait Event Histogram
• Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
• % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
• % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
• Ordered by Event (idle events last)
% of Waits
Event Total Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
ADR block file read 173 80.3 5.2 2.3 5.8 1.7 4.6
ADR block file write 99 96.0 3.0 1.0
ADR file lock 102 100.0
Disk file operations I/O 165 100.0
LGWR wait for redo copy 11 90.9 9.1
Parameter File I/O 24 100.0
SGA: allocation forcing component growth 8 100.0
SQL*Net break/reset to client 6 100.0
SQL*Net message to client 4992 100.0
SQL*Net more data from client 20 100.0
asynch descriptor resize 541 100.0
change tracking file synchronous read 56 83.9 1.8 14.3
change tracking file synchronous write 56 80.4 7.1 1.8 10.7
control file parallel write 1262 80.3 1.7 .6 .6 .8 1.3 14.7
control file sequential read 6866 94.1 .9 .7 .7 .3 .4 2.9
db file parallel write 628 94.3 2.1 1.0 .8 .3 .3 1.3
db file scattered read 3457 72.6 7.2 5.4 6.9 5.7 .5 1.6
db file sequential read 4525 78.7 2.7 1.8 9.6 5.3 .4 1.5
direct path read temp 764 40.2 18.6 9.4 6.2 11.0 5.8 8.9
direct path sync 1 100.0
direct path write 6 83.3 16.7
direct path write temp 259 .4 1.2 88.8 .4 9.3
enq: RO - fast object reuse 14 42.9 42.9 7.1 7.1
latch free 1 100.0
latch: cache buffers lru chain 2 100.0
latch: checkpoint queue latch 2 100.0
latch: messages 2 100.0
latch: object queue header operation 2 100.0
latch: redo allocation 1 100.0
latch: row cache objects 1 100.0
local write wait 44 100.0
log buffer space 2 50.0 50.0
log file parallel write 383 92.4 .8 1.0 5.7
log file sync 39 82.1 2.6 2.6 12.8
os thread startup 17 100.0
process diagnostic dump 94 34.0 63.8 2.1
reliable message 7 100.0
utl_file I/O 12 100.0
DIAG idle wait 7204 100.0
PX Idle Wait 1802 100.0
SQL*Net message from client 5067 87.1 6.6 1.0 .5 .5 .1 .5 3.7
Space Manager: slave idle wait 726 .6 99.4
Streams AQ: qmn coordinator idle wait 256 49.2 .8 50.0
Streams AQ: qmn slave idle wait 128 100.0
Streams AQ: waiting for messages in the queue 721 100.0
Streams AQ: waiting for time management or cleanup tasks 5 40.0 20.0 40.0
class slave wait 17 100.0
pmon timer 1212 .9 99.1
rdbms ipc message 17.6K 1.8 .4 .2 .2 .1 .1 21.0 76.2
smon timer 12 100.0
Back to Wait Events Statistics
Back to Top
I couldnt add the rest of the report here since it is telling me I have exceeded 30000 characters. If you want to see the full report, please email me at [email protected]Unless your database is strictly a DSS-type of database, your AWR report exposes loads of issues with it. And I think none of the time during the AWR window was spent on database. Look at the DB time (with all those multi cores) compared with the elapsed time of the AWR.
As you are on 11g, why not make use of MEMORY_TARGET (a single parameter to manage both SGA and PGA)? If you are already on it, ignore this as I can't see it anywhere. If not, get rid of SGA_TARGET and PGA_AGGREGATE_TARGET and replace it with a single MEMORY_TARGET parameter. However you may have a minimum threshold set for different SGA pools so that they won't shrink beyond that point.
Having said that, setting MEMORY_TARGET is not a guarantee to avoid ORA-4030. Just a single bad PL/SQL code could go and exploit the untunable part of your process memory and even go and blow up the physical memory. If you are using FORALL and BULK load, see if you can cut it down into few chunks rather than running as a single process.
What does your V$PGASTAT say? -
Automatic Parallelism causes Merge statement to take longer.
We have a problem in a new project as part of the ETL load into the Oracle datawarehouse we perform a merge statement to update rows in a global temporary table then load
the results into a permanant table, when testing with automatic parallel execution enabled the plan changes and the merge never finishes and consumes vast amounts of resources.
The database version is:-
Database version :11.2.0.3
OS: redhat 64bit
three node rac 20 cores per node
when executing serially the query response is typically similar to the following:
MERGE /*+ gather_plan_statistics no_parallel */ INTO T_GTTCHARGEVALUES USING
(SELECT
CASTACCOUNTID,
CHARGESCHEME,
MAX(CUMULATIVEVALUE) AS MAXMONTHVALUE,
MAX(CUMULATIVECOUNT) AS MAXMONTHCOUNT
FROM
V_CACHARGESALL
WHERE
CHARGEDATE >= TRUNC(TO_DATE(:B1,'YYYY-MM-DD'),'MM')
AND CHARGEDATE < TO_DATE(:B1,'YYYY-MM-DD')
GROUP BY
CASTACCOUNTID,
CHARGESCHEME
HAVING MAX(CUMULATIVECOUNT) IS NOT NULL ) MTOTAL
ON
(T_GTTCHARGEVALUES.CASTACCOUNTID=MTOTAL.CASTACCOUNTID AND
T_GTTCHARGEVALUES.CHARGESCHEME=MTOTAL.CHARGESCHEME)
WHEN MATCHED
THEN UPDATE SET
CUMULATIVEVALUE=CUMULATIVEVALUE+MTOTAL.MAXMONTHVALUE ,
CUMULATIVECOUNT=CUMULATIVECOUNT+MTOTAL.MAXMONTHCOUNT;
1448340 rows merged.
select * from table(dbms_xplan.display_cursor(null,null,'ALLSTATS LAST'));
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem |
| 0 | MERGE STATEMENT | | 1 | | 0 |00:03:08.43 | 2095K| 186K| | | |
| 1 | MERGE | T_GTTCHARGEVALUES | 1 | | 0 |00:03:08.43 | 2095K| 186K| | | |
| 2 | VIEW | | 1 | | 1448K|00:02:53.14 | 619K| 177K| | | |
|* 3 | HASH JOIN | | 1 | 1 | 1448K|00:02:52.70 | 619K| 177K| 812K| 812K| 1218K (0)|
| 4 | VIEW | | 1 | 1 | 203 |00:02:51.26 | 608K| 177K| | | |
|* 5 | FILTER | | 1 | | 203 |00:02:51.26 | 608K| 177K| | | |
| 6 | SORT GROUP BY | | 1 | 1 | 480 |00:02:51.26 | 608K| 177K| 73728 | 73728 | |
|* 7 | FILTER | | 1 | | 21M|00:02:56.04 | 608K| 177K| | | |
| 8 | PARTITION RANGE ITERATOR| | 1 | 392K| 21M|00:02:51.32 | 608K| 177K| | | |
|* 9 | TABLE ACCESS FULL | T_CACHARGES | 24 | 392K| 21M|00:02:47.48 | 608K| 177K| | | |
| 10 | TABLE ACCESS FULL | T_GTTCHARGEVALUES | 1 | 1451K| 1451K|00:00:00.48 | 10980 | 0 | | | |
Predicate Information (identified by operation id):
3 - access("T_GTTCHARGEVALUES"."CASTACCOUNTID"="MTOTAL"."CASTACCOUNTID" AND "T_GTTCHARGEVALUES"."CHARGESCHEME"="MTOTAL"."CHARGESCHEME")
5 - filter(MAX("CUMULATIVECOUNT") IS NOT NULL)
7 - filter(TRUNC(TO_DATE(:B1,'YYYY-MM-DD'),'fmmm')<TO_DATE(:B1,'YYYY-MM-DD'))
9 - filter(("LOGICALLYDELETED"=0 AND "CHARGEDATE">=TRUNC(TO_DATE(:B1,'YYYY-MM-DD'),'fmmm') AND "CHARGEDATE"<TO_DATE(:B1,'YYYY-MM-DD')))removing the no_parallel hint results in the following, (this is pulled from the sql monitoring report and editied to remove the lines relating to individual parallel servers)
I understand that the query is considered for parallel execution due to the estimated length of time it will run for and although the degree of parallleism seems excessive
it is the default maximum for the server configuration, what we are tryig to understand is which statistics could be inacurate or missing and could cause this kind of problem.
In this case we can add the no_parallel hint in the etl package as a workaround but would really liek to identify the root cause to avoid similar problems elsewhere.
SQL Monitoring Report
SQL Text
MERGE INTO T_GTTCHARGEVALUES USING (SELECT CASTACCOUNTID, CHARGESCHEME, MAX(CUMULATIVEVALUE) AS MAXMONTHVALUE,
MAX(CUMULATIVECOUNT) AS MAXMONTHCOUNT FROM V_CACHARGESALL WHERE CHARGEDATE >= TRUNC(TO_DATE(:B1,'YYYY-MM-DD'),'MM')
AND CHARGEDATE < to_date(:B1,'YYYY-MM-DD')
GROUP BY CASTACCOUNTID, CHARGESCHEME HAVING MAX(CUMULATIVECOUNT) IS NOT NULL ) MTOTAL
ON (T_GTTCHARGEVALUES.CASTACCOUNTID=MTOTAL.CASTACCOUNTID AND
T_GTTCHARGEVALUES.CHARGESCHEME=MTOTAL.CHARGESCHEME) WHEN MATCHED THEN UPDATE SET
CUMULATIVEVALUE=CUMULATIVEVALUE+MTOTAL.MAXMONTHVALUE ,
CUMULATIVECOUNT=CUMULATIVECOUNT+MTOTAL.MAXMONTHCOUNT
Error: ORA-1013
ORA-01013: user requested cancel of current operation
Global Information
Status : DONE (ERROR)
Instance ID : 1
Session : XXXX(2815:12369)
SQL ID : 70kzttjbyyspt
SQL Execution ID : 16777216
Execution Started : 04/27/2012 09:43:27
First Refresh Time : 04/27/2012 09:43:27
Last Refresh Time : 04/27/2012 09:48:43
Duration : 316s
Module/Action : SQL*Plus/-
Service : SYS$USERS
Program : sqlplus@XXXX (TNS V1-V3)
Binds
========================================================================================================================
| Name | Position | Type | Value |
========================================================================================================================
| :B1 | 1 | VARCHAR2(32) | 2012-04-25 |
========================================================================================================================
Global Stats
====================================================================================================================
| Elapsed | Queuing | Cpu | IO | Application | Concurrency | Cluster | Other | Buffer | Read | Read |
| Time(s) | Time(s) | Time(s) | Waits(s) | Waits(s) | Waits(s) | Waits(s) | Waits(s) | Gets | Reqs | Bytes |
====================================================================================================================
| 7555 | 0.00 | 4290 | 2812 | 0.08 | 27 | 183 | 243 | 3M | 294K | 7GB |
====================================================================================================================
SQL Plan Monitoring Details (Plan Hash Value=323941584)
==========================================================================================================================================================================================================
| Id | Operation | Name | Rows | Cost | Time | Start | Execs | Rows | Read | Read | Mem | Activity | Activity Detail |
| | | | (Estim) | | Active(s) | Active | | (Actual) | Reqs | Bytes | (Max) | (%) | (# samples) |
==========================================================================================================================================================================================================
| 0 | MERGE STATEMENT | | | | | | 1 | | | | | | |
| 1 | MERGE | T_GTTCHARGEVALUES | | | | | 1 | | | | | | |
| 2 | PX COORDINATOR | | | | 57 | +1 | 481 | 0 | 317 | 5MB | | 4.05 | latch: shared pool (40) |
| | | | | | | | | | | | | | os thread startup (17) |
| | | | | | | | | | | | | | Cpu (7) |
| | | | | | | | | | | | | | DFS lock handle (36) |
| | | | | | | | | | | | | | SGA: allocation forcing component growth (14) |
| | | | | | | | | | | | | | latch: parallel query alloc buffer (200) |
| 3 | PX SEND QC (RANDOM) | :TQ10003 | 1 | 19054 | | | | | | | | | |
| 4 | VIEW | | | | | | | | | | | | |
| 5 | FILTER | | | | | | | | | | | | |
| 6 | SORT GROUP BY | | 1 | 19054 | | | | | | | | | |
| 7 | PX RECEIVE | | 1 | 19054 | | | | | | | | | |
| 8 | PX SEND HASH | :TQ10002 | 1 | 19054 | | | 240 | | | | | | |
| 9 | SORT GROUP BY | | 1 | 19054 | 246 | +70 | 240 | 0 | | | 228M | 49.32 | Cpu (3821) |
| 10 | FILTER | | | | 245 | +71 | 240 | 3G | | | | 0.08 | Cpu (6) |
| 11 | HASH JOIN | | 1 | 19054 | 259 | +57 | 240 | 3G | | | 276M | 4.31 | Cpu (334) |
| 12 | PX RECEIVE | | 1M | 5 | 259 | +57 | 240 | 1M | | | | 0.04 | Cpu (3) |
| 13 | PX SEND HASH | :TQ10000 | 1M | 5 | 6 | +56 | 240 | 1M | | | | 0.01 | Cpu (1) |
| 14 | PX BLOCK ITERATOR | | 1M | 5 | 6 | +56 | 240 | 1M | | | | 0.03 | Cpu (1) |
| | | | | | | | | | | | | | PX Deq: reap credit (1) |
| 15 | TABLE ACCESS FULL | T_GTTCHARGEVALUES | 1M | 5 | 7 | +55 | 5486 | 1M | 5487 | 86MB | | 2.31 | gc cr grant 2-way (3) |
| | | | | | | | | | | | | | gc current block lost (7) |
| | | | | | | | | | | | | | Cpu (7) |
| | | | | | | | | | | | | | db file sequential read (162) |
| 16 | PX RECEIVE | | 78M | 19047 | 255 | +61 | 240 | 801K | | | | 0.03 | IPC send completion sync (2) |
| 17 | PX SEND HASH | :TQ10001 | 78M | 19047 | 250 | +66 | 240 | 3M | | | | 0.06 | Cpu (5) |
| 18 | PX BLOCK ITERATOR | | 78M | 19047 | 250 | +66 | 240 | 4M | | | | | |
| 19 | TABLE ACCESS FULL | T_CACHARGES | 78M | 19047 | 254 | +62 | 1016 | 4M | 288K | 6GB | | 37.69 | gc buffer busy acquire (104) |
| | | | | | | | | | | | | | gc cr block 2-way (1) |
| | | | | | | | | | | | | | gc cr block lost (9) |
| | | | | | | | | | | | | | gc cr grant 2-way (14) |
| | | | | | | | | | | | | | gc cr multi block request (1) |
| | | | | | | | | | | | | | gc current block 2-way (3) |
| | | | | | | | | | | | | | gc current block 3-way (2) |
| | | | | | | | | | | | | | gc current block busy (1) |
| | | | | | | | | | | | | | gc current grant busy (2) |
| | | | | | | | | | | | | | Cpu (58) |
| | | | | | | | | | | | | | latch: gc element (1) |
| | | | | | | | | | | | | | db file parallel read (26) |
| | | | | | | | | | | | | | db file scattered read (207) |
| | | | | | | | | | | | | | db file sequential read (2433) |
| | | | | | | | | | | | | | direct path read (1) |
| | | | | | | | | | | | | | read by other session (57) |
==========================================================================================================================================================================================================
Parallel Execution Details (DOP=240 , Servers Allocated=480)
Instances : 3chris_c wrote:
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem |
|* 9 | TABLE ACCESS FULL | T_CACHARGES | 24 | 392K| 21M|00:02:47.48 | 608K| 177K| | | |
Based on the discrepancy between the estimated number of rows and the actual, and the below posted bind value of 2012-04-25 i'd first be checking if the statistics on T_CACHARGES are up to date.
As a reference
http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:4399338600346902127
So that would be my first avenue of exploration.
Cheers, -
Here's is the contents of v$system_event..
Is this
EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT
enq: TX - index contention 40564851 214701526 5.29
enq: TX - row lock contention 188846 12454614 65.95
enq: SQ - contention 141971 70568 0.5
cause for concern?
EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT
SQL*Net message to client 6015051449 607254 0
SQL*Net message from client 6015048542 178177969892 29.62
gcs remote message 2948555287 2633481757 0.89
CGS wait for IPC msg 1517805027 634397 0
db file sequential read 1500615188 816364485 0.54
ges remote message 1247679701 1407300224 1.13
gc cr multi block request 778432813 9913464 0.01
gc current block 2-way 747852637 38030616 0.05
db file scattered read 709428365 460939295 0.65
rdbms ipc message 708473316 37650068633 53.14
gc buffer busy acquire 671285134 1033621285 1.54
PX Deq: reap credit 667784615 484449 0
gcs log flush sync 592376026 171712257 0.29
gc cr block 2-way 530861847 19607062 0.04
library cache pin 437937120 15126237 0.03
log file sync 379523272 797193932 2.1
DIAG idle wait 359607166 2822108755 7.85
log file parallel write 351225436 259263769 0.74
LNS ASYNC end of log 350170653 1398410516 3.99
LNS wait on SENDREQ 321652621 3209301 0.01
PX qref latch 297396661 94308 0
read by other session 289140108 148440270 0.51
buffer deadlock 163505781 983055 0.01
gc current block busy 119223825 467716658 3.92
PX Deq: Table Q Normal 117332841 23574867 0.2
ksxr poll remote instances 110480324 90333 0
buffer busy waits 106938153 19933900 0.19
direct path read 93429599 108427028 1.16
SQL*Net more data from client 86471785 23026529 0.27
gc current grant busy 84978157 28215346 0.33
control file sequential read 82646297 23694583 0.29
PX Deq Credit: send blkd 78641669 9569299 0.12
latch: cache buffers chains 74218671 690277 0.01
gc current grant 2-way 72557796 1920419 0.03
library cache: mutex X 71106697 75993 0
DFS lock handle 70722498 2716407 0.04
gc cr grant 2-way 64558237 1633004 0.03
PX Deq: Execution Msg 61706261 314222076 5.09
gc cr block busy 61469863 119850802 1.95
library cache lock 52428649 3773354 0.07
PX Deq: Slave Session Stats 48040224 1886805 0.04
db file parallel read 46415188 118467902 2.55
IPC send completion sync 46250594 965101 0.02
enq: TX - index contention 40564851 214701526 5.29
PX Deq: Execute Reply 39689685 17243721 0.43
gc buffer busy release 36976909 242714774 6.56
SQL*Net more data to client 36627952 44167 0
PX Deq: Msg Fragment 30501244 343397 0.01
rdbms ipc reply 29725302 1352370 0.05
RMAN backup & recovery I/O 28824547 37722662 1.31
reliable message 27892263 3082134 0.11
PX Idle Wait 27356097 4651277341 170.03
ASM file metadata operation 25098749 8850323 0.35
gc object scan 22705857 7485 0
db file parallel write 19896252 52152606 2.62
latch: ges resource hash list 19336183 427451 0.02
enq: PS - contention 19143961 707455 0.04
PX Deq: Parse Reply 19093356 895799 0.05
gc cr disk read 17816846 448909 0.03
ASM background timer 16101806 1383957874 85.95
PX Deq: Slave Join Frag 16044789 233149 0.01
wait for unread message on broadcast channel 15056320 1413552546 93.88
cursor: mutex X 13435193 24140 0
KJC: Wait for msg sends to complete 13268497 11397 0
PX Deq: Signal ACK RSG 13214824 101941 0.01
KSV master wait 13206286 4235645 0.32
direct path read temp 12617694 5487608 0.43
PX Deq Credit: need buffer 11675868 879967 0.08
row cache lock 11536185 398216 0.03
PX Deq Credit: Session Stats 9480862 78910 0.01
SQL*Net message to dblink 9312894 1538 0
SQL*Net message from dblink 9312894 6279631 0.67
control file parallel write 7760982 11854435 1.53
pmon timer 7558889 1412576090 186.88
PX Deq: Join ACK 7548816 498931 0.07
gc current multi block request 6035173 155898 0.03
PING 5706961 1413230267 247.63
enq: XR - database force logging 4662671 198813 0.04
class slave wait 4561877 7097429006 1555.81
Streams AQ: waiting for messages in the queue 4495828 1543411682 343.3
SQL*Net more data from dblink 3696582 444575 0.12
LGWR wait for redo copy 3655353 17840 0
log file sequential read 3387305 6610414 1.95
Log archive I/O 2990486 276772 0.09
SQL*Net break/reset to client 2971976 2385935 0.8
direct path write temp 2839390 2522114 0.89
Space Manager: slave idle wait 2827526 1412987186 499.73
latch: shared pool 2808517 298150 0.11
latch: gc element 2421717 24688 0.01
SGA: MMAN sleep for component shrink 2336447 2458094 1.05
latch: enqueue hash chains 2279645 15435 0.01
latch free 2089418 78732 0.04
gc current split 2044784 1864009 0.91
PX Deq: Signal ACK EXT 1976164 19263 0.01
enq: FB - contention 1473469 61036 0.04
cursor: pin S wait on X 1313129 1464789 1.12
SQL*Net more data to dblink 1232891 986 0
Streams AQ: RAC qmn coordinator idle wait 1211300 788 0
enq: HW - contention 1175390 2077008 1.77
latch: session allocation 1167768 21883 0.02
Streams AQ: qmn coordinator idle wait 1144699 1412546634 1233.99
Streams AQ: qmn slave idle wait 1031585 2227183681 2158.99
lock deadlock retry 962937 4698 0
enq: CF - contention 956154 609647 0.64
latch: cache buffers lru chain 902764 37552 0.04
latch: object queue header operation 817911 27717 0.03
global enqueue expand wait 768633 654105 0.85
Data file init write 756191 329758 0.44
latch: gcs resource hash 647021 4147 0.01
local write wait 603007 286191 0.47
latch: row cache objects 599358 6453 0.01
ges lmd/lmses to freeze in rcfg - mrcvr 481759 156345 0.32
shared server idle wait 471190 1413238589 2999.3
enq: RF - DG Broker Current File ID 469833 23209 0.05
smon timer 432383 1411851085 3265.28
SGA: allocation forcing component growth 363333 379008 1.04
gc current retry 341104 1121252 3.29
enq: RF - synch: DG Broker metadata 319143 588290 1.84
enq: PG - contention 313659 14830 0.05
enq: TT - contention 260134 11207172 43.08
enq: KO - fast object checkpoint 236745 820808 3.47
dispatcher timer 236637 1413242481 5972.2
direct path write 231382 191008 0.83
cursor: pin S 229011 394 0
Streams AQ: waiting for time management or cleanup tasks 199981 1413148548 7066.41
enq: TX - row lock contention 188846 12454614 65.95
enq: TX - allocate ITL entry 153703 54252 0.35
enq: SQ - contention 141971 70568 0.5
ksdxexeother 141885 56 0
latch: redo allocation 138912 1858 0.01
recovery area: computing applied logs 126415 45925 0.36
gc current block congested 126318 21768 0.17
resmgr:cpu quantum 123074 151384 1.23
jobq slave wait 120678 35574221 294.79
Datapump dump file I/O 90431 9127 0.1
ges inquiry response 89402 4041 0.05
os thread startup 83809 222586 2.66
cr request retry 80062 71896 0.9
PX Deq: Table Q Sample 79665 133402 1.67
gc cr block congested 79026 14792 0.19
gc cr failure 77521 25019 0.32
enq: WF - contention 73983 825388 11.16
enq: TQ - TM contention 72871 3319 0.05
lock escalate retry 65714 1574 0.02
buffer exterminate 59775 64919 1.09
fbar timer 47136 1413183353 29980.98
log file switch completion 46911 452097 9.64
recovery area: computing obsolete files 45699 8547 0.19
enq: US - contention 40401 8805 0.22
enq: TM - contention 39149 5435032 138.83
library cache load lock 36311 382575 10.54
kjbdrmcvtq lmon drm quiesce: ping completion 31668 47443 1.5
enq: TD - KTF dump entries 31468 1424 0.05
enq: RO - fast object reuse 28422 31772 1.12
parallel recovery slave wait for change 27558 3163 0.11
name-service call wait 23694 181533 7.66
control file single write 22375 1624 0.07
kksfbc child completion 21239 106926 5.03
PX Deq: Table Q qref 19325 245 0.01
enq: TX - contention 18805 113253 6.02
latch: messages 17203 181 0.01
enq: RS - prevent file delete 16913 1013 0.06
enq: RS - prevent aging list update 15682 642 0.04
PX Deq: Table Q Get Keys 14322 42935 3
gc current grant congested 14292 2192 0.15
cursor: mutex S 13285 8 0
log file single write 13164 5371 0.41
latch: undo global data 12649 178 0.01
kksfbc research 11894 12680 1.07
parallel recovery slave idle wait 11193 5872 0.52
wait list latch free 11026 11794 1.07
enq: CT - state 11001 417 0.04
latch: checkpoint queue latch 10526 132 0.01
enq: PE - contention 10506 1139 0.11
ARCH wait on SENDREQ 9957 216480 21.74
gc cr grant congested 9465 1584 0.17
wait for scn ack 9377 3155 0.34
enq: TA - contention 8856 324 0.04
log buffer space 8777 89323 10.18
enq: TK - Auto Task Serialization 8542 343 0.04
enq: DR - contention 7842 323 0.04
process diagnostic dump 7707 2072 0.27
JOX Jit Process Sleep 7612 11286431 1482.72
enq: TC - contention 7357 340817 46.33
ges global resource directory to be frozen 7140 12299 1.72
enq: CO - master slave det 6850 312 0.05
enq: JS - job run lock - synchronize 6704 397 0.06
gcs drm freeze in enter server mode 6542 40742 6.23
enq: TS - contention 5959 89332 14.99
ARCH wait for archivelog lock 5600 36 0.01
PX Nsq: PQ load info query 5377 104798 19.49
db file single write 5373 3452 0.64
gc remaster 5315 50625 9.52
latch: parallel query alloc buffer 4939 1906 0.39
enq: TO - contention 4799 143 0.03
enq: AF - task serialization 4395 161 0.04
enq: PI - contention 4251 163 0.04
ges2 LMON to wake up LMD - mrcvr 4210 28 0.01
enq: DL - contention 3889 239 0.06
kjctssqmg: quick message send wait 3408 22 0.01
LNS wait on DETACH 3275 741 0.23
ksfd: async disk IO 3274 1 0
LNS wait on ATTACH 3273 51940 15.87
ARCH wait on DETACH 3231 714 0.22
ARCH wait on ATTACH 3226 43238 13.4
enq: BR - file shrink 2787 116 0.04
write complete waits 2631 1070 0.41
enq: MD - contention 2596 67 0.03
enq: WL - contention 2198 266518 121.25
single-task message 2098 25896 12.34
enq: OD - Serializing DDLs 2054 66 0.03
resmgr:internal state change 2001 14735 7.36
ARCH wait on c/f tx acquire 2 1751 175230 100.07
enq: WR - contention 1636 69 0.04
latch: cache buffer handles 1610 29 0.02
statement suspended, wait error to be cleared 1497 22626 15.11
Streams AQ: qmn coordinator waiting for slave to start 1214 678966 559.28
enq: PD - contention 1182 33 0.03
JS kgl get object wait 1096 4922 4.49
undo segment extension 1070 10065 9.41
PL/SQL lock timer 949 8739819 9209.5
enq: AE - lock 937 28 0.03
LGWR-LNS wait on channel 832 913 1.1
ges DFS hang analysis phase 2 acks 816 495 0.61
latch: redo writing 729 9 0.01
gc quiesce 665 564 0.85
enq: JS - queue lock 482 2111 4.38
PX Deq: Test for credit 442 13 0.03
enq: SS - contention 386 274 0.71
recovery area: computing dropped files 328 1400 4.27
recovery area: computing backed up files 328 496 1.51
ksdxexeotherwait 279 10592 37.97
log switch/archive 250 137570 550.28
gc domain validation 223 39964 179.21
auto-sqltune: wait graph update 195 96514 494.95
wait for a undo record 170 1214 7.14
parallel recovery coord send blocked 168 4 0.02
enq: JS - wdw op 168 3741 22.27
enq: KT - contention 165 5 0.03
switch logfile command 156 6290 40.32
gcs resource directory to be unfrozen 149 12839 86.17
Data Guard Broker Wait 139 10906 78.46
enq: SK - contention 129 4 0.03
enq: JS - job recov lock 128 4 0.03
gc cr block lost 125 6772 54.17
virtual circuit wait 122 3 0.03
ges LMON to get to FTDONE 100 187 1.87
enq: CU - contention 80 242 3.02
enq: JQ - contention 78 7 0.09
cursor: pin X 73 83 1.14
parallel recovery coord wait for reply 70 510 7.29
PX Deq: Txn Recovery Start 67 3436 51.29
SQL*Net break/reset to dblink 60 0 0
gc current block lost 57 2869 50.33
ges LMD suspend for testing event 51 709 13.89
inactive session 46 4550 98.91
recovery read 45 5 0.11
JS kill job wait 41 3548 86.53
enq: AS - service activation 40 1 0.03
enq: TL - contention 35 2 0.05
enq: UL - contention 34 524 15.42
gcs enter server mode 33 1559 47.24
wait for stopper event to be increased 30 218 7.27
enq: TQ - DDL contention 24 300 12.52
enq: MR - contention 21 1 0.03
ges reconfiguration to start 20 54 2.72
ges enter server mode 20 502 25.08
buffer latch 18 1337 74.26
enq: SR - contention 18 1 0.05
Streams: RAC waiting for inter instance ack 18 3748 208.21
enq: PR - contention 17 46 2.72
kupp process wait 16 166 10.39
checkpoint completed 15 3678 245.19
PX Deque wait 14 68 4.87
enq: BF - allocation contention 14 1 0.08
enq: XL - fault extent map 14 51 3.66
enq: FU - contention 14 17 1.18
enq: TH - metric threshold evaluation 13 114 8.78
enq: MW - contention 12 0 0.04
enq: DD - contention 10 0 0.04
process terminate 8 41 5.08
ges cgs registration 8 151 18.9
buffer resize 7 0 0
ktm: instance recovery 7 698 99.66
LNS wait on LGWR 6 0 0
ASM background starting 6 381 63.43
gc cr block 3-way 5 0 0.08
enq: PV - syncstart 5 9 1.74
Global transaction acquire instance locks 4 4 1.09
enq: RS - read alert level 4 0 0.04
LGWR wait on LNS 3 0 0
gc recovery 3 540 179.85
Streams AQ: enqueue blocked on low memory 3 544 181.2
DBWR range invalidation sync 3 17 5.83
enq: DM - contention 3 0 0.03
enq: RF - FSFO Observer Heartbeat 3 0 0.03
enq: JS - q mem clnup lck 3 0 0
DG Broker configuration file I/O 2 0 0
enq: RC - Result Cache: Contention 2 493 246.6
enq: KM - contention 2 0 0.03
enq: RT - contention 2 0 0.04
instance state change 2 0 0.12
kkdlgon 2 10 5.11
enq: TQ - INI contention 2 292 146.07
enq: JS - contention 2 0 0
ARCH wait for netserver start 1 400 400.02
log file switch (checkpoint incomplete) 1 3 3.44
JS coord start wait 1 50 50.09
ges lmd and pmon to attach 1 1 1.26
wait for tmc2 to complete 1 3 3.03
control file heartbeat 1 400 400.02
enq: SW - contention 1 0 0.04
enq: PW - perwarm status in dbw0 1 0 0.09
enq: FS - contention 1 0 0.04
enq: XR - quiesce database 1 0 0.04
enq: RS - write alert level 1 0 0.02
enq: CN - race with init 1 0 0.03
enq: FE - contention 1 4 3.77
Wait for shrink lock2 1 10 10.03
enq: IA - contention 1 0 0.02
enq: RF - atomicity 1 0 0.05
enq: RF - synchronization: aifo master 1 0 0.02
enq: RF - RF - Database Automatic Disable 1 0 0.06
enq: WP - contention 1 0 0.02
enq: TB - SQL Tuning Base Cache Load 1 0 0.05
enq: JS - evt notify 1 0 0.02Edited by: steffi on Mar 20, 2011 12:21 AM
Edited by: steffi on Mar 20, 2011 8:18 AM
Edited by: steffi on Mar 20, 2011 8:19 AMText can be formatted by tagging the beginning and end of the block of text with the code ta
\Formatted text goes here.
\Example:
This is formatted.When cutting and pasting text such as execution plans, excerpts from AWR reports, etc, it will maintain spacing and formatting, and make for much easier reading.
As to the content, well, dumping the contents of v$system_event is pretty close to useless.
As to the first three events you listed, 'enq: TX - index contention', 'enq: TX - row lock contention', 'enq: SQ - contention', well, all of those are easily tunable.
First, for 'enq: SQ - contention', check your sequences. Do you have any NOCACHE sequences? Or sequences with small caches?
As for 'enq: TX - row lock contention', well that's fairly self-explanatory. You have multiple sessions trying to lock the same row in the same table at the same time.
Last, 'enq: TX - index contention', that's non-row level contention on an index. For example, if you have a unique index, insert a row w/ column value 1, but don't commit, then try to insert that same value from another session.
But, before you do any of that, I think you need to clearly understand where the bottlenecks are. Try taking some AWR snapshots, about 5 minutes apart, when you're having performance problems. Look at the AWR report for that 5 minute period. In particular, look at your Top 5 timed events.
Hope that helps,
-Mark -
Hi All,
using 10gR2 on Sun-Solaris. Getting consistently "db file parallel read" over 35 ms as an average wait for the past few months. No performance issues as such.
Using RAID 1+0. DB Size is 2 TB. Transactions are OLTP/Batch
Is this metric high or normal. How to justify that..
Please advise. Statspack excerpt as below;
Database DB Id Instance Inst Num Startup Time Release RAC
~~~~~~~~ ----------- ------------ -------- --------------- ----------- ---
817713220 jdedb 1 20-Jan-13 12:04 10.2.0.4.0 NO
Host Name: AVIJDEDB Num CPUs: 32 Phys Memory (MB): 32,047
~~~~
Snapshot Snap Id Snap Time Sessions Curs/Sess Comment
~~~~~~~~ ---------- ------------------ -------- --------- -------------------
Begin Snap: 1112 04-Feb-13 11:00:03 542 21.8
End Snap: 1113 04-Feb-13 12:00:05 551 22.0
Elapsed: 60.03 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 20,960M 21,008M Std Block Size: 8K
Shared Pool Size: 1,424M 1,376M Log Buffer: 13,909K
Load Profile Per Second Per Transaction
~~~~~~~~~~~~ --------------- ---------------
Redo size: 63,672.15 3,896.95
Logical reads: 22,421.78 1,372.29
Block changes: 237.54 14.54
Physical reads: 8,628.59 528.10
Physical writes: 58.18 3.56
User calls: 1,438.67 88.05
Parses: 363.68 22.26
Hard parses: 0.71 0.04
Sorts: 111.80 6.84
Logons: 0.32 0.02
Executes: 904.41 55.35
Transactions: 16.34
% Blocks changed per Read: 1.06 Recursive Call %: 14.04
Rollback per transaction %: 2.36 Rows per Sort: 25.35
Instance Efficiency Percentages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 61.53 In-memory Sort %: 100.00
Library Hit %: 99.77 Soft Parse %: 99.80
Execute to Parse %: 59.79 Latch Hit %: 99.99
Parse CPU to Parse Elapsd %: 90.27 % Non-Parse CPU: 99.05
Shared Pool Statistics Begin End
Memory Usage %: 52.21 52.29
% SQL with executions>1: 85.63 87.51
% Memory for SQL w/exec>1: 82.23 86.58
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time
CPU time 8,073 61.6
db file sequential read 764,853 2,671 3 20.4
db file scattered read 487,789 1,661 3 12.7
db file parallel read 8,823 333 38 2.5
direct path write temp 5,478 91 17 .7
Host CPU (CPUs: 32)
~~~~~~~~ Load Average
Begin End User System Idle WIO WCPU
2.65 2.25 6.25 2.87 90.88 0.00 0.38
Instance CPU
~~~~~~~~~~~~
% of total CPU for Instance: 7.11
% of busy CPU for Instance: 77.95
%DB time waiting for CPU - Resource Mgr:
Memory Statistics Begin End
~~~~~~~~~~~~~~~~~ ------------ ------------
Host Mem (MB): 32,046.5 32,046.5
SGA use (MB): 22,528.0 22,528.0
PGA use (MB): 1,473.7 1,334.7
% Host Mem used for SGA+PGA: 74.9 74.5
Time Model System Stats -> Ordered by % of DB time desc, Statistic name
Statistic Time (s) % of DB time
sql execute elapsed time 7,939.4 90.8
DB CPU 5,811.5 66.4
parse time elapsed 183.9 2.1
hard parse elapsed time 109.4 1.3
connection management call elapsed 10.7 .1
PL/SQL execution elapsed time 4.2 .0
repeated bind elapsed time 2.0 .0
hard parse (sharing criteria) elaps 0.4 .0
PL/SQL compilation elapsed time 0.1 .0
hard parse (bind mismatch) elapsed 0.1 .0
sequence load elapsed time 0.0 .0
failed parse elapsed time 0.0 .0
DB time 8,746.6
background elapsed time 2,621.5
background cpu time 2,381.4
Wait Events
-> s - second, cs - centisecond, ms - millisecond, us - microsecond
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by Total Wait Time desc, Waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
db file sequential read 764,853 0 2,671 3 13.0
db file scattered read 487,789 0 1,661 3 8.3
db file parallel read 8,823 0 333 38 0.1
direct path write temp 5,478 0 91 17 0.1
SQL*Net more data to client 1,324,395 0 82 0 22.5
log file sync 57,449 0 67 1 1.0
log file parallel write 59,434 0 61 1 1.0
db file parallel write 19,065 0 27 1 0.3
direct path read 9,621 0 16 2 0.2
os thread startup 49 0 5 103 0.0
control file parallel write 1,706 0 3 2 0.0
control file sequential read 4,935 0 1 0 0.1
read by other session 691 0 1 2 0.0
direct path write 2,523 0 1 0 0.0
direct path read temp 1,218 0 1 0 0.0
cursor: pin S wait on X 32 100 0 10 0.0
SGA: MMAN sleep for component shr 87 8 0 3 0.0
SQL*Net break/reset to client 1,064 0 0 0 0.0
SGA: allocation forcing component 12 83 0 10 0.0
latch free 17 0 0 7 0.0
rdbms ipc reply 4 0 0 11 0.0
LGWR wait for redo copy 228 0 0 0 0.0
enq: HW - contention 4 0 0 4 0.0
latch: cache buffers lru chain 12 0 0 1 0.0
latch: shared pool 2 0 0 3 0.0
undo segment extension 403 100 0 0 0.0
buffer busy waits 36 0 0 0 0.0
enq: TX - row lock contention 2 0 0 1 0.0
SQL*Net message from client 5,041,043 0 1,604,869 318 85.7
Streams AQ: qmn slave idle wait 382 0 7,003 18332 0.0
virtual circuit status 121 100 3,540 29258 0.0
wait for unread message on broadc 3,599 100 3,514 976 0.1
Streams AQ: waiting for messages 720 100 3,513 4880 0.0
Streams AQ: qmn coordinator idle 259 50 3,501 13518 0.0
jobq slave wait 1,167 99 3,402 2916 0.0
Streams AQ: waiting for time mana 14 100 3,077 ###### 0.0
SQL*Net more data from client 185,834 0 15 0 3.2
SQL*Net message to client 5,041,055 0 10 0 85.7
------------------------------------------------------------->
>
using 10gR2 on Sun-Solaris. Getting consistently "db file parallel read" over 35 ms as an average wait for the past few months. No performance issues as such.
Using RAID 1+0. DB Size is 2 TB. Transactions are OLTP/Batch
Is this metric high or normal. How to justify that..
Looking at your results it's not really possible to say.
db file parallel read is a request for a number of randomly distirbuted blocks, and the time for a read is the time for the last block of the set to be returned.
Without knowing how many blocks are being requested at a time you can't really determine what constitutes a reasonable time. Given that you say OLTP + Batch, and have a large volume of scattered reads, it's quite possible that you have some queries on the Batch side doing very large index range scans - which would allow for some very large db file parallel reads.
I take it from the use of statspack that you're not licensed for the diagnostic and performance packs; it would be easy to query v$active_session_history to get some idea of the number of blocks per request as this is given by the P2 parameter. As it is. you may be able to get a rough idea by messing about with the various "physical read" numbers in the Instance Activity section of statspack.
Regards
Jonathan Lewis -
My wait events - can anyone see a problem?
hi,
this is what i have, can anyone see a problem?
thanks
EVENT TOTAL_WAITS PCT_WAITS TIME_WAIT_SEC PCT_TIME_WAITED TOTAL_TIMEOUTS PCT_TIMEOUTS AVERAGE_WAIT_SEC
Streams AQ: qmn slave idle wait 148147 .3 4051461.88 38.04 3478 .07 27.35
Streams AQ: qmn coordinator idle wa 291006 .59 3962890.53 37.21 148370 3.13 13.62
it
Streams AQ: waiting for time manage 948 0 2021434.2 18.98 948 .02 2132.31
ment or cleanup tasks
control file parallel write 1292057 2.64 266839.64 2.51 0 0 .21
log file parallel write 28433394 58.02 134658.55 1.26 0 0 0
db file sequential read 8307195 16.95 69830.07 .66 0 0 .01
free buffer waits 3117839 6.36 43374.04 .41 3106093 65.55 .01
log buffer space 55520 .11 20810.2 .2 20235 .43 .37
db file scattered read 583604 1.19 18169.58 .17 0 0 .03
write complete waits 17946 .04 17536.66 .16 17941 .38 .98
log file sync 282268 .58 10005.35 .09 9369 .2 .04
enq: RO - fast object reuse 26602 .05 6623.44 .06 2171 .05 .25
enq: CF - contention 1839 0 5178.14 .05 1723 .04 2.82
Streams AQ: qmn coordinator waiting 999 0 4311.01 .04 883 .02 4.32
for slave to start
buffer busy waits 32464 .07 3898.51 .04 3950 .08 .12
control file sequential read 2199199 4.49 3558.34 .03 0 0 0
SGA: MMAN sleep for component shrin 234330 .48 2523.65 .02 234216 4.94 .01
k
buffer exterminate 1583 0 1539.72 .01 1573 .03 .97
library cache pin 317 0 927.71 .01 316 .01 2.93
enq: CI - contention 1829 0 570.84 .01 159 0 .31
log file switch completion 1658 0 517.18 0 425 .01 .31
enq: TX - row lock contention 257 0 438.8 0 149 0 1.71
read by other session 27269 .06 355.17 0 52 0 .01
os thread startup 3869 .01 338.67 0 98 0 .09
latch: shared pool 760 0 285.87 0 0 0 .38
latch: row cache objects 664 0 250 0 0 0 .38
Data file init write 16324 .03 231.59 0 0 0 .01
reliable message 19189 .04 218.16 0 170 0 .01
latch: library cache 483 0 172.51 0 0 0 .36
SQL*Net message from dblink 1143086 2.33 128.69 0 0 0 0
latch free 6091 .01 121.1 0 0 0 .02
library cache load lock 90 0 89.48 0 18 0 .99
log file single write 1894 0 69.76 0 0 0 .04
cursor: pin S wait on X 5183 .01 55.87 0 5165 .11 .01
local write wait 6732 .01 42.58 0 2 0 .01
log file switch (checkpoint incompl 95 0 42.11 0 30 0 .44
ete)
row cache lock 119 0 30.96 0 10 0 .26
SQL*Net more data from dblink 17198 .04 25.92 0 0 0 0
log file switch (private strand flu 69 0 17.54 0 5 0 .25
sh incomplete)
enq: HW - contention 180 0 16.53 0 5 0 .09
enq: PR - contention 9 0 14.5 0 2 0 1.61
enq: JS - queue lock 51 0 12.36 0 0 0 .24
SQL*Net more data to client 48311 .1 11.66 0 0 0 0
enq: TM - contention 12 0 10.66 0 3 0 .89
class slave wait 3128 .01 7.03 0 1 0 0
JS coord start wait 68 0 6.42 0 68 0 .09
direct path write 92712 .19 6.06 0 0 0 0
control file heartbeat 1 0 3.91 0 1 0 3.91
PX Deq: Par Recov Execute 100 0 3.8 0 0 0 .04
log file sequential read 1900 0 2.88 0 0 0 0
single-task message 50 0 2.61 0 0 0 .05
enq: TX - contention 11 0 2.38 0 0 0 .22
undo segment extension 1181001 2.41 1.95 0 1180981 24.92 0
db file single write 165 0 1.3 0 0 0 .01
enq: TX - index contention 97 0 1.27 0 0 0 .01
LGWR wait for redo copy 20840 .04 .66 0 0 0 0
JS kgl get object wait 8 0 .63 0 8 0 .08
SQL*Net message to dblink 1143086 2.33 .55 0 0 0 0
kksfbc child completion 14 0 .55 0 11 0 .04
direct path read temp 217237 .44 .41 0 0 0 0
latch: cache buffers chains 2138 0 .37 0 0 0 0
latch: messages 1245 0 .27 0 0 0 0
latch: redo writing 786 0 .15 0 0 0 0
PX Deq: Par Recov Reply 65 0 .09 0 0 0 0
latch: checkpoint queue latch 171 0 .09 0 0 0 0
latch: redo allocation 1029 0 .08 0 0 0 0
latch: cache buffers lru chain 268 0 .07 0 0 0 0
SGA: allocation forcing component g 5 0 .05 0 2 0 .01
rowth
db file parallel read 83 0 .04 0 0 0 0
latch: In memory undo latch 558 0 .04 0 0 0 0
latch: object queue header operatio 338 0 .04 0 0 0 0
n
direct path read 5042 .01 .03 0 0 0 0
PX Deque wait 7 0 .02 0 0 0 0
direct path write temp 4691 .01 .02 0 0 0 0
enq: SQ - contention 1 0 .02 0 0 0 .02
latch: session allocation 190 0 .02 0 0 0 0
PX Deq: Join ACK 15 0 .01 0 0 0 0
cursor: pin S 894 0 .01 0 0 0 0
enq: TX - allocate ITL entry 37 0 .01 0 0 0 0
kkdlgon 15 0 .01 0 0 0 0
latch: enqueue hash chains 37 0 .01 0 0 0 0
library cache lock 1 0 .01 0 0 0 .01
Log archive I/O 1 0 0 0 0 0 0
PX Deq: Par Recov Change Vector 2 0 0 0 0 0 0
PX Deq: Signal ACK 3 0 0 0 0 0 0
PX Deq: Test for msg 1 0 0 0 0 0 0
PX qref latch 1 0 0 0 1 0 0
SQL*Net break/reset to dblink 5 0 0 0 0 0 0
SQL*Net more data to dblink 1 0 0 0 0 0 0
buffer deadlock 27 0 0 0 27 0 0
checkpoint completed 4 0 0 0 0 0 0
cursor: mutex S 3 0 0 0 0 0 0
cursor: mutex X 1 0 0 0 0 0 0
enq: JS - q mem clnup lck 1 0 0 0 0 0 0
enq: PS - contention 2 0 0 0 0 0 0
enq: US - contention 1 0 0 0 0 0 0
instance state change 2 0 0 0 0 0 0
latch: library cache lock 4 0 0 0 0 0 0
latch: library cache pin 1 0 0 0 0 0 0
latch: object queue header heap 8 0 0 0 0 0 0
latch: undo global data 3 0 0 0 0 0 0
recovery read 39 0 0 0 0 0 0Hi,
If its for a week than I won't bother. Probably you should try to get teh same report for these wait events in a much smaller period , like 20-30minutes of time period when your db is fully operational. If still at that time the wait events,these or any other, shoot up to high wait times, things can be investigated more deeply.
HTH
Aman.... -
Update Statement Simply hanged but doing db file sequential read
Hi,
Last night we had issue with one of the prod server where we updating one of table which contains large number records in millions.Same identical machine completed in1 hour and other box never completed but doing db file sequential read but in the long ops the last statement it was done 20:16 after that nothing is happening but i ran few trace on that user.
/u01/app/oracle/admin/SURV2/udump/surv2_ora_10048.trc
Oracle Database 10g Release 10.2.0.4.0 - Production
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
System name: SunOS
Node name: prdfa001
Release: 5.10
Version: Generic_139556-08
Machine: i86pc
Instance name: SURV2
Redo thread mounted by this instance: 1
Oracle process number: 18
Unix process pid: 10048, image: oracle@prdfa001
*** 2010-09-09 23:37:07.484
*** ACTION NAME:() 2010-09-09 23:37:07.473
*** MODULE NAME:(SQL*Plus) 2010-09-09 23:37:07.473
*** SERVICE NAME:(SURV2) 2010-09-09 23:37:07.473
*** SESSION ID:(289.54) 2010-09-09 23:37:07.473
Received ORADEBUG command 'unlimit' from process Unix process pid: 3983, image:
*** 2010-09-09 23:37:20.315
Received ORADEBUG command 'event 10046 trace name context forever, level 12' from process Unix process pid: 3983, image:
WAIT #7: nam='db file sequential read' ela= 11160 file#=13 block#=2252349 blocks=1 obj#=166421 tim=12499462835161
WAIT #7: nam='db file sequential read' ela= 2857 file#=13 block#=2249751 blocks=1 obj#=166421 tim=12499462838137
WAIT #7: nam='db file sequential read' ela= 3810 file#=13 block#=2251361 blocks=1 obj#=166421 tim=12499462842048
WAIT #7: nam='db file sequential read' ela= 4459 file#=13 block#=2247059 blocks=1 obj#=166421 tim=12499462846564
WAIT #7: nam='db file sequential read' ela= 2841 file#=13 block#=2247507 blocks=1 obj#=166421 tim=12499462849468
WAIT #7: nam='db file sequential read' ela= 427 file#=13 block#=2247568 blocks=1 obj#=166421 tim=12499462850032
WAIT #7: nam='db file sequential read' ela= 1187 file#=13 block#=2248264 blocks=1 obj#=166421 tim=12499462851327
WAIT #7: nam='db file sequential read' ela= 2687 file#=13 block#=2250707 blocks=1 obj#=166421 tim=12499462854178
WAIT #7: nam='db file sequential read' ela= 3657 file#=13 block#=2249697 blocks=1 obj#=166421 tim=12499462857896
WAIT #7: nam='db file sequential read' ela= 4139 file#=13 block#=2247074 blocks=1 obj#=166421 tim=12499462862093
WAIT #7: nam='db file sequential read' ela= 4180 file#=47 block#=3649690 blocks=1 obj#=166421 tim=12499509270445
WAIT #7: nam='db file sequential read' ela= 4802 file#=47 block#=3649309 blocks=1 obj#=166421 tim=12499509275327
WAIT #7: nam='db file sequential read' ela= 2459 file#=47 block#=3652697 blocks=1 obj#=166421 tim=12499509277859
WAIT #7: nam='db file sequential read' ela= 4015 file#=47 block#=3652826 blocks=1 obj#=166421 tim=12499509281948
WAIT #7: nam='db file sequential read' ela= 2248 file#=47 block#=3651610 blocks=1 obj#=166421 tim=12499509284269
WAIT #7: nam='db file sequential read' ela= 4824 file#=47 block#=3654297 blocks=1 obj#=166421 tim=12499509289166
WAIT #7: nam='db file sequential read' ela= 2008 file#=47 block#=3652312 blocks=1 obj#=166421 tim=12499509291248
WAIT #7: nam='db file sequential read' ela= 1925 file#=47 block#=3654490 blocks=1 obj#=166421 tim=12499509293246
WAIT #7: nam='db file sequential read' ela= 2859 file#=47 block#=3648458 blocks=1 obj#=166421 tim=12499509296178
WAIT #7: nam='db file sequential read' ela= 1740 file#=47 block#=3648212 blocks=1 obj#=166421 tim=12499509297991
WAIT #7: nam='db file sequential read' ela= 2566 file#=47 block#=3648411 blocks=1 obj#=166421 tim=12499509300631
WAIT #7: nam='db file sequential read' ela= 50772 file#=5 block#=480749 blocks=1 obj#=166421 tim=12499509351477
WAIT #7: nam='db file sequential read' ela= 12928 file#=5 block#=477177 blocks=1 obj#=166421 tim=12499509364482
WAIT #7: nam='db file sequential read' ela= 11116 file#=5 block#=479412 blocks=1 obj#=166421 tim=12499509375672
WAIT #7: nam='db file sequential read' ela= 4803 file#=5 block#=483440 blocks=1 obj#=166421 tim=12499509380549
WAIT #7: nam='db file sequential read' ela= 6900 file#=5 block#=481454 blocks=1 obj#=166421 tim=12499509387522
Received ORADEBUG command 'event 10046 trace name context off' from process Unix process pid: 3983, image:
/u01/app/oracle/admin/SURV2/udump/surv2_ora_1545.trc
Oracle Database 10g Release 10.2.0.4.0 - Production
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
System name: SunOS
Node name: prdfa001
Release: 5.10
Version: Generic_139556-08
Machine: i86pc
Instance name: SURV2
Redo thread mounted by this instance: 1
Oracle process number: 22
Unix process pid: 1545, image: oracle@prdfa001 (TNS V1-V3)
*** ACTION NAME:() 2010-09-09 23:20:13.485
*** MODULE NAME:(sqlplus@prdfa001 (TNS V1-V3)) 2010-09-09 23:20:13.485
*** SERVICE NAME:(SYS$USERS) 2010-09-09 23:20:13.485
*** SESSION ID:(290.697) 2010-09-09 23:20:13.485
===================================================
SYSTEM STATE
System global information:
processes: base 47819b480, size 300, cleanup 4781a5638
allocation: free sessions 47f1d6148, free calls 0
control alloc errors: 0 (process), 0 (session), 0 (call)
PMON latch cleanup depth: 0
seconds since PMON's last scan for dead processes: 20
system statistics:
1171 logons cumulative
19 logons current
89219 opened cursors cumulative
86 opened cursors current
15095069 user commits
5 user rollbacks
58632904 user calls
44023255 recursive calls
224311 recursive cpu usage
201424173 session logical reads
0 session stored procedure space
901812 CPU used when call started
995437 CPU used by this session
6814196 DB time
0 cluster wait time
22542300822 concurrency wait time
3095 application wait time
16479074661 user I/O wait time
1284052668 session connect time
1284067190 process last non-idle time
189018343568 session uga memory
1249667216 session uga memory max
26059216 messages sent
26059220 messages received
239739 background timeouts
162399896 session pga memory
189662872 session pga memory max
4 enqueue timeouts
901146 enqueue waits
0 enqueue deadlocks
32122711 enqueue requests
17819 enqueue conversions
32122676 enqueue releases
0 global enqueue gets sync
0 global enqueue gets async
0 global enqueue get time
0 global enqueue releases
2865667 physical read total IO requests
262620 physical read total multi block requests
270093476864 physical read total bytes
select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'SERVICE_NAME'), INSTANCE_NUMBER, STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN') from v$instance where INSTANCE_NAME=SYS_CONTEXT('USERENV', 'INSTANCE_NAME')
hash=550c95f3d0cfa8290e60ea8382d3a2ca timestamp=09-09-2010 04:24:19
namespace=CRSR flags=RON/KGHP/TIM/PN0/LRG/KST/DBN/MTX/[100100d1]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=9 hpc=0582 hlc=0582
lwt=47df576e8[47df576e8,47df576e8] ltm=47df576f8[47df576f8,47df576f8]
pwt=47df576b0[47df576b0,47df576b0] ptm=47df576c0[47df576c0,47df576c0]
ref=47df57718[47df57718,47df57718] lnd=47df57730[47df57730,47df57730]
LIBRARY OBJECT: object=471ee1d38
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
CHILDREN: size=16
child# table reference handle
0 471ee1800 471ee1470 47df7dce0
DATA BLOCKS:
data# heap pointer status pins change whr
0 47df7de48 471ee1e50 I/P/A/-/- 0 NONE 00
SO: 473691d60, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=473691d60 handle=47bb22fa0 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=473691de0[4735dbcb8,476cfbf58] htb=476cfbf58 ssga=476cfb6a0
user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x0
LIBRARY OBJECT HANDLE: handle=47bb22fa0 mtx=47bb230d0(0) cdp=0
namespace=CRSR flags=RON/KGHP/PN0/EXP/[10010100]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=fd84 hlc=fd84
lwt=47bb23048[47bb23048,47bb23048] ltm=47bb23058[47bb23058,47bb23058]
pwt=47bb23010[47bb23010,47bb23010] ptm=47bb23020[47bb23020,47bb23020]
ref=47bb23078[472f8de18,472f8de18] lnd=47bb23090[47bb23090,47bb23090]
LIBRARY OBJECT: object=472f8d9d8
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
DEPENDENCIES: count=1 size=16
AUTHORIZATIONS: count=1 size=16 minimum entrysize=16
ACCESSES: count=1 size=16
TRANSLATIONS: count=1 size=16
DATA BLOCKS:
data# heap pointer status pins change whr
0 47bb22ee0 472f8daf0 I/P/A/-/- 0 NONE 00
6 472f8e508 46be86250 I/-/A/-/E 0 NONE 00
SO: 4735dbc38, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=4735dbc38 handle=47bb231c8 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=4735dbcb8[476cfbf58,473691de0] htb=476cfbf58 ssga=476cfb6a0
user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x4c894f8b
LIBRARY OBJECT HANDLE: handle=47bb231c8 mtx=47bb232f8(1) cdp=1
name=select value$ from props$ where name = 'GLOBAL_DB_NAME'
hash=4bb432d65c5a391a42a5c3fa74472c7a timestamp=09-09-2010 04:24:12
namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/KST/DBN/MTX/[120100d0]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=0584 hlc=0584
lwt=47bb23270[47bb23270,47bb23270] ltm=47bb23280[47bb23280,47bb23280]
pwt=47bb23238[47bb23238,47bb23238] ptm=47bb23248[47bb23248,47bb23248]
ref=47bb232a0[47bb232a0,47bb232a0] lnd=47bb232b8[47bb232b8,47bb232b8]
LIBRARY OBJECT: object=472f8e6e0
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
CHILDREN: size=16
child# table reference handle
0 472f8e1a8 472f8de18 47bb22fa0
DATA BLOCKS:
data# heap pointer status pins change whr
0 47bb23108 472f8e7f8 I/P/A/-/- 0 NONE 00
SO: 473644348, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=473644348 handle=47bbde418 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=4736443c8[476cfc0b8,476cfc0b8] htb=476cfc0b8 ssga=476cfb6a0
user=47924e810 session=47924e810 count=1 flags=[0000] savepoint=0x4c894f8b
LIBRARY OBJECT HANDLE: handle=47bbde418 mtx=47bbde548(0) cdp=0
name=ALTER SESSION SET TIME_ZONE='+02:00'
hash=3878dff8839e71e3dd05a2e75fbd6390 timestamp=09-09-2010 04:24:04
namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/DBN/[12010040]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=11 hpc=04e8 hlc=04e8
lwt=47bbde4c0[47bbde4c0,47bbde4c0] ltm=47bbde4d0[47bbde4d0,47bbde4d0]
pwt=47bbde488[47bbde488,47bbde488] ptm=47bbde498[47bbde498,47bbde498]
ref=47bbde4f0[47bbde4f0,47bbde4f0] lnd=47bbde508[47bbde508,47bbde508]
LIBRARY OBJECT: object=472fffc08
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
DATA BLOCKS:
data# heap pointer status pins change whr
0 47bbde320 472fffd20 I/P/A/-/- 0 NONE 00
SO: 47aecf9e8, type: 41, owner: 47924e810, flag: INIT/-/-/0x00
(dummy) nxc=0, nlb=0
SO: 47f290540, type: 11, owner: 4781a7dc0, flag: INIT/-/-/0x00
(broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: 4781a7dc0,
event: 1132, last message event: 1132,
last message waited event: 1132, next message: 0(0), messages read: 0
channel: (47a2df4f8) system events broadcast channel
scope: 2, event: 1132, last mesage event: 18,
publishers/subscribers: 0/17,
messages published: 1
SO: 47826b228, type: 3, owner: 4781a7dc0, flag: INIT/-/-/0x00
(call) sess: cur 47924e810, rec 0, usr 47924e810; depth: 0
SO: 476c52968, type: 16, owner: 4781a7dc0, flag: INIT/-/-/0x00
(osp req holder)
PSEUDO PROCESS for group DEFAULT:
SO: 47a1eb7d0, type: 2, owner: 0, flag: INIT/-/-/0x00
(process) Oracle pid=0, calls cur/top: 0/0, flag: (20) PSEUDO
int error: 0, call error: 0, sess error: 0, txn error 0
(post info) last post received: 0 0 0
last post received-location: No post
last process to post me: none
last post sent: 0 0 0
last post sent-location: No post
last process posted by me: none
(latch info) wait_event=0 bits=0
Process Group: DEFAULT, pseudo proc: 47a1eb7d0
O/S info: user: , term: , ospid: (DEAD)
OSD pid info: Unix process pid: 0, image: PSEUDO
Dump of memory from 0x00000004791BF538 to 0x00000004791BF740
4791BF530 00000000 00000000 [........]
4791BF540 00000000 00000000 00000000 00000000 [................]
Repeat 31 times
NO DETACHED BRANCHES.
NO DETACHED NETWORK CONNECTIONS.
CLEANUP STATE OBJECTS:
SO: 47f0cd038, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: instance enqueue anchor state
latch: 0x380009890
SO: 4782cf080, type: 5, owner: 47f0cd038, flag: INIT/-/-/0x00
(enqueue) TA-00000006-00000001 DID: 0001-000F-0000000B
lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 res_flag: 0x2
res: 0x47a28d020, mode: X, lock_flag: 0x0
own: 0x0, sess: 0x0, prv: 0x47a28d030
SO: 47f0cd098, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: switchable channel handle anch
latch: 0x38000ac98
SO: 47f28f868, type: 11, owner: 47f0cd098, flag: INIT/-/-/0x00
(broadcast handle) flag: (c2) ACTIVE SUBSCRIBER, owner: 0,
event: 1, last message event: 1,
last message waited event: 1, next message: 0(0), messages read: 0
channel: (47a2e4190) KPON channel
scope: 2, event: 1, last mesage event: 0,
publishers/subscribers: 0/1,
messages published: 0
SO: 47f0cd0f8, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: TT shared object cleanup SO
latch: 0x38001c6b8
SO: 47f0cd158, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: SS shared object cleanup SO
latch: 0x38001cd48
END OF SYSTEM STATE
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
db file sequential read 2,347,652 9,215 4 64.5 User I/O
db file scattered read 245,687 4,199 17 29.4 User I/O
CPU time 974 6.8
db file parallel write 50,082 408 8 2.9 System I/O
log file parallel write 6,963 52 7 0.4 System I/O
Time Model Statistics DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> Total time in database user-calls (DB Time): 14286.4s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 14,280.3 100.0
DB CPU 974.5 6.8
PL/SQL execution elapsed time 531.8 3.7
parse time elapsed 30.5 .2
hard parse elapsed time 27.1 .2
connection management call elapsed time 14.9 .1
hard parse (sharing criteria) elapsed time 3.4 .0
hard parse (bind mismatch) elapsed time 3.1 .0
PL/SQL compilation elapsed time 2.4 .0
failed parse elapsed time 0.0 .0
repeated bind elapsed time 0.0 .0
sequence load elapsed time 0.0 .0
DB time 14,286.4 N/A
background elapsed time 670.2 N/A
background cpu time 186.1 N/A
Wait Class DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc
Avg
%Time Total Wait wait Waits
Wait Class Waits -outs Time (s) (ms) /txn
User I/O 2,593,484 .0 13,415 5 150.0
System I/O 87,506 .0 515 6 5.1
Other 839 11.4 6 7 0.0
Commit 3,225 .1 6 2 0.2
Concurrency 1,033 .0 5 5 0.1
Configuration 2,514 99.4 0 0 0.1
Network 47,559 .0 0 0 2.8
Application 7 .0 0 0 0.0
Wait Events DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
db file sequential read 2,347,652 .0 9,215 4 135.8
db file scattered read 245,687 .0 4,199 17 14.2
db file parallel write 50,082 .0 408 8 2.9
log file parallel write 6,963 .0 52 7 0.4
control file parallel write 6,203 .0 44 7 0.4
control file sequential read 24,242 .0 11 0 1.4
log file sync 3,225 .1 6 2 0.2
latch free 84 .0 4 47 0.0
os thread startup 25 .0 3 120 0.0
latch: session allocation 39 .0 1 33 0.0
db file parallel read 12 .0 1 92 0.0
enq: TX - index contention 186 .0 1 3 0.0
latch: shared pool 47 .0 1 11 0.0
LGWR wait for redo copy 319 3.1 0 1 0.0
library cache load lock 2 .0 0 172 0.0
buffer busy waits 590 .0 0 0 0.0
log file switch completion 6 .0 0 29 0.0
SGA: allocation forcing comp 11 54.5 0 14 0.0
latch: library cache lock 50 .0 0 3 0.0
read by other session 38 .0 0 4 0.0
direct path read 42 .0 0 3 0.0
SQL*Net message to client 44,807 .0 0 0 2.6
rdbms ipc reply 207 .0 0 0 0.0
SQL*Net more data from clien 1,014 .0 0 0 0.1
latch: cache buffers chains 24 .0 0 1 0.0
latch: library cache 29 .0 0 1 0.0
log file sequential read 8 .0 0 3 0.0
direct path write 50 .0 0 0 0.0
SQL*Net more data to client 398 .0 0 0 0.0
latch: object queue header o 12 .0 0 1 0.0
latch: In memory undo latch 78 .0 0 0 0.0
undo segment extension 2,507 99.7 0 0 0.1
latch: cache buffers lru cha 4 .0 0 1 0.0
log file single write 8 .0 0 0 0.0
local write wait 3 .0 0 1 0.0
enq: RO - fast object reuse 3 .0 0 1 0.0
buffer deadlock 87 92.0 0 0 0.0
enq: JS - queue lock 1 .0 0 1 0.0
cursor: pin S 70 .0 0 0 0.0
latch: row cache objects 2 .0 0 1 0.0
SQL*Net message to dblink 1,338 .0 0 0 0.1
latch: checkpoint queue latc 2 .0 0 0 0.0
reliable message 3 .0 0 0 0.0
log buffer space 1 .0 0 1 0.0
SQL*Net break/reset to clien 4 .0 0 0 0.0
SQL*Net more data from dblin 2 .0 0 0 0.0
SQL*Net message from client 44,949 .0 155,701 3464 2.6
virtual circuit status 621 100.0 18,156 29237 0.0
Streams AQ: qmn slave idle w 664 .0 18,127 27299 0.0
Streams AQ: qmn coordinator 1,339 50.4 18,099 13517 0.1
Streams AQ: waiting for time 12 100.0 8,741 728394 0.0
jobq slave wait 130 100.0 380 2927 0.0
PL/SQL lock timer 1 100.0 1 978 0.0
SQL*Net message from dblink 1,338 .0 0 0 0.1
single-task message 1 .0 0 38 0.0
class slave wait 11 .0 0 1 0.0
SQL ordered by Elapsed Time DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> Resources reported for PL/SQL code includes the resources used by all SQL
statements called by the code.
-> % Total DB Time is the Elapsed Time of the SQL statement divided
into the Total Database Time multiplied by 100
Elapsed CPU Elap per % Total
Time (s) Time (s) Executions Exec (s) DB Time SQL Id
13,664 906 0 N/A 95.6 gr2cx6athc5j5
Module: SQL*Plus
BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
8,792 195 0 N/A 61.5 986fzxtzr52u5
Module: SQL*Plus
UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
2,524 368 1 2524.1 17.7 c4uf0x6hdgnwq
Module: SQL*Plus
UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
SYS_B_1"
1,414 177 1 1414.4 9.9 cbg09ma34kq8w
Module: SQL*Plus
SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
742 137 1 742.2 5.2 g0sg6v994wssq
Module: SQL*Plus
SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
274 11 1 274.2 1.9 6mcpb06rctk0x
Module: DBMS_SCHEDULER
call dbms_space.auto_space_advisor_job_proc ( )
264 8 27 9.8 1.8 8szmwam7fysa3
Module: DBMS_SCHEDULER
insert into wri$_adv_objspace_trend_data select timepoint, space_usage, space_a
lloc, quality from table(dbms_space.object_growth_trend(:1, :2, :3, :4, NULL, N
ULL, NULL, 'FALSE', :5, 'FALSE'))
99 1 1 99.4 0.7 1z0x41f66nvjr
Module: SQL*Plus
UPDATE TIBEX_INSTRUMENTADMIN SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
SYS_B_1"
21 10 1 21.5 0.2 bbc1ck8594kvj
Module: SQL*Plus
UPDATE TIBEX_INSTRUMENTDAILYHIST SET ADJOPEN=NVL(ADJOPEN,OPEN), ADJHIGH=NVL(ADJH
IGH,HIGH), ADJLOW=NVL(ADJLOW,LOW), ADJMID=NVL(ADJMID,MID), ADJCLOSE=NVL(ADJCLOSE
,CLOSE), ADJVOLUME=NVL(ADJVOLUME,VOLUME), ADJCLOSINGBID=NVL(ADJCLOSINGBID,CLOSIN
GBID), ADJCLOSINGOFFER=NVL(ADJCLOSINGOFFER,CLOSINGOFFER)
12 0 1 12.5 0.1 6xm9p9uy5kaap
Module: SQL*Plus
SELECT count(*) FROM TIBEX_INSTRUMENTSTATE WHERE INSTRUMENTID=:"SYS_B_0"
SQL ordered by CPU Time DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> Resources reported for PL/SQL code includes the resources used by all SQL
statements called by the code.
-> % Total DB Time is the Elapsed Time of the SQL statement divided
into the Total Database Time multiplied by 100
CPU Elapsed CPU per % Total
Time (s) Time (s) Executions Exec (s) DB Time SQL Id
906 13,664 0 N/A 95.6 gr2cx6athc5j5
Module: SQL*Plus
BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
368 2,524 1 367.51 17.7 c4uf0x6hdgnwq
Module: SQL*Plus
UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
SYS_B_1"
195 8,792 0 N/A 61.5 986fzxtzr52u5
Module: SQL*Plus
UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
177 1,414 1 176.93 9.9 cbg09ma34kq8w
Module: SQL*Plus
SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
137 742 1 137.38 5.2 g0sg6v994wssq
Module: SQL*Plus
SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
11 274 1 10.82 1.9 6mcpb06rctk0x
Module: DBMS_SCHEDULER
call dbms_space.auto_space_advisor_job_proc ( )
10 21 1 9.65 0.2 bbc1ck8594kvjEdited by: NM on 10-Sep-2010 07:39Hi,
Last night we had issue with one of the prod server where we updating one of table which contains large number records in millions.Same identical machine completed in1 hour and other box never completed but doing db file sequential read but in the long ops the last statement it was done 20:16 after that nothing is happening but i ran few trace on that user.
/u01/app/oracle/admin/SURV2/udump/surv2_ora_10048.trc
Oracle Database 10g Release 10.2.0.4.0 - Production
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
System name: SunOS
Node name: prdfa001
Release: 5.10
Version: Generic_139556-08
Machine: i86pc
Instance name: SURV2
Redo thread mounted by this instance: 1
Oracle process number: 18
Unix process pid: 10048, image: oracle@prdfa001
*** 2010-09-09 23:37:07.484
*** ACTION NAME:() 2010-09-09 23:37:07.473
*** MODULE NAME:(SQL*Plus) 2010-09-09 23:37:07.473
*** SERVICE NAME:(SURV2) 2010-09-09 23:37:07.473
*** SESSION ID:(289.54) 2010-09-09 23:37:07.473
Received ORADEBUG command 'unlimit' from process Unix process pid: 3983, image:
*** 2010-09-09 23:37:20.315
Received ORADEBUG command 'event 10046 trace name context forever, level 12' from process Unix process pid: 3983, image:
WAIT #7: nam='db file sequential read' ela= 11160 file#=13 block#=2252349 blocks=1 obj#=166421 tim=12499462835161
WAIT #7: nam='db file sequential read' ela= 2857 file#=13 block#=2249751 blocks=1 obj#=166421 tim=12499462838137
WAIT #7: nam='db file sequential read' ela= 3810 file#=13 block#=2251361 blocks=1 obj#=166421 tim=12499462842048
WAIT #7: nam='db file sequential read' ela= 4459 file#=13 block#=2247059 blocks=1 obj#=166421 tim=12499462846564
WAIT #7: nam='db file sequential read' ela= 2841 file#=13 block#=2247507 blocks=1 obj#=166421 tim=12499462849468
WAIT #7: nam='db file sequential read' ela= 427 file#=13 block#=2247568 blocks=1 obj#=166421 tim=12499462850032
WAIT #7: nam='db file sequential read' ela= 1187 file#=13 block#=2248264 blocks=1 obj#=166421 tim=12499462851327
WAIT #7: nam='db file sequential read' ela= 2687 file#=13 block#=2250707 blocks=1 obj#=166421 tim=12499462854178
WAIT #7: nam='db file sequential read' ela= 3657 file#=13 block#=2249697 blocks=1 obj#=166421 tim=12499462857896
WAIT #7: nam='db file sequential read' ela= 4139 file#=13 block#=2247074 blocks=1 obj#=166421 tim=12499462862093
WAIT #7: nam='db file sequential read' ela= 4180 file#=47 block#=3649690 blocks=1 obj#=166421 tim=12499509270445
WAIT #7: nam='db file sequential read' ela= 4802 file#=47 block#=3649309 blocks=1 obj#=166421 tim=12499509275327
WAIT #7: nam='db file sequential read' ela= 2459 file#=47 block#=3652697 blocks=1 obj#=166421 tim=12499509277859
WAIT #7: nam='db file sequential read' ela= 4015 file#=47 block#=3652826 blocks=1 obj#=166421 tim=12499509281948
WAIT #7: nam='db file sequential read' ela= 2248 file#=47 block#=3651610 blocks=1 obj#=166421 tim=12499509284269
WAIT #7: nam='db file sequential read' ela= 4824 file#=47 block#=3654297 blocks=1 obj#=166421 tim=12499509289166
WAIT #7: nam='db file sequential read' ela= 2008 file#=47 block#=3652312 blocks=1 obj#=166421 tim=12499509291248
WAIT #7: nam='db file sequential read' ela= 1925 file#=47 block#=3654490 blocks=1 obj#=166421 tim=12499509293246
WAIT #7: nam='db file sequential read' ela= 2859 file#=47 block#=3648458 blocks=1 obj#=166421 tim=12499509296178
WAIT #7: nam='db file sequential read' ela= 1740 file#=47 block#=3648212 blocks=1 obj#=166421 tim=12499509297991
WAIT #7: nam='db file sequential read' ela= 2566 file#=47 block#=3648411 blocks=1 obj#=166421 tim=12499509300631
WAIT #7: nam='db file sequential read' ela= 50772 file#=5 block#=480749 blocks=1 obj#=166421 tim=12499509351477
WAIT #7: nam='db file sequential read' ela= 12928 file#=5 block#=477177 blocks=1 obj#=166421 tim=12499509364482
WAIT #7: nam='db file sequential read' ela= 11116 file#=5 block#=479412 blocks=1 obj#=166421 tim=12499509375672
WAIT #7: nam='db file sequential read' ela= 4803 file#=5 block#=483440 blocks=1 obj#=166421 tim=12499509380549
WAIT #7: nam='db file sequential read' ela= 6900 file#=5 block#=481454 blocks=1 obj#=166421 tim=12499509387522
Received ORADEBUG command 'event 10046 trace name context off' from process Unix process pid: 3983, image:
/u01/app/oracle/admin/SURV2/udump/surv2_ora_1545.trc
Oracle Database 10g Release 10.2.0.4.0 - Production
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
System name: SunOS
Node name: prdfa001
Release: 5.10
Version: Generic_139556-08
Machine: i86pc
Instance name: SURV2
Redo thread mounted by this instance: 1
Oracle process number: 22
Unix process pid: 1545, image: oracle@prdfa001 (TNS V1-V3)
*** ACTION NAME:() 2010-09-09 23:20:13.485
*** MODULE NAME:(sqlplus@prdfa001 (TNS V1-V3)) 2010-09-09 23:20:13.485
*** SERVICE NAME:(SYS$USERS) 2010-09-09 23:20:13.485
*** SESSION ID:(290.697) 2010-09-09 23:20:13.485
===================================================
SYSTEM STATE
System global information:
processes: base 47819b480, size 300, cleanup 4781a5638
allocation: free sessions 47f1d6148, free calls 0
control alloc errors: 0 (process), 0 (session), 0 (call)
PMON latch cleanup depth: 0
seconds since PMON's last scan for dead processes: 20
system statistics:
1171 logons cumulative
19 logons current
89219 opened cursors cumulative
86 opened cursors current
15095069 user commits
5 user rollbacks
58632904 user calls
44023255 recursive calls
224311 recursive cpu usage
201424173 session logical reads
0 session stored procedure space
901812 CPU used when call started
995437 CPU used by this session
6814196 DB time
0 cluster wait time
22542300822 concurrency wait time
3095 application wait time
16479074661 user I/O wait time
1284052668 session connect time
1284067190 process last non-idle time
189018343568 session uga memory
1249667216 session uga memory max
26059216 messages sent
26059220 messages received
239739 background timeouts
162399896 session pga memory
189662872 session pga memory max
4 enqueue timeouts
901146 enqueue waits
0 enqueue deadlocks
32122711 enqueue requests
17819 enqueue conversions
32122676 enqueue releases
0 global enqueue gets sync
0 global enqueue gets async
0 global enqueue get time
0 global enqueue releases
2865667 physical read total IO requests
262620 physical read total multi block requests
270093476864 physical read total bytes
select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'SERVICE_NAME'), INSTANCE_NUMBER, STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN') from v$instance where INSTANCE_NAME=SYS_CONTEXT('USERENV', 'INSTANCE_NAME')
hash=550c95f3d0cfa8290e60ea8382d3a2ca timestamp=09-09-2010 04:24:19
namespace=CRSR flags=RON/KGHP/TIM/PN0/LRG/KST/DBN/MTX/[100100d1]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=9 hpc=0582 hlc=0582
lwt=47df576e8[47df576e8,47df576e8] ltm=47df576f8[47df576f8,47df576f8]
pwt=47df576b0[47df576b0,47df576b0] ptm=47df576c0[47df576c0,47df576c0]
ref=47df57718[47df57718,47df57718] lnd=47df57730[47df57730,47df57730]
LIBRARY OBJECT: object=471ee1d38
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
CHILDREN: size=16
child# table reference handle
0 471ee1800 471ee1470 47df7dce0
DATA BLOCKS:
data# heap pointer status pins change whr
0 47df7de48 471ee1e50 I/P/A/-/- 0 NONE 00
SO: 473691d60, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=473691d60 handle=47bb22fa0 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=473691de0[4735dbcb8,476cfbf58] htb=476cfbf58 ssga=476cfb6a0
user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x0
LIBRARY OBJECT HANDLE: handle=47bb22fa0 mtx=47bb230d0(0) cdp=0
namespace=CRSR flags=RON/KGHP/PN0/EXP/[10010100]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=fd84 hlc=fd84
lwt=47bb23048[47bb23048,47bb23048] ltm=47bb23058[47bb23058,47bb23058]
pwt=47bb23010[47bb23010,47bb23010] ptm=47bb23020[47bb23020,47bb23020]
ref=47bb23078[472f8de18,472f8de18] lnd=47bb23090[47bb23090,47bb23090]
LIBRARY OBJECT: object=472f8d9d8
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
DEPENDENCIES: count=1 size=16
AUTHORIZATIONS: count=1 size=16 minimum entrysize=16
ACCESSES: count=1 size=16
TRANSLATIONS: count=1 size=16
DATA BLOCKS:
data# heap pointer status pins change whr
0 47bb22ee0 472f8daf0 I/P/A/-/- 0 NONE 00
6 472f8e508 46be86250 I/-/A/-/E 0 NONE 00
SO: 4735dbc38, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=4735dbc38 handle=47bb231c8 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=4735dbcb8[476cfbf58,473691de0] htb=476cfbf58 ssga=476cfb6a0
user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x4c894f8b
LIBRARY OBJECT HANDLE: handle=47bb231c8 mtx=47bb232f8(1) cdp=1
name=select value$ from props$ where name = 'GLOBAL_DB_NAME'
hash=4bb432d65c5a391a42a5c3fa74472c7a timestamp=09-09-2010 04:24:12
namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/KST/DBN/MTX/[120100d0]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=0584 hlc=0584
lwt=47bb23270[47bb23270,47bb23270] ltm=47bb23280[47bb23280,47bb23280]
pwt=47bb23238[47bb23238,47bb23238] ptm=47bb23248[47bb23248,47bb23248]
ref=47bb232a0[47bb232a0,47bb232a0] lnd=47bb232b8[47bb232b8,47bb232b8]
LIBRARY OBJECT: object=472f8e6e0
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
CHILDREN: size=16
child# table reference handle
0 472f8e1a8 472f8de18 47bb22fa0
DATA BLOCKS:
data# heap pointer status pins change whr
0 47bb23108 472f8e7f8 I/P/A/-/- 0 NONE 00
SO: 473644348, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=473644348 handle=47bbde418 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=4736443c8[476cfc0b8,476cfc0b8] htb=476cfc0b8 ssga=476cfb6a0
user=47924e810 session=47924e810 count=1 flags=[0000] savepoint=0x4c894f8b
LIBRARY OBJECT HANDLE: handle=47bbde418 mtx=47bbde548(0) cdp=0
name=ALTER SESSION SET TIME_ZONE='+02:00'
hash=3878dff8839e71e3dd05a2e75fbd6390 timestamp=09-09-2010 04:24:04
namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/DBN/[12010040]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=11 hpc=04e8 hlc=04e8
lwt=47bbde4c0[47bbde4c0,47bbde4c0] ltm=47bbde4d0[47bbde4d0,47bbde4d0]
pwt=47bbde488[47bbde488,47bbde488] ptm=47bbde498[47bbde498,47bbde498]
ref=47bbde4f0[47bbde4f0,47bbde4f0] lnd=47bbde508[47bbde508,47bbde508]
LIBRARY OBJECT: object=472fffc08
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
DATA BLOCKS:
data# heap pointer status pins change whr
0 47bbde320 472fffd20 I/P/A/-/- 0 NONE 00
SO: 47aecf9e8, type: 41, owner: 47924e810, flag: INIT/-/-/0x00
(dummy) nxc=0, nlb=0
SO: 47f290540, type: 11, owner: 4781a7dc0, flag: INIT/-/-/0x00
(broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: 4781a7dc0,
event: 1132, last message event: 1132,
last message waited event: 1132, next message: 0(0), messages read: 0
channel: (47a2df4f8) system events broadcast channel
scope: 2, event: 1132, last mesage event: 18,
publishers/subscribers: 0/17,
messages published: 1
SO: 47826b228, type: 3, owner: 4781a7dc0, flag: INIT/-/-/0x00
(call) sess: cur 47924e810, rec 0, usr 47924e810; depth: 0
SO: 476c52968, type: 16, owner: 4781a7dc0, flag: INIT/-/-/0x00
(osp req holder)
PSEUDO PROCESS for group DEFAULT:
SO: 47a1eb7d0, type: 2, owner: 0, flag: INIT/-/-/0x00
(process) Oracle pid=0, calls cur/top: 0/0, flag: (20) PSEUDO
int error: 0, call error: 0, sess error: 0, txn error 0
(post info) last post received: 0 0 0
last post received-location: No post
last process to post me: none
last post sent: 0 0 0
last post sent-location: No post
last process posted by me: none
(latch info) wait_event=0 bits=0
Process Group: DEFAULT, pseudo proc: 47a1eb7d0
O/S info: user: , term: , ospid: (DEAD)
OSD pid info: Unix process pid: 0, image: PSEUDO
Dump of memory from 0x00000004791BF538 to 0x00000004791BF740
4791BF530 00000000 00000000 [........]
4791BF540 00000000 00000000 00000000 00000000 [................]
Repeat 31 times
NO DETACHED BRANCHES.
NO DETACHED NETWORK CONNECTIONS.
CLEANUP STATE OBJECTS:
SO: 47f0cd038, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: instance enqueue anchor state
latch: 0x380009890
SO: 4782cf080, type: 5, owner: 47f0cd038, flag: INIT/-/-/0x00
(enqueue) TA-00000006-00000001 DID: 0001-000F-0000000B
lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 res_flag: 0x2
res: 0x47a28d020, mode: X, lock_flag: 0x0
own: 0x0, sess: 0x0, prv: 0x47a28d030
SO: 47f0cd098, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: switchable channel handle anch
latch: 0x38000ac98
SO: 47f28f868, type: 11, owner: 47f0cd098, flag: INIT/-/-/0x00
(broadcast handle) flag: (c2) ACTIVE SUBSCRIBER, owner: 0,
event: 1, last message event: 1,
last message waited event: 1, next message: 0(0), messages read: 0
channel: (47a2e4190) KPON channel
scope: 2, event: 1, last mesage event: 0,
publishers/subscribers: 0/1,
messages published: 0
SO: 47f0cd0f8, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: TT shared object cleanup SO
latch: 0x38001c6b8
SO: 47f0cd158, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: SS shared object cleanup SO
latch: 0x38001cd48
END OF SYSTEM STATE
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
db file sequential read 2,347,652 9,215 4 64.5 User I/O
db file scattered read 245,687 4,199 17 29.4 User I/O
CPU time 974 6.8
db file parallel write 50,082 408 8 2.9 System I/O
log file parallel write 6,963 52 7 0.4 System I/O
Time Model Statistics DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> Total time in database user-calls (DB Time): 14286.4s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 14,280.3 100.0
DB CPU 974.5 6.8
PL/SQL execution elapsed time 531.8 3.7
parse time elapsed 30.5 .2
hard parse elapsed time 27.1 .2
connection management call elapsed time 14.9 .1
hard parse (sharing criteria) elapsed time 3.4 .0
hard parse (bind mismatch) elapsed time 3.1 .0
PL/SQL compilation elapsed time 2.4 .0
failed parse elapsed time 0.0 .0
repeated bind elapsed time 0.0 .0
sequence load elapsed time 0.0 .0
DB time 14,286.4 N/A
background elapsed time 670.2 N/A
background cpu time 186.1 N/A
Wait Class DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc
Avg
%Time Total Wait wait Waits
Wait Class Waits -outs Time (s) (ms) /txn
User I/O 2,593,484 .0 13,415 5 150.0
System I/O 87,506 .0 515 6 5.1
Other 839 11.4 6 7 0.0
Commit 3,225 .1 6 2 0.2
Concurrency 1,033 .0 5 5 0.1
Configuration 2,514 99.4 0 0 0.1
Network 47,559 .0 0 0 2.8
Application 7 .0 0 0 0.0
Wait Events DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
db file sequential read 2,347,652 .0 9,215 4 135.8
db file scattered read 245,687 .0 4,199 17 14.2
db file parallel write 50,082 .0 408 8 2.9
log file parallel write 6,963 .0 52 7 0.4
control file parallel write 6,203 .0 44 7 0.4
control file sequential read 24,242 .0 11 0 1.4
log file sync 3,225 .1 6 2 0.2
latch free 84 .0 4 47 0.0
os thread startup 25 .0 3 120 0.0
latch: session allocation 39 .0 1 33 0.0
db file parallel read 12 .0 1 92 0.0
enq: TX - index contention 186 .0 1 3 0.0
latch: shared pool 47 .0 1 11 0.0
LGWR wait for redo copy 319 3.1 0 1 0.0
library cache load lock 2 .0 0 172 0.0
buffer busy waits 590 .0 0 0 0.0
log file switch completion 6 .0 0 29 0.0
SGA: allocation forcing comp 11 54.5 0 14 0.0
latch: library cache lock 50 .0 0 3 0.0
read by other session 38 .0 0 4 0.0
direct path read 42 .0 0 3 0.0
SQL*Net message to client 44,807 .0 0 0 2.6
rdbms ipc reply 207 .0 0 0 0.0
SQL*Net more data from clien 1,014 .0 0 0 0.1
latch: cache buffers chains 24 .0 0 1 0.0
latch: library cache 29 .0 0 1 0.0
log file sequential read 8 .0 0 3 0.0
direct path write 50 .0 0 0 0.0
SQL*Net more data to client 398 .0 0 0 0.0
latch: object queue header o 12 .0 0 1 0.0
latch: In memory undo latch 78 .0 0 0 0.0
undo segment extension 2,507 99.7 0 0 0.1
latch: cache buffers lru cha 4 .0 0 1 0.0
log file single write 8 .0 0 0 0.0
local write wait 3 .0 0 1 0.0
enq: RO - fast object reuse 3 .0 0 1 0.0
buffer deadlock 87 92.0 0 0 0.0
enq: JS - queue lock 1 .0 0 1 0.0
cursor: pin S 70 .0 0 0 0.0
latch: row cache objects 2 .0 0 1 0.0
SQL*Net message to dblink 1,338 .0 0 0 0.1
latch: checkpoint queue latc 2 .0 0 0 0.0
reliable message 3 .0 0 0 0.0
log buffer space 1 .0 0 1 0.0
SQL*Net break/reset to clien 4 .0 0 0 0.0
SQL*Net more data from dblin 2 .0 0 0 0.0
SQL*Net message from client 44,949 .0 155,701 3464 2.6
virtual circuit status 621 100.0 18,156 29237 0.0
Streams AQ: qmn slave idle w 664 .0 18,127 27299 0.0
Streams AQ: qmn coordinator 1,339 50.4 18,099 13517 0.1
Streams AQ: waiting for time 12 100.0 8,741 728394 0.0
jobq slave wait 130 100.0 380 2927 0.0
PL/SQL lock timer 1 100.0 1 978 0.0
SQL*Net message from dblink 1,338 .0 0 0 0.1
single-task message 1 .0 0 38 0.0
class slave wait 11 .0 0 1 0.0
SQL ordered by Elapsed Time DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> Resources reported for PL/SQL code includes the resources used by all SQL
statements called by the code.
-> % Total DB Time is the Elapsed Time of the SQL statement divided
into the Total Database Time multiplied by 100
Elapsed CPU Elap per % Total
Time (s) Time (s) Executions Exec (s) DB Time SQL Id
13,664 906 0 N/A 95.6 gr2cx6athc5j5
Module: SQL*Plus
BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
8,792 195 0 N/A 61.5 986fzxtzr52u5
Module: SQL*Plus
UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
2,524 368 1 2524.1 17.7 c4uf0x6hdgnwq
Module: SQL*Plus
UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
SYS_B_1"
1,414 177 1 1414.4 9.9 cbg09ma34kq8w
Module: SQL*Plus
SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
742 137 1 742.2 5.2 g0sg6v994wssq
Module: SQL*Plus
SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
274 11 1 274.2 1.9 6mcpb06rctk0x
Module: DBMS_SCHEDULER
call dbms_space.auto_space_advisor_job_proc ( )
264 8 27 9.8 1.8 8szmwam7fysa3
Module: DBMS_SCHEDULER
insert into wri$_adv_objspace_trend_data select timepoint, space_usage, space_a
lloc, quality from table(dbms_space.object_growth_trend(:1, :2, :3, :4, NULL, N
ULL, NULL, 'FALSE', :5, 'FALSE'))
99 1 1 99.4 0.7 1z0x41f66nvjr
Module: SQL*Plus
UPDATE TIBEX_INSTRUMENTADMIN SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
SYS_B_1"
21 10 1 21.5 0.2 bbc1ck8594kvj
Module: SQL*Plus
UPDATE TIBEX_INSTRUMENTDAILYHIST SET ADJOPEN=NVL(ADJOPEN,OPEN), ADJHIGH=NVL(ADJH
IGH,HIGH), ADJLOW=NVL(ADJLOW,LOW), ADJMID=NVL(ADJMID,MID), ADJCLOSE=NVL(ADJCLOSE
,CLOSE), ADJVOLUME=NVL(ADJVOLUME,VOLUME), ADJCLOSINGBID=NVL(ADJCLOSINGBID,CLOSIN
GBID), ADJCLOSINGOFFER=NVL(ADJCLOSINGOFFER,CLOSINGOFFER)
12 0 1 12.5 0.1 6xm9p9uy5kaap
Module: SQL*Plus
SELECT count(*) FROM TIBEX_INSTRUMENTSTATE WHERE INSTRUMENTID=:"SYS_B_0"
SQL ordered by CPU Time DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> Resources reported for PL/SQL code includes the resources used by all SQL
statements called by the code.
-> % Total DB Time is the Elapsed Time of the SQL statement divided
into the Total Database Time multiplied by 100
CPU Elapsed CPU per % Total
Time (s) Time (s) Executions Exec (s) DB Time SQL Id
906 13,664 0 N/A 95.6 gr2cx6athc5j5
Module: SQL*Plus
BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
368 2,524 1 367.51 17.7 c4uf0x6hdgnwq
Module: SQL*Plus
UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
SYS_B_1"
195 8,792 0 N/A 61.5 986fzxtzr52u5
Module: SQL*Plus
UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
177 1,414 1 176.93 9.9 cbg09ma34kq8w
Module: SQL*Plus
SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
137 742 1 137.38 5.2 g0sg6v994wssq
Module: SQL*Plus
SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
11 274 1 10.82 1.9 6mcpb06rctk0x
Module: DBMS_SCHEDULER
call dbms_space.auto_space_advisor_job_proc ( )
10 21 1 9.65 0.2 bbc1ck8594kvjEdited by: NM on 10-Sep-2010 07:39 -
JDBC, SQL*Net wait interface, performance degradation on 10g vs. 9i
Hi All,
I came across performance issue that I think results from mis-configuration of something between Oracle and JDBC. The logic of my system executes 12 threads in java. Each thread performs simple 'select a,b,c...f from table_xyz' on different tables. (so I have 12 different tables with cardinality from 3 to 48 millions and one working thread per table).
In each thread I'm creating result set that is explicitly marked as forward_only, transaction is set read only, fetch size is set to 100000 records. Java logic processes records in standard while(rs.next()) {...} loop.
I'm experiencing performance degradation between execution on Oracle 9i and Oracle 10g of the same java code, on the same machine, on the same data. The difference is enormous, 9i execution takes 26 hours while 10g execution takes 39 hours.
I have collected statspack for 9i and awr report for 10g. Below I've enclosed top wait events for 9i and 10g
===== 9i ===================
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
db file sequential read 22,939,988 0 6,240 0 0.7
control file parallel write 6,152 0 296 48 0.0
SQL*Net more data to client 2,877,154 0 280 0 0.1
db file scattered read 26,842 0 91 3 0.0
log file parallel write 3,528 0 83 23 0.0
latch free 94,845 0 50 1 0.0
process startup 93 0 5 50 0.0
log file sync 34 0 2 46 0.0
log file switch completion 2 0 0 215 0.0
db file single write 9 0 0 33 0.0
control file sequential read 4,912 0 0 0 0.0
wait list latch free 15 0 0 12 0.0
LGWR wait for redo copy 84 0 0 1 0.0
log file single write 2 0 0 18 0.0
async disk IO 263 0 0 0 0.0
direct path read 2,058 0 0 0 0.0
slave TJ process wait 1 1 0 12 0.0
===== 10g ==================
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
db file scattered read 268,314 .0 2,776 10 0.0
SQL*Net message to client 278,082,276 .0 813 0 7.1
io done 20,715 .0 457 22 0.0
control file parallel write 10,971 .0 336 31 0.0
db file parallel write 15,904 .0 294 18 0.0
db file sequential read 66,266 .0 257 4 0.0
log file parallel write 3,510 .0 145 41 0.0
SQL*Net more data to client 2,221,521 .0 102 0 0.1
SGA: allocation forcing comp 2,489 99.9 27 11 0.0
log file sync 564 .0 23 41 0.0
os thread startup 176 4.0 19 106 0.0
latch: shared pool 372 .0 11 29 0.0
latch: library cache 537 .0 5 10 0.0
rdbms ipc reply 57 .0 3 49 0.0
log file switch completion 5 40.0 3 552 0.0
latch free 4,141 .0 2 0 0.0
I put full blame for the slowdown on SQL*Net message to client wait event. All I could find about this event is that it is a network related problem. I assume it would be true if database and client were on different machines.. However in my case they are on them very same machine.
I'd be very very grateful if someone could point me in the right direction, i.e. give a hint what statistics should I analyze further? what might cause this event to appear? why probable cause (that is said be outside db) affects only 10g instance?
Thanks in advance,
Rafi.Hi Steven,
Thanks for the input. It's a fact that I did not gather statistics on my tables. My understanding is that statistics are useful for queries more complex than simple select * from table_xxx. In my case tables don't have indexes. There's no filtering condition as well. Full table scan is what I actually want as all software logic is inside the java code.
Explain plans are as follows:
======= 10g ================================
PLAN_TABLE_OUTPUT
Plan hash value: 1141003974
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 1 | 259 | 2 (0)| 00:00:01 |
| 1 | TABLE ACCESS FULL| xxx | 1 | 259 | 2 (0)| 00:00:01 |
In sqlplus I get:
SQL> set autotrace traceonly explain statistics;
SQL> select * from xxx;
36184384 rows selected.
Elapsed: 00:38:44.35
Execution Plan
Plan hash value: 1141003974
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 1 | 259 | 2 (0)| 00:00:01 |
| 1 | TABLE ACCESS FULL| xxx | 1 | 259 | 2 (0)| 00:00:01 |
Statistics
1 recursive calls
0 db block gets
3339240 consistent gets
981517 physical reads
116 redo size
26535700 bytes received via SQL*Net from client
2412294 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
36184384 rows processed
======= 9i =================================
PLAN_TABLE_OUTPUT
| Id | Operation | Name | Rows | Bytes | Cost |
| 0 | SELECT STATEMENT | | | | |
| 1 | TABLE ACCESS FULL | xxx | | | |
Note: rule based optimization
In sqlplus I get:
SQL> set autotrace traceonly explain statistics;
SQL> select * from xxx;
36184384 rows selected.
Elapsed: 00:17:43.06
Execution Plan
0 SELECT STATEMENT Optimizer=CHOOSE
1 0 TABLE ACCESS (FULL) OF 'xxx'
Statistics
0 recursive calls
1 db block gets
3306118 consistent gets
957515 physical reads
100 redo size
23659424 bytes sent via SQL*Net to client
26535867 bytes received via SQL*Net from client
2412294 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
36184384 rows processed
Thanks for pointing me difference in table scans. I infer that 9i is doing single-block full table scan (db file sequential read) while 10g is using multi-block full table scan (db file scattered read).
I now have theory that 9i is faster because sequential reads use continuous buffer space while scattered reads use discontinuous buffer space. Since I'm accessing data 'row by row' in jdbc 10g might have an overhead in providing data from discontinuous buffer space. This overhead shows itself as SQL*Net message to client wait. Is that making any sense?
Is there any way I could force 10g (i.e. with hint) to use sequential reads instead of scattered reads for full table scan?
I'll experiment with FTS tuning in 10g by enabling automatic multi-block reads tuning (i.e. db_file_multiblock_read_count=0 instead of 32 as it is now). I'll also check if response time improves after statistics are gathered.
Please advice if you have any other ideas.
Thanks & regards,
Rafi. -
Hi All,
System - Oracle 10.2.0.4 on SOLARIS 10. Application is JDE..(Lot of adhoc queries, lot of batch/ETL jobs at Night, normal OLTP)
Using RAID 1+0
In my AWR report for some time I notice the following;
1> High db file parallel read for past few months
2> High log file switch completion
3> Very low Buffer Hit Ratio from 38-55 %
Can anyone please explain on this; What needs to be done ?
AWR Report and Findings as below; Have posted only the high values
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 55.66 In-memory Sort %: 100.00
Library Hit %: 98.34 Soft Parse %: 99.25
Execute to Parse %: 57.56 Latch Hit %: 99.97
Parse CPU to Parse Elapsd %: 90.02 % Non-Parse CPU: 98.34
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
CPU time 4,472 67.5
db file parallel read 8,003 166 21 2.5 User I/O
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
db file parallel read 8,003 .0 166 21 0.0
lSQL*Net more data to client 834,185 .0 52 0 4.0
SQL*Net more data from clien 186,379 .0 16 0 0.9
log file sequential read 423 .0 4 8 0.0
cursor: pin S wait on X 50 100.0 0 10 0.0
SGA: allocation forcing comp 39 79.5 0 10 0.0
log file switch completion 9 .0 0 35 0.0
latch free 11 .0 0 8 0.0
latch: shared pool 11 .0 0 7 0.0
SQL*Net message from client 2,691,087 .0 1,471,156 547 12.8
Streams AQ: qmn slave idle w 385 .0 7,065 18351 0.0
virtual circuit status 121 100.0 3,542 29273 0.0
Streams AQ: qmn coordinator 264 50.4 3,532 13380 0.0
Streams AQ: waiting for mess 721 100.0 3,520 4882 0.0
wait for unread message on b 3,601 100.0 3,516 976 0.0
jobq slave wait 1,192 98.9 3,473 2914 0.0
Streams AQ: waiting for time 4 100.0 1,058 264428 0.0
class slave wait 2 .0 0 0 0.0
Background Wait Events DB/Inst: HQJDDB/hqjddb Snaps: 66170-66171
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
os thread startup 48 .0 4 83 0.0
log file sequential read 423 .0 4 8 0.0
control file parallel write 1,808 .0 3 2 0.0
latch: shared pool 1 .0 0 28 0.0
log file single write 6 .0 0 4 0.0
lStreams AQ: qmn slave idle w 385 .0 7,065 18351 0.0
Streams AQ: qmn coordinator 264 50.4 3,532 13380 0.0
smon timer 317 .0 3,507 11064 0.0
pmon timer 1,689 100.0 3,499 2071 0.0
Streams AQ: waiting for time 4 100.0 1,058 264428 0.0
init.ora Parameters
End value
Parameter Name Begin value (if different)
_addm_auto_enable TRUE
_optimizer_mjc_enabled FALSE
_spin_count 6000
db_32k_cache_size 4294967296
db_block_size 8192
db_keep_cache_size 1073741824
db_recovery_file_dest_size 2147483648
db_writer_processes 4
filesystemio_options ASYNCH
hash_area_size 20971520
job_queue_processes 10
large_pool_size 83886080
log_buffer 14242816
max_dump_file_size UNLIMITED
open_cursors 300
parallel_adaptive_multi_user FALSE
parallel_execution_message_si 8192
parallel_max_servers 640
parallel_threads_per_cpu 2
pga_aggregate_target 3221225472
processes 800
recyclebin ON
remote_login_passwordfile EXCLUSIVE
session_cached_cursors 300
sga_max_size 23622320128
sga_target 23622320128
undo_management AUTO
My log switches over past 2 weeks;;
Date Day 00:00 1am 2am 3am 4am 5am 6am 7am 8am 9am 10am 11am 12:00 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm 11pm
02-MAY-12 Wed 1 2 11 6 4 1 1 1 1 2 3 7 1
01-MAY-12 Tue 2 4 13 6 5 1 1 1 3 1 1 2 1 2 1
30-APR-12 Mon 1 2 10 8 3 1 1 2 3 2 6 2 1 2 2 2 4 2 2 1 2 1
29-APR-12 Sun 2 2 12 6 5 1 1 2 1 3 2 2 2 1 2 1 1 1 1
28-APR-12 Sat 1 3 15 4 3 8 5 2 1 2 2 4 3 1 2 1 2 1 3 2 4 1
27-APR-12 Fri 1 4 14 4 3 1 1 2 1 3 6 8 4 1 3 2 2 2 2 1 2 2
26-APR-12 Thu 2 3 11 5 5 1 1 16 6 4 2 1 1 5 5 5 2 1 1 2 1
25-APR-12 Wed 2 2 14 4 5 1 1 3 2 2 2 1 1 1 1 2 2 1 1 2 1
24-APR-12 Tue 2 5 12 5 4 1 1 2 2 2 3 1 4 2 2 2 2 1 2 2 1
23-APR-12 Mon 1 2 10 4 2 1 1 1 2 2 4 6 3 2 2 2 2 2 2 3 2 1
22-APR-12 Sun 2 2 12 4 2 1 1 1 39 1 1 2 2 1 2 1 1 1 1 1
21-APR-12 Sat 2 2 13 4 3 7 5 2 1 1 3 4 3 1 2 1 2 2 2 1 2 1
20-APR-12 Fri 1 3 11 3 3 2 1 1 2 3 1 2 1 4 2 3 3 2 1 2 2 1
19-APR-12 Thu 2 4 14 3 4 1 1 1 3 1 2 2 2 1 8 3 1 2 1
18-APR-12 Wed 2 2 11 3 4 1 1 1 2 3 6 8 1 2 2 3 2 2 1 1 2 1
My redo log file sizes;
SQL> select group#, members, bytes/1024/1024 "Size (MB)" from v$log;
GROUP# MEMBERS Size (MB)
1 1 150
2 1 150
3 1 150
4 1 150Iordan Iotzov wrote:
1> High db file parallel read for past few monthsWhat are the criteria for “high”? This event could show up during recovery and buffer pre-fetching. At 2.5% of your total call time, why do you think this event presents a systemic issue?
In general the response time has decreased. Is there any way to reduce the wait..
Did not find much info on it from Net..
2> High log file switch completion The report shows 9 waits each lasting around 35 ms., so I do not see this as a serious performance issue at this time. Many would find 38 log switches per hour too high. You could reduce that number by increasing the size of the redo log files. From the report I see, this would likely not help your performance now, but it could be a good preventive measure.
Thinking on the same line. But how to set the optimal size of redo logs..?
SQL> select optimal_logfile_size from v$instance_recovery;
OPTIMAL_LOGFILE_SIZE
-- NO DATA.
3> Very low Buffer Hit Ratio from 38-55 %
Buffer hit ratio is pretty much meaningless. This Jonathan Lewis’ blog entry sums it up nicely - http://jonathanlewis.wordpress.com/2007/09/05/hit-ratios-2/
Concur with you. But was looking into potential problems, since the BCHR decreased from a earlier value of 65-75% -
Oracle 8.1.7.4 32 bit and SGA size
hi gurus,
OS : solaris 5.8 64 bit
Oracle: 8.1.7.4 32 bit
i have seen documents saying that maximum SGA allocation possible is only 1.75 GB due to 32 bit application limitation.
is there a way in which we can allocate > 1.75 GB for the SGA granules?
any option like /AWE in windows?
thanks,
charlesOS : solaris 5.8 64 bit
Oracle: 8.1.7.4 32 bitAs far as i know mixing 32-bit database and 64-bit operating system is not supported - as 8i is.
Anyway with a 32-bit database you are basically limited to 2 GB per process. You can use "indirect_data_buffer" which will
enable you to increase your database block buffers > 2 GB....
Ronny Egner
My blog: http://blog.ronnyegner-consulting.de -
JSP 2.0 as JSF component encoders - continued
This is in continuation to this topic:
http://forum.java.sun.com/thread.jsp?forum=427&thread=414850&tstart=15&trange=15
As I look further into JSP 2.0, I am more and more sold to the idea of encoding JSF components thru JSP 2.0 tag files and/or SimpleTag handlers ...
Like, I was trying to figure out how could I derive the functionality of JSF facets from JSP tags - and the answer was amazingly straightforward - fragment attributes ... the tag could treat the fragments as facets while evaluating the body with its normal course ...
If the benefits of tag files/simple tag handlers are felt among the JSF team, I was wondering if the tag file/simple tag handler could be made part of the JSF rendererType paradigm?
I was wondering if it were a feasible idea to work in encoder-decoder pairs? The declarations in faces-config.xml for the renderers could be something along these lines:
If the encoding was with a simple tag handler:
<render-kit>
<renderer>
<renderer-type>Component1Renderer</renderer-type>
<decoder>decoder.Component1Decoder</decoder>
<encoder type="SimpleTagHandler">encoder.Component1EncoderTag</encoder>
</renderer>
</render-kit>
If the encoding was with a tag file:
<render-kit>
<renderer>
<renderer-type>Component1Renderer</renderer-type>
<decoder>decoder.Component1Decoder</decoder>
<encoder type="TagFile">/WEB-INF/tags/component1Encoder.tag</encoder>
</renderer>
</render-kit>
The above may all be very far fetched but I thought I'd just put across what I had in mind.
Thanks!This is in continuation to this topic:
http://forum.java.sun.com/thread.jsp?forum=427&thread=4
4850&tstart=15&trange=15
As I look further into JSP 2.0, I am more and more
sold to the idea of encoding JSF components thru JSP
2.0 tag files and/or SimpleTag handlers ...
As has been pointed out by others, this isn't feasible at all unless JavaServer Faces were to choose JSP 2.0 as the minimum platform. However, even if that were to happen, there are some crucial problems in at least two dimensions:
* JavaServer Faces integrates nicely with JSP, but does not
require it. Therefore, the encoding functionality of a
component (or renderer) needs to operate correctly even for
a non-JSP rendering environment. How can that be reconciled
with a desire to use JSP-specific rendering capabilities?
* JSP 2.0 is very cool, but it doesn't deal at all with the
issue of handling an incoming request. Therefore,
a solution that uses JSP as a templating environment (which
violates the "don't require JSP requirement as mentioned in
the previous paragraph"), will STILL require some non-JSP
functionality to do the decoding, validation, and event handling
parts of the application. To me, it does NOT seem like forcing
component writers to master two technologies (instead of one)
would make JavaServer Faces easier to use.
Implementation technology aside, the logic of decoding (what request parameters do I look at?) is very tightly coupled with the logic of encoding (what markup do I generate?). Splitting those two aspects of the problem into different source files, implemented with different technologies, and requiring changes in both placeds when you make changes, seems like something that goes against all the best practices design principles we've all been learning over the past several decades.
If JSP had a way to express the functional logic needed for handling events, validations, and calls to business logic, I might be more sympathetic to this approach -- but it doesn't. It is also interesting to note that Struts (among many other frameworks) has embedded markup generation for user interface components in Java logic (in the case of Struts, it's in the HTML tag library) from the very beginning. Why should it be OK there (and 75,000 downloads per month says it's pretty popular) and not in JavaServer Faces?
Further, why should a page author that is just using components give a rip what the implementation technology is? Isn't that a concern solely of the component writer? I would contend that anyone who builds a non-trivial component, in any technology framework, is going to need functional programming knowledge (be it Java, VB, C#, some XML-based variant, JavaScript, whatever) anyway. Rendering is about 20% of the problem -- the 80% that still needs to be solved is going to require programming knowledge anyway. And JSP, even with all the fantastic things added in 2.0, doesn't address any of those needs.
Craig McClanahan -
Hi All,
I'm running 10g on Linux. I'm trying to optimize the SGA in our environment. I occationally get alerts that our memory on the db server is at 99%. I'd like to not get those anymore. Currently i have the following set for our memory parameters:
Total SGA size: 850M
Max SGA size is 1.2G
PGA: 200M
Questions is this. When i boot up oracle with the above parameters does oracle use/claim the Max SGA size from the server or does it only take the Total SGA size.
The reason i ask is because i want to reduce the total memory used on the machine and hoping to do this by reducing the Total SGA size w/o rebooting the db. If i have to change the Max SGA then i have to reboot the db.
If i run a "top" command i see that oracle is using roughly 1.2G of virtual memory so it seems that oracle has the Max SGA allocated to it even though the Total SGA is only 850M. Does this make sense.
Basically if i want to reduce the memory usage on my machine do i reduce Max SGA or Total SGA parameter?
Thanks, daveThanks for the response Pierre. So if i run that i get:
oracle@db1:~$ ipcs -m | grep oracle
0x42d513ec 622593 oracle 640 1258295296 49
Shows 1200M in the shared memory segment. That tells me that the Max SGA setting dictates how much memory is used on the server. Is that right?
My Max is set to 1200M
My Total is set to 850M
Maybe you are looking for
-
I recently changed my computer software from Vista to Window & Ultimate. Since changing, I cannot see the print button which should be at the bottom. I have tried to hide the bottom toolbar but it still does not allow me to see the print button. Plea
-
I receive a "Problem Loading Page" message, "The Connection Was Reset." I've tried all your tip.
I receive the "connection was reset" message on 90% of my attempts to visit websites. All sites are bookmarked and safe. I printed out all help information and tried the forum. I am using Windows 8.1. Some of the help instructions seem to be for XP o
-
How to use an HTML file with Parameters to make an audioClip
HI guys, I am still working on getting simple sound to work in my program! I have learned that I can use Java.Applet's soundClip object, even when not using an Applet, but I do not know how to work with the URL object. Here is a SCCE so you can see w
-
Imported Objects possible for third party SWC
Hello, I have come across a scenario where SWC which is installed on third party system and need imported objects from SAP. Can i do it or not. i am bit confused. My third party SWC needs a RFC because i have to create interface to expose as web serv
-
Why is my color picker not showing up in the gradient panel?
When I try to change the colors to a gradient I am not able to change the selected color. When I double click on the color slider I want to change, the pop up panel gives me grayed out CMYK sliders. And all of the fill and stroke icons go dark and di