Performance problem - event : cursor: pin S wait on X
Hi,
Bellow is 17 min awr report , of oracle PeopleSoft DB on 10204 instance on HPUX machine.
During this time the customers complained on poor performance.
There were 4,104.23 execution per second and 3,784.95 parses
which mean that almost any statment was parsed. since the Soft Parse %= 99.77
its seems that most of the parses are soft parse.
During those 17 min , the DB Time = 721.74 min and the "Top 5 Timed Events"
shows : "cursor: pin S wait on X" at the top of the Timed Events
Attached some details for the awr report
Could you please suggest where to focus ?
Thanks
WORKLOAD REPOSITORY report for
DB Name DB Id Instance Inst Num Release RAC Host
xxxx 2993006132 xxxx 1 10.2.0.4.0 NO xxxx
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 18085 25-Mar-10 10:30:41 286 14.9
End Snap: 18086 25-Mar-10 10:48:39 301 15.1
Elapsed: 17.96 (mins)
DB Time: 721.74 (mins)
Cache Sizes
~~~~~~~~~~~ Begin End
Buffer Cache: 4,448M 4,368M Std Block Size: 8K
Shared Pool Size: 2,736M 2,816M Log Buffer: 2,080K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 3,831,000.13 271,096.84
Logical reads: 164,733.47 11,657.20
Block changes: 17,757.42 1,256.59
Physical reads: 885.19 62.64
Physical writes: 504.92 35.73
User calls: 5,775.09 408.67
Parses: 3,784.95 267.84
Hard parses: 8.55 0.60
Sorts: 212.37 15.03
Logons: 0.77 0.05
Executes: 4,104.23 290.43
Transactions: 14.13
% Blocks changed per Read: 10.78 Recursive Call %: 24.14
Rollback per transaction %: 0.18 Rows per Sort: 57.86
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.98 Redo NoWait %: 99.97
Buffer Hit %: 99.47 In-memory Sort %: 100.00
Library Hit %: 99.73 Soft Parse %: 99.77
Execute to Parse %: 7.78 Latch Hit %: 99.77
Parse CPU to Parse Elapsd %: 3.06 % Non-Parse CPU: 89.23
Shared Pool Statistics Begin End
Memory Usage %: 34.44 34.78
% SQL with executions>1: 76.52 60.40
% Memory for SQL w/exec>1: 73.75 99.18
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
cursor: pin S wait on X 1,378,354 13,462 10 31.1 Concurrenc
db file sequential read 878,684 8,779 10 20.3 User I/O
CPU time 4,998 11.5
local write wait 2,692 2,442 907 5.6 User I/O
cursor: pin S 1,932,830 2,270 1 5.2 Other
Time Model Statistics DB/Inst: xxxx/xxxx Snaps: 18085-18086
Statistic Name Time (s) % of DB Time
sql execute elapsed time 21,690.6 50.1
parse time elapsed 17,504.9 40.4
DB CPU 4,998.0 11.5
hard parse elapsed time 372.1 .9
connection management call elapsed time 183.9 .4
sequence load elapsed time 125.8 .3
PL/SQL execution elapsed time 89.2 .2
PL/SQL compilation elapsed time 9.2 .0
inbound PL/SQL rpc elapsed time 5.5 .0
hard parse (sharing criteria) elapsed time 5.5 .0
hard parse (bind mismatch) elapsed time 0.5 .0
failed parse elapsed time 0.1 .0
repeated bind elapsed time 0.0 .0
DB time 43,304.1 N/A
background elapsed time 3,742.3 N/A
background cpu time 114.8 N/A
Avg
%Time Total Wait wait Waits
Wait Class Waits -outs Time (s) (ms) /txn
Concurrency 1,413,633 97.5 14,283 10 92.8
User I/O 925,010 .3 11,485 12 60.7
Other 1,984,969 .2 2,858 1 130.3
Application 1,342 46.4 1,873 1396 0.1
Configuration 12,116 63.6 1,857 153 0.8
System I/O 582,094 .0 1,444 2 38.2
Commit 17,253 .6 1,057 61 1.1
Network 6,180,701 .0 68 0 405.9
Wait Events DB/Inst: xxxx/xxxx Snaps: 18085-18086
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
cursor: pin S wait on X 1,378,354 100.0 13,462 10 90.5
db file sequential read 878,684 .0 8,779 10 57.7
local write wait 2,692 91.2 2,442 907 0.2
cursor: pin S 1,932,830 .0 2,270 1 126.9
log file switch (checkpoint 2,669 49.1 1,510 566 0.2
enq: RO - fast object reuse 542 86.5 1,398 2580 0.0
log file sync 17,253 .6 1,057 61 1.1
control file sequential read 450,043 .0 579 1 29.6
log file parallel write 17,903 .0 558 31 1.2
enq: TX - row lock contentio 295 52.2 475 1610 0.0
buffer busy waits 7,338 4.4 348 47 0.5
buffer exterminate 322 92.5 302 938 0.0
read by other session 24,694 .0 183 7 1.6
library cache lock 59 94.9 167 2825 0.0
log file sequential read 109,494 .0 161 1 7.2
latch: cache buffers chains 18,662 .0 149 8 1.2
log buffer space 2,493 .0 139 56 0.2
Log archive I/O 3,592 .0 131 37 0.2
free buffer waits 6,420 99.1 130 20 0.4
latch free 42,812 .0 121 3 2.8
Streams capture: waiting for 845 6.0 106 125 0.1
latch: library cache 2,074 .0 96 46 0.1
db file scattered read 12,437 .0 80 6 0.8
enq: SQ - contention 150 14.0 71 471 0.0
SQL*Net more data from clien 331,961 .0 41 0 21.8
latch: shared pool 320 .0 32 100 0.0
LGWR wait for redo copy 5,307 49.1 29 5 0.3
SQL*Net more data to client 254,217 .0 17 0 16.7
control file parallel write 1,038 .0 15 14 0.1
latch: library cache lock 477 .4 14 29 0.0
latch: row cache objects 6,013 .0 10 2 0.4
SQL*Net message to client 5,587,878 .0 10 0 366.9
latch: redo allocation 1,274 .0 9 7 0.1
log file switch completion 62 .0 6 92 0.0
Streams AQ: qmn coordinator 1 100.0 5 4882 0.0
latch: cache buffers lru cha 434 .0 4 9 0.0
block change tracking buffer 111 .0 4 35 0.0
wait list latch free 135 .0 3 21 0.0
enq: TX - index contention 132 .0 2 17 0.0
latch: session allocation 139 .0 2 14 0.0
latch: object queue header o 379 .0 2 4 0.0
row cache lock 15 .0 2 107 0.0
latch: redo copy 56 .0 1 17 0.0
latch: library cache pin 184 .0 1 5 0.0
write complete waits 14 28.6 1 51 0.0
latch: redo writing 251 .0 1 3 0.0
enq: MN - contention 3 .0 1 206 0.0
enq: CF - contention 16 .0 0 23 0.0
log file single write 24 .0 0 13 0.0
os thread startup 3 .0 0 102 0.0
reliable message 66 .0 0 4 0.0
enq: JS - queue lock 2 .0 0 136 0.0
latch: cache buffer handles 46 .0 0 5 0.0
buffer deadlock 65 100.0 0 4 0.0
latch: undo global data 73 .0 0 3 0.0
change tracking file synchro 24 .0 0 6 0.0
change tracking file synchro 30 .0 0 3 0.0
kksfbc child completion 2 100.0 0 52 0.0
SQL*Net break/reset to clien 505 .0 0 0 0.0
db file parallel read 3 .0 0 30 0.0
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
SQL*Net more data from dblin 127 .0 0 0 0.0
SQL*Net more data to dblink 319 .0 0 0 0.0
latch: enqueue hash chains 20 .0 0 2 0.0
latch: checkpoint queue latc 5 .0 0 5 0.0
SQL*Net message to dblink 6,199 .0 0 0 0.4
enq: TX - allocate ITL entry 1 .0 0 22 0.0
direct path read 5,316 .0 0 0 0.3
latch: messages 24 .0 0 1 0.0
enq: US - contention 3 .0 0 4 0.0
direct path write 1,178 .0 0 0 0.1
rdbms ipc reply 1 .0 0 1 0.0
library cache load lock 2 .0 0 0 0.0
direct path write temp 3 .0 0 0 0.0
direct path read temp 3 .0 0 0 0.0
SQL*Net message from client 5,587,890 .0 135,002 24 366.9
wait for unread message on b 7,809 21.8 3,139 402 0.5
LogMiner: client waiting for 262,604 .1 3,021 12 17.2
LogMiner: wakeup event for b 1,405,104 2.4 2,917 2 92.3
Streams AQ: qmn slave idle w 489 .0 2,650 5420 0.0
LogMiner: wakeup event for p 123,723 32.1 2,453 20 8.1
Streams AQ: waiting for time 9 55.6 1,790 198928 0.0
LogMiner: reader waiting for 45,193 51.3 1,526 34 3.0
Streams AQ: waiting for mess 297 99.3 1,052 3542 0.0
Streams AQ: qmn coordinator 470 33.8 1,050 2233 0.0
Streams AQ: delete acknowled 405 32.3 1,049 2591 0.0
jobq slave wait 379 77.8 958 2529 0.0
LogMiner: wakeup event for r 16,591 10.6 125 8 1.1
SGA: MMAN sleep for componen 3,928 99.3 35 9 0.3
SQL*Net message from dblink 6,199 .0 31 5 0.4
single-task message 108 .0 8 74 0.0
class slave wait 3 .0 0 0 0.0
Background Wait Events DB/Inst: xxxx/xxxx Snaps: 18085-18086
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
log file parallel write 17,916 .0 558 31 1.2
Log archive I/O 3,592 .0 131 37 0.2
log file sequential read 3,636 .0 47 13 0.2
events in waitclass Other 6,149 42.4 40 7 0.4
log file switch (checkpoint 30 53.3 19 619 0.0
control file parallel write 1,038 .0 15 14 0.1
db file sequential read 1,166 .0 6 5 0.1
control file sequential read 2,986 .0 6 2 0.2
latch: shared pool 4 .0 4 917 0.0
latch: library cache 5 .0 3 646 0.0
free buffer waits 160 98.8 2 10 0.0
buffer busy waits 2 .0 1 404 0.0
latch: redo writing 19 .0 0 23 0.0
log file single write 24 .0 0 13 0.0
os thread startup 3 .0 0 102 0.0
log buffer space 7 .0 0 35 0.0
latch: cache buffers chains 16 .0 0 8 0.0
log file switch completion 1 .0 0 71 0.0
latch: library cache lock 3 66.7 0 11 0.0
latch: redo copy 1 .0 0 20 0.0
direct path read 5,316 .0 0 0 0.3
latch: row cache objects 3 .0 0 1 0.0
direct path write 1,174 .0 0 0 0.1
latch: library cache pin 3 .0 0 0 0.0
rdbms ipc message 20,401 24.2 11,112 545 1.3
Streams AQ: qmn slave idle w 489 .0 2,650 5420 0.0
Streams AQ: waiting for time 9 55.6 1,790 198928 0.0
pmon timer 379 94.5 1,050 2771 0.0
Streams AQ: delete acknowled 406 32.3 1,050 2586 0.0
Streams AQ: qmn coordinator 470 33.8 1,050 2233 0.0
smon timer 146 .0 1,039 7118 0.0
SGA: MMAN sleep for componen 3,928 99.3 35 9 0.3
Operating System Statistics DB/Inst: xxxx/xxxx Snaps: 18085-18086
Statistic Total
AVG_BUSY_TIME 68,992
AVG_IDLE_TIME 37,988
AVG_IOWAIT_TIME 28,529
AVG_SYS_TIME 11,748
AVG_USER_TIME 57,214
BUSY_TIME 552,209
IDLE_TIME 304,181
IOWAIT_TIME 228,489
SYS_TIME 94,253
USER_TIME 457,956
LOAD 2
OS_CPU_WAIT_TIME 147,872,604,500
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 49,152
VM_OUT_BYTES 0
PHYSICAL_MEMORY_BYTES 25,630,269,440
NUM_CPUS 8
NUM_CPU_SOCKETS 8
mbobak wrote:
So, this is a parsing related wait. You already mentioned that you're doing lots of parsing, mostly soft. Do you have session_cursor_cache parameter set to a reasonable value? 10g, I believe the default is 50, which is probably not a bad starting point. You may get additional benefits with moderate increases, perhaps to 100-200 range. It can be costly to do so, but can the extra parsing be addressed in the application? Is there anything you can do to reduce parsing in the application? When the problem occurs, how is the CPU consumption on the box? Are the CPUs pegged? Are you bottlenecked on CPU resources? Finally, there are bugs around 10.2.0.x and mutexes, so, you may want to open an SR w/ Oracle support, and determine if the root cause is actually a bug.
Mark,
I think you might read a little more into the stats than you have done - averaging etc. notwithstanding.
There are 8.55 "hard" parses per second - which in 17.96 minutes is about 9,500 hard parses - and there are 1.3M pin S wait on X: which is about 130 per hard parse (and 1.9M pin S). So the average statistics might be showing an interesting impact on individual actions.
The waits on "local write wait" are worth nothing. There are various reasons for this, one of which is the segment header block writes and index root block writes when you truncate a table - which could also be a cause of the "enq: RO - fast object reuse" waits in the body of the report.
Truncating tables tends to invalidate cursors and cause hard parsing.
So I would look for code that is popular, executed from a number of sessions, and truncates tables.
There were some bugs in this area relating to global temporay tables - but they should have been fixed in 10.2.0.4.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
To post code, statspack/AWR report, execution plans or trace files, start and end the section with the tag {noformat}{noformat} (lowercase, curly brackets, no spaces) so that the text appears in fixed format.
There is a +"Preview"+ tab at the top of the text entry panel. Use this to check what your message will look like before you post the message. If it looks a complete mess you're unlikely to get a response. (Click on the +"Plain text"+ tab if you want to edit the text to tidy it up.)
+"Science is more than a body of knowledge; it is a way of thinking"+
+Carl Sagan+
Similar Messages
-
Query performance problem - events 2505-read cache and 2510-write cache
Hi,
I am experiencing severe performance problems with a query, specifically with events 2505 (Read Cache) and 2510 (Write Cache) which went up to 11000 seconds on some executions. Data Manager (400 s), OLAP data selection (90 s) and OLAP user exit (250 s) are other the other event with noticeable times. All other events are very quick.
The query settings (RSRT) are
persistent cache across each app server -> cluster table,
update cache in delta process is checked ->group on infoprovider type
use cache despite virtual characteristics/key figs checked (one info-cube has1 virtual key figure which should have a static result for a day)
=>Do you know how I can get more details than what's in 0TCT_C02 to break down the read and write cache events time or do you have any recommandation?
I have checked and no dataloads were in progres on the info-providers and no master data loads (change run). Overall system performance was acceptable for other queries.
ThanksHi,
Looks like you're using BDB, not BDB JE, and this is the BDB JE forum. Could you please repost here?:
Berkeley DB
Thanks,
mark -
环境：db：10.2.0.4.0 system：hp-ux 11.31
近期系统在访问高峰时，性能严重下降，分析awr得到信息为
Elapsed: 60.20 (mins)
DB Time: 1,916.95 (mins) 系统负担非常严重
Library Hit %: 99.60 Soft Parse %: 99.44
Execute to Parse %: -22.89 重解析现象很严重
Event Waits Time(s) Avg Wait(ms) % Total Call Time Wait Class
CPU time 26,661 23.2
cursor: pin S 5,346,339 11,727 2 10.2 Other
latch: cache buffers chains 25,963 5,205 200 4.5 Concurrency
db file scattered read 710,601 2,698 4 2.3 User I/O
latch: library cache 6,743 2,087 309 1.8 Concurrenc
cursor: pin S事件为等待时间第一， 一般判读为系统并发sql语句过多，由于没有之前正常状态的awr报告作为对比，所以对于这个系统问题的根源和解决办法不是很清楚，对于应用方面，减少并发sql语句，绑定变量等，在数据库方面调整ession_cached_cursors参数，增加sharepool空间，由于水平有限只能知道这么多，希望得到更加深入的分析。谢谢
附部分awr（如何上传完整报告？）WORKLOAD REPOSITORY report for
DB Name DB Id Instance Inst num Release RAC Host
ORCL 1183953527 orcl 1 10.2.0.4.0 NO HP-UX-1
Snap Id Snap Time Sessions Cursors/Session
Begin Snap: 36825 14-8ÔÂ -12 09:00:16 1426 1.5
End Snap: 36826 14-8ÔÂ -12 10:00:29 1630 1.4
Elapsed: 60.20 (mins)
DB Time: 1,916.95 (mins)
Report Summary
Cache Sizes
Begin End
Buffer Cache: 9,216M 9,216M Std Block Size: 8K
Shared Pool Size: 6,144M 6,144M Log Buffer: 14,348K
Load Profile
Per Second Per Transaction
Redo size: 85,077.91 5,632.74
Logical reads: 323,798.90 21,437.69
Block changes: 489.98 32.44
Physical reads: 4,190.38 277.43
Physical writes: 36.21 2.40
User calls: 7,273.13 481.53
Parses: 1,499.45 99.27
Hard parses: 8.39 0.56
Sorts: 150.73 9.98
Logons: 1.72 0.11
Executes: 1,220.19 80.78
Transactions: 15.10
% Blocks changed per Read: 0.15 Recursive Call %: 9.01
Rollback per transaction %: 12.68 Rows per Sort: 118.78
Instance Efficiency Percentages (Target 100%)
Buffer Nowait %: 99.99 Redo NoWait %: 99.99
Buffer Hit %: 98.71 In-memory Sort %: 100.00
Library Hit %: 99.60 Soft Parse %: 99.44
Execute to Parse %: -22.89 Latch Hit %: 99.95
Parse CPU to Parse Elapsd %: 6.61 % Non-Parse CPU: 94.79
Shared Pool Statistics
Begin End
Memory Usage %: 92.12 92.19
% SQL with executions>1: 90.33 87.07
% Memory for SQL w/exec>1: 90.37 88.32
Top 5 Timed Events
Event Waits Time(s) Avg Wait(ms) % Total Call Time Wait Class
CPU time 26,661 23.2
cursor: pin S 5,346,339 11,727 2 10.2 Other
latch: cache buffers chains 25,963 5,205 200 4.5 Concurrency
db file scattered read 710,601 2,698 4 2.3 User I/O
latch: library cache 6,743 2,087 309 1.8 Concurrency
Time Model Statistics
Total time in database user-calls (DB Time): 115017.1s
Statistics including the word "background" measure background process time, and so do not contribute to the DB time statistic
Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 89,472.10 77.79
DB CPU 26,660.94 23.18
parse time elapsed 20,759.09 18.05
PL/SQL execution elapsed time 322.11 0.28
hard parse elapsed time 256.97 0.22
connection management call elapsed time 90.04 0.08
failed parse elapsed time 44.23 0.04
hard parse (sharing criteria) elapsed time 27.70 0.02
sequence load elapsed time 13.43 0.01
PL/SQL compilation elapsed time 10.03 0.01
hard parse (bind mismatch) elapsed time 2.82 0.00
repeated bind elapsed time 0.10 0.00
inbound PL/SQL rpc elapsed time 0.05 0.00
DB time 115,017.09
background elapsed time 451.46
background cpu time 47.64
Back to Wait Events Statistics
Back to Top
Wait Class
s - second
cs - centisecond - 100th of a second
ms - millisecond - 1000th of a second
us - microsecond - 1000000th of a second
ordered by wait time desc, waits desc
Wait Class Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn
Other 5,359,517 0.06 12,985 2 98.23
Concurrency 41,787 7.97 7,899 189 0.77
User I/O 1,541,553 0.00 4,385 3 28.25
Network 28,263,892 0.00 515 0 518.03
Commit 31,974 1.02 489 15 0.59
Application 7,028 1.04 232 33 0.13
System I/O 41,367 0.00 118 3 0.76
Configuration 48 0.00 8 177 0.00
Back to Wait Events Statistics
Back to Top
Wait Events
s - second
cs - centisecond - 100th of a second
ms - millisecond - 1000th of a second
us - microsecond - 1000000th of a second
ordered by wait time desc, waits desc (idle events last)
Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn
cursor: pin S 5,346,339 0.00 11,727 2 97.99
latch: cache buffers chains 25,963 0.00 5,205 200 0.48
db file scattered read 710,601 0.00 2,698 4 13.02
latch: library cache 6,743 0.00 2,087 309 0.12
db file sequential read 657,512 0.00 1,191 2 12.05
latch free 7,839 0.10 973 124 0.14
read by other session 153,818 0.04 489 3 2.82
log file sync 31,974 1.02 489 15 0.59
SQL*Net more data to client 2,454,333 0.00 308 0 44.98
enq: TX - row lock contention 82 89.02 227 2762 0.00
latch: library cache lock 968 0.00 211 218 0.02
latch: session allocation 1,369 0.00 196 143 0.03
latch: shared pool 1,801 0.00 177 98 0.03
SQL*Net message to client 25,709,776 0.00 130 0 471.22
latch: row cache objects 2,772 0.00 113 41 0.05
log file parallel write 33,654 0.00 105 3 0.62
SQL*Net more data from client 99,783 0.00 77 1 1.83
buffer busy waits 198 24.24 64 325 0.00
latch: cache buffers lru chain 204 0.00 35 173 0.00
cursor: pin S wait on X 3,284 99.97 32 10 0.06
LGWR wait for redo copy 3,493 91.24 32 9 0.06
latch: object queue header operation 91 0.00 8 91 0.00
latch: In memory undo latch 43 0.00 8 191 0.00
db file parallel read 530 0.00 7 14 0.01
enq: SQ - contention 8 0.00 5 670 0.00
log file sequential read 400 0.00 5 13 0.01
SQL*Net break/reset to client 6,942 0.00 5 1 0.13
Streams AQ: qmn coordinator waiting for slave to start 1 100.00 5 4889 0.00
control file parallel write 1,844 0.00 5 2 0.03
latch: enqueue hash chains 55 0.00 4 76 0.00
Log archive I/O 344 0.00 3 9 0.01
wait list latch free 104 0.00 2 20 0.00
latch: undo global data 9 0.00 2 191 0.00
log file switch (checkpoint incomplete) 11 0.00 2 150 0.00
log file switch completion 28 0.00 1 53 0.00
os thread startup 11 0.00 1 132 0.00
control file sequential read 5,093 0.00 0 0 0.09
latch: redo allocation 4 0.00 0 68 0.00
enq: RO - fast object reuse 4 0.00 0 59 0.00
local write wait 50 0.00 0 2 0.00
latch: library cache pin 1 0.00 0 95 0.00
latch: cache buffer handles 1 0.00 0 80 0.00
enq: CF - contention 2 0.00 0 19 0.00
direct path read temp 18,314 0.00 0 0 0.34
log file single write 32 0.00 0 1 0.00
enq: TX - index contention 3 0.00 0 5 0.00
direct path write temp 25 0.00 0 0 0.00
reliable message 4 0.00 0 0 0.00
direct path read 350 0.00 0 0 0.01
Background Wait Events
ordered by wait time desc, waits desc (idle events last)
Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn
events in waitclass Other 2,441 130.93 140 57 0.04
log file parallel write 33,654 0.00 105 3 0.62
log file sequential read 400 0.00 5 13 0.01
control file parallel write 1,844 0.00 5 2 0.03
buffer busy waits 4 75.00 3 832 0.00
Log archive I/O 344 0.00 3 9 0.01
os thread startup 11 0.00 1 132 0.00
latch: library cache 3 0.00 1 220 0.00
control file sequential read 4,390 0.00 0 0 0.08
latch: shared pool 1 0.00 0 309 0.00
db file sequential read 82 0.00 0 4 0.00
db file scattered read 56 0.00 0 4 0.00
log file single write 32 0.00 0 1 0.00
latch: library cache lock 1 0.00 0 26 0.00
direct path read 344 0.00 0 0 0.01
direct path write 343 0.00 0 0 0.01
latch: In memory undo latch 1 0.00 0 0 0.00
Back to Wait Events Statistics
Back to Top
Operating System Statistics
Statistic Total
AVG_BUSY_TIME 355,404
AVG_IDLE_TIME 5,624
AVG_IOWAIT_TIME 1,350
AVG_SYS_TIME 24,626
AVG_USER_TIME 330,677
BUSY_TIME 2,844,143
IDLE_TIME 45,639
IOWAIT_TIME 11,363
SYS_TIME 197,893
USER_TIME 2,646,250
LOAD 8
OS_CPU_WAIT_TIME ###############
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 0
VM_OUT_BYTES 0
PHYSICAL_MEMORY_BYTES 51,504,857,088
NUM_CPUS 8
NUM_CPU_SOCKETS 8
Back to Wait Events Statistics
Back to Top
Service Statistics
ordered by DB Time
Service Name DB Time (s) DB CPU (s) Physical Reads Logical Reads
orcl 108,264.60 24,956.90 12,796,071 1,110,466,840
SYS$USERS 6,748.20 1,707.20 2,350,206 59,196,825
SYS$BACKGROUND 0.00 0.00 1,188 41,361
Back to Wait Events Statistics
Back to Top
Service Wait Class Stats
Wait Class info for services in the Service Statistics section.
Total Waits and Time Waited displayed for the following wait classes: User I/O, Concurrency, Administrative, Network
Time Waited (Wt Time) in centisecond (100th of a second)
Service Name User I/O Total Wts User I/O Wt Time Concurcy Total Wts Concurcy Wt Time Admin Total Wts Admin Wt Time Network Total Wts Network Wt Time
orcl 1354220 373545 40097 757402 0 0 26777629 50042
SYS$USERS 186115 64780 1563 27129 0 0 1456144 1238
SYS$BACKGROUND 1221 183 23 584 0 0 0 0
Back to Wait Events Statistics
Back to Top
SQL ordered by Elapsed Time
Resources reported for PL/SQL code includes the resources used by all SQL statements called by the code.
% Total DB Time is the Elapsed Time of the SQL statement divided into the Total Database Time multiplied by 100
Elapsed Time (s) CPU Time (s) Executions Elap per Exec (s) % Total DB Time SQL Id SQL Module SQL Text
17,352 1,151 282,032 0.06 15.09 58nqcgq93qdpp w3wp.exe SELECT sysdate FROM dual
13,412 3,344 3,516 3.81 11.66 gpz6zsfjvvgx7 obilling.exe select sum ( costs ) , sum ( ...
9,176 2,771 393 23.35 7.98 ajp712h079d57 dosage.exe select DIAG_DESC from outp_mr ...
8,351 2,411 3,163 2.64 7.26 9skkhx9qfmzjy outpdoct.exe SELECT "OUTP_TREAT_REC"."VIS...
6,278 2,042 2,598 2.42 5.46 f9nfgywd1dzbc outpdoct.exe SELECT :"SYS_B_0" del_indica...
3,406 1,070 729 4.67 2.96 3fgaurmna0hx9 obilling.exe SELECT "OUTP_PRESC_T"."VISIT...
3,228 867 9,221 0.35 2.81 3vy04kpknkqgk w3wp.exe select T1.REMIND_DATETIME, T1....
2,802 514 214 13.09 2.44 f4q13jvt6n8r8 mzcx.exe SELECT DISTINCT "OUTPBILL"."...
2,406 533 725 3.32 2.09 87mazvgqfas1z obilling.exe SELECT "OUTP_TREAT_REC_T"."V...
2,340 535 724 3.23 2.03 f4xrxypn2cpkm obilling.exe SELECT "OUTP_PRESC_T"."VISIT...
2,303 781 15,143 0.15 2.00 89yw129xmhchc w3wp.exe select a.PATIENT_ID, a.VISIT_I...
1,572 449 4,748 0.33 1.37 0nwg8cxxzcb0m w3wp.exe select T1.REMIND_DATETIME, T1....
1,258 135 44 28.58 1.09 cnuj5dstf2559 outpdoct.exe SELECT "DOCT_DRUG_PRESC_DE...
Back to SQL Statistics
Back to Top
SQL ordered by CPU Time
Resources reported for PL/SQL code includes the resources used by all SQL statements called by the code.
% Total DB Time is the Elapsed Time of the SQL statement divided into the Total Database Time multiplied by 100
CPU Time (s) Elapsed Time (s) Executions CPU per Exec (s) % Total DB Time SQL Id SQL Module SQL Text
3,344 13,412 3,516 0.95 11.66 gpz6zsfjvvgx7 obilling.exe select sum ( costs ) , sum ( ...
2,771 9,176 393 7.05 7.98 ajp712h079d57 dosage.exe select DIAG_DESC from outp_mr ...
2,411 8,351 3,163 0.76 7.26 9skkhx9qfmzjy outpdoct.exe SELECT "OUTP_TREAT_REC"."VIS...
2,042 6,278 2,598 0.79 5.46 f9nfgywd1dzbc outpdoct.exe SELECT :"SYS_B_0" del_indica...
1,151 17,352 282,032 0.00 15.09 58nqcgq93qdpp w3wp.exe SELECT sysdate FROM dual
1,070 3,406 729 1.47 2.96 3fgaurmna0hx9 obilling.exe SELECT "OUTP_PRESC_T"."VISIT...
867 3,228 9,221 0.09 2.81 3vy04kpknkqgk w3wp.exe select T1.REMIND_DATETIME, T1....
781 2,303 15,143 0.05 2.00 89yw129xmhchc w3wp.exe select a.PATIENT_ID, a.VISIT_I...
540 825 1,790 0.30 0.72 cw74b4gm99xd5 ORACLE.EXE SELECT "A4"."PATIENT_ID", TO_C...
535 2,340 724 0.74 2.03 f4xrxypn2cpkm obilling.exe SELECT "OUTP_PRESC_T"."VISIT...
533 2,406 725 0.73 2.09 87mazvgqfas1z obilling.exe SELECT "OUTP_TREAT_REC_T"."V...
514 2,802 214 2.40 2.44 f4q13jvt6n8r8 mzcx.exe SELECT DISTINCT "OUTPBILL"."...
449 1,572 4,748 0.09 1.37 0nwg8cxxzcb0m w3wp.exe select T1.REMIND_DATETIME, T1....
135 1,258 44 3.06 1.09 cnuj5dstf2559 outpdoct.exe SELECT "DOCT_DRUG_PRESC_DE...
Back to SQL Statistics
Back to Top
SQL ordered by Gets
Resources reported for PL/SQL code includes the resources used by all SQL statements called by the code.
Total Buffer Gets: 1,169,640,302
Captured SQL account for 71.4% of Total
Buffer Gets Executions Gets per Exec %Total CPU Time (s) Elapsed Time (s) SQL Id SQL Module SQL Text
158,325,351 3,516 45,029.96 13.54 3344.37 13412.17 gpz6zsfjvvgx7 obilling.exe select sum ( costs ) , sum ( ...
142,486,001 3,163 45,047.74 12.18 2410.79 8351.24 9skkhx9qfmzjy outpdoct.exe SELECT "OUTP_TREAT_REC"."VIS...
107,954,245 2,598 41,552.83 9.23 2041.61 6278.17 f9nfgywd1dzbc outpdoct.exe SELECT :"SYS_B_0" del_indica...
54,640,906 729 74,953.23 4.67 1069.81 3405.85 3fgaurmna0hx9 obilling.exe SELECT "OUTP_PRESC_T"."VISIT...
54,464,191 15,143 3,596.66 4.66 781.43 2302.60 89yw129xmhchc w3wp.exe select a.PATIENT_ID, a.VISIT_I...
52,300,799 9,221 5,671.92 4.47 866.73 3228.09 3vy04kpknkqgk w3wp.exe select T1.REMIND_DATETIME, T1....
36,350,381 1,790 20,307.48 3.11 539.85 825.10 cw74b4gm99xd5 ORACLE.EXE SELECT "A4"."PATIENT_ID", TO_C...
28,097,571 1,132 24,821.18 2.40 105.62 304.33 44cpsg5hgwtv3 outpdoct.exe SELECT "CLINIC_MASTER"."NAME...
27,329,179 724 37,747.48 2.34 534.54 2339.79 f4xrxypn2cpkm obilling.exe SELECT "OUTP_PRESC_T"."VISIT...
27,276,384 725 37,622.60 2.33 532.84 2405.61 87mazvgqfas1z obilling.exe SELECT "OUTP_TREAT_REC_T"."V...
26,913,555 4,748 5,668.40 2.30 449.02 1571.69 0nwg8cxxzcb0m w3wp.exe select T1.REMIND_DATETIME, T1....
25,335,084 214 118,388.24 2.17 514.25 2802.12 f4q13jvt6n8r8 mzcx.exe SELECT DISTINCT "OUTPBILL"."...
12,756,154 4 3,189,038.50 1.09 73.99 592.57 4v3jxnrj4duq0 ºǫ́ÅäÒ©ÐÂ2012.07.31.exe SELECT "DRUG_PRESC_MASTER"....
Back to SQL Statistics
Back to Top
SQL ordered by Reads
Total Disk Reads: 15,136,657
Captured SQL account for 81.4% of Total
Physical Reads Executions Reads per Exec %Total CPU Time (s) Elapsed Time (s) SQL Id SQL Module SQL Text
6,716,909 214 31,387.43 44.38 514.25 2802.12 f4q13jvt6n8r8 mzcx.exe SELECT DISTINCT "OUTPBILL"."...
3,642,877 44 82,792.66 24.07 134.64 1257.56 cnuj5dstf2559 outpdoct.exe SELECT "DOCT_DRUG_PRESC_DE...
1,707,395 12 142,282.92 11.28 56.42 470.82 bkkm26gd9ydjp prescent.exe SELECT "DRUG_PRESC_MASTER"....
208,221 4 52,055.25 1.38 73.99 592.57 4v3jxnrj4duq0 ºǫ́ÅäÒ©ÐÂ2012.07.31.exe SELECT "DRUG_PRESC_MASTER"....
137,856 0 0.91 383.09 1041.12 b1y7hxw5s572b ORACLE.EXE SELECT /*+ OPAQUE_TRANSFORM */...
45,210 1 45,210.00 0.30 136.24 617.92 9k2mtm738ffbk ORACLE.EXE SELECT "A1"."PATIENT_ID", "A1"...
21,692 0 0.14 44.82 288.67 5w6u29xnk192r ORACLE.EXE SELECT "A1"."TEST_NO", "A1"."I...
2,078 17,010 0.12 0.01 6.78 13.35 3ts97u1my91n4 testprn.exe SELECT "LAB"."LAB_TEST_ITEM...
539 1,732 0.31 0.00 124.85 289.44 0a96nnbzvb78j sjcj.exe select diag_desc , doctor fro...
224 14,499 0.02 0.00 1.75 2.81 adr24vcznhjhz C:\Documents and Settings\yj.ÐÄÄÚʵÑéÊÒ1.000\×À? SELECT COSTS , BILLING_INDICA...
Back to SQL Statistics
Back to Top
SQL ordered by Executions
Total Executions: 4,407,615
Captured SQL account for 43.0% of Total
Executions Rows Processed Rows per Exec CPU per Exec (s) Elap per Exec (s) SQL Id SQL Module SQL Text
324,063 324,042 1.00 0.00 0.00 1jzhjx524wyfm presdisp.exe select TRADE_PRICE , retail_p...
282,032 281,745 1.00 0.00 0.06 58nqcgq93qdpp w3wp.exe SELECT sysdate FROM dual
197,067 197,006 1.00 0.00 0.00 f0wzs9nc663bn obilling.exe select sysdate from dual
162,033 112,966 0.70 0.00 0.00 ak8wcxnbv0mq7 presdisp.exe select location from drug_stoc...
162,032 162,023 1.00 0.00 0.00 g982zswg09n71 presdisp.exe SELECT sum ( DRUG_STOCK.QUANTI...
152,651 1,296 0.01 0.00 0.00 ctc7t080mcrpr PACSVR.exe Select exam_master.*, exam_ma...
89,187 89,186 1.00 0.00 0.00 054n8y7sfgsjc anesmgr.exe select 1 from dual
53,088 24,076 0.45 0.00 0.00 fhzdwv4sgvurz ORACLE.EXE SELECT "A1"."END_DATE_TIME", S...
43,074 991 0.02 0.00 0.00 8kj11spfu1w5k ORACLE.EXE SELECT DISTINCT "A1"."LOG_DATE...
40,962 4,803 0.12 0.00 0.00 664crkjw8n47y doctws.exe select dept_code , ward_code ...
Back to SQL Statistics
Back to Top
SQL ordered by Parse Calls
Total Parse Calls: 5,416,394
Captured SQL account for 41.3% of Total
Parse Calls Executions % Total Parses SQL Id SQL Module SQL Text
457,941 152,651 8.45 ctc7t080mcrpr PACSVR.exe Select exam_master.*, exam_ma...
324,064 324,063 5.98 1jzhjx524wyfm presdisp.exe select TRADE_PRICE , retail_p...
281,992 282,032 5.21 58nqcgq93qdpp w3wp.exe SELECT sysdate FROM dual
197,067 197,067 3.64 f0wzs9nc663bn obilling.exe select sysdate from dual
162,033 162,033 2.99 ak8wcxnbv0mq7 presdisp.exe select location from drug_stoc...
162,032 162,032 2.99 g982zswg09n71 presdisp.exe SELECT sum ( DRUG_STOCK.QUANTI...
113,451 0 2.09 gbztx1wqkp1zw ORACLE.EXE SELECT * FROM "OPERATION_MASTE...
109,344 0 2.02 1gk864rffswjx ORACLE.EXE SELECT * FROM "PAT_VISIT"
104,957 0 1.94 dw9ggd0h86cw4 ORACLE.EXE SELECT * FROM "ADT_LOG"
89,187 89,187 1.65 054n8y7sfgsjc anesmgr.exe select 1 from dual
Back to SQL Statistics
Back to Top
SQL ordered by Sharable Memory
Only Statements with Sharable Memory greater than 1048576 are displayed
Sharable Mem (b) Executions % Total SQL Id SQL Module SQL Text
14,285,976 0.22 998p6b7d1q9bz ** SQL Text Not Available **
14,265,528 0.22 dsxd21s5bw5h3 ** SQL Text Not Available **
4,621,517 45 0.07 18h8s79av5v45 w3wp.exe select T1.REMIND_DATETIME, T1....
3,949,845 20 0.06 bn69cjprha24w w3wp.exe select T1.REMIND_DATETIME, T1....
3,407,101 41 0.05 5syt1sugfkg7c w3wp.exe select T1.REMIND_DATETIME, T1....
2,623,013 65 0.04 gh6391dujc994 w3wp.exe select T1.REMIND_DATETIME, T1....
2,314,741 30 0.04 49husy8txxw6h w3wp.exe select T1.REMIND_DATETIME, T1....
2,184,421 87 0.03 5a77mhab8fmq4 w3wp.exe select T1.REMIND_DATETIME, T1....
2,029,197 24 0.03 57wp7yk0ajprt w3wp.exe select T1.REMIND_DATETIME, T1....
2,028,837 5 0.03 csd4pjvrfpxry w3wp.exe select T1.REMIND_DATETIME, T1....
2,024,581 44 0.03 890zsd9cju097 w3wp.exe select T1.REMIND_DATETIME, T1....
1,925,949 39 0.03 cw4y41z3vnz8y w3wp.exe select T1.REMIND_DATETIME, T1....
1,793,389 71 0.03 gwx893aphkzw8 w3wp.exe select T1.REMIND_DATETIME, T1....
1,747,565 19 0.03 4p0vbj8qba0sj w3wp.exe select T1.REMIND_DATETIME, T1....
1,747,253 75 0.03 6hhka0kpch5bq w3wp.exe select T1.REMIND_DATETIME, T1....
1,746,061 60 0.03 cqbyt6m5m9f6g w3wp.exe select T1.REMIND_DATETIME, T1....
1,745,909 47 0.03 aatwks74mtqkx w3wp.exe select T1.REMIND_DATETIME, T1....
1,718,005 20 0.03 cmws54r2xv332 w3wp.exe select T1.REMIND_DATETIME, T1....
1,709,493 24 0.03 71zxrhv747290 w3wp.exe select T1.REMIND_DATETIME, T1....
1,265,905 1,585 0.02 a7c5z25vb0kvz obilling.exe SELECT COLUMN_NAME, DATA_TYPE...
1,132,605 30 0.02 9s6btksh172dz w3wp.exe select T1.REMIND_DATETIME, T1....
1,115,157 30 0.02 9s6btksh172dz w3wp.exe select T1.REMIND_DATETIME, T1....
1,055,029 105 0.02 da0pcbq6qxxqx w3wp.exe select T1.REMIND_DATETIME, T1....
Back to SQL Statistics
Back to Top
SQL ordered by Version Count
Only Statements with Version Count greater than 20 are displayed
Version Count Executions SQL Id SQL Module SQL Text
698 998p6b7d1q9bz ** SQL Text Not Available **
697 dsxd21s5bw5h3 ** SQL Text Not Available **
57 192 9mrhw217pc809 DOCTWS.EXE select MR_CODE , mr_attr , m...
43 1,307 53mkjqn5tdptj examapt.exe SELECT to_char ( last_updt_dat...
32 125 399071urkdx7n admit.exe SELECT to_char ( last_updt_dat...
26 564 14fz7s749tx24 prescent.exe Select drug_spec , batch_no , ...
Back to SQL Statistics
Back to Top
Complete List of SQL Text
SQL Id SQL Text
054n8y7sfgsjc select 1 from dual
0a96nnbzvb78j select diag_desc , doctor from outp_mr where patient_id =:1 and visit_date like visit_date and visit_no =:2
0nwg8cxxzcb0m select T1.REMIND_DATETIME, T1.START_DATETIME, T1.VISIT_DATE, T1.REMIND_TYPE, T1.REMIND_ID, T1.DELETE_FLAG, T1.REPEAT_INTERVAL, T1.VISIT_ID, T1.VISIT_NO, T1.DELETE_USER_ID, T1.DELETE_USER_NAME, T1.PATIENT_ID, T1.REMIND_DEPICT, T1.REMIND_INDEX, T2.REMIND_TYPE_NAME, T1.REMIND_USER_ID, T1.REMIND_USER_NAME from REMIND T1, REMIND_TYPE_DICT T2 where T1.REMIND_TYPE=T2.REMIND_TYPE_CODE(+) and T1.Remind_Type in (:"SYS_B_0") and T1.Delete_Flag=:"SYS_B_1" and T1.Remind_User_Name=:"SYS_B_2" order by T1.Remind_Id
14fz7s749tx24 Select drug_spec , batch_no , quantity , supply_indicator from drug_stock where drug_code =:1 and firm_id =:2 and package_spec =:3 and package_units =:4 and storage =:5 order by batch_no DESC
18h8s79av5v45
sql--省略
Back to SQL Statistics
Back to Top
SGA breakdown difference
ordered by Pool, Name
N/A value for Begin MB or End MB indicates the size of that Pool/Name was insignificant, or zero in that snapshot
Pool Name Begin MB End MB % Diff
java free memory 26.47 26.47 0.00
java joxlod exec hp 5.34 5.34 0.00
large free memory 32.00 32.00 0.00
shared CCursor 870.37 871.80 0.16
shared Cursor Stats 65.30 65.30 0.00
shared PCursor 506.48 505.60 -0.17
shared db_block_hash_buckets 90.00 90.00 0.00
shared free memory 484.07 480.10 -0.82
shared kglsim object batch 90.30 90.30 0.00
shared library cache 352.64 351.46 -0.33
shared sql area 3,290.12 3,291.57 0.04
streams free memory 16.00 16.00 0.00
buffer_cache 9,216.00 9,216.00 0.00
fixed_sga 1.99 1.99 0.00
log_buffer 14.01 14.01 0.00
Back to Memory Statistics
Back to Top
Resource Limit Stats
only rows with Current or Maximum Utilization > 80% of Limit are shown
ordered by resource name
Resource Name Current Utilization Maximum Utilization Initial Allocation Limit
processes 1,631 1,824 2000 2000
sessions 1,639 1,884 2205 2205
Back to Top
init.ora Parameters
Parameter Name Begin value End value (if different)
audit_file_dest /oracle/product/10.2.0/admin/orcl/adump
background_dump_dest /oracle/product/10.2.0/admin/orcl/bdump
compatible 10.2.0.1.0
control_files /oradata/orcl/control01.ctl, /oradata/orcl/control02.ctl, /oradata/orcl/control03.ctl, /oralog/orcl/control04.ctl
core_dump_dest /oracle/product/10.2.0/admin/orcl/cdump
cursor_sharing FORCE
db_block_size 8192
db_cache_size 9663676416
db_domain
db_file_multiblock_read_count 32
db_name orcl
db_recovery_file_dest /oracle/product/10.2.0/flash_recovery_area
db_recovery_file_dest_size 2147483648
java_pool_size 33554432
job_queue_processes 10
large_pool_size 33554432
log_archive_dest_1 LOCATION=/oralog/arch
nls_language SIMPLIFIED CHINESE
nls_length_semantics CHAR
nls_territory CHINA
open_cursors 300
pga_aggregate_target 2147483648
processes 2000
remote_login_passwordfile EXCLUSIVE
sessions 2205
shared_pool_size 6442450944
streams_pool_size 16777216
undo_management AUTO
undo_tablespace UNDOTBS1
user_dump_dest /oracle/product/10.2.0/admin/orcl/udump
Back to Top
End of Report焦点在于解析：
Parses: 1,499.45 99.27
Hard parses: 8.39 0.56
Execute to Parse %: -22.89每秒软解析1500次 ，Execute to Parse 解析是负值 ， 这说明 执行解析比非常差。
以下SQL parse次数频繁：
SQL ordered by Parse Calls
Total Parse Calls: 5,416,394
Captured SQL account for 41.3% of Total
Parse Calls Executions % Total Parses SQL Id SQL Module SQL Text
457,941 152,651 8.45 ctc7t080mcrpr PACSVR.exe Select exam_master.*, exam_ma...
324,064 324,063 5.98 1jzhjx524wyfm presdisp.exe select TRADE_PRICE , retail_p...SQL ID ctc7t080mcrpr 1jzhjx524wyfm
ctc7t080mcrpr 实际执行只有 152,651 ，解析倒有457,941次
建议分析以上语句为何产生了大量软解析
Advice:
1. 分析SQL语句，减少软解析的数量
2. 利用open cursor、session cached cursors等技术减少软解析
3. 考虑设置_kks_use_mutex_pin=FALSE 禁用10g中的MUTEX PIN CURSOR特性
*<font color="red" size="2" face="courier">如果觉得本回复有意义，请点击本条回复右手边的Correct按钮，谢谢！</font>*
Maclean Liu
Oracle Database Administrator
Oracle Certified 10g/11g Master
www.askmaclean.com
Edited by: Liu Maclean on 2012-8-23 下午11:48 -
100 %CPU utilizationis , cache buffers chains and cursor: pin S
Hi every one ,
we have incident causing system response very slow with very bad response time, below top 5 wait events from AWR (RAC database)
Event Waits Time(s) Avg Wait(ms) % Total Call Time Wait Class
latch: cache buffers chains 122,492 198,139 1,618 16.8 Concurrency
gc buffer busy 119,903 83,248 694 7.1 Cluster
cursor: pin S 18,674,280 72,651 4 6.2 Other
log file sync 639,867 66,673 104 5.7
Commit latch free 143,519 54,239 378 4.6 Other
Oracle support clearly identified the issue with latch cache buffer chains as SQL statement executed around 35000 times which is too high based on execution plan . and they suggest to tune SQL statements .
my question is cursor: pin S wait on X and library cache lock related ot it is just a symptoms , and is document 742599.1 applicable to us or not as we have 10.2.0.5 (suggest disable automatic memory management)
As I know high CPU utilization as result of latch: cache buffers chains , the cursor Pin S Wait should not .
Thank you in advanceHi,
All these 4 top events (excluding log file sync) are quite unusual and in your case, if all these are comming atop, these quite well be related. So, you can't say that cursor pin s wait on x should not be dealt saperatly, but, still you can try out suggestion in the note. First find out from v$sgasta about current allocation of shared pool, then after disabling automatic memory management, increase shared_pool significantly as compared to current value, and then monito the system
Definitely you should tune your SQL also, as suggested by support.
Salman -
Performance problem on wait event PX Deq: Execute Reply
Hi everybody
I encounter some performance problem, I've made a tkprof on a select statement and I saw that more than 95% of the elapsed time is due to event PX Deq: Execute Reply.
This request is not CPU or paging consuming. What is this event and how could I reduce it ? Could it be a disk problem ?
Thanks a lot, best regards
Greg
Here is a sample of my tkprof:
call count cpu elapsed disk query current rows
Parse 1 0.03 0.03 0 0 0 0
Execute 1 0.22 2.16 68 177 12 0
Fetch 2 0.17 511.97 38 40 0 1
total 4 0.42 514.16 106 217 12 1
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 38
Rows Row Source Operation
1 PX COORDINATOR (cr=202 pr=103 pw=0 time=513984636 us)
0 PX SEND QC (RANDOM) :TQ10003 (cr=0 pr=0 pw=0 time=0 us)
0 HASH GROUP BY (cr=0 pr=0 pw=0 time=0 us)
0 PX RECEIVE (cr=0 pr=0 pw=0 time=0 us)
0 PX SEND HASH :TQ10002 (cr=0 pr=0 pw=0 time=0 us)
0 HASH GROUP BY (cr=0 pr=0 pw=0 time=0 us)
0 HASH JOIN (cr=0 pr=0 pw=0 time=0 us)
0 BUFFER SORT (cr=0 pr=0 pw=0 time=0 us)
0 PX RECEIVE (cr=0 pr=0 pw=0 time=0 us)
0 PX SEND BROADCAST :TQ10000 (cr=0 pr=0 pw=0 time=0 us)
473 TABLE ACCESS FULL DIM_CALL_DISTANCE (cr=8 pr=7 pw=0 time=27259 us)
0 HASH JOIN (cr=0 pr=0 pw=0 time=0 us)
0 BUFFER SORT (cr=0 pr=0 pw=0 time=0 us)
0 PX RECEIVE (cr=0 pr=0 pw=0 time=0 us)
0 PX SEND BROADCAST :TQ10001 (cr=0 pr=0 pw=0 time=0 us)
4 TABLE ACCESS FULL DIM_AUDIT_CALL (cr=32 pr=31 pw=0 time=35037 us)
0 PX BLOCK ITERATOR PARTITION: 1 16 (cr=0 pr=0 pw=0 time=0 us)
0 TABLE ACCESS FULL FACT_CALL PARTITION: 1 48 (cr=0 pr=0 pw=0 time=0 us)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file sequential read 67 0.05 0.95
os thread startup 4 0.21 0.80
PX Deq: Join ACK 4 0.00 0.00
PX Deq: Parse Reply 3 0.13 0.17
SQL*Net message to client 2 0.00 0.00
PX Deq: Execute Reply 304 1.96 511.68
db file scattered read 6 0.01 0.03
PX qref latch 12 0.00 0.00
SQL*Net message from client 2 94.93 94.94
PX Deq: Signal ACK 6 0.10 0.11
enq: PS - contention 1 0.00 0.00
********************************************************************************PX Deq: Execute Reply is an idle event associated with Parallel Query. Are your tables partitioned or have a degree greater then 1?
The tables appear to be small in size. The overhead associated with parallel query generally hinders response time on queries involving small tables. -
Problem with JFrame and busy/wait Cursor
Hi -- I'm trying to set a JFrame's cursor to be the busy cursor,
for the duration of some operation (usually just a few seconds).
I can get it to work in some situations, but not others.
Timing does seem to be an issue.
There are thousands of posts on the BugParade, but
in general Sun indicates this is not a bug. I just need
a work-around.
I've written a test program below to demonstrate the problem.
I have the problem on Solaris, running with both J2SE 1.3 and 1.4.
I have not tested on Windows yet.
When you run the following code, three JFrames will be opened,
each with the same 5 buttons. The first "F1" listens to its own
buttons, and works fine. The other two (F2 and F3) listen
to each other's buttons.
The "BUSY" button simply sets the cursor on its listener
to the busy cursor. The "DONE" button sets it to the
default cursor. These work fine.
The "SLEEP" button sets the cursor, sleeps for 3 seconds,
and sets it back. This does not work.
The "INVOKE LATER" button does the same thing,
except is uses invokeLater to sleep and set the
cursor back. This also does not work.
The "DELAY" button sleeps for 3 seconds (giving you
time to move the mouse into the other (listerner's)
window, and then it behaves just like the "SLEEP"
button. This works.
Any ideas would be appreciated, thanks.
-J
import java.awt.BorderLayout;
import java.awt.Cursor;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import javax.swing.JButton;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.SwingUtilities;
public class BusyFrameTest implements ActionListener
private static Cursor busy = Cursor.getPredefinedCursor (Cursor.WAIT_CURSOR);
private static Cursor done = Cursor.getDefaultCursor();
JFrame frame;
JButton[] buttons;
public BusyFrameTest (String title)
frame = new JFrame (title);
buttons = new JButton[5];
buttons[0] = new JButton ("BUSY");
buttons[1] = new JButton ("DONE");
buttons[2] = new JButton ("SLEEP");
buttons[3] = new JButton ("INVOKE LATER");
buttons[4] = new JButton ("DELAY");
JPanel buttonPanel = new JPanel();
for (int i = 0; i < buttons.length; i++)
buttonPanel.add (buttons);
frame.getContentPane().add (buttonPanel);
frame.pack();
frame.setVisible (true);
public void addListeners (ActionListener listener)
for (int i = 0; i < buttons.length; i++)
buttons[i].addActionListener (listener);
public void actionPerformed (ActionEvent e)
System.out.print (frame.getTitle() + ": " + e.getActionCommand());
if (e.getActionCommand().equals ("BUSY"))
frame.setCursor (busy);
else if (e.getActionCommand().equals ("DONE"))
frame.setCursor (done);
else if (e.getActionCommand().equals ("SLEEP"))
frame.setCursor (busy);
try { Thread.sleep (3000); } catch (Exception ex) { }
frame.setCursor (done);
System.out.print (" finished");
else if (e.getActionCommand().equals ("INVOKE LATER"))
frame.setCursor (busy);
SwingUtilities.invokeLater (thread);
else if (e.getActionCommand().equals ("DELAY"))
try { Thread.sleep (3000); } catch (Exception ex) { }
frame.setCursor (busy);
try { Thread.sleep (3000); } catch (Exception ex) { }
frame.setCursor (done);
System.out.print (" finished");
System.out.println();
Runnable thread = new Runnable()
public void run()
try { Thread.sleep (3000); } catch (Exception ex) { }
frame.setCursor (done);
System.out.println (" finished");
public static void main (String[] args)
BusyFrameTest f1 = new BusyFrameTest ("F1");
f1.addListeners (f1);
BusyFrameTest f2 = new BusyFrameTest ("F2");
BusyFrameTest f3 = new BusyFrameTest ("F3");
f2.addListeners (f3); // 2 listens to 3
f3.addListeners (f2); // 3 listens to 2I've had the same problems with cursors and repaints in a swing application, and I was thinking of if I could use invokeLater, but I never got that far with it.
I still believe you would need a thread for the time consuming task, and that invokeLater is something you only need to use in a thread different from the event thread. -
Event ID 218 Copy of database 'Mailbox Database'- experienced a performance problem
Hi,
We have 16 GB of RAM for EXchange 2010 and 4 Processor and using iSCSI Starwind LUN.
Event ID : 218
Event Source : ExchangeStoreDB
Event Category :Database Recovery
the copy of database 'Mailbox Database - NYM' on this server experienced a performance problem. Failover returned the following
error: here is only one copy of this mailbox database (Mailbox Database - NYM). Automatic recovery is not available.
It occurs specially when the exchange backup start ; we are using window backup for taking exchange backup
I can see warnning of ESE below the event 218 - Information Store (3912) Mailbox Database - NYM: A request to write to the file "D:\Program Files\Microsoft\Exchange Server\V14\Mailbox\Mailbox Database 0959355037\Mailbox Database 0959355037.edb" at
offset 150801350656 (0x000000231c760000) for 32768 (0x00008000) bytes has not completed for 68 second(s). This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.
Also,
1. How can I enable verbose logging to expert level for exchange MailboxDatabase.
2. What type performace counter need to be set for exchange database in Performance Monitor
Any Help ?
Thanks
PrakashWe had the same issue on our mbx server with Exchange 2010 Ent SP3, Win Svr 2008 R2 SP1 running on VMware v5.1 with Exchange dbs mounted to the VM via iSCSI LUNs on our NetApp SAN.
We escalated the ticket, and the Microsoft Exchange escalation team stated that the EIDs of the database corruption and automatic recovery seem to point to a hardware issue. And, they told us not to panic and that there was no need to rebuild the environment
and migrate the databases immediately. They instead asked us to focus all our efforts on solving the iSCSI environment issues, since each Exchange EID db corruption/ autorecovery would be preceded by some type of corresponding iSCI System EID.
We hence opened tickets with MS Storage Team and with NetApp support and worked on this ticket with input from all 3 groups.
After about a month an a half of troubleshooting & trial error- with tickets open with NetApp, MS Storage Team, and MS Exchange team, we finally seem to have applied a configuration change that worked.
NetApp support referred us to the following article relating to VMware:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2039495
So, per NetApp support and MS Storage team, do the following:
1. Manually dismount Exchange Databases on the mailbox server.
2. Open Device Manager, Network Adapters, VMXNET3 Ethernet Adapter #3, Properties
3. Change Small RX Buffers from the default 512 to the maximum 8192 using the drop down selector.
4. Change the RX Ring #1 from the default 1024 to the maximum of 4096 using the drop down selector.
5. Click OK and save the change.
6. Re-mount the Exchange Databases
And, to our surprise, this seems to have done the trick. It seems like the issue is an iSCSI-related NIC setting all along.
Thanks,
Detrich -
Performance problems on a Oracle 11G with Windows 2008 64bits.
Hi everyone,
I have noticed that our db is going low and low every week. My server has 16GB RAM and 10GB are dedicated to the Oracle database, this is a 11.2.0.1 with Windows 2008 R2 SP1 64bits. I like to know acording to the nexts values what you guys recommend to adjust in the init.ora:
orcl.__db_cache_size=5402263552
orcl.__java_pool_size=33554432
orcl.__large_pool_size=33554432
orcl.__pga_aggregate_target=3657433088
orcl.__sga_target=6878658560
orcl.__shared_io_pool_size=0
orcl.__shared_pool_size=1308622848
orcl.__streams_pool_size=33554432
*.memory_target=10511974400
*.open_cursors=5000
*.optimizer_mode='RULE'
*.processes=300
Acording to the memory target on how values can be increased the processes, pga_agregate_target, etc.
Also we have problems related to the bug Bug 9593134 that “Connections to Oracle 11g are slow and can take anywhere from 10 seconds to 2 minutes.” there is a fix on linux by removing the dns names on it but anyone have experience on windows platforms?
Thanks for all and sorry for my english.
Regards.
Arturo.Regarding the long connection times, have you tried using network packet capture software (such as Wireshark) to determine what the client computer is doing when a connection attempt is initiated?
The Oracle Database time model statistics, along with the system wide wait events may help you diagnose the non-connection related performance issues (you should not just look at the statistics, but instead capture the current values, wait a period of time, capture the statistics again, and compare the changes in the statistic values). A statspack report might also help you - but a 10046 trace at level 8 or 12 is more appropriate if you are able to identify a couple of sessions that experience performance problems.
I do not suggest just blindly modifying parameters, although I am curious to know:
* Why the session level parameter OPEN_CURSORS is set to 5000 - do you expect a single session to hold open 5,000 cursors?
* Why are you using the deprecated RULE based optimizer?
* Why is the MEMORY_TARGET parameter used when the SGA_TARGET and PGA_AGGREGATE target are specified?
Charles Hooper
http://hoopercharles.wordpress.com/
IT Manager/Oracle DBA
K&M Machine-Fabricating, Inc. -
Hi all,
I'm having some performance problems and i have generated an AWR of a day and i have seen this following things:
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
CPU time 50,318 41.7
db file sequential read 6,688,472 32,711 5 27.1 User I/O
Backup: sbtwrite2 1,068,309 7,903 7 6.6 Administra
db file scattered read 1,012,065 6,999 7 5.8 User I/O
PX Deq Credit: send blkd 231,401 4,989 22 4.1 Other
Operating System Statistics DB/Inst: CAPDB14P/capdb14p1 Snaps: 15710-15778
Statistic Total
AVG_BUSY_TIME 3,221,704
AVG_IDLE_TIME 4,923,831
AVG_IOWAIT_TIME 2,302,776
AVG_SYS_TIME 537,429
AVG_USER_TIME 2,682,900
BUSY_TIME 6,446,121
IDLE_TIME 9,850,381
IOWAIT_TIME 4,608,322
SYS_TIME 1,077,598
USER_TIME 5,368,523
LOAD 0
OS_CPU_WAIT_TIME 1,999,898,469,700
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 12,201,893,888
VM_OUT_BYTES 476,655,616
PHYSICAL_MEMORY_BYTES 8,568,512,512
NUM_CPUS 2
NUM_CPU_SOCKETS 2
###########################3
I think that we are having CPU problems here !!
All my memory caches are good, 99% hit.
Anybody agree with me???
Tks,
PauloI have problems on some queries that have another wait event related to RAC.
"gc cs multi block request" is taking a lot of time on some queries. These queries run very fast at another databas that isn't a RAC database.
Example:
1-Tables has the same number of rows!!!!!
2-Both tables and indexes are analyzed using the same tool (DBMA_STATS)
####RAC DATABASE####
SELECT 1 from dual
WHERE NOT EXISTS (SELECT 1
FROM mensalidade a
WHERE data_vencimento >= CHAR_TO_DATE('20070201'));
----Explain
SELECT STATEMENT, GOAL = ALL_ROWS 4 1
FILTER
FAST DUAL 2 1
PX COORDINATOR FORCED SERIAL
PX SEND QC (RANDOM) SYS :TQ10000 2 1 7
PX BLOCK ITERATOR 2 1 7
INDEX FAST FULL SCAN BRCAPDB2 IMENSALIDADE1 2 1 7
----It takes more than 500 seconds to run
####STANDALONE DATABASE####
SELECT 1 from dual
WHERE NOT EXISTS (SELECT 1
FROM mensalidade a
WHERE data_vencimento >= CHAR_TO_DATE('20070201'));
----Explain
SELECT STATEMENT, GOAL = ALL_ROWS 4 1
FILTER
FAST DUAL 2 1
PX COORDINATOR FORCED SERIAL
PX SEND QC (RANDOM) SYS :TQ10000 2 2 16
PX BLOCK ITERATOR 2 2 16
TABLE ACCESS FULL BRCAPDB2 MENSALIDADE 2 2 16
----It takes 0.1 seconds to run -
Dear,
I encountered performance problem in my production databases, running Oracle RAC (2 nodes) on version 10.2.0.2.0. The scenario is that the database suddenly slows down at a particular time. In reviewing the alert log, I found the following message:
Node 1: Ora-3136 inbound connection timed out (occurs around 20 times)
Node 2: SQL*net time-out error (occurs few times)
In addition, in my statspack report, at that particular time interval, the following messages appear:
Node 1:
For the top 5 timed events, 'log file switch completion' is the first on the list, with the following figures:
Waits Time(s) Avg Wait (ms) Total Call time
4143 4036 974 25.8
Node 2:
For the top 5 timed events, 'gc cr block busy' is the first on the list, with the following figures:
Waits Time(s) Avg Wait (ms) Total Call time
3387 1939 573 27.2
I just guess 'log file switch completion' should not always be on the 'wait list'. Also, Archive log seems not full at the moment. There are also a lot of I/O at that moment. Not understand what the DB is doing.
Do anyone have any ideas on the cause of the problem?
ThanksYou need to define a service to run preferably on node 2 with failover to node 1, and give a your batch jobs a connect string that connects to that service:
srvctl add service -d whatever -s batchserv -r node2 -a node1
and in your tnsnames.ora,
batchserv=
(description=
(address_list=
(failover=on)(load_balance=on)
(address=(node1....)(address=(node2...))
(connect_data=(service_name=batchserv))
I've recorded many demonstrations of this sort thing,
Free Oracle Databse 12c and 11g Tutorials for Administration and Developers SkillBuilders
John Watson
Oracle Certified Master DBA -
Question for ORACLE EXPERTS!!! PERFORMANCE PROBLEM!!!
I have 2 nodes on RAC. On node1 the following query run in 15 seconds and on node2 the same query run in 40 minutes. This is a big problem because we are migrating a SQL Server database to Oracle RAC 10gR2 (10.2.0.2) on HP-UX Itanium and we can´t finish the project because of this performance problem!!! PLEASE HELP ME ORACLE GURUS !!! I have opened a TAR at Metalink a 3 month ago,but they can´t help me with this. Can anyone explain this event to me please??? " gc cr multi block request 92011 0.33 737.49
This is the query that i run on both nodes and this is the tkprof of the trace file when i run this on second node:
select /*+ NO_PARALLEL(t) */ sum(t.saldo_em_real)
from
brcapdb2.titulo t
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1 13.76 790.26 0 107219 0 0
total 3 13.76 790.26 0 107219 0 0
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 31
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
SQL*Net message to client 2 0.00 0.00
SQL*Net message from client 2 0.02 0.02
gc cr multi block request 92011 0.33 737.49
gc current grant busy 1 0.00 0.00
latch: KCL gc element parent latch 4 0.00 0.00
latch: object queue header operation 17 0.00 0.01
latch free 6 0.00 0.00
gc remaster 9 1.94 9.00
gcs drm freeze in enter server mode 19 1.97 30.52
gc current block 2-way 10 0.61 2.72
SQL*Net break/reset to client 1 0.01 0.01
Please help me if you can!!
Tks,
Paulo.Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
SQL*Net message to client 2 0.00 0.00
SQL*Net message from client 2 0.02 0.02
gc cr multi block request 92011 0.33 737.49
gc current grant busy 1 0.00 0.00
latch: KCL gc element parent latch 4 0.00 0.00
latch: object queue header operation 17 0.00 0.01
latch free 6 0.00 0.00
gc remaster 9 1.94 9.00
gcs drm freeze in enter server mode 19 1.97 30.52
gc current block 2-way 10 0.61 2.72
SQL*Net break/reset to client 1 0.01 0.01There's lots of global cache traffic going on. All those multiblock transfers seem to be saturating the interconnect. Some things to consider:
- Have a look at metalink note 3951017.8 for a possible fix
- Check that your interconnect is properly configured
- Try running the query with PARALLEL hint. This would mean doing disk I/O instead of buffer & interconnect I/O.
edit: changed "without NOPARALLEL" to "with PARALLEL"
Message was edited by:
antti.koskinen -
Problem Event Name:_LiveKernelEvent OS Version:_6.1.7601.2.1.0.256.1 Locale ID:_1033
Problem Event Name: LiveKernelEvent
OS Version: 6.1.7601.2.1.0.256.1
Locale ID: 1033Hi,
In order to assist you, we will need the .DMP files to analyze what exactly occurred at the time of the crash, etc.
If you don't know where .DMP files are located, here's how to get to them:
1. Navigate to the %systemroot%\Minidump folder.
2. Copy any and all DMP files in the Minidump folder to your Desktop and then zip up these files.
3. Upload the zip containing the .DMP files to Onedrive or a hosting site of your choice and paste in your reply. Prefered sites: Onedrive, Mediafire, Dropbox, etc. Nothing with wait-timers.
4 (optional): The type of .DMP files located in the Minidump folder are known as Small Memory Dumps. In %systemroot% there will be what is known as a Kernel-Dump (if your system is set to generate). It is labeled MEMORY.DMP. The difference
between Small Memory Dumps and Kernel-Dumps in the simplest definition is a Kernel-Dump contains
much more information at the time of the crash, therefore allowing further debugging of your issue. If your upload speed permits it, and you aren't going against any strict bandwidth and/or usage caps, etc, the Kernel-Dump is the best
choice. Do note that Kernel-Dumps are much larger in size due to containing much more info, which is why I mentioned upload speed, etc.
If you are going to use Onedrive but don't know how to upload to it, please visit the following:
Upload photos and files to Onedrive.
Please note that any "cleaner" programs such as TuneUp Utilities, CCleaner, etc, by default will delete .DMP files upon use.
If your computer is not generating .DMP files, please do the following:
1. Start > type %systemroot% which should show the Windows folder, click on it. Once inside that folder, ensure there is a Minidump folder created. If not, CTRL-SHIFT-N to make a New Folder and name it Minidump.
2. Windows key + Pause key. This should bring up System. Click Advanced System Settings on the left > Advanced > Performance > Settings > Advanced > Ensure there's a check-mark for 'Automatically manage paging file size for all
drives'.
3. Windows key + Pause key. This should bring up System. Click Advanced System Settings on the left > Advanced > Startup and Recovery > Settings > System Failure > ensure there is a check mark next to 'Write an event to the system
log'.
Ensure Small Memory Dump is selected and ensure the path is %systemroot%\Minidump.
4. Double check that the WERS is ENABLED:
Start > Search > type services.msc > Under the name tab, find Windows Error Reporting Service > If the status of the service is not Started then right click it and select Start. Also ensure that under Startup Type it is set to Automatic rather than
Manual. You can do this by right clicking it, selecting properties, and under General selecting startup type to 'Automatic', and then click Apply.
If you cannot get into normal mode to do any of this, please do this via Safe Mode.
Regards,
Patrick
“Be kind whenever possible. It is always possible.” - Dalai Lama -
Erratic performance problems in Oracle 8.0.x
Hi all,
We are having a performance problem that appeared somewhere between 8.0.6 and 8.1.5 when using embedded SQL and the ProC compiler under Linux and Solaris.
The moment we use client libraries > 8.0.x, things seem to grind to a halt. We are currently using 8.0.6 client against 8, 9 and 10g databases. Using 8.1.5 or 10g clients against Oracle 9 or 10 databases triggers the problem.
The problem also isn't tied to any specific query. On the latest run we tried, the timings for a specific problem query are as follows: Oracle 10 server, Oracle 8.0.6 client - 40 seconds; Oracle 10 server (same database), Oracle 10 client - 14 hours
Explain plan doesn't show anything funny with the query. On occasion, the query does get through quickly. Subsequent runs are then also quick.
The query also runs fine in SQLPlus.
What we have noticed is that the server process flatlines it at 100% CPU usage for the entire duration. The client, on the other hand is just sleeping, waiting for data from the server. Stopping the client in the debugger shows that the client is waiting in the sqlcxt() call when opening the cursor, not actually fetching data.
We are at our wits end as to where look next, and we can't stay on 8.0.6 client libraries for ever as this is starting to cause us other hassles now.
Did something significant perhaps change between 8.0.x and 8.1.x that we need to cater for in our apps?
Any help/ideas would be greatly appreciated.
Regards,
GerhardCheck Metalink
Client / Server / Interoperability Support Between Different Oracle Versions
Doc ID: Note:207303.1
Looks like Client version 8.1.5 has some problem, it was never designed to support Oracle version higher than 8.1.7
On the other hand, 8.0.6 was supported up to 9.2
I would stay with 8.0.6 if I have to use Oracle 8 client. Client version 8.1.7 seems much better. -
Performance problem while CPU is 80% Idel ?
Hi,
My end users are claim for performance problem during execution of batch process.
As you can see there are 1,745 statement executing each second.
Awr report shows 98.1% of the time , waits on CPU .
Also Awr report shows that Host CPU is :79.9% Idel.
The second wait event shows only 212 seconds waits on db file sequential read.
Yet , 4 minute in 1 hour period is seems not an issue.
Please advise
DB Name DB Id Instance Inst Num Startup Time Release RAC
QERP xxx erp 1 21-Jan-13 15:40 11.2.0.2.0 ; NO
Host Name Platform CPUs Cores Sockets Memory(GB)
erptst HP-UX IA (64-bit) 16 16 4 127.83
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 40066 22-Jan-13 20:00:52 207 9.6
End Snap: 40067 22-Jan-13 21:00:05 210 9.6
Elapsed: 59.21 (mins)
DB Time: 189.24 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 8,800M 8,800M Std Block Size: 8K
Shared Pool Size: 1,056M 1,056M Log Buffer: 49,344K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 3.2 ; 0.1 ; 0.00 ; 0.05
DB CPU(s): 3.1 ; 0.1 ; 0.00 ; 0.05
Redo size: 604,285.1 ; 27,271.3
Logical reads: 364,792.3 ; 16,463.0
Block changes: 3,629.5 ; 163.8
Physical reads: 21.5 ; 1.0
Physical writes: 95.3 ; 4.3
User calls: 68.7 ; 3.1
Parses: 212.9 ; 9.6
Hard parses: 0.3 ; 0.0
W/A MB processed: 1.2 ; 0.1
Logons: 0.3 ; 0.0
Executes: 1,745.2 ; 78.8
Rollbacks: 1.2 ; 0.1
Transactions: 22.2
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 ; Redo NoWait %: 100.00
Buffer Hit %: 99.99 ; In-memory Sort %: 100.00
Library Hit %: 99.95 ; Soft Parse %: 99.85
Execute to Parse %: 87.80 ; Latch Hit %: 99.99
Parse CPU to Parse Elapsd %: 74.76 ; % Non-Parse CPU: 99.89
Shared Pool Statistics Begin End
Memory Usage %: 75.37 ; 76.85
% SQL with executions>1: 95.31 ; 85.98
% Memory for SQL w/exec>1: 90.33 ; 82.84
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
DB CPU 11,144 98.1
db file sequential read 52,714 214 4 1.9 User I/O
SQL*Net break/reset to client 29,050 6 0 .1 Applicatio
log file sync 2,536 6 2 .0 Commit
buffer busy waits 4,338 2 1 .0 Concurrenc
Host CPU (CPUs: 16 Cores: 16 Sockets: 4)
~~~~~~~~ Load Average
Begin End %User %System %WIO %Idle
0.34 ; 0.33 ; 19.7 ; 0.4 ; 1.8 ; 79.9Nikolay Savvinov wrote:
if the users are complaining about performance of the batch process, then that's what you should be looking at, not the entire system.I find it strange to see "end users" and "the batch process" in the same sentence (as it was in the first post). "End users" gives me the feeling of a significant number of concurrent sessions with people waiting for results in real time at the far end, while "batch process" carries the image a small number of large scale processes running overnight to prepare the data for the following morning.
I mention this because my first view of the AWR output was: you've got 16 CPUs, only three in use, virtually no users, and doing very little work, how can the users complain. (One answer, of course, is that the 13 CPUs could be locked out of use as far as Oracle is concerned). On the second read I decided that the "users" had gone home, and the complaint was simply that the batch process wasn't completing in time.
In this case I think "the entire system" IS "the batch process"
Determine which stored procedures and/or SQL statements took longer than usual and then find out why. Most likely you'll be able to find
everything you need in AWR views (DBA_HIST_SQL%) and ASH archive (DBA_HIST_ACTIVE_SESS_HISTORY).
If the batch process has changed dramatically and recently, then a simple first step might be to look at the current AWR report, find the few most time-consuming SQL statements, and use the awrsqrpt.sql script to find their history of execution plans.
But I'd also just look at the expensive SQL - bearing in mind, particularly, that there are very few user calls per second, yet many hundred executions per second: it strikes me that there could be quite a lot of PL/SQL going on doing something a little bit expensive many times or some PL/SQL function that calls some SQL that used to be called rarely from an SQL statement but is now (due, perhaps to a change in plan) being called much more frequently - so check SQL Ordered by Executions.
Regards
Jonathan Lewis -
MS SQL Server 2008 performance problem
We use TopLink 10.1.3.5 to connect to MS SQL Server 2008.
What we are seeing is that when a query is being run by TopLink a lot of cursors open up and remain open. Our database CPU usage goes up and it affects the whole application.
Our DBA took a look at it and said the database shows FETCH_APICURSOR* being used for select statements.
Is there a way to tell TopLink not to use cursors for queries?
Thanks.Can you pin point a particular TopLink query tied to the " FETCH_APICURSOR* " call in the app and post how it is being created?
My guess is that the application is specifying the TopLink query object to return a cursor or stream and not closing it in all cases, or keeping them open for a long period - did you say they were leaking, or is it just that a large number are open at a time leading to performance problems?
This streams+cursors are described in the TopLink docs here
http://docs.oracle.com/cd/E21764_01/web.1111/b32441/qryadv.htm#CJGJBHGJ
or the 10g docs here:
http://sqltech.cl/doc/oas10gR3/web.1013/b13593/qryadv010.htm
If this is the case, you might want to use a different strategy such as pagination instead of cursors, described here:
http://docs.oracle.com/cd/E17904_01/web.1111/b32441/optimiz.htm#CHDIBGFE
Best Regards,
Chris
Maybe you are looking for
-
MacBook (2007) Hard Drive failure?
Half an hour ago my MacBook started to have quite a large clicking noise, somewhat kind of like the optical disk checker that happens when you start up your Mac, but slightly different. 20 seconds later, everything started freezing. I forced-shutdown
-
Hi experts; May I know how to transfer EE Income by multi-banks? Say for Example: EE Income = 8,000 USD. He has 2 Bank Accounts HSBC & Standard Chartered. He wants to be paid via 2 Banks by 50% (HSBC:4,000 USD, other: 4,000 USD). So how do I configur
-
After a recent Server OS Rebuild I have a few Computers with Trust Issues
Hi, After a Server OS Rebuild I have tried to the best of my ability to recreate everything as it was. I have a few Remote computers that are having ERROR ID 5513 (IE Reestablish trust) I have usually simple deleted the account on client workstation
-
I keep on adjusting the duration of the IMOVIE clips, hit Done , and when I go check the clip, it reverts to the original duration. Any suggestions??
-
Size limitation in Sticky Notes?
We have a user who, upon opening her PDF in Adobe Acrobat 7, has difficulty opening and editing some sticky notes. I noted that one in particular had a large amount of text, and was wondering if there is a size/text limitation in sticky notes? Thank