ARCH wait on SENDREQ
Dear Experts,
I found following wait event in my ADDM report(TOP most)
wait event ARCH wait on SENDREQ in wait class network was consuming significant amount of time.We a standby database with follwing setting:
log_archive_dest_2=service=aaaaa reopen=60 ARCH NOAFFIRM
How to reduce this wait event.
Thanks&Regards
Sunil Kumar
:)
Hi,
Re: LNS wait on SENDREQ
Similar Messages
-
"ARCH wait on SENDREQ" is top in top 5 timed events
my database is two node rac (10.2.0.4)database.my database performance is very slow.
please suggest how to reduce the "ARCH wait on SENDREQ".
thanks in advance
karthiThis wait event will not impact on user sessions. No end user has ever telephoned the help desk to complain that "the wait event such-and-such is too high". What are they complaining about? For example, is it a query that takes too long? Or a batch job that doesn't finish in time? You need to focus on the problem before trying to find a solution.
-
Arch wait on sendreq is always very high
Hi all,
we have a DG system ,
a primary and 1 standby both 10.2.0.3 , the arch wait on sendreq is always very high
i can't see any recommendation from Grid control?
should i worry about that?
ThanksHi,
there is doc http://download.oracle.com/docs/cd/B19306_01/server.102/b14239/log_transport.htm#i1227137
http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_DataGuardNetworkBestPractices.pdf
Regards,
Tom
http://oracledba.cz -
DATAGUARD - LNS wait on SENDREQ
Hi Guys,
Is this normal in the primary database? I have one physical standby database.
3,300(s) an hour is it some problem?
wait events - LNS wait on SENDREQ
version 10.2.0.4
max performance
lgrw - async mode
not using standby redo log
thanksTue Jul 5 10:21:26 2011
RFS[2]: Archived Log: '/oraarch/PROD/arch/PROD_1_34768_644538341.arc'
Primary database is in MAXIMUM PERFORMANCE mode
Tue Jul 5 12:16:38 2011
RFS[2]: Archived Log: '/oraarch/PROD/arch/PROD_1_34769_644538341.arc'
Primary database is in MAXIMUM PERFORMANCE mode
Tue Jul 5 12:56:42 2011
RFS[2]: Archived Log: '/oraarch/PROD/arch/PROD_1_34770_644538341.arc'
Primary database is in MAXIMUM PERFORMANCE mode
RFS[2]: Archived Log: '/oraarch/PROD/arch/PROD_1_34771_644538341.arc'
Primary database is in MAXIMUM PERFORMANCE mode
Tue Jul 5 14:22:56 2011
RFS[1]: Archived Log: '/oraarch/PROD/arch/PROD_1_34773_644538341.arc'
Tue Jul 5 14:23:16 2011
RFS[2]: Archived Log: '/oraarch/PROD/arch/PROD_1_34772_644538341.arc'
Primary database is in MAXIMUM PERFORMANCE mode
Tue Jul 5 14:34:01 2011
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
Tue Jul 5 14:34:09 2011
MRP0: Background Media Recovery cancelled with status 16037
Tue Jul 5 14:34:09 2011
Errors in file /oraarch/PROD/admin/bdump/prod_mrp0_20948.trc:
ORA-16037: user requested cancel of managed recovery operation
Recovery interrupted!
Tue Jul 5 14:34:10 2011
Errors in file /oraarch/PROD/admin/bdump/prod_mrp0_20948.trc:
ORA-16037: user requested cancel of managed recovery operation
Tue Jul 5 14:34:10 2011
MRP0: Background Media Recovery process shutdown (PROD)
Tue Jul 5 14:34:10 2011
Managed Standby Recovery Canceled (PROD)
Tue Jul 5 14:34:10 2011
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
Tue Jul 5 14:34:22 2011
alter database recover managed standby database disconnect
Tue Jul 5 14:34:22 2011
Attempt to start background Managed Standby Recovery process (PROD)
MRP0 started with pid=36, OS id=9864
Tue Jul 5 14:34:22 2011
MRP0: Background Managed Standby Recovery process started (PROD)
Managed Standby Recovery not using Real Time Apply
parallel recovery started with 7 processes
Tue Jul 5 14:34:27 2011
Errors in file /oraarch/PROD/admin/bdump/prod_p000_8301.trc:
Tue Jul 5 14:34:27 2011
Warning: OS async I/O limit 128 is lower than recovery batch 1024
Tue Jul 5 14:34:27 2011
Errors in file /oraarch/PROD/admin/bdump/prod_p001_8303.trc:
Tue Jul 5 14:34:27 2011
Warning: OS async I/O limit 128 is lower than recovery batch 1024
Tue Jul 5 14:34:27 2011
Errors in file /oraarch/PROD/admin/bdump/prod_p003_8307.trc:
Tue Jul 5 14:34:27 2011
Warning: OS async I/O limit 128 is lower than recovery batch 1024
Tue Jul 5 14:34:27 2011
Errors in file /oraarch/PROD/admin/bdump/prod_p002_8305.trc:
Tue Jul 5 14:34:27 2011
Warning: OS async I/O limit 128 is lower than recovery batch 1024
Tue Jul 5 14:34:27 2011
Errors in file /oraarch/PROD/admin/bdump/prod_p004_8309.trc:
Tue Jul 5 14:34:27 2011
Warning: OS async I/O limit 128 is lower than recovery batch 1024
Tue Jul 5 14:34:27 2011
Errors in file /oraarch/PROD/admin/bdump/prod_p006_8313.trc:
Tue Jul 5 14:34:27 2011
Warning: OS async I/O limit 128 is lower than recovery batch 1024
Tue Jul 5 14:34:27 2011
Errors in file /oraarch/PROD/admin/bdump/prod_p005_8311.trc:
Tue Jul 5 14:34:27 2011
Warning: OS async I/O limit 128 is lower than recovery batch 1024
Tue Jul 5 14:34:27 2011
Waiting for all non-current ORLs to be archived...
Media Recovery Waiting for thread 1 sequence 34712
Fetching gap sequence in thread 1, gap sequence 34712-34712
Tue Jul 5 14:34:28 2011
Completed: alter database recover managed standby database disconnect
Tue Jul 5 14:39:30 2011
FAL[client]: Failed to request gap sequence
GAP - thread 1 sequence 34712-34712
DBID 14011163 branch 644538341
FAL[client]: All defined FAL servers have been attempted. -
Hi,
What is the problem to cause this wait event. How to improve the performance of DB. Kindly let me knowFrom: http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_DataGuardNetworkBestPractices.pdf
LGWR SYNC Tuning Example
As described in Data Guard documentation [5], LGWR SYNC redo transport is a synchronous process that causes the database to delay acknowledgment of user commits until the following actions have been completed:
1. LGWR writes redo to the online redo log at the production database (when LGWR SYNC is not used, user commits are acknowledged once this step completes except when the parameter COMMIT NOWAIT is used, as described in Appendix C).
2. The Data Guard LNS process on the production database performs a network send to the Data Guard RFS process on the standby database. For redo write sizes larger than 1MB LNS will issue multiple network sends to the RFS process on the standby.
3. The RFS process receives the redo being sent by LNS and completes the I/O into the standby redo log.
4. RFS process sends acknowledgment back to LNS that the redo has been received and written to disk.
5. LNS posts LGWR that the all the redo has been successfully received and written to disk by the standby.
On the production database the time for step one is represented with the “log file parallel write” wait event. Steps 2 through 5 are represented with the “LNS wait on SENDREQ” wait event. You can further divide the “LNS wait on SENDREQ” wait event into network time and RFS I/O time by subtracting the “RFS write” wait event obtained on the standby. These wait events can be assessed on the standby by using the query described in the “Data Guard Specific Wait Events” section above for a physical standby or by using AWR for a logical standby.
---------- -
Hi
ORACLE10gR2/Redhat4
RAC database.
I check my vg$session and I found that I have an event :ARCH wait on ATTACH that is waiting for more than 5 days to be executed!!!
How I can stop this session?
I try to kill it by ALTER SESSIOn but it's not working...I get an ORA-00029.
SQL> ed
écrit fichier afiedt.buf
1 select INST_ID,SID,EVENT,SECONDS_IN_WAIT from gv$session_wait
2* where sid=305
SQL> /
INST_ID SID EVENT SECONDS_IN_WAIT
1 305 ARCH wait on ATTACH 443613
2 305 rdbms ipc message 201
Thanck you for help
Edited by: vittel on Apr 22, 2009 2:33 PMHi,
Yes whe have a dataguard and I checked it's synchron with the primary.
Primary
SQL> select inst_id,max(sequence#) from gv$log_history
2 group by inst_id;
INST_ID MAX(SEQUENCE#)
1 11668
2 11668
Dataguard
SQL> select max(sequence#) from v$log_history;
MAX(SEQUENCE#)
11668
Regards
Edited by: vittel on Apr 22, 2009 2:39 PM
Edited by: vittel on Apr 22, 2009 2:41 PM -
Hi,
As per documents In general, the addition of wait classes helps direct the DBA more quickly toward the root cause of performance problems.
How could i trace the root cause of performence problems if it is related to wait class?
Thanks,userpat wrote:
Hi,
As per documents In general, the addition of wait classes helps direct the DBA more quickly toward the root cause of performance problems.
How could i trace the root cause of performence problems if it is related to wait class?
Thanks,I am not completely sure that I understand your question. The wait class gives you an approximate idea of where the performance problem will be found. You must then further investigate the wait events in that wait class. There are of course potential problems with starting at the wait class (some wait classes have 2 wait events, while others have many - that could throw off the search for the problem that is impacting performance the most), but at least it provides a starting point. To give you an idea of the wait events in each wait class, here is a SQL statement that was executed on Oracle Database 11.1.0.7:
SQL> DESC V$EVENT_NAME
Name Null? Type
EVENT# NUMBER
EVENT_ID NUMBER
NAME VARCHAR2(64)
PARAMETER1 VARCHAR2(64)
PARAMETER2 VARCHAR2(64)
PARAMETER3 VARCHAR2(64)
WAIT_CLASS_ID NUMBER
WAIT_CLASS# NUMBER
WAIT_CLASS VARCHAR2(64)
SELECT
SUBSTR(NAME,1,30) EVEMT_NAME,
SUBSTR(WAIT_CLASS,1,20) WAIT_CLASS
FROM
V$EVENT_NAME
ORDER BY
SUBSTR(WAIT_CLASS,1,20),
SUBSTR(NAME,1,30);
EVEMT_NAME WAIT_CLASS
ASM COD rollback operation com Administrative
ASM mount : wait for heartbeat Administrative
Backup: sbtbackup Administrative
Backup: sbtbufinfo Administrative
Backup: sbtclose Administrative
Backup: sbtclose2 Administrative
OLAP DML Sleep Application
SQL*Net break/reset to client Application
SQL*Net break/reset to dblink Application
Streams capture: filter callba Application
Streams: apply reader waiting Application
WCR: replay lock order Application
Wait for Table Lock Application
enq: KO - fast object checkpoi Application
enq: PW - flush prewarm buffer Application
enq: RC - Result Cache: Conten Application
enq: RO - contention Application
enq: RO - fast object reuse Application
enq: TM - contention Application
enq: TX - row lock contention Application
enq: UL - contention Application
ASM PST query : wait for [PM][ Cluster
gc assume Cluster
gc block recovery request Cluster
enq: BB - 2PC across RAC insta Commit
log file sync Commit
Shared IO Pool Memory Concurrency
Streams apply: waiting for dep Concurrency
buffer busy waits Concurrency
cursor: mutex S Concurrency
cursor: mutex X Concurrency
cursor: pin S wait on X Concurrency
Global transaction acquire ins Configuration
Streams apply: waiting to comm Configuration
checkpoint completed Configuration
enq: HW - contention Configuration
enq: SQ - contention Configuration
enq: SS - contention Configuration
enq: ST - contention Configuration
enq: TX - allocate ITL entry Configuration
free buffer waits Configuration
ASM background timer Idle
DIAG idle wait Idle
EMON slave idle wait Idle
HS message to agent Idle
IORM Scheduler Slave Idle Wait Idle
JOX Jit Process Sleep Idle
ARCH wait for flow-control Network
ARCH wait for net re-connect Network
ARCH wait for netserver detach Network
ARCH wait for netserver init 1 Network
ARCH wait for netserver init 2 Network
ARCH wait for netserver start Network
ARCH wait on ATTACH Network
ARCH wait on DETACH Network
ARCH wait on SENDREQ Network
LGWR wait on ATTACH Network
LGWR wait on DETACH Network
LGWR wait on LNS Network
LGWR wait on SENDREQ Network
LNS wait on ATTACH Network
LNS wait on DETACH Network
LNS wait on LGWR Network
LNS wait on SENDREQ Network
SQL*Net message from dblink Network
SQL*Net message to client Network
SQL*Net message to dblink Network
SQL*Net more data from client Network
SQL*Net more data from dblink Network
AQ propagation connection Other
ARCH wait for archivelog lock Other
ARCH wait for process death 1 Other
ARCH wait for process death 2 Other
ARCH wait for process death 3 Other
ARCH wait for process death 4 Other
ARCH wait for process death 5 Other
ARCH wait for process start 1 Other
Streams AQ: enqueue blocked du Queueing
Streams AQ: enqueue blocked on Queueing
Streams capture: waiting for s Queueing
Streams: flow control Queueing
Streams: resolve low memory co Queueing
resmgr:I/O prioritization Scheduler
resmgr:become active Scheduler
resmgr:cpu quantum Scheduler
ARCH random i/o System I/O
ARCH sequential i/o System I/O
Archiver slave I/O System I/O
DBWR slave I/O System I/O
LGWR random i/o System I/O
BFILE read User I/O
DG Broker configuration file I User I/O
Data file init write User I/O
Datapump dump file I/O User I/O
Log file init write User I/O
Shared IO Pool IO Completion User I/O
buffer read retry User I/O
cell multiblock physical read User I/O
cell single block physical rea User I/O
cell smart file creation User I/O
cell smart index scan User I/O
cell smart table scan User I/O
cell statistics gather User I/O
db file parallel read User I/O
db file scattered read User I/O
db file sequential read User I/O
db file single write User I/O
...So, if the User I/O wait class floats to the top of the wait classes between a known start time and end time, and the Commit wait class is at the bottom of the wait classes when comparing accumulated time, it probably would not make much sense to spend time investigating the wait events in the Commit class... until you realize that there is a single event in the Commit wait class that typically contributes wait time, while there are many in the User I/O wait class.
Charles Hooper
Co-author of "Expert Oracle Practices: Oracle Database Administration from the Oak Table"
http://hoopercharles.wordpress.com/
IT Manager/Oracle DBA
K&M Machine-Fabricating, Inc. -
Statspack Report of My DB instance
Hey Guys:
I generated a statspack report of my db instance and here is what are my findings.
I would like to hear from you experts on where I should be focussing if at all some thing is wrong. I am new to using statspack and hence this help.
DB Name DB Id Instance Inst Num Release Cluster Host
Test 3378348462 Test 1 9.2.0.7.0 NO Sonar
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 21 07-Nov-07 09:57:54 2,163 50.6
End Snap: 31 07-Nov-07 13:05:35 2,394 49.3
Elapsed: 187.68 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 2,048M Std Block Size: 4K
Shared Pool Size: 912M Log Buffer: 2,048K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 92,813.08 3,394.20
Logical reads: 28,731.61 1,050.72
Block changes: 639.05 23.37
Physical reads: 1,817.04 66.45
Physical writes: 39.44 1.44
User calls: 2,688.74 98.33
Parses: 553.54 20.24
Hard parses: 3.63 0.13
Sorts: 219.64 8.03
Logons: 3.94 0.14
Executes: 1,045.60 38.24
Transactions: 27.34
% Blocks changed per Read: 2.22 Recursive Call %: 38.00
Rollback per transaction %: 32.76 Rows per Sort: 17.22
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.97 Redo NoWait %: 100.00
Buffer Hit %: 93.69 In-memory Sort %: 100.00
Library Hit %: 99.60 Soft Parse %: 99.34
Execute to Parse %: 47.06 Latch Hit %: 99.92
Parse CPU to Parse Elapsd %: 70.22 % Non-Parse CPU: 96.20
Shared Pool Statistics Begin End
Memory Usage %: 100.00 100.00
% SQL with executions>1: 57.52 52.53
% Memory for SQL w/exec>1: 59.44 51.50
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
CPU time 19,121 37.80
db file sequential read 8,259,408 14,317 28.31
SQL*Net message from dblink 135,187 10,801 21.36
db file scattered read 792,239 2,091 4.13
ARCH wait on SENDREQ 1,385 1,077 2.13
Tablespace IO Stats for DB: EAST Instance: Test Snaps: 21 -31
->ordered by IOs (Reads + Writes) desc
Tablespace
Av Av Av Av Buffer Av Buf
Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)
SALES
4,234,917 376 1.4 1.3 12,952 1 36,425 1.2
RETAIL
1,580,550 140 2.5 4.7 18,922 2 24,786 1.7
AR
1,272,815 113 2.1 3.4 4,044 0 17,387 1.4
CATALYST
467,941 42 1.6 3.0 30,109 3 648 0.9
SALES_IDX
471,258 42 2.2 1.0 23,052 2 2,592 3.0
CATIDX
283,058 25 4.1 1.2 101,820 9 7,552 3.4
CORP
235,608 21 2.1 1.2 7,480 1 2,025 2.7
RETAILLRGIDX
201,128 18 1.8 1.0 20,337 2 1,979 3.2
ARIDX2
194,451 17 2.1 1.0 6,102 1 248 1.5
UNDOTBS
1,876 0 1.7 1.0 112,541 10 269 0.1
RETAILSMLIDX
55,340 5 2.2 1.0 5,936 1 397 3.2
SYSTEM
49,147 4 2.2 1.5 11,311 1 111 2.5
PERFSTAT
3,042 0 0.5 1.0 659 0 0 0.0
TEMP
1,687 0 552.1 36.0 1,635 0 0 0.0
IFFM
5 0 10.0 1.0 5 0 0 0.0
TOOLS
5 0 4.0 1.0 5 0 0 0.0
USERS
5 0 6.0 1.0 5 0 0 0.0
Thanks in Advance,
Regards,
- BMPWell, it depends... very much , what do you want to get? ...two situations may arise:
1) You have a performance issue
2) You're trying to be proactive and find out any 'performance gain' oportunity (or you, neither, your users have realized there is a problem...which is very rare)
My suggestions, if your case is (2)
a) Focus on Wait events
b) Analyze what is going on with excesive reads, mostly if you see disk IO and waits.
c) Try to reduce excesive executions: sometimes developers think is cheaper to ask the database 1 million times, than add 100 lines of code (Yep...it's easier for them to charge the bill on us)
... there are more suggestions, only those poped right now... sorry,I need a larger buffer cache.
Regards
http://oracledisect.blogspot.com -
Here's is the contents of v$system_event..
Is this
EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT
enq: TX - index contention 40564851 214701526 5.29
enq: TX - row lock contention 188846 12454614 65.95
enq: SQ - contention 141971 70568 0.5
cause for concern?
EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT
SQL*Net message to client 6015051449 607254 0
SQL*Net message from client 6015048542 178177969892 29.62
gcs remote message 2948555287 2633481757 0.89
CGS wait for IPC msg 1517805027 634397 0
db file sequential read 1500615188 816364485 0.54
ges remote message 1247679701 1407300224 1.13
gc cr multi block request 778432813 9913464 0.01
gc current block 2-way 747852637 38030616 0.05
db file scattered read 709428365 460939295 0.65
rdbms ipc message 708473316 37650068633 53.14
gc buffer busy acquire 671285134 1033621285 1.54
PX Deq: reap credit 667784615 484449 0
gcs log flush sync 592376026 171712257 0.29
gc cr block 2-way 530861847 19607062 0.04
library cache pin 437937120 15126237 0.03
log file sync 379523272 797193932 2.1
DIAG idle wait 359607166 2822108755 7.85
log file parallel write 351225436 259263769 0.74
LNS ASYNC end of log 350170653 1398410516 3.99
LNS wait on SENDREQ 321652621 3209301 0.01
PX qref latch 297396661 94308 0
read by other session 289140108 148440270 0.51
buffer deadlock 163505781 983055 0.01
gc current block busy 119223825 467716658 3.92
PX Deq: Table Q Normal 117332841 23574867 0.2
ksxr poll remote instances 110480324 90333 0
buffer busy waits 106938153 19933900 0.19
direct path read 93429599 108427028 1.16
SQL*Net more data from client 86471785 23026529 0.27
gc current grant busy 84978157 28215346 0.33
control file sequential read 82646297 23694583 0.29
PX Deq Credit: send blkd 78641669 9569299 0.12
latch: cache buffers chains 74218671 690277 0.01
gc current grant 2-way 72557796 1920419 0.03
library cache: mutex X 71106697 75993 0
DFS lock handle 70722498 2716407 0.04
gc cr grant 2-way 64558237 1633004 0.03
PX Deq: Execution Msg 61706261 314222076 5.09
gc cr block busy 61469863 119850802 1.95
library cache lock 52428649 3773354 0.07
PX Deq: Slave Session Stats 48040224 1886805 0.04
db file parallel read 46415188 118467902 2.55
IPC send completion sync 46250594 965101 0.02
enq: TX - index contention 40564851 214701526 5.29
PX Deq: Execute Reply 39689685 17243721 0.43
gc buffer busy release 36976909 242714774 6.56
SQL*Net more data to client 36627952 44167 0
PX Deq: Msg Fragment 30501244 343397 0.01
rdbms ipc reply 29725302 1352370 0.05
RMAN backup & recovery I/O 28824547 37722662 1.31
reliable message 27892263 3082134 0.11
PX Idle Wait 27356097 4651277341 170.03
ASM file metadata operation 25098749 8850323 0.35
gc object scan 22705857 7485 0
db file parallel write 19896252 52152606 2.62
latch: ges resource hash list 19336183 427451 0.02
enq: PS - contention 19143961 707455 0.04
PX Deq: Parse Reply 19093356 895799 0.05
gc cr disk read 17816846 448909 0.03
ASM background timer 16101806 1383957874 85.95
PX Deq: Slave Join Frag 16044789 233149 0.01
wait for unread message on broadcast channel 15056320 1413552546 93.88
cursor: mutex X 13435193 24140 0
KJC: Wait for msg sends to complete 13268497 11397 0
PX Deq: Signal ACK RSG 13214824 101941 0.01
KSV master wait 13206286 4235645 0.32
direct path read temp 12617694 5487608 0.43
PX Deq Credit: need buffer 11675868 879967 0.08
row cache lock 11536185 398216 0.03
PX Deq Credit: Session Stats 9480862 78910 0.01
SQL*Net message to dblink 9312894 1538 0
SQL*Net message from dblink 9312894 6279631 0.67
control file parallel write 7760982 11854435 1.53
pmon timer 7558889 1412576090 186.88
PX Deq: Join ACK 7548816 498931 0.07
gc current multi block request 6035173 155898 0.03
PING 5706961 1413230267 247.63
enq: XR - database force logging 4662671 198813 0.04
class slave wait 4561877 7097429006 1555.81
Streams AQ: waiting for messages in the queue 4495828 1543411682 343.3
SQL*Net more data from dblink 3696582 444575 0.12
LGWR wait for redo copy 3655353 17840 0
log file sequential read 3387305 6610414 1.95
Log archive I/O 2990486 276772 0.09
SQL*Net break/reset to client 2971976 2385935 0.8
direct path write temp 2839390 2522114 0.89
Space Manager: slave idle wait 2827526 1412987186 499.73
latch: shared pool 2808517 298150 0.11
latch: gc element 2421717 24688 0.01
SGA: MMAN sleep for component shrink 2336447 2458094 1.05
latch: enqueue hash chains 2279645 15435 0.01
latch free 2089418 78732 0.04
gc current split 2044784 1864009 0.91
PX Deq: Signal ACK EXT 1976164 19263 0.01
enq: FB - contention 1473469 61036 0.04
cursor: pin S wait on X 1313129 1464789 1.12
SQL*Net more data to dblink 1232891 986 0
Streams AQ: RAC qmn coordinator idle wait 1211300 788 0
enq: HW - contention 1175390 2077008 1.77
latch: session allocation 1167768 21883 0.02
Streams AQ: qmn coordinator idle wait 1144699 1412546634 1233.99
Streams AQ: qmn slave idle wait 1031585 2227183681 2158.99
lock deadlock retry 962937 4698 0
enq: CF - contention 956154 609647 0.64
latch: cache buffers lru chain 902764 37552 0.04
latch: object queue header operation 817911 27717 0.03
global enqueue expand wait 768633 654105 0.85
Data file init write 756191 329758 0.44
latch: gcs resource hash 647021 4147 0.01
local write wait 603007 286191 0.47
latch: row cache objects 599358 6453 0.01
ges lmd/lmses to freeze in rcfg - mrcvr 481759 156345 0.32
shared server idle wait 471190 1413238589 2999.3
enq: RF - DG Broker Current File ID 469833 23209 0.05
smon timer 432383 1411851085 3265.28
SGA: allocation forcing component growth 363333 379008 1.04
gc current retry 341104 1121252 3.29
enq: RF - synch: DG Broker metadata 319143 588290 1.84
enq: PG - contention 313659 14830 0.05
enq: TT - contention 260134 11207172 43.08
enq: KO - fast object checkpoint 236745 820808 3.47
dispatcher timer 236637 1413242481 5972.2
direct path write 231382 191008 0.83
cursor: pin S 229011 394 0
Streams AQ: waiting for time management or cleanup tasks 199981 1413148548 7066.41
enq: TX - row lock contention 188846 12454614 65.95
enq: TX - allocate ITL entry 153703 54252 0.35
enq: SQ - contention 141971 70568 0.5
ksdxexeother 141885 56 0
latch: redo allocation 138912 1858 0.01
recovery area: computing applied logs 126415 45925 0.36
gc current block congested 126318 21768 0.17
resmgr:cpu quantum 123074 151384 1.23
jobq slave wait 120678 35574221 294.79
Datapump dump file I/O 90431 9127 0.1
ges inquiry response 89402 4041 0.05
os thread startup 83809 222586 2.66
cr request retry 80062 71896 0.9
PX Deq: Table Q Sample 79665 133402 1.67
gc cr block congested 79026 14792 0.19
gc cr failure 77521 25019 0.32
enq: WF - contention 73983 825388 11.16
enq: TQ - TM contention 72871 3319 0.05
lock escalate retry 65714 1574 0.02
buffer exterminate 59775 64919 1.09
fbar timer 47136 1413183353 29980.98
log file switch completion 46911 452097 9.64
recovery area: computing obsolete files 45699 8547 0.19
enq: US - contention 40401 8805 0.22
enq: TM - contention 39149 5435032 138.83
library cache load lock 36311 382575 10.54
kjbdrmcvtq lmon drm quiesce: ping completion 31668 47443 1.5
enq: TD - KTF dump entries 31468 1424 0.05
enq: RO - fast object reuse 28422 31772 1.12
parallel recovery slave wait for change 27558 3163 0.11
name-service call wait 23694 181533 7.66
control file single write 22375 1624 0.07
kksfbc child completion 21239 106926 5.03
PX Deq: Table Q qref 19325 245 0.01
enq: TX - contention 18805 113253 6.02
latch: messages 17203 181 0.01
enq: RS - prevent file delete 16913 1013 0.06
enq: RS - prevent aging list update 15682 642 0.04
PX Deq: Table Q Get Keys 14322 42935 3
gc current grant congested 14292 2192 0.15
cursor: mutex S 13285 8 0
log file single write 13164 5371 0.41
latch: undo global data 12649 178 0.01
kksfbc research 11894 12680 1.07
parallel recovery slave idle wait 11193 5872 0.52
wait list latch free 11026 11794 1.07
enq: CT - state 11001 417 0.04
latch: checkpoint queue latch 10526 132 0.01
enq: PE - contention 10506 1139 0.11
ARCH wait on SENDREQ 9957 216480 21.74
gc cr grant congested 9465 1584 0.17
wait for scn ack 9377 3155 0.34
enq: TA - contention 8856 324 0.04
log buffer space 8777 89323 10.18
enq: TK - Auto Task Serialization 8542 343 0.04
enq: DR - contention 7842 323 0.04
process diagnostic dump 7707 2072 0.27
JOX Jit Process Sleep 7612 11286431 1482.72
enq: TC - contention 7357 340817 46.33
ges global resource directory to be frozen 7140 12299 1.72
enq: CO - master slave det 6850 312 0.05
enq: JS - job run lock - synchronize 6704 397 0.06
gcs drm freeze in enter server mode 6542 40742 6.23
enq: TS - contention 5959 89332 14.99
ARCH wait for archivelog lock 5600 36 0.01
PX Nsq: PQ load info query 5377 104798 19.49
db file single write 5373 3452 0.64
gc remaster 5315 50625 9.52
latch: parallel query alloc buffer 4939 1906 0.39
enq: TO - contention 4799 143 0.03
enq: AF - task serialization 4395 161 0.04
enq: PI - contention 4251 163 0.04
ges2 LMON to wake up LMD - mrcvr 4210 28 0.01
enq: DL - contention 3889 239 0.06
kjctssqmg: quick message send wait 3408 22 0.01
LNS wait on DETACH 3275 741 0.23
ksfd: async disk IO 3274 1 0
LNS wait on ATTACH 3273 51940 15.87
ARCH wait on DETACH 3231 714 0.22
ARCH wait on ATTACH 3226 43238 13.4
enq: BR - file shrink 2787 116 0.04
write complete waits 2631 1070 0.41
enq: MD - contention 2596 67 0.03
enq: WL - contention 2198 266518 121.25
single-task message 2098 25896 12.34
enq: OD - Serializing DDLs 2054 66 0.03
resmgr:internal state change 2001 14735 7.36
ARCH wait on c/f tx acquire 2 1751 175230 100.07
enq: WR - contention 1636 69 0.04
latch: cache buffer handles 1610 29 0.02
statement suspended, wait error to be cleared 1497 22626 15.11
Streams AQ: qmn coordinator waiting for slave to start 1214 678966 559.28
enq: PD - contention 1182 33 0.03
JS kgl get object wait 1096 4922 4.49
undo segment extension 1070 10065 9.41
PL/SQL lock timer 949 8739819 9209.5
enq: AE - lock 937 28 0.03
LGWR-LNS wait on channel 832 913 1.1
ges DFS hang analysis phase 2 acks 816 495 0.61
latch: redo writing 729 9 0.01
gc quiesce 665 564 0.85
enq: JS - queue lock 482 2111 4.38
PX Deq: Test for credit 442 13 0.03
enq: SS - contention 386 274 0.71
recovery area: computing dropped files 328 1400 4.27
recovery area: computing backed up files 328 496 1.51
ksdxexeotherwait 279 10592 37.97
log switch/archive 250 137570 550.28
gc domain validation 223 39964 179.21
auto-sqltune: wait graph update 195 96514 494.95
wait for a undo record 170 1214 7.14
parallel recovery coord send blocked 168 4 0.02
enq: JS - wdw op 168 3741 22.27
enq: KT - contention 165 5 0.03
switch logfile command 156 6290 40.32
gcs resource directory to be unfrozen 149 12839 86.17
Data Guard Broker Wait 139 10906 78.46
enq: SK - contention 129 4 0.03
enq: JS - job recov lock 128 4 0.03
gc cr block lost 125 6772 54.17
virtual circuit wait 122 3 0.03
ges LMON to get to FTDONE 100 187 1.87
enq: CU - contention 80 242 3.02
enq: JQ - contention 78 7 0.09
cursor: pin X 73 83 1.14
parallel recovery coord wait for reply 70 510 7.29
PX Deq: Txn Recovery Start 67 3436 51.29
SQL*Net break/reset to dblink 60 0 0
gc current block lost 57 2869 50.33
ges LMD suspend for testing event 51 709 13.89
inactive session 46 4550 98.91
recovery read 45 5 0.11
JS kill job wait 41 3548 86.53
enq: AS - service activation 40 1 0.03
enq: TL - contention 35 2 0.05
enq: UL - contention 34 524 15.42
gcs enter server mode 33 1559 47.24
wait for stopper event to be increased 30 218 7.27
enq: TQ - DDL contention 24 300 12.52
enq: MR - contention 21 1 0.03
ges reconfiguration to start 20 54 2.72
ges enter server mode 20 502 25.08
buffer latch 18 1337 74.26
enq: SR - contention 18 1 0.05
Streams: RAC waiting for inter instance ack 18 3748 208.21
enq: PR - contention 17 46 2.72
kupp process wait 16 166 10.39
checkpoint completed 15 3678 245.19
PX Deque wait 14 68 4.87
enq: BF - allocation contention 14 1 0.08
enq: XL - fault extent map 14 51 3.66
enq: FU - contention 14 17 1.18
enq: TH - metric threshold evaluation 13 114 8.78
enq: MW - contention 12 0 0.04
enq: DD - contention 10 0 0.04
process terminate 8 41 5.08
ges cgs registration 8 151 18.9
buffer resize 7 0 0
ktm: instance recovery 7 698 99.66
LNS wait on LGWR 6 0 0
ASM background starting 6 381 63.43
gc cr block 3-way 5 0 0.08
enq: PV - syncstart 5 9 1.74
Global transaction acquire instance locks 4 4 1.09
enq: RS - read alert level 4 0 0.04
LGWR wait on LNS 3 0 0
gc recovery 3 540 179.85
Streams AQ: enqueue blocked on low memory 3 544 181.2
DBWR range invalidation sync 3 17 5.83
enq: DM - contention 3 0 0.03
enq: RF - FSFO Observer Heartbeat 3 0 0.03
enq: JS - q mem clnup lck 3 0 0
DG Broker configuration file I/O 2 0 0
enq: RC - Result Cache: Contention 2 493 246.6
enq: KM - contention 2 0 0.03
enq: RT - contention 2 0 0.04
instance state change 2 0 0.12
kkdlgon 2 10 5.11
enq: TQ - INI contention 2 292 146.07
enq: JS - contention 2 0 0
ARCH wait for netserver start 1 400 400.02
log file switch (checkpoint incomplete) 1 3 3.44
JS coord start wait 1 50 50.09
ges lmd and pmon to attach 1 1 1.26
wait for tmc2 to complete 1 3 3.03
control file heartbeat 1 400 400.02
enq: SW - contention 1 0 0.04
enq: PW - perwarm status in dbw0 1 0 0.09
enq: FS - contention 1 0 0.04
enq: XR - quiesce database 1 0 0.04
enq: RS - write alert level 1 0 0.02
enq: CN - race with init 1 0 0.03
enq: FE - contention 1 4 3.77
Wait for shrink lock2 1 10 10.03
enq: IA - contention 1 0 0.02
enq: RF - atomicity 1 0 0.05
enq: RF - synchronization: aifo master 1 0 0.02
enq: RF - RF - Database Automatic Disable 1 0 0.06
enq: WP - contention 1 0 0.02
enq: TB - SQL Tuning Base Cache Load 1 0 0.05
enq: JS - evt notify 1 0 0.02Edited by: steffi on Mar 20, 2011 12:21 AM
Edited by: steffi on Mar 20, 2011 8:18 AM
Edited by: steffi on Mar 20, 2011 8:19 AMText can be formatted by tagging the beginning and end of the block of text with the code ta
\Formatted text goes here.
\Example:
This is formatted.When cutting and pasting text such as execution plans, excerpts from AWR reports, etc, it will maintain spacing and formatting, and make for much easier reading.
As to the content, well, dumping the contents of v$system_event is pretty close to useless.
As to the first three events you listed, 'enq: TX - index contention', 'enq: TX - row lock contention', 'enq: SQ - contention', well, all of those are easily tunable.
First, for 'enq: SQ - contention', check your sequences. Do you have any NOCACHE sequences? Or sequences with small caches?
As for 'enq: TX - row lock contention', well that's fairly self-explanatory. You have multiple sessions trying to lock the same row in the same table at the same time.
Last, 'enq: TX - index contention', that's non-row level contention on an index. For example, if you have a unique index, insert a row w/ column value 1, but don't commit, then try to insert that same value from another session.
But, before you do any of that, I think you need to clearly understand where the bottlenecks are. Try taking some AWR snapshots, about 5 minutes apart, when you're having performance problems. Look at the AWR report for that 5 minute period. In particular, look at your Top 5 timed events.
Hope that helps,
-Mark -
Log file switch (checkpoint not complete)
HI,
I am using Oracle 9.2 on rhel
IN the statspack report I am getting one of the event I.e log file switch (checkpoint not complete).Statspack duration is about 1.5 hrs...any suggestion
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.97 Redo NoWait %: 98.31
Buffer Hit %: 95.84 In-memory Sort %: 100.00
Library Hit %: 99.57 Soft Parse %: 98.51
Execute to Parse %: 72.70 Latch Hit %: 99.71
Parse CPU to Parse Elapsd %: 53.15 % Non-Parse CPU: 99.10
Shared Pool Statistics Begin End
Memory Usage %: 93.66 93.74
% SQL with executions>1: 60.41 60.94
% Memory for SQL w/exec>1: 60.89 61.66
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
log file switch (checkpoint incomplete) 35,936 35,100 42.81
enqueue 6,144 16,684 20.35
buffer busy waits 17,190 13,346 16.28
wait for a undo record 51,967 4,931 6.01
ARCH wait on SENDREQ 877 4,813 5.87
-------------------------------------------------------------Please find the whole stats[pack report
{code}
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
ICAI 1504443695 icai 1 9.2.0.8.0 NO icaidb.icai.
org
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 70 04-Aug-10 14:27:14 162 34.7
End Snap: 73 04-Aug-10 15:30:43 254 55.4
Elapsed: 63.48 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 4,928M Std Block Size: 8K
Shared Pool Size: 1,312M Log Buffer: 1,024K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 96,260.76 53,769.94
Logical reads: 13,998.20 7,819.20
Block changes: 1,227.83 685.85
Physical reads: 592.13 330.76
Physical writes: 19.93 11.13
User calls: 313.12 174.91
Parses: 31.41 17.55
Hard parses: 0.47 0.26
Sorts: 11.61 6.49
Logons: 0.11 0.06
Executes: 115.04 64.26
Transactions: 1.79
% Blocks changed per Read: 8.77 Recursive Call %: 26.28
Rollback per transaction %: 5.43 Rows per Sort: 472.17
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.97 Redo NoWait %: 98.31
Buffer Hit %: 95.84 In-memory Sort %: 100.00
Library Hit %: 99.57 Soft Parse %: 98.51
Execute to Parse %: 72.70 Latch Hit %: 99.71
Parse CPU to Parse Elapsd %: 53.15 % Non-Parse CPU: 99.10
Shared Pool Statistics Begin End
Memory Usage %: 93.66 93.74
% SQL with executions>1: 60.41 60.94
% Memory for SQL w/exec>1: 60.89 61.66
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
log file switch (checkpoint incomplete) 35,936 35,100 42.81
enqueue 6,144 16,684 20.35
buffer busy waits 17,190 13,346 16.28
wait for a undo record 51,967 4,931 6.01
ARCH wait on SENDREQ 877 4,813 5.87
Wait Events for DB: ICAI Instance: icai Snaps: 70 -73
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
log file switch (checkpoint 35,936 35,886 35,100 977 5.3
enqueue 6,144 5,660 16,684 2716 0.9
buffer busy waits 17,190 5,325 13,346 776 2.5
wait for a undo record 51,967 49,137 4,931 95 7.6
ARCH wait on SENDREQ 877 0 4,813 5489 0.1
log file switch (archiving n 3,705 3,653 3,600 972 0.5
db file sequential read 600,718 0 621 1 88.1
log file sync 6,826 140 561 82 1.0
log file parallel write 7,052 0 421 60 1.0
log buffer space 1,361 16 230 169 0.2
db file scattered read 289,115 0 212 1 42.4
switch logfile command 116 23 160 1377 0.0
wait for stopper event to be 1,924 1,111 123 64 0.3
control file parallel write 1,355 0 63 46 0.2
PX Deq: Txn Recovery Start 1,253 0 36 29 0.2
SQL*Net break/reset to clien 560 0 20 36 0.1
local write wait 18 15 17 918 0.0
log file switch completion 21 7 9 442 0.0
control file sequential read 237,021 0 6 0 34.8
log file sequential read 437 0 6 13 0.1
BFILE get length 297 0 2 7 0.0
latch free 485 67 2 4 0.1
BFILE read 1,023 0 1 1 0.2
log file single write 18 0 0 16 0.0
SQL*Net more data to client 13,785 0 0 0 2.0
process startup 10 0 0 9 0.0
control file single write 10 0 0 4 0.0
row cache lock 34 0 0 0 0.0
db file single write 1 0 0 14 0.0
LGWR wait for redo copy 89 0 0 0 0.0
PX Deq: Signal ACK 3 0 0 4 0.0
PX Deq: Join ACK 5 0 0 1 0.0
BFILE open 106 0 0 0 0.0
db file parallel read 25 0 0 0 0.0
async disk IO 1,383 0 0 0 0.2
db file parallel write 255 0 0 0 0.0
BFILE internal seek 1,023 0 0 0 0.2
direct path read 843 0 0 0 0.1
BFILE closure 106 0 0 0 0.0
undo segment extension 844 844 0 0 0.1
direct path write 96 0 0 0 0.0
SQL*Net message from client 1,188,764 0 445,926 375 174.3
virtual circuit status 125 125 3,660 29277 0.0
wakeup time manager 86 86 2,451 28506 0.0
PX Idle Wait 755 750 1,466 1941 0.1
jobq slave wait 60 60 176 2930 0.0
SQL*Net more data from clien 3,035 0 1 0 0.4
SQL*Net message to client 1,188,882 0 1 0 174.3
Wait Events for DB: ICAI Instance: icai Snaps: 70 -73
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
Background Wait Events for DB: ICAI Instance: icai Snaps: 70 -73
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
ARCH wait on SENDREQ 877 0 4,813 5489 0.1
buffer busy waits 2,164 1,176 1,184 547 0.3
log file parallel write 7,052 0 421 60 1.0
wait for stopper event to be 1,924 1,111 123 64 0.3
control file parallel write 1,301 0 57 44 0.2
enqueue 393 0 10 26 0.1
control file sequential read 234,452 0 6 0 34.4
log file sequential read 431 0 6 13 0.1
db file scattered read 124 0 1 10 0.0
log buffer space 40 0 1 13 0.0
log file single write 15 0 0 20 0.0
db file sequential read 20 0 0 9 0.0
process startup 7 0 0 8 0.0
latch free 19 3 0 2 0.0
LGWR wait for redo copy 89 0 0 0 0.0
PX Deq: Signal ACK 3 0 0 4 0.0
PX Deq: Join ACK 5 0 0 1 0.0
db file parallel write 255 0 0 0 0.0
async disk IO 793 0 0 0 0.1
rdbms ipc reply 1 0 0 0 0.0
direct path read 88 0 0 0 0.0
direct path write 88 0 0 0 0.0
rdbms ipc message 50,357 3,296 16,467 327 7.4
pmon timer 1,271 1,251 3,653 2874 0.2
smon timer 64 0 2,302 35963 0.0
SQL ordered by Gets for DB: ICAI Instance: icai Snaps: 70 -73
-> End Buffer Gets Threshold: 10000
-> Note that resources reported for PL/SQL includes the resources used by
all SQL statements called within the PL/SQL code. As individual SQL
statements are also reported, it is possible and valid for the summed
total % to exceed 100
CPU Elapsd
Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value
5,652,833 456 12,396.6 10.6 20.04 1062.67 285625578
SELECT /*+ INDEX(OT_DAK_ENTRY_DETL IDM_DED_NAME) */DEH_TXN_CODE
,DEH_NO FROM OT_DAK_ENTRY_HEAD,OT_DAK_ENTRY_DETL WHERE DEH_SY
S_ID = DED_DEH_SYS_ID AND TRUNC(DEH_APPLICATION_DT) = :b1 AND
DED_DAK_CODE = :b2 AND DED_NAME LIKE LTRIM(RTRIM(:b3)) || '%'
AND NVL(DED_INSTR_NO,'XXXXXX') = NVL(:b4,'XXXXXX') AND TRUNC(
5,096,348 189 26,964.8 9.6 23.64 23.25 1772835295
select decode(level,1,'',2,' ',3,' ',4,' ',5,'
', ' ') || decode(:1,'ENG',menu_option_desc,menu_opt
ion_desc_bl) "OPTION", menu_parent_id "PARENT", menu_action_type
"TYPE",menu_action "ACTION", decode(level,1,'',2,' ',3,' ',4,'
',5,' ', ' ') ||decode(menu_action_type, 'M', '+', 'o'
4,894,185 96 50,981.1 9.2 7.89 10.11 23088203
INSERT INTO OT_MEM_FEE_COL_DETL(MFCD_FEE_TYPE,MFCD_CONDON_FEE_YN
,MFCD_EXCESS_USED_YN,MFCD_CONDN_CODE,MFCD_PM_CODE,MFCD_CURR_CODE
,MFCD_INSTR_NO,MFCD_INSTR_DT,MFCD_AMT,MFCD_BANK_CODE,MFCD_INSTR_
TYPE,MFCD_BRANCH,MFCD_COLLECTION,MFCD_FM_DT,MFCD_TO_DT,MFCD_RES_
CODE,MFCD_CR_UID,MFCD_CR_DT,MFCD_UPD_UID,MFCD_UPD_DT,MFCD_CONDON
4,885,684 152 32,142.7 9.2 7.64 8.87 1007886847
SELECT MIN(MFCH_NO) FROM OT_MEM_FEE_COL_HEAD, OT_MEM_FEE_COL_DET
L WHERE MFCH_SYS_ID = MFCD_MFCH_SYS_ID AND MFCH_REF_NO = :B4 AND
MFCH_REF_TXN_CODE = :B3 AND MFCD_INSTR_NO = :B2 AND MFCD_BANK_C
ODE = :B1 AND MFCD_AMT > 0
2,680,356 446 6,009.8 5.0 95.57 125.49 197211170
SELECT /*+ INDEX(OT_STUDENT_FEE_COL_IPCC_HEAD OT_STUDENT_FEE_COL
IPCCHEAD_UK01) */SFCH_SYS_ID,SFCH_DT,DECODE(NVL(SSTN_SRN,SFCH
TEMPREF_NO),SSTN_SRN, NULL ,SFCH_TEMP_REF_NO) SFCH_TEMP_REF_NO
,NVL(SFCH_STUD_SRN,SSTN_SRN) SFCH_STUD_SRN,SFCH_COURSE_CODE,SFCH
SCHEMECODE,SFCH_EXMP_STUD_YN,SFCH_EXMP_STUD_REASON,DEH_APPLICA
2,288,204 1 2,288,204.0 4.3 54.31 59.36 3103356680
DECLARE job BINARY_INTEGER := :job; next_date DATE := :mydate;
broken BOOLEAN := FALSE; BEGIN BEGIN /*Quest SOO PPCM job */ qu
est_ppcm_snapshot_pkg.take_snapshot; END; :mydate := next_date;
IF broken THEN :b := 1; ELSE :b := 0; END IF; END;
2,253,877 1 2,253,877.0 4.2 52.70 56.21 579012758
DELETE FROM QUEST_PPCM_SQL_TEXT TXT WHERE INSTANCE_ID >= 0 AND N
OT EXISTS (SELECT 1 FROM QUEST_PPCM_SQL_SNAPSHOT SNAP WHERE SNAP
.SNAPSHOT_ID > 0 AND SNAP.INSTANCE_ID= TXT.INSTANCE_ID AND SNAP.
SQL_ID = TXT.SQL_ID)
1,656,006 24 69,000.3 3.1 10.00 24.26 4081782417
SELECT PISH_COURSE_CODE FROM OV_STU_PAYINSLIP_IPCC_DTL WHERE
PISH_BANK_CODE = :b1 AND PISH_NO BETWEEN :b2 AND :b3 AND PISD_
CURR_CODE = :b4 AND PISH_REGION_CODE = :b5 ORDER BY 1
SQL ordered by Gets for DB: ICAI Instance: icai Snaps: 70 -73
-> End Buffer Gets Threshold: 10000
-> Note that resources reported for PL/SQL includes the resources used by
all SQL statements called within the PL/SQL code. As individual SQL
statements are also reported, it is possible and valid for the summed
total % to exceed 100
CPU Elapsd
Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value
1,567,946 12 130,662.2 2.9 21.99 47.39 1585476974
SELECT NVL(TO_CHAR(A.DEH_APPLICATION_DT,'DD/MM/RRRR'), NULL ),NV
L(TO_CHAR(A.DEH_DT,'DD/MM/RRRR'), NULL ) FROM OT_DAK_ENTRY_HEA
D A,OT_DAK_ENTRY_DETL B,OT_FIRM_NAME_APPR_HEAD C WHERE A.DEH_SY
S_ID = B.DED_DEH_SYS_ID AND B.DED_DAK_SYS_ID = C.FNAH_DED_SYS_I
D AND C.FNAH_SYS_ID = (SELECT MAX(B.FNAH_SYS_ID) FROM OT_FIR
1,216,226 4 304,056.5 2.3 9.90 54.25 937031003
SELECT TRIM(STUD_SRN)
Q1_REGNO, TR
IM(STUD_TEMP_REF_NO)
Q1_TEMPNO, STUD_TITLE
1,138,801 178 6,397.8 2.1 18.13 1009.80 1617597
SELECT SRN,ACTIVITYDESCRIPTION,STATUS,DOCUMENTNO,DOCUMENTDATE FR
OM OV_ART_TRANS_STATUS WHERE (SRN=:1) order by DOCUMENTDATE
1,029,221 230 4,474.9 1.9 20.27 20.76 1838125769
SELECT MRH_DT,MRH_FIRST_NAME,MRH_MIDDLE_NAME,MRH_SUR_NAME,MRH_ST
ATUS FROM OM_MEM_REG_HEAD WHERE DECODE(:b1,1,MRH_MRN,MRH_MFCH
TEMPREF_NO) = :b2
778,949 52 14,979.8 1.5 20.32 21.85 4142254844
SELECT LTRIM(RTRIM(DECODE(TIT_NAME,'MR.','CA.','MS.','CA.','MRS.
','CA.') || ' ' || MRH_FIRST_NAME || ' ' || MRH_MIDDLE_NAME
|| ' ' || MRH_SUR_NAME || ' ' || DECODE(MRH_APPR_UID, NULL ,
NULL ,DECODE(MRH_MEM_STATUS,2, NULL ,DECODE(MRH_FELLOW_STATUS_YN
,'Y','FCA','ACA'))) || DECODE(MRH_RESI_STATUS,'A','
755,893 517 1,462.1 1.4 90.43 89.07 1033584013
SELECT DECODE(MFCD_FEE_TYPE,'M08',1,'M05',2,'M06',3,'M09',4,'M10
',5,'M11',6,'M12',7,'M07',8,'M13',9,'M14',10,'M15',11,'M04',12,'
M03',13,'M02',14,'M01',15,'M21',16,'M22',17,'M23',18,'EXCESS',19
,20) FEE_SEQ,MFCD_FEE_TYPE FEE_TYPE,SUM(MFCD_AMT) AMOUNT FROM
OT_MEM_FEE_COL_HEAD,OT_MEM_FEE_COL_DETL WHERE MFCH_SYS_ID = MFC
751,010 1,090 689.0 1.4 28.61 31.41 1734754400
SELECT ROWID,PIIPD_ICAI_EXAM_APPEARED,PIIPD_REG_NO,PIIPD_MTH,PII
PD_YR,PIIPD_ROLL_NO,PIIPD_EXT_EXAM_APPEARED,PIIPD_EXAM_CODE_1,PI
IPD_SUBJ_CODE_1,PIIPD_GROUP_1,PIIPD_ROLL_NO_1,PIIPD_LAST_PAPER_D
T_1,PIIPD_EXAM_CODE_2,PIIPD_SUBJ_CODE_2,PIIPD_GROUP_2,PIIPD_ROLL
SQL ordered by Reads for DB: ICAI Instance: icai Snaps: 70 -73
-> End Disk Reads Threshold: 1000
CPU Elapsd
Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
227,513 2 113,756.5 10.1 22.04 79.36 2837394537
SELECT STUD_SRN,STUD_FIRST_NAME,STUD_MIDDLE_NAME,STUD_MAIDEN_NAM
E,STUD_SURNAME,STUD_FATHER_NAME,STUD_BIRTH_DT,STUD_COMM_CODE,STU
D_SEX,STUD_HANDICAPPED_YN,STUD_HANDICAPPED_REASON,STUD_LANG_CODE
,STUD_NATIONALITY_CODE,STUD_EMAIL,STUD_PERMNT_ADDR_LINE_1,STUD_P
ERMNT_ADDR_LINE_2,STUD_PERMNT_ADDR_LINE_3,STUD_PERMNT_ADDR_LINE_
220,939 12 18,411.6 9.8 21.99 47.39 1585476974
SELECT NVL(TO_CHAR(A.DEH_APPLICATION_DT,'DD/MM/RRRR'), NULL ),NV
L(TO_CHAR(A.DEH_DT,'DD/MM/RRRR'), NULL ) FROM OT_DAK_ENTRY_HEA
D A,OT_DAK_ENTRY_DETL B,OT_FIRM_NAME_APPR_HEAD C WHERE A.DEH_SY
S_ID = B.DED_DEH_SYS_ID AND B.DED_DAK_SYS_ID = C.FNAH_DED_SYS_I
D AND C.FNAH_SYS_ID = (SELECT MAX(B.FNAH_SYS_ID) FROM OT_FIR
198,343 2 99,171.5 8.8 5.82 46.25 1414719916
UPDATE OM_MEM_REG_HEAD SET MRH_MRN=:b1 WHERE MRH_SYS_ID = :b2
198,343 2 99,171.5 8.8 5.81 46.10 1414796677
UPDATE OT_DAK_ACTV_HISTORY SET DAH_REG_NO = :B1 WHERE DAH_REG_NO
= :B3 AND TRUNC(DAH_ACTV_ED_DT ) <= TRUNC(:B2 ) AND DAH_ACTV_ST
ATUS = 'C'
173,892 2 86,946.0 7.7 13.85 16.34 3262067067
SELECT STUD_SRN,STUD_FIRST_NAME,STUD_MIDDLE_NAME,STUD_MAIDEN_NAM
E,STUD_SURNAME,STUD_FATHER_NAME,STUD_BIRTH_DT,STUD_COMM_CODE,STU
D_SEX,STUD_HANDICAPPED_YN,STUD_HANDICAPPED_REASON,STUD_LANG_CODE
,STUD_NATIONALITY_CODE,STUD_EMAIL,STUD_PERMNT_ADDR_LINE_1,STUD_P
ERMNT_ADDR_LINE_2,STUD_PERMNT_ADDR_LINE_3,STUD_PERMNT_ADDR_LINE_
112,038 9 12,448.7 5.0 9.23 10.14 2058267852
SELECT ROWID,STUD_DT,STUD_TXN_CODE,STUD_NO,STUD_AMD_NO,STUD_REF_
FROM,STUD_REF_TXN_CODE,STUD_REF_NO,STUD_TEMP_REF_NO,STUD_SRN,STU
D_TITLE,STUD_STATUS,STUD_FIRST_NAME,STUD_MIDDLE_NAME,STUD_SURNAM
E,STUD_MAIDEN_NAME,STUD_NAME_STATUS,STUD_FATHER_NAME,STUD_NATION
ALITY_CODE,STUD_NATION_PROOF_ENCL_YN,STUD_SEX,STUD_HANDICAPPED_Y
102,583 1 102,583.0 4.5 4.51 285.99 802587273
SELECT ROWID,DEH_DT,DEH_TXN_CODE,DEH_NO,DEH_AMD_NO,DEH_REF_FROM,
DEH_REF_TXN_CODE,DEH_REF_NO,DEH_REF_SYS_ID,DEH_REGION_CODE,DEH_A
PPLICATION_DT,DEH_DOC_STATUS,DEH_STATUS,DEH_PRINT_STATUS,DEH_CLO
STATUS,DEHSYS_ID,DEH_COMP_CODE,DEH_ACNT_YR,DEH_AMD_DT,DEH_AMD_
UID,DEH_AMD_RES_CODE,DEH_REF_FROM_NUM,DEH_CR_UID,DEH_CR_DT,DEH_U
98,825 3 32,941.7 4.4 6.70 6.66 2078892348
/*SELECT STUD_SRN
Q1_RE
GNO, O_GET_OLD_REG_NO(STUD_SRN, :BP_COURSE) OLD_NO, TIT_NAME||'
'||STUD_FIRST_NAME||' '||STUD_MIDDLE_NAME||' '||STUD_SURNAME
Q1_NAME, LTRIM(RTRIM(A.STUD_CORRES_ADDR_LINE_1 ))||DECO
96,187 2 48,093.5 4.3 10.78 16.53 3301514821
SELECT MFCD_PAYIN_SLIP_NO MFCD_PAYIN_SLIP_NO
, DECODE(MFCD_BANK_CODE ,'ICI',1,2) ICI_FIRST
, DECODE(MFCD_INSTR_TYPE,'S',1,'L
SQL ordered by Reads for DB: ICAI Instance: icai Snaps: 70 -73
-> End Disk Reads Threshold: 1000
CPU Elapsd
Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
',2,'O',3,4) TYPE_FIRST , MFCD_PAYIN_SLIP_DT , FM_BANK
.BANK_NAME , BAD_ADDR1 ADD1
80,683 4 20,170.8 3.6 9.90 54.25 937031003
SELECT TRIM(STUD_SRN)
Q1_REGNO, TR
IM(STUD_TEMP_REF_NO)
Q1_TEMPNO, STUD_TITLE
77,972 8 9,746.5 3.5 8.92 8.84 2241526944
SELECT ROWID,STUD_DT,STUD_TXN_CODE,STUD_NO,STUD_AMD_NO,STUD_REF_
FROM,STUD_REF_TXN_CODE,STUD_REF_NO,STUD_TEMP_REF_NO,STUD_SRN,STU
D_TITLE,STUD_STATUS,STUD_FIRST_NAME,STUD_MIDDLE_NAME,STUD_SURNAM
E,STUD_MAIDEN_NAME,STUD_NAME_STATUS,STUD_FATHER_NAME,STUD_NATION
ALITY_CODE,STUD_NATION_PROOF_ENCL_YN,STUD_SEX,STUD_HANDICAPPED_Y
75,667 3 25,222.3 3.4 3.34 25.09 3345305231
SELECT DISTINCT SFCH_STUD_SRN FROM OT_STUDENT_FEE_COL_HEAD A,O
T_STUDENT_FEE_COL_DETL B WHERE B.SFCD_SFCH_SYS_ID = A.SFCH_SYS_
ID AND B.SFCD_INSTR_BANK_CODE = :b1 AND B.SFCD_INSTR_NO = :b2
72,658 52 1,397.3 3.2 20.32 21.85 4142254844
SELECT LTRIM(RTRIM(DECODE(TIT_NAME,'MR.','CA.','MS.','CA.','MRS.
','CA.') || ' ' || MRH_FIRST_NAME || ' ' || MRH_MIDDLE_NAME
|| ' ' || MRH_SUR_NAME || ' ' || DECODE(MRH_APPR_UID, NULL ,
NULL ,DECODE(MRH_MEM_STATUS,2, NULL ,DECODE(MRH_FELLOW_STATUS_YN
,'Y','FCA','ACA'))) || DECODE(MRH_RESI_STATUS,'A','
48,619 3 16,206.3 2.2 4.19 4.11 496772197
SELECT ROWID,STUD_DT,STUD_TXN_CODE,STUD_NO,STUD_AMD_NO,STUD_REF_
FROM,STUD_REF_TXN_CODE,STUD_REF_NO,STUD_TEMP_REF_NO,STUD_SRN,STU
D_TITLE,STUD_STATUS,STUD_FIRST_NAME,STUD_MIDDLE_NAME,STUD_SURNAM
E,STUD_MAIDEN_NAME,STUD_NAME_STATUS,STUD_FATHER_NAME,STUD_NATION
ALITY_CODE,STUD_NATION_PROOF_ENCL_YN,STUD_SEX,STUD_HANDICAPPED_Y
48,063 230 209.0 2.1 20.27 20.76 1838125769
SELECT MRH_DT,MRH_FIRST_NAME,MRH_MIDDLE_NAME,MRH_SUR_NAME,MRH_ST
ATUS FROM OM_MEM_REG_HEAD WHERE DECODE(:b1,1,MRH_MRN,MRH_MFCH
SQL ordered by Executions for DB: ICAI Instance: icai Snaps: 70 -73
-> End Executions Threshold: 100
CPU per Elap per
Executions Rows Processed Rows per Exec Exec (s) Exec (s) Hash Value
38,614 38,614 1.0 0.00 0.00 1741347688
SELECT SYSDATE FROM SYS.DUAL
12,490 12,488 1.0 0.00 0.00 2614155871
SELECT DECODE(:b1,'L',DECODE(:b2,'ENG',STATUS_NAME,STATUS_BL_NAM
E),DECODE(:b2,'ENG',STATUS_SHORT_NAME,STATUS_BL_SHORT_NAME)) STA
TUS_DESC,STATUS_FRZ_FLAG_NUM FROM OM_STATUS WHERE STATUS_CODE
= :b4
8,629 8,628 1.0 0.00 0.00 1644340447
SELECT DECODE(:b1,'L',DECODE(:b2,'ENG',FEE_TYPE_NAME,FEE_TYPE_BL
NAME),DECODE(:b2,'ENG',FEETYPE_SHORT_NAME,FEE_TYPE_BL_SHORT_NA
ME)) FEE_NAME,FEE_TYPE_FRZ_FLAG_NUM FROM OM_FEE_TYPE WHERE FE
E_TYPE_CODE = :b4
7,275 7,272 1.0 0.00 0.34 3716207873
update seq$ set increment$=:2,minvalue=:3,maxvalue=:4,cycle#=:5,
order$=:6,cache=:7,highwater=:8,audit$=:9,flags=:10 where obj#=:
1
6,293 6,283 1.0 0.00 0.00 2804237544
SELECT DECODE(:b1,'L',DECODE(:b2,'ENG',CITY_NAME,CITY_BL_NAME),D
ECODE(:b2,'ENG',CITY_SHORT_NAME,CITY_BL_SHORT_NAME)) CITY_NAME,C
ITY_TALUK_CODE,CITY_DIST_CODE,CITY_STATE_CODE,CITY_REGION_CODE,C
ITY_FRZ_FLAG_NUM FROM OM_CITY WHERE CITY_CODE = :b4
6,221 6,221 1.0 0.00 0.00 484036617
SELECT DAH_SYS_ID.NEXTVAL FROM DUAL
6,221 6,221 1.0 0.00 0.00 2945494810
SELECT COUNT(DAH_SYS_ID) FROM OT_DAK_ACTV_HISTORY WHERE DAH_ACTV
TYPE = :B2 AND DAHTXN_SYS_ID = :B1
5,979 5,979 1.0 0.00 0.00 35936114
SELECT STUD_DOC_STATUS FROM OM_STUDENT_HEAD WHERE STUD_SYS_ID
= :b1
4,637 4,637 1.0 0.00 0.00 1237293873
SELECT DECODE(:b1,'L',DECODE(:b2,'ENG',COU_NAME,COU_BL_NAME),DEC
ODE(:b2,'ENG',COU_SHORT_NAME,COU_BL_SHORT_NAME)) COU_NAME,COU_FR
Z_FLAG_NUM FROM OM_COUNTRY WHERE COU_CODE = :b4
4,404 1,276 0.3 0.00 0.00 1829426463
SELECT NVL(TAU_FROM_VALUE,0), NVL(TAU_TO_VALUE,0) FROM IM_TXN_AU
TH_USER WHERE TAU_TA_TYPE = :B3 AND TAU_TXN_CODE = :B2 AND TAU_A
UTH_UID = :B1
4,220 4,220 1.0 0.00 0.00 1006906503
UPDATE OT_DAK_ACTV_HISTORY SET DAH_ACTV_ED_DT = :B3 , DAH_ACTV_S
TATUS = 'C' WHERE DAH_ACTV_TYPE = :B2 AND DAH_TXN_SYS_ID = :B1
3,874 66 0.0 0.00 0.00 4284733339
SELECT TIT_NAME ||' '|| AR_FIRST_NAME||' '||AR_MIDDLE_NAME||' '|
|AR_SUR_NAME FROM OT_ARTICLE_REGISTRATION, OM_TITLE WHERE TRIM(A
SQL ordered by Executions for DB: ICAI Instance: icai Snaps: 70 -73
-> End Executions Threshold: 100 -
Physical standby crash leads to peculiar process behavior on primary nodes
Hey all, we had an issue early this morning with our physical standby database in our production environment. It's an odd problem and I thought I'd share it to see if anyone else has come across a similar problem.
Our environment consists of a two node RAC cluster running 10.2.0.2 Enterprise on Red-Hat x86-64 Linux boxes. We have implemented a single physical standby at a remote site using real-time log apply.
Upon arrival this morning I discovered that a large portion of the primary database activity was a wait class event(ARCH wait on SENDREQ). The problem was quickly identified as a network issue between our site and the remote standby. However, the physical standby seemed to be running normal, with no messages in the alert logs regarding network disconnects. Further investigation showed that our log-apply had stopped all together early that morning. It seems the DBWR process crashed around 630am, as per this trace dump:
*** 2008-02-27 06:24:53.667
Waited for detached process: DBW0 for 300 seconds:
*** 2008-02-27 06:24:53.667
Dumping diagnostic information for DBW0:
OS pid = 10587
loadavg : 12.11 11.67 8.95
memory info: free memory = 0.00M
swap info: free = 0.00M alloc = 0.00M total = 0.00M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S oracle 2448 31813 0 78 0 - 13190 wait 06:07 ? 00:00:00 /bin/sh -c /usr/bin/gdb -quiet -x
/tmp/stacknvvano
/proc/10587/exe 10587 < /dev/null 2>&1
0 S oracle 2449 2448 0 78 0 - 38780 wait 06:07 ? 00:00:00 /usr/bin/gdb -quiet -x
/tmp/stacknvvano /proc/10587
/exe 10587
0 D oracle 10587 1 0 76 0 - 43072 - Jan04 ? 00:04:09 ora_dbw0_schclsdb
So what we did was restart the standby server and the database came up fine. We reinitiated log apply services and both databases did sync up. However
we are still seeing the "ARCH wait on SENDREQ" wait event from ARC0 and ARC1 on both the RAC instances.
We tried reseting the connection to the standby database, however did this not clear up the wait event. The system itself, as far as I can tell is not suffering, and redo logs are going across to the standby database without issue. What could be the problem with the ARC0 and ARC1 processes?
It would probably be helpful to mention that we don't actually use the ARCH processes for the log apply unless the standby falls behind. Archiving is handled by the local LNS process and remote RFS service. A quick instance restart would fix it, to be sure, but since this is happening in a production deployment in a 24x7 environment, I'd rather not have any downtime. A service request is of course filed, but since they have not gotten back to me yet and Metalink hasn't been of much help, I thought I'd see if any of you have seen this before. This giant gray blob on my grid control histogram is quite obnoxious. =) Thanks for taking the time to read. Cheers.
-Jim HenaghanJust as an update to anyone else who may have this problem, we haven't found a good way to clear up our Archiver issue, so a instance restart will take place. Further research indicates the issue came about from a MRP process crash due to a known bug fixed in 10.2.0.4 (https://metalink.oracle.com/metalink/plsql/f?p=130:14:316259464254960300::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,5894403.8,1,0,1,helvetica)
-
Archive log wating..............
hi,
When i hv fire this query archive is wating ,what is a problem i can't understand,
my senior says that find out SCN of archive wating , but which vew
i hv try v$log ,v$log_history,v$logfile
but no get answer pls suggest which view give me information of archive wait event.
SQL> select event,sid,state,p1,p2,p3 from v$session_wait
where event='ARCH wait on SENDREQ';
2
EVENT SID STATE P1 P2 P3
ARCH wait on SENDREQ 428 WAITING 0 0 0You have seen here V$ARCHIVE_DEST. Was it achiving at the defined locations?
LOG_ARCHIVE_MAX_PROCESSES initialization parameter specifies the number of ARCn processes that the database initially invokes.
However, to avoid any runtime overhead of invoking additional ARCn processes, you can set the LOG_ARCHIVE_MAX_PROCESSES initialization parameter to specify up to ten ARCn processes to be started at instance startup.
http://download.oracle.com/docs/cd/B19306_01/server.102/b14231/archredo.htm#sthref1057
You can also this
V$ARCHIVE_PROCESSES
Displays information about the state of the various archive processes for an instance.
V$LOG_HISTORY
Contains log history information such as which logs have been archived and the SCN range for each archived log.
Adith -
Hi: I'm analyzing this STATSPACK report: it is "volume test" on our UAT server, so most input is from 'bind variables'. Our shared pool is well utilized in oracle. Oracle redo logs is not appropriately configured on this server, as in 'Top 5 wait events' there are 2 for redos.
I need to know what else information can be dig-out from 'foreground wait events' & 'background wait events', and what can assist us to better understanding, in combination of 'Top 5 wait event's, that how the server/test went? it could be overwelming No. of wait events, so appreciate any helpful diagnostic or analysis. Database is oracle 11.2.0.4 upgraded from 11.2.0.3, on IBM AIX power system 64bit, level 6.x
STATSPACK report for
Database DB Id Instance Inst Num Startup Time Release RAC
~~~~~~~~ ----------- ------------ -------- --------------- ----------- ---
700000XXX XXX 1 22-Apr-15 12:12 11.2.0.4.0 NO
Host Name Platform CPUs Cores Sockets Memory (G)
~~~~ ---------------- ---------------------- ----- ----- ------- ------------
dXXXX_XXX AIX-Based Systems (64- 2 1 0 16.0
Snapshot Snap Id Snap Time Sessions Curs/Sess Comment
~~~~~~~~ ---------- ------------------ -------- --------- ------------------
Begin Snap: 5635 22-Apr-15 13:00:02 114 4.6
End Snap: 5636 22-Apr-15 14:00:01 128 8.8
Elapsed: 59.98 (mins) Av Act Sess: 0.6
DB time: 35.98 (mins) DB CPU: 19.43 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 2,064M Std Block Size: 8K
Shared Pool: 3,072M Log Buffer: 13,632K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ ------------------ ----------------- ----------- -----------
DB time(s): 0.6 0.0 0.00 0.00
DB CPU(s): 0.3 0.0 0.00 0.00
Redo size: 458,720.6 8,755.7
Logical reads: 12,874.2 245.7
Block changes: 1,356.4 25.9
Physical reads: 6.6 0.1
Physical writes: 61.8 1.2
User calls: 2,033.7 38.8
Parses: 286.5 5.5
Hard parses: 0.5 0.0
W/A MB processed: 1.7 0.0
Logons: 1.2 0.0
Executes: 801.1 15.3
Rollbacks: 6.1 0.1
Transactions: 52.4
Instance Efficiency Indicators
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 99.98 Optimal W/A Exec %: 100.00
Library Hit %: 99.77 Soft Parse %: 99.82
Execute to Parse %: 64.24 Latch Hit %: 99.98
Parse CPU to Parse Elapsd %: 53.15 % Non-Parse CPU: 98.03
Shared Pool Statistics Begin End
Memory Usage %: 10.50 12.79
% SQL with executions>1: 69.98 78.37
% Memory for SQL w/exec>1: 70.22 81.96
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time
CPU time 847 50.2
enq: TX - row lock contention 4,480 434 97 25.8
log file sync 284,169 185 1 11.0
log file parallel write 299,537 164 1 9.7
log file sequential read 698 16 24 1.0
Host CPU (CPUs: 2 Cores: 1 Sockets: 0)
~~~~~~~~ Load Average
Begin End User System Idle WIO WCPU
1.16 1.84 19.28 14.51 66.21 1.20 82.01
Instance CPU
~~~~~~~~~~~~ % Time (seconds)
Host: Total time (s): 7,193.8
Host: Busy CPU time (s): 2,430.7
% of time Host is Busy: 33.8
Instance: Total CPU time (s): 1,203.1
% of Busy CPU used for Instance: 49.5
Instance: Total Database time (s): 2,426.4
%DB time waiting for CPU (Resource Mgr): 0.0
Memory Statistics Begin End
~~~~~~~~~~~~~~~~~ ------------ ------------
Host Mem (MB): 16,384.0 16,384.0
SGA use (MB): 7,136.0 7,136.0
PGA use (MB): 282.5 361.4
% Host Mem used for SGA+PGA: 45.3 45.8
Foreground Wait Events DB/Inst: XXXXXs Snaps: 5635-5636
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by Total Wait Time desc, Waits desc (idle events last)
Avg %Total
%Tim Total Wait wait Waits Call
Event Waits out Time (s) (ms) /txn Time
enq: TX - row lock contentio 4,480 0 434 97 0.0 25.8
log file sync 284,167 0 185 1 1.5 11.0
Disk file operations I/O 8,741 0 4 0 0.0 .2
direct path write 13,247 0 3 0 0.1 .2
db file sequential read 6,058 0 1 0 0.0 .1
buffer busy waits 1,800 0 1 1 0.0 .1
SQL*Net more data to client 29,161 0 1 0 0.2 .1
direct path read 7,696 0 1 0 0.0 .0
db file scattered read 316 0 1 2 0.0 .0
latch: shared pool 144 0 0 2 0.0 .0
CSS initialization 30 0 0 3 0.0 .0
cursor: pin S 10 0 0 9 0.0 .0
row cache lock 41 0 0 2 0.0 .0
latch: row cache objects 19 0 0 3 0.0 .0
log file switch (private str 8 0 0 7 0.0 .0
library cache: mutex X 28 0 0 2 0.0 .0
latch: cache buffers chains 54 0 0 1 0.0 .0
latch free 290 0 0 0 0.0 .0
control file sequential read 1,568 0 0 0 0.0 .0
log file switch (checkpoint 4 0 0 6 0.0 .0
direct path sync 8 0 0 3 0.0 .0
latch: redo allocation 60 0 0 0 0.0 .0
SQL*Net break/reset to clien 34 0 0 1 0.0 .0
latch: enqueue hash chains 45 0 0 0 0.0 .0
latch: cache buffers lru cha 7 0 0 2 0.0 .0
latch: session allocation 5 0 0 1 0.0 .0
latch: object queue header o 6 0 0 1 0.0 .0
ASM file metadata operation 30 0 0 0 0.0 .0
latch: In memory undo latch 15 0 0 0 0.0 .0
latch: undo global data 8 0 0 0 0.0 .0
SQL*Net message from client 6,362,536 0 278,225 44 33.7
jobq slave wait 7,270 100 3,635 500 0.0
SQL*Net more data from clien 7,976 0 15 2 0.0
SQL*Net message to client 6,362,544 0 8 0 33.7
Background Wait Events DB/Inst: XXXXXs Snaps: 5635-5636
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by Total Wait Time desc, Waits desc (idle events last)
Avg %Total
%Tim Total Wait wait Waits Call
Event Waits out Time (s) (ms) /txn Time
log file parallel write 299,537 0 164 1 1.6 9.7
log file sequential read 698 0 16 24 0.0 1.0
db file parallel write 9,556 0 13 1 0.1 .8
os thread startup 146 0 10 70 0.0 .6
control file parallel write 2,037 0 2 1 0.0 .1
Log archive I/O 35 0 1 30 0.0 .1
LGWR wait for redo copy 2,447 0 0 0 0.0 .0
db file async I/O submit 9,556 0 0 0 0.1 .0
db file sequential read 145 0 0 2 0.0 .0
Disk file operations I/O 349 0 0 0 0.0 .0
db file scattered read 30 0 0 4 0.0 .0
control file sequential read 5,837 0 0 0 0.0 .0
ADR block file read 19 0 0 4 0.0 .0
ADR block file write 5 0 0 15 0.0 .0
direct path write 14 0 0 2 0.0 .0
direct path read 3 0 0 7 0.0 .0
latch: shared pool 3 0 0 6 0.0 .0
log file single write 56 0 0 0 0.0 .0
latch: redo allocation 53 0 0 0 0.0 .0
latch: active service list 1 0 0 3 0.0 .0
latch free 11 0 0 0 0.0 .0
rdbms ipc message 314,523 5 57,189 182 1.7
Space Manager: slave idle wa 4,086 88 18,996 4649 0.0
DIAG idle wait 7,185 100 7,186 1000 0.0
Streams AQ: waiting for time 2 50 4,909 ###### 0.0
Streams AQ: qmn slave idle w 129 0 3,612 28002 0.0
Streams AQ: qmn coordinator 258 50 3,612 14001 0.0
smon timer 43 2 3,605 83839 0.0
pmon timer 1,199 99 3,596 2999 0.0
SQL*Net message from client 17,019 0 31 2 0.1
SQL*Net message to client 12,762 0 0 0 0.1
class slave wait 28 0 0 0 0.0
thank you very much!Hi: just know it now: it is a large amount of 'concurrent transaction' designed in this "Volume Test" - to simulate large incoming transaction volme, so I guess wait in eq:TX - row is expected.
The fact: (1) redo logs at uat server is known to not well-tune for configurations (2) volume test slow 5%, however data amount in its test is kept the same by each time import production data, by the team. So why it slowed 5% this year?
The wait histogram is pasted below, any one interest to take a look? any ideas?
Wait Event Histogram DB/Inst: XXXX/XXXX Snaps: 5635-5636
-> Total Waits - units: K is 1000, M is 1000000, G is 1000000000
-> % of Waits - column heading: <=1s is truly <1024ms, >1s is truly >=1024ms
-> % of Waits - value: .0 indicates value was <.05%, null is truly 0
-> Ordered by Event (idle events last)
Total ----------------- % of Waits ------------------
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
ADR block file read 19 26.3 5.3 10.5 57.9
ADR block file write 5 40.0 60.0
ADR file lock 6 100.0
ARCH wait for archivelog l 14 100.0
ASM file metadata operatio 30 100.0
CSS initialization 30 100.0
Disk file operations I/O 9090 97.2 1.4 .6 .4 .2 .1 .1
LGWR wait for redo copy 2447 98.5 .5 .4 .2 .2 .2 .1
Log archive I/O 35 40.0 8.6 25.7 2.9 22.9
SQL*Net break/reset to cli 34 85.3 8.8 5.9
SQL*Net more data to clien 29K 99.9 .0 .0 .0 .0 .0
buffer busy waits 1800 96.8 .7 .7 .6 .3 .4 .5
control file parallel writ 2037 90.7 5.0 2.1 .8 1.0 .3 .1
control file sequential re 7405 100.0 .0
cursor: pin S 10 10.0 90.0
db file async I/O submit 9556 99.9 .0 .0 .0
db file parallel read 1 100.0
db file parallel write 9556 62.0 32.4 1.7 .8 1.5 1.3 .1
db file scattered read 345 72.8 3.8 2.3 11.6 9.0 .6
db file sequential read 6199 97.2 .2 .3 1.6 .7 .0 .0
direct path read 7699 99.1 .4 .2 .1 .1 .0
direct path sync 8 25.0 37.5 12.5 25.0
direct path write 13K 97.8 .9 .5 .4 .3 .1 .0
enq: TX - row lock content 4480 .4 .7 1.3 3.0 6.8 12.3 75.4 .1
latch free 301 98.3 .3 .7 .7
latch: In memory undo latc 15 93.3 6.7
latch: active service list 1 100.0
latch: cache buffers chain 55 94.5 3.6 1.8
latch: cache buffers lru c 9 88.9 11.1
latch: call allocation 6 100.0
latch: checkpoint queue la 3 100.0
latch: enqueue hash chains 45 97.8 2.2
latch: messages 4 100.0
latch: object queue header 7 85.7 14.3
latch: redo allocation 113 97.3 1.8 .9
latch: row cache objects 19 89.5 5.3 5.3
latch: session allocation 5 80.0 20.0
latch: shared pool 147 90.5 1.4 2.7 1.4 .7 1.4 2.0
latch: undo global data 8 100.0
library cache: mutex X 28 89.3 3.6 3.6 3.6
log file parallel write 299K 95.6 2.6 1.0 .4 .3 .2 .0
log file sequential read 698 29.5 .1 4.6 46.8 18.9
log file single write 56 100.0
log file switch (checkpoin 4 25.0 50.0 25.0
log file switch (private s 8 12.5 37.5 50.0
log file sync 284K 93.3 3.7 1.4 .7 .5 .3 .1
os thread startup 146 100.0
row cache lock 41 85.4 9.8 2.4 2.4
DIAG idle wait 7184 100.0
SQL*Net message from clien 6379K 86.6 5.1 2.9 1.3 .7 .3 2.8 .3
SQL*Net message to client 6375K 100.0 .0 .0 .0 .0 .0 .0
Wait Event Histogram DB/Inst: XXXX/xxxx Snaps: 5635-5636
-> Total Waits - units: K is 1000, M is 1000000, G is 1000000000
-> % of Waits - column heading: <=1s is truly <1024ms, >1s is truly >=1024ms
-> % of Waits - value: .0 indicates value was <.05%, null is truly 0
-> Ordered by Event (idle events last)
Total ----------------- % of Waits ------------------
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
SQL*Net more data from cli 7976 99.7 .1 .1 .0 .1
Space Manager: slave idle 4086 .1 .2 .0 .0 .3 3.2 96.1
Streams AQ: qmn coordinato 258 49.2 .8 50.0
Streams AQ: qmn slave idle 129 100.0
Streams AQ: waiting for ti 2 50.0 50.0
class slave wait 28 92.9 3.6 3.6
jobq slave wait 7270 .0 100.0
pmon timer 1199 100.0
rdbms ipc message 314K 10.3 7.3 39.7 15.4 10.6 5.3 8.2 3.3
smon timer 43 100.0 -
Performance issue showing read by other session Event
Hi All,
we are having a severe performance issue in my database when we are running batch jobs.
This was a new database(11.2.0.2) and we are testing the performance by running some batch jobs. These batch jobs included some inserts and updates.
I am seeing read by other session in top 5 timed events and cache buffers chains in Latch Miss Sources section.
Please help me to solve this out.
Inst Num Startup Time Release RAC
1 27-Feb-12 09:03 11.2.0.2.0 NO
Platform CPUs Cores Sockets Memory(GB)
Linux x86 64-bit 8 8 8 48.00
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 5605 29-Feb-12 03:00:27 63 4.5
End Snap: 5614 29-Feb-12 12:00:47 63 4.3
Elapsed: 540.32 (mins)
DB Time: 1,774.23 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 1,952M 1,952M Std Block Size: 16K
Shared Pool Size: 1,024M 1,024M Log Buffer: 18,868K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 3.3 0.8 0.02 0.05
DB CPU(s): 1.1 0.3 0.01 0.02
Redo size: 55,763.8 13,849.3
Logical reads: 23,906.6 5,937.4
Block changes: 325.7 80.9
Physical reads: 665.6 165.3
Physical writes: 40.4 10.0
User calls: 60.7 15.1
Parses: 10.6 2.6
Hard parses: 1.1 0.3
W/A MB processed: 0.6 0.2
Logons: 0.1 0.0
Executes: 151.2 37.6
Rollbacks: 0.0 0.0
Transactions: 4.0
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.94 Redo NoWait %: 100.00
Buffer Hit %: 97.90 In-memory Sort %: 100.00
Library Hit %: 98.06 Soft Parse %: 90.16
Execute to Parse %: 92.96 Latch Hit %: 100.00
Parse CPU to Parse Elapsd %: 76.71 % Non-Parse CPU: 98.57
Shared Pool Statistics Begin End
Memory Usage %: 89.38 87.96
% SQL with executions>1: 97.14 95.15
% Memory for SQL w/exec>1: 96.05 92.46
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
db file sequential read 14,092,706 65,613 5 61.6 User I/O
DB CPU 34,819 32.7
read by other session 308,534 1,260 4 1.2 User I/O
direct path read 97,454 987 10 .9 User I/O
db file scattered read 71,870 910 13 .9 User I/O
Host CPU (CPUs: 8 Cores: 8 Sockets: 8)
~~~~~~~~ Load Average
Begin End %User %System %WIO %Idle
0.43 0.36 13.7 0.6 9.7 85.7
Instance CPU
~~~~~~~~~~~~
% of total CPU for Instance: 13.5
% of busy CPU for Instance: 94.2
%DB time waiting for CPU - Resource Mgr: 0.0
Memory Statistics
~~~~~~~~~~~~~~~~~ Begin End
Host Mem (MB): 49,152.0 49,152.0
SGA use (MB): 3,072.0 3,072.0
PGA use (MB): 506.5 629.1
% Host Mem used for SGA+PGA: 7.28 7.53
Time Model Statistics
-> Total time in database user-calls (DB Time): 106453.8s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 105,531.1 99.1
DB CPU 34,818.8 32.7
parse time elapsed 714.7 .7
hard parse elapsed time 684.8 .6
PL/SQL execution elapsed time 161.9 .2
PL/SQL compilation elapsed time 44.2 .0
connection management call elapsed time 16.9 .0
hard parse (sharing criteria) elapsed time 10.2 .0
hard parse (bind mismatch) elapsed time 9.4 .0
sequence load elapsed time 2.9 .0
repeated bind elapsed time 0.5 .0
failed parse elapsed time 0.0 .0
DB time 106,453.8
background elapsed time 1,753.9
background cpu time 61.7
Operating System Statistics
-> *TIME statistic values are diffed.
All others display actual values. End Value is displayed if different
-> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name
Statistic Value End Value
BUSY_TIME 3,704,415
IDLE_TIME 22,203,740
IOWAIT_TIME 2,517,864
NICE_TIME 3
SYS_TIME 145,696
USER_TIME 3,557,758
LOAD 0 0
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 358,813,045,760
VM_OUT_BYTES 29,514,830,848
PHYSICAL_MEMORY_BYTES 51,539,607,552
NUM_CPUS 8
NUM_CPU_CORES 8
NUM_CPU_SOCKETS 8
GLOBAL_RECEIVE_SIZE_MAX 4,194,304
GLOBAL_SEND_SIZE_MAX 1,048,586
TCP_RECEIVE_SIZE_DEFAULT 87,380
TCP_RECEIVE_SIZE_MAX 4,194,304
TCP_RECEIVE_SIZE_MIN 4,096
TCP_SEND_SIZE_DEFAULT 16,384
TCP_SEND_SIZE_MAX 4,194,304
TCP_SEND_SIZE_MIN 4,096
Operating System Statistics -
Snap Time Load %busy %user %sys %idle %iowait
29-Feb 03:00:27 0.4 N/A N/A N/A N/A N/A
29-Feb 04:00:35 1.4 11.9 11.2 0.6 88.1 14.3
29-Feb 05:00:41 1.7 13.8 13.2 0.6 86.2 15.8
29-Feb 06:00:48 1.5 14.0 13.5 0.6 86.0 12.3
29-Feb 07:01:00 1.8 16.3 15.8 0.5 83.7 10.4
29-Feb 08:00:12 2.6 23.2 22.5 0.6 76.8 12.6
29-Feb 09:00:26 1.3 16.6 16.0 0.5 83.4 5.7
29-Feb 10:00:33 1.2 13.8 13.3 0.5 86.2 2.0
29-Feb 11:00:43 1.3 14.5 14.0 0.5 85.5 3.8
29-Feb 12:00:47 0.4 4.9 4.2 0.7 95.1 10.6
Foreground Wait Class
-> s - second, ms - millisecond - 1000th of a second
-> ordered by wait time desc, waits desc
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
-> Captured Time accounts for 97.9% of Total DB time 106,453.79 (s)
-> Total FG Wait Time: 69,415.64 (s) DB CPU time: 34,818.79 (s)
Avg
%Time Total Wait wait
Wait Class Waits -outs Time (s) (ms) %DB time
User I/O 14,693,843 0 69,222 5 65.0
DB CPU 34,819 32.7
Commit 40,629 0 119 3 0.1
System I/O 26,504 0 57 2 0.1
Network 1,945,010 0 11 0 0.0
Other 125,200 99 4 0 0.0
Application 2,673 0 2 1 0.0
Concurrency 3,059 0 1 0 0.0
Configuration 31 19 0 15 0.0
Foreground Wait Events
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
db file sequential read 14,092,706 0 65,613 5 108.0 61.6
read by other session 308,534 0 1,260 4 2.4 1.2
direct path read 97,454 0 987 10 0.7 .9
db file scattered read 71,870 0 910 13 0.6 .9
db file parallel read 35,001 0 372 11 0.3 .3
log file sync 40,629 0 119 3 0.3 .1
control file sequential re 26,504 0 57 2 0.2 .1
direct path read temp 14,499 0 49 3 0.1 .0
direct path write temp 9,186 0 28 3 0.1 .0
SQL*Net message to client 1,923,973 0 5 0 14.7 .0
SQL*Net message from dblin 1,056 0 5 5 0.0 .0
Disk file operations I/O 8,848 0 2 0 0.1 .0
ASM file metadata operatio 36 0 2 54 0.0 .0
SQL*Net break/reset to cli 2,636 0 1 1 0.0 .0
ADR block file read 472 0 1 1 0.0 .0
os thread startup 8 0 1 74 0.0 .0
SQL*Net more data to clien 17,656 0 1 0 0.1 .0
asynch descriptor resize 123,852 100 0 0 0.9 .0
local write wait 110 0 0 4 0.0 .0
utl_file I/O 55,635 0 0 0 0.4 .0
log file switch (private s 8 0 0 52 0.0 .0
cursor: pin S wait on X 2 0 0 142 0.0 .0
enq: KO - fast object chec 13 0 0 20 0.0 .0
PX Deq: Slave Session Stat 248 0 0 1 0.0 .0
enq: RO - fast object reus 18 0 0 11 0.0 .0
latch: cache buffers chain 2,511 0 0 0 0.0 .0
latch: shared pool 195 0 0 1 0.0 .0
CSS initialization 12 0 0 8 0.0 .0
PX qref latch 54 100 0 2 0.0 .0
SQL*Net more data from cli 995 0 0 0 0.0 .0
SQL*Net more data from dbl 300 0 0 0 0.0 .0
kksfbc child completion 1 100 0 56 0.0 .0
library cache: mutex X 244 0 0 0 0.0 .0
PX Deq: Signal ACK RSG 124 0 0 0 0.0 .0
undo segment extension 6 100 0 7 0.0 .0
PX Deq: Signal ACK EXT 124 0 0 0 0.0 .0
library cache load lock 3 0 0 9 0.0 .0
ADR block file write 45 0 0 1 0.0 .0
CSS operation: action 12 0 0 2 0.0 .0
reliable message 28 0 0 1 0.0 .0
CSS operation: query 72 0 0 0 0.0 .0
latch: row cache objects 14 0 0 1 0.0 .0
enq: SQ - contention 17 0 0 0 0.0 .0
latch free 32 0 0 0 0.0 .0
buffer busy waits 52 0 0 0 0.0 .0
enq: PS - contention 16 0 0 0 0.0 .0
enq: TX - row lock content 6 0 0 1 0.0 .0
SQL*Net message to dblink 1,018 0 0 0 0.0 .0
cursor: pin S 23 0 0 0 0.0 .0
latch: cache buffers lru c 8 0 0 0 0.0 .0
SQL*Net message from clien 1,923,970 0 944,508 491 14.7
jobq slave wait 66,732 100 33,334 500 0.5
Streams AQ: waiting for me 6,481 100 32,412 5001 0.0
wait for unread message on 32,858 98 32,411 986 0.3
PX Deq: Execution Msg 1,448 0 190 131 0.0
PX Deq: Execute Reply 1,196 0 74 62 0.0
HS message to agent 228 0 4 19 0.0
single-task message 42 0 4 97 0.0
PX Deq Credit: send blkd 904 0 2 3 0.0
PX Deq Credit: need buffer 205 0 1 3 0.0
Foreground Wait Events
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
PX Deq: Table Q Normal 4,291 0 1 0 0.0
PX Deq: Join ACK 124 0 0 1 0.0
PX Deq: Parse Reply 124 0 0 0 0.0
KSV master wait 256 0 0 0 0.0
Latch Miss Sources
-> only latches with sleeps are shown
-> ordered by name, sleeps desc
NoWait Waiter
Latch Name Where Misses Sleeps Sleeps
ASM map operation freeli kffmTranslate2 0 2 0
DML lock allocation ktadmc 0 2 0
FOB s.o list latch ksfd_allfob 0 2 2
In memory undo latch ktiFlushMe 0 5 0
In memory undo latch ktichg: child 0 3 0
PC and Classifier lists No latch 0 6 0
Real-time plan statistic keswxAddNewPlanEntry 0 20 20
SQL memory manager worka qesmmIRegisterWorkArea:1 0 1 1
active service list kswslogon: session logout 0 23 12
active service list kswssetsvc: PX session swi 0 6 1
active service list kswsite: service iterator 0 1 0
archive process latch kcrrgpll 0 3 3
cache buffers chains kcbgtcr_2 0 1,746 573
cache buffers chains kcbgtcr: fast path (cr pin 0 1,024 2,126
cache buffers chains kcbgcur_2 0 60 8
cache buffers chains kcbchg1: kslbegin: bufs no 0 16 3
cache buffers chains kcbgtcr: fast path 0 14 20
cache buffers chains kcbzibmlt: multi-block rea 0 10 0
cache buffers chains kcbrls_2 0 9 53
cache buffers chains kcbgtcr: kslbegin shared 0 8 1
cache buffers chains kcbrls_1 0 7 84
cache buffers chains kcbgtcr: kslbegin excl 0 6 14
cache buffers chains kcbnew: new latch again 0 6 0
cache buffers chains kcbzgb: scan from tail. no 0 6 0
cache buffers chains kcbzwb 0 5 8
cache buffers chains kcbgcur: fast path (shr) 0 3 0
cache buffers chains kcbget: pin buffer 0 3 0
cache buffers chains kcbzhngcbk2_1 0 1 0
cache buffers lru chain kcbzgws 0 19 0
cache buffers lru chain kcbo_link_q 0 3 0
call allocation ksuxds 0 14 10
call allocation ksudlp: top call 0 2 3
enqueue hash chains ksqgtl3 0 2 1
enqueue hash chains ksqrcl 0 1 2
enqueues ksqgel: create enqueue 0 1 0
object queue header oper kcbo_unlink_q 0 5 2
object queue header oper kcbo_sw_buf 0 2 0
object queue header oper kcbo_link_q 0 1 2
object queue header oper kcbo_switch_cq 0 1 2
object queue header oper kcbo_switch_mq_bg 0 1 4
parallel query alloc buf kxfpbalo 0 1 1
process allocation ksucrp:1 0 2 0
process queue reference kxfpqrsnd 0 1 0
qmn task queue latch kwqmnmvtsks: delay to read 0 1 0
redo allocation kcrfw_redo_gen: redo alloc 0 17 0
row cache objects kqreqd: reget 0 6 0
row cache objects kqrpre: find obj 0 6 13
row cache objects kqrso 0 2 0
row cache objects kqreqd 0 1 2
row cache objects kqrpre: init complete 0 1 1
shared pool kghalo 0 199 106
shared pool kghupr1 0 39 109
shared pool kghfre 0 18 19
shared pool kghalp 0 7 29
space background task la ktsj_grab_task 0 21 27
Mutex Sleep Summary
-> ordered by number of sleeps desc
Wait
Mutex Type Location Sleeps Time (ms)
Library Cache kglhdgn2 106 338 12
Library Cache kgllkc1 57 259 10
Library Cache kgllkdl1 85 123 21
Cursor Pin kkslce [KKSCHLPIN2] 70 286
Library Cache kglget2 2 31 1
Library Cache kglhdgn1 62 31 2
Library Cache kglpin1 4 26 1
Library Cache kglpnal1 90 18 0
Library Cache kglpndl1 95 15 2
Library Cache kgllldl2 112 6 0
Library Cache kglini1 32 1 0
-------------------------------------------------------------Thanks in advance.Hi,
Thanks for reply.
I provided one hour report.
Inst Num Startup Time Release RAC
1 27-Feb-12 09:03 11.2.0.2.0 NO
Platform CPUs Cores Sockets Memory(GB)
Linux x86 64-bit 8 8 8 48.00
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 5606 29-Feb-12 04:00:35 63 3.7
End Snap: 5607 29-Feb-12 05:00:41 63 3.6
Elapsed: 60.11 (mins)
DB Time: 382.67 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 1,952M 1,952M Std Block Size: 16K
Shared Pool Size: 1,024M 1,024M Log Buffer: 18,868K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 6.4 0.8 0.03 0.03
DB CPU(s): 1.0 0.1 0.00 0.00
Redo size: 84,539.3 10,425.6
Logical reads: 23,345.6 2,879.1
Block changes: 386.5 47.7
Physical reads: 1,605.0 197.9
Physical writes: 7.1 0.9
User calls: 233.9 28.9
Parses: 4.0 0.5
Hard parses: 0.1 0.0
W/A MB processed: 0.1 0.0
Logons: 0.1 0.0
Executes: 210.9 26.0
Rollbacks: 0.0 0.0
Transactions: 8.1
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.62 Redo NoWait %: 100.00
Buffer Hit %: 95.57 In-memory Sort %: 100.00
Library Hit %: 99.90 Soft Parse %: 98.68
Execute to Parse %: 98.10 Latch Hit %: 99.99
Parse CPU to Parse Elapsd %: 32.08 % Non-Parse CPU: 99.90
Shared Pool Statistics Begin End
Memory Usage %: 89.25 89.45
% SQL with executions>1: 96.79 97.52
% Memory for SQL w/exec>1: 95.67 96.56
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
db file sequential read 3,054,464 17,002 6 74.0 User I/O
DB CPU 3,748 16.3
read by other session 199,603 796 4 3.5 User I/O
direct path read 46,301 439 9 1.9 User I/O
db file scattered read 21,113 269 13 1.2 User I/O
Host CPU (CPUs: 8 Cores: 8 Sockets: 8)
~~~~~~~~ Load Average
Begin End %User %System %WIO %Idle
1.45 1.67 13.2 0.6 15.8 86.2
Instance CPU
~~~~~~~~~~~~
% of total CPU for Instance: 13.0
% of busy CPU for Instance: 94.7
%DB time waiting for CPU - Resource Mgr: 0.0
Memory Statistics
~~~~~~~~~~~~~~~~~ Begin End
Host Mem (MB): 49,152.0 49,152.0
SGA use (MB): 3,072.0 3,072.0
PGA use (MB): 513.5 467.7
% Host Mem used for SGA+PGA: 7.29 7.20
Time Model Statistics
-> Total time in database user-calls (DB Time): 22960.5s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 22,835.9 99.5
DB CPU 3,748.4 16.3
parse time elapsed 15.4 .1
hard parse elapsed time 14.3 .1
PL/SQL execution elapsed time 7.5 .0
PL/SQL compilation elapsed time 6.0 .0
connection management call elapsed time 1.6 .0
sequence load elapsed time 0.4 .0
hard parse (sharing criteria) elapsed time 0.0 .0
repeated bind elapsed time 0.0 .0
failed parse elapsed time 0.0 .0
DB time 22,960.5
background elapsed time 238.1
background cpu time 4.9
Operating System Statistics
-> *TIME statistic values are diffed.
All others display actual values. End Value is displayed if different
-> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name
Statistic Value End Value
BUSY_TIME 396,506
IDLE_TIME 2,483,725
IOWAIT_TIME 455,495
NICE_TIME 0
SYS_TIME 16,163
USER_TIME 380,052
LOAD 1 2
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 95,646,943,232
VM_OUT_BYTES 1,686,059,008
PHYSICAL_MEMORY_BYTES 51,539,607,552
NUM_CPUS 8
NUM_CPU_CORES 8
NUM_CPU_SOCKETS 8
GLOBAL_RECEIVE_SIZE_MAX 4,194,304
GLOBAL_SEND_SIZE_MAX 1,048,586
TCP_RECEIVE_SIZE_DEFAULT 87,380
TCP_RECEIVE_SIZE_MAX 4,194,304
TCP_RECEIVE_SIZE_MIN 4,096
TCP_SEND_SIZE_DEFAULT 16,384
TCP_SEND_SIZE_MAX 4,194,304
TCP_SEND_SIZE_MIN 4,096
Operating System Statistics -
Snap Time Load %busy %user %sys %idle %iowait
29-Feb 04:00:35 1.4 N/A N/A N/A N/A N/A
29-Feb 05:00:41 1.7 13.8 13.2 0.6 86.2 15.8
Foreground Wait Class
-> s - second, ms - millisecond - 1000th of a second
-> ordered by wait time desc, waits desc
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
-> Captured Time accounts for 97.6% of Total DB time 22,960.46 (s)
-> Total FG Wait Time: 18,651.75 (s) DB CPU time: 3,748.35 (s)
Avg
%Time Total Wait wait
Wait Class Waits -outs Time (s) (ms) %DB time
User I/O 3,327,253 0 18,576 6 80.9
DB CPU 3,748 16.3
Commit 23,882 0 69 3 0.3
System I/O 1,035 0 3 3 0.0
Network 842,393 0 2 0 0.0
Other 10,120 99 0 0 0.0
Configuration 3 0 0 58 0.0
Application 264 0 0 1 0.0
Concurrency 1,482 0 0 0 0.0
Foreground Wait Events
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
db file sequential read 3,054,464 0 17,002 6 104.5 74.0
read by other session 199,603 0 796 4 6.8 3.5
direct path read 46,301 0 439 9 1.6 1.9
db file scattered read 21,113 0 269 13 0.7 1.2
log file sync 23,882 0 69 3 0.8 .3
db file parallel read 4,727 0 68 14 0.2 .3
control file sequential re 1,035 0 3 3 0.0 .0
SQL*Net message to client 840,792 0 2 0 28.8 .0
direct path read temp 95 0 2 18 0.0 .0
local write wait 79 0 0 4 0.0 .0
Disk file operations I/O 870 0 0 0 0.0 .0
ASM file metadata operatio 4 0 0 50 0.0 .0
log file switch (private s 3 0 0 58 0.0 .0
ADR block file read 36 0 0 3 0.0 .0
enq: RO - fast object reus 5 0 0 16 0.0 .0
latch: cache buffers chain 1,465 0 0 0 0.1 .0
SQL*Net break/reset to cli 256 0 0 0 0.0 .0
asynch descriptor resize 10,059 100 0 0 0.3 .0
SQL*Net more data to clien 1,510 0 0 0 0.1 .0
enq: KO - fast object chec 3 0 0 8 0.0 .0
SQL*Net more data from cli 91 0 0 0 0.0 .0
latch: shared pool 14 0 0 0 0.0 .0
ADR block file write 5 0 0 1 0.0 .0
reliable message 8 0 0 0 0.0 .0
direct path write temp 1 0 0 2 0.0 .0
SQL*Net message from clien 840,794 0 68,885 82 28.8
jobq slave wait 7,365 100 3,679 499 0.3
Streams AQ: waiting for me 721 100 3,605 5000 0.0
wait for unread message on 3,648 98 3,603 988 0.1
KSV master wait 20 0 0 0 0.0
Background Wait Events
-> ordered by wait time desc, waits desc (idle events last)
-> Only events with Total Wait Time (s) >= .001 are shown
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % bg
Event Waits -outs Time (s) (ms) /txn time
log file parallel write 29,353 0 83 3 1.0 34.8
db file parallel write 5,753 0 17 3 0.2 6.9
db file sequential read 1,638 0 15 9 0.1 6.1
control file sequential re 5,142 0 13 2 0.2 5.4
os thread startup 140 0 8 58 0.0 3.4
control file parallel writ 1,440 0 8 6 0.0 3.4
log file sequential read 304 0 8 26 0.0 3.3
db file scattered read 214 0 2 9 0.0 .8
ASM file metadata operatio 1,199 0 1 1 0.0 .3
direct path write 35 0 0 6 0.0 .1
direct path read 41 0 0 5 0.0 .1
kfk: async disk IO 6 0 0 9 0.0 .0
Disk file operations I/O 1,266 0 0 0 0.0 .0
ADR block file read 16 0 0 2 0.0 .0
read by other session 3 0 0 8 0.0 .0
Log archive I/O 2 0 0 10 0.0 .0
log file sync 3 0 0 5 0.0 .0
asynch descriptor resize 341 100 0 0 0.0 .0
CSS initialization 1 0 0 6 0.0 .0
log file single write 4 0 0 1 0.0 .0
latch: redo allocation 3 0 0 1 0.0 .0
ADR block file write 5 0 0 1 0.0 .0
LGWR wait for redo copy 45 0 0 0 0.0 .0
CSS operation: query 6 0 0 0 0.0 .0
CSS operation: action 1 0 0 1 0.0 .0
SQL*Net message to client 420 0 0 0 0.0 .0
rdbms ipc message 47,816 39 61,046 1277 1.6
DIAG idle wait 7,200 100 7,200 1000 0.2
Space Manager: slave idle 1,146 98 5,674 4951 0.0
class slave wait 284 0 3,983 14026 0.0
dispatcher timer 61 100 3,660 60006 0.0
Streams AQ: qmn coordinato 258 50 3,613 14003 0.0
Streams AQ: qmn slave idle 130 0 3,613 27789 0.0
Streams AQ: waiting for ti 7 71 3,608 515430 0.0
wait for unread message on 3,605 100 3,606 1000 0.1
pmon timer 1,201 100 3,604 3001 0.0
smon timer 15 73 3,603 240207 0.0
ASM background timer 754 0 3,602 4777 0.0
shared server idle wait 120 100 3,601 30006 0.0
SQL*Net message from clien 554 0 4 7 0.0
KSV master wait 101 0 0 2 0.0
Wait Event Histogram
-> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
-> % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
-> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
-> Ordered by Event (idle events last)
% of Waits
Total
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
ADR block file read 52 73.1 1.9 9.6 13.5 1.9
ADR block file write 10 100.0
ADR file lock 12 100.0
ARCH wait for archivelog l 3 100.0
ASM file metadata operatio 1203 97.3 .5 .7 .3 .2 .9
CSS initialization 1 100.0
CSS operation: action 1 100.0
CSS operation: query 6 83.3 16.7
Disk file operations I/O 2118 95.4 4.5 .1
LGWR wait for redo copy 45 100.0
Log archive I/O 2 100.0
SQL*Net break/reset to cli 256 99.6 .4
SQL*Net message to client 839.9 100.0 .0
SQL*Net more data from cli 91 100.0
SQL*Net more data to clien 1503 100.0
asynch descriptor resize 10.4K 100.0
buffer busy waits 2 100.0
control file parallel writ 1440 5.7 35.1 24.0 16.3 12.0 5.5 1.5
control file sequential re 6177 69.4 7.5 5.9 8.1 7.1 1.7 .3
db file parallel read 4727 1.7 3.2 3.2 10.1 46.6 33.3 1.8
db file parallel write 5755 42.3 21.3 18.6 11.2 4.6 1.4 .5
db file scattered read 21.5K 8.4 4.3 11.9 18.9 26.3 25.3 4.9
db file sequential read 3053. 28.7 15.1 11.1 17.9 21.5 5.4 .3 .0
direct path read 46.3K 9.9 8.8 18.5 21.7 22.8 15.7 2.7
direct path read temp 95 9.5 9.5 23.2 49.5 8.4
direct path write 35 11.4 31.4 17.1 22.9 11.4 2.9 2.9
direct path write temp 1 100.0
enq: KO - fast object chec 3 66.7 33.3
enq: RO - fast object reus 5 20.0 20.0 20.0 20.0 20.0
kfk: async disk IO 6 50.0 16.7 16.7 16.7
latch free 3 100.0
latch: cache buffers chain 1465 100.0
latch: cache buffers lru c 1 100.0
latch: object queue header 2 100.0
latch: redo allocation 3 33.3 33.3 33.3
latch: row cache objects 2 100.0
latch: shared pool 15 93.3 6.7
local write wait 79 35.4 34.2 21.5 8.9
log file parallel write 29.4K 47.8 21.7 11.9 9.9 6.8 1.6 .3
log file sequential read 304 6.3 3.0 3.6 10.2 23.4 24.3 29.3
log file single write 4 25.0 75.0
log file switch (private s 3 100.0
log file sync 23.9K 40.9 28.0 12.9 9.7 6.7 1.5 .3
os thread startup 140 100.0
read by other session 199.6 37.1 19.9 12.9 13.1 13.8 3.1 .2
reliable message 8 100.0
ASM background timer 755 2.9 .4 .1 .1 .3 .1 .3 95.8
DIAG idle wait 7196 100.0
KSV master wait 121 88.4 2.5 3.3 2.5 .8 .8 1.7
SQL*Net message from clien 840.1 97.1 1.8 .5 .2 .2 .1 .0 .1
Space Manager: slave idle 1147 .1 .5 99.4
Streams AQ: qmn coordinato 258 49.6 .4 50.0
Streams AQ: qmn slave idle 130 .8 99.2
Streams AQ: waiting for me 721 100.0
Streams AQ: waiting for ti 7 28.6 42.9 28.6
class slave wait 283 39.9 2.5 2.5 3.5 4.9 9.2 15.2 22.3
dispatcher timer 60 100.0
jobq slave wait 7360 .0 .0 .0 99.9
pmon timer 1201 100.0
rdbms ipc message 47.8K 2.7 31.6 17.4 1.1 1.1 .9 20.9 24.3
Wait Event Histogram DB/Inst: I2KPROD/I2KPROD Snaps: 5606-5607
-> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
-> % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
-> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
-> Ordered by Event (idle events last)
% of Waits
Total
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
shared server idle wait 120 100.0
smon timer 16 6.3 93.8
wait for unread message on 7250 .1 99.9
Latch Miss Sources
-> only latches with sleeps are shown
-> ordered by name, sleeps desc
NoWait Waiter
Latch Name Where Misses Sleeps Sleeps
In memory undo latch ktichg: child 0 1 0
active service list kswslogon: session logout 0 2 0
cache buffers chains kcbgtcr_2 0 1,123 483
cache buffers chains kcbgtcr: fast path (cr pin 0 496 1,131
cache buffers chains kcbrls_2 0 5 6
cache buffers chains kcbgcur_2 0 4 0
cache buffers chains kcbgtcr: fast path 0 3 1
cache buffers chains kcbzwb 0 2 4
cache buffers chains kcbchg1: kslbegin: bufs no 0 1 0
cache buffers chains kcbnew: new latch again 0 1 0
cache buffers chains kcbrls_1 0 1 6
cache buffers chains kcbzgb: scan from tail. no 0 1 0
cache buffers lru chain kcbzgws 0 1 0
object queue header oper kcbo_switch_cq 0 1 0
object queue header oper kcbo_switch_mq_bg 0 1 2
redo allocation kcrfw_redo_gen: redo alloc 0 3 0
row cache objects kqrpre: find obj 0 1 1
row cache objects kqrso 0 1 0
shared pool kghalo 0 13 3
shared pool kghupr1 0 4 15
shared pool kghalp 0 1 0
space background task la ktsj_grab_task 0 2 2
------------------------------------------------------------- -
Oracle 10.2.0.3 performance issue
Hi all,
I have a performance issue in the database I currently maintain.
Here's the specifications:
- Windows 2003 Server 64bit
- Oracle 10g 10.2.0.3 patch 31
- Application Server 10gR2 OC4J (Forms and Report Services)
The server was re-installed about 3 weeks ago after it got viruses.
I believe my setup on the memory parameters were fine because for two weeks the system ran fine and the end of day process is also increasing in terms of time.
However, starting 2 days ago in the middle of no where, the end of day process started to get really slow (from 11 minutes became 1 hour).
The IT standby in my client will normally do an analyze schema and after that every thing will be back to normal again.
I turned off the GATHER_STATS_JOB 2 weeks ago and replaced it with DBMS_STATS.GATHER_DATABASE_STATS which I scheduled to execute every Friday.
This problem occurred even before the server was re-installed and as usual, analyze schema (GATHER_SCHEMA_STATS) will fix it.
I don't have much options currently, and analyzing the schema seems to be the only solution which normally we never do for other clients (at not everyday analyzing schema).
I hope any of you could help me out with any solution on how to trace the exact problem on this.
Thank you,
AdhikaHi Satish,
This end of day process basically will insert some values to certain tables and also generating reports afterward.
I managed to get the AWR report within the time of the end of day process.
sql execute elapsed time is on the top of Time Model Statistics.
What I don't understand here is why is this issue happened only after 2 weeks?
I suspected that this might be because Oracle pick the wrong execution plan, and normally a wrong execution plan caused by outdated statistics on the indexes and tables, but what I cannot understand is why this is happening on Monday morning where in Friday night GATHER_DATABASE_STATS ran successfully.
Yesterday when the analyze schema was executed again, this morning I got another email saying that the performance issue occurred again.
The top most wait events are: db file sequential read, db file scattered read, log file parallel write, LNS Wait on SENDREQ, and log file sequential read.
Thank you,
Adhika
Maybe you are looking for
-
Unable to connect to iTunes after iOS 6.0.1 update on iPad 3
After I updated the other day I can no longer get new apps or update my current apps. I keep getting a message saying I cannot connect to iTunes. I have tried some solutions that I found (moving the date to 2013 & updating my banking info on iTunes)
-
Goods return without reference to a Material Document or delivery note
Hi Is there a way to do Goods return without a material document. Please suggest. The situation is that a material will go to the field (material is being maintained against a serial number profile) and let suppose that it is returned after a few yea
-
When I get a new USB headset, how do I set up the voice so that it is optimal?
I have a Mac Mini desktop computer and don't know how to make sure my sound is set up and optimized when I get a new headset. Would someone please tell me how to do this?
-
Hi there I am trying to do a table lookup from XI in my message mapping by calling an remote RFC. The return values must return ALL the value for a specific column. The code I have here returns a blank value. Can someone see what my problem may be? M
-
HelloDoes anybody know the port number/range a client PC has to be able to "see" through a firewall to access Essbase Server? I saw indication about Agentport and Serverport, what is the right and what is the default value?Thank You & CheersMaurice L