DFS lock handle
What means DFS lock handle?
It is in TOP 5 Wait Events of Oracle 10G database running on Windows.
Thanks
Why are you asking a basic docs question here?
I went to tahiti.oracle.com and put in your subject and it immediately
returned this link:
http://download.oracle.com/docs/cd/B19306_01/server.102/b14237/waitevents003.htm#sthref4395
Please check the docs before asking for help.
Similar Messages
-
Hi all,
does anyone can help understanding the DFS lock handle Event ?.
The reference does not consult clear information why this can happen
http://download.oracle.com/docs/cd/B19306_01/server.102/b14237/waitevents003.htm#sthref3033
Thanks
*TActually it does. It discusses the lock is a global lock, so cluster-wide.
It also discusses communication with the DLM, an acronym which you can look up and which means Distributed Lock Manager.
So no doubt DFS means Distrihbuted File System.
Sybrand Bakker
Senior Oracle DBA -
RAC processes (10.1.0.3) spinning forever on a DFS lock handle. Any SELECT on a specific table runs into this wait state and waits forever (until killed).
Potentially bug 3520103, but the causes described in Metalink Note 3520103.8 is not applicable in our case.
Metalink provides very little usable data about this bug. It would seem that we have now run into this problem twice in so many months on 10.1.0.3. Upgrade to 10.2 is planned but a few month away still.
The suggest workaround in Note 3520103.8 says "Restart one instance". This fails to work and does not make much sense to me either. Surely the ASM instance owning the unreleased DFS lock should be restarted and not just simply any (random) instance in the cluster?
Is it perhaps possible to identify these locks using SQL against an ASM instance?
Current workarounds I have are
- restarting the entire cluster (a very large hammer approach I don't like)
- dropping and re-creating the table in question (not always feasible and does it cause the dfs lock to be released, or does it instead merely hides the problem?)
If anyone has any details to share about this DFS lock handle problem, potential causes (other than what Metalink thinks), alternative workarounds, etc, I will appreciate it.
Thanks.Hi Bkboudreau,
If I am not misunderstand your question, you are finding the method for iSCSI target servers synchronize iSCSI data.
Replication of data between sites is very important in a multi-site cluster, and is accomplished in different ways by different hardware vendors. You can choose the hardware
storage vendor solution when you want to synchronize data between two shared storage.
You cannot use the feature in Windows Server 2008 called Distributed File System Replication (DFS-R) as your data replication method in a multi-site cluster. DFS-R only performs
its data replication after a file is closed. This works well for files such as documents, presentations, or spreadsheets, but it will not work for files that are held open, such as databases or virtual machines. You must choose a replication option other than
DFS-R.
The related KB:
Requirements and Recommendations for a Multi-Site Failover Cluster
http://technet.microsoft.com/zh-cn/library/dd197575(v=ws.10).aspx
The similar thread:
how to sync the data between the two iSCSI target server
http://social.technet.microsoft.com/Forums/windowsserver/en-US/067ba543-369c-4d22-9f00-1f41d7aedc00/how-to-sync-the-data-between-the-two-iscsi-target-server?forum=winserverClustering
Hope this helps.
We
are trying to better understand customer views on social support experience, so your participation in this
interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place. -
Livecache lock handler not available in APO
Hello Experts,
We are getting the error "Livecache lock handler not available" error when we are running planning jobs in SAP APO.
We are using parallel processing profiles for DP background processing jobs (Macro calculations) and I am getting this error only when a job is run with PP profile.
Below are the parameters of the parallel processing profiles:
Application: DP background processing
No of parallel processes: 2
Block Size: 500
The error message is written in the spool generated for the background job and the job fails. Eventually the process chain fails in which the job is included.
Please provide some pointers if anyone has come across the same error.
Regards,
RaghuHi JB,
Thanks for the note, Basis check the solution in the note and system seems to have met the requirements.
But one thing is likely to increase the MaxSQLLocks in the system as the demand of the locks is more than the availability. So basis is looking at this dimension to increase the number of locks in the system.
Will update soon in case of any conclusion.
Regards,
Raghu -
Live cache lock handler not available
Hi we are frequently getting this error Live cache lock handler not available and this causes our process chains to fail.
Can anyone please explain in details what this error refers to and direct to any good links for error resolution.Hello,
1. As SAP customer, you could create the SAP message to the component BC-DB-LCA and get SAP support.
2. If you set the flag 'activate livecache lock' in tcode /SAPAPO/MSDP_ADMIN for your planning area on SCM system, please run the connection test for the LEA connection in /ndb59 to check the LEA connection settings.
The liveCache installation guide has additional information about LEA connection.
Regards, Natalia Khlopina -
Livecache lock handler not available
Hello Experts,
We are getting the error "Livecache lock handler not available" error when we are running planning jobs in SAP APO.
We are using parallel processing profiles for DP background processing jobs (Macro calculations) and I am getting this error only when a job is run with PP profile.
Below are the parameters of the parallel processing profiles:
Application: DP background processing
No of parallel processes: 2
Block Size: 500
The error message is written in the spool generated for the background job and the job fails. Eventually the process chain fails in which the job is included.
Please provide some pointers if anyone has come across the same error.
Regards,
RaghuHi JB,
Thanks for the note, Basis check the solution in the note and system seems to have met the requirements.
But one thing is likely to increase the MaxSQLLocks in the system as the demand of the locks is more than the availability. So basis is looking at this dimension to increase the number of locks in the system.
Will update soon in case of any conclusion.
Regards,
Raghu -
Hi, I have a big problem... after setting up SSL (only some changes in strust and adding parameter in rz10) I cannot use Sap Logon anymore.. Log in is ok, but when I try to run any transaction I see only error:
'Unknown error calling profile editor'
and in syslog after restart I have a lot of errors like that:
'SAP-Basis System: Error in Central Lock Handler'
please help.after system restart I see this when calling any transaction:
http://img441.imageshack.us/img441/5152/errorso.jpg
ok, I found backup files of profile in sap directory and replace it with actual. After restart of whole machine it started working.
Edited by: pw.skiba on Jun 17, 2010 1:35 PM -
Hi
I am running a little test that runs two threads that update the same table. Each thread tries to update several documents in the within a single transaction. The update is done by retrieving the document, modifying it, adding the updated document and deleting the existing document.
I have enable dead lock detection and when a dead lock exception is thrown I abort the current transaction.
However after a few iteration, deleteDocument causes a core dump and DBXML prints to stderr : "Previous deadlock return not resolved".
Is there anything else to resolve when a dead lock occurs other than abort the transaction?
This is the relevant stack trace:
#0 0x00421780 in Dbc::get () from /u/yoava/dbxml-2.2.13/install/lib/libdb_cxx-4.3.so
(gdb) where
#0 0x00421780 in Dbc::get () from /u/yoava/dbxml-2.2.13/install/lib/libdb_cxx-4.3.so
#1 0x00934289 in DbXml::SyntaxDatabase::updateStatistics (this=0x2000001c,
context=@0x8641c34, key=@0x8628070, statistics=@0x8628088) at Cursor.hpp:48
#2 0x00901d10 in DbXml::StatisticsWriteCache::updateContainer (this=0x8641bcc,
context=@0x8641c34, container=Internal: global symbol `Container' found in Container.cpp psymtab but not in symtab.
Container may be an inlined function, or may be a template function
(if a template, try specifying an instantiation: Container<type>).
) at /usr/include/c++/3.2.3/bits/stl_tree.h:202
#3 0x0093bacf in DbXml::KeyStash::updateIndex (this=0x8634df8, context=@0x8641c34,
container=0x8630e68) at KeyStash.cpp:210
#4 0x008c72b7 in DbXml::Container::deleteDocument (this=0x8630e68, txn=0x86417f8,
document=@0x8671f38, context=Internal: global symbol `UpdateContext' found in UpdateContext.cpp psymtab but not in symtab.
UpdateContext may be an inlined function, or may be a template function
(if a template, try specifying an instantiation: UpdateContext<type>).
) at Container.cpp:679
#5 0x008d3475 in DeleteDocumentFunctor2::method (this=0x2000001c, container=@0x8630e68,
txn=Internal: global symbol `Transaction' found in Transaction.cpp psymtab but not in symtab.
Transaction may be an inlined function, or may be a template function
(if a template, try specifying an instantiation: Transaction<type>).
) at TransactedContainer.cpp:121
#6 0x008d3149 in DbXml::TransactedContainer::transactedMethod (this=0x8630e68,
txn=0x3d71c8, flags=0, f=@0xb176b470) at TransactedContainer.cpp:217
#7 0x008d2fe8 in DbXml::TransactedContainer::deleteDocument (this=0x8630e68,
txn=0x86417f8, document=Internal: global symbol `Document' found in Document.cpp psymtab but not in symtab.
Document may be an inlined function, or may be a template function
(if a template, try specifying an instantiation: Document<type>).
) at TransactedContainer.cpp:26
#8 0x00906940 in DbXml::XmlContainer::deleteDocument (this=0xbfff8c94, txn=@0xb176b5a0,
document=Internal: global symbol `XmlDocument' found in XmlDocument.cpp psymtab but not in symtab.
XmlDocument may be an inlined function, or may be a template function
(if a template, try specifying an instantiation: XmlDocument<type>).
) at /u/yoava/dbxml-2.2.13/dbxml/include/dbxml/XmlDocument.hpp:72
#9 0x0804a5c0 in DoUpdates (arg=0xbfff8c90) at dbxml_test_6.cpp:99
#10 0x003dedec in start_thread () from /lib/tls/libpthread.so.0
#11 0x0037ea2a in clone () from /lib/tls/libc.so.6
thanksI have applied them. When applying patch 6 I got an error:
compile3.server 46% patch < patch.2.2.13.6
patch.2.2.13.6: No such file or directory.
compile3.server 47% patch < patch.2.2.13.6
(Stripping trailing CRs from patch.)
can't find file to patch at input line 3
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
|*** NsEventGenerator.cpp.orig Thu Dec 8 14:50:50 2005
|--- dbxml/src/dbxml/nodeStore/NsEventGenerator.cppThu Sep 28 17:24:57 2006
File to patch: dbxml/src/dbxml/DbWrapper.hpp
patching file dbxml/src/dbxml/DbWrapper.hpp
Hunk #1 FAILED at 357.
1 out of 1 hunk FAILED -- saving rejects to file dbxml/src/dbxml/DbWrapper.hpp.rej
However this patch does not seem to be lock related. -
ROW CACHE ENQUEUE LOCK/ibrary cache load lock leads to database hung
(lowercase, curly brackets, no spaces)
We faced database hung on 3 node 11i erp 9i rac database.
We saw the library cache load lock timed out events reported in alert log.
Then few ora-600 and later ROW CACHE ENQUEUE LOCK timed out event. Eventually database was hung and we had to bounce the services .
we created support sr 7845542.992 for RCA.
The support says to increase shared pool size to avoid shared pool fragmentation and avoid reload ,additionaly to upgrade to 10g database.
I am not covinced adding additional pool size would solve this or upgrade to 10 .furthermore even 10g has such issues reported.
I saw couple of bugs mentioned such issue can happen due deadlock of session holding latches .
kindly let me know your view on issue
If required i can attach statspack for more information. (lowercase, curly brackets, no spaces)Many Thanks, i was keen to have your update .
There are 8 cpus on each node . Reloads very high during time period ,but normally there are not high reloads.
Statspack details for 3 nodes
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
PROD 21184234 PROD1 1 9.2.0.8.0 YES npi-or-db-p-
11.npi.corp
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 149817 30-Oct-09 13:00:09 574 #########
End Snap: 149837 30-Oct-09 14:00:17 602 #########
Elapsed: 60.13 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 8,192M Std Block Size: 8K
Shared Pool Size: 1,024M Log Buffer: 10,240K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 122,414.93 11,449.13
Logical reads: 69,550.76 6,504.89
Block changes: 928.41 86.83
Physical reads: 196.24 18.35
Physical writes: 28.65 2.68
User calls: 343.97 32.17
Parses: 558.61 52.25
Hard parses: 43.48 4.07
Sorts: 467.24 43.70
Logons: 0.63 0.06
Executes: 2,046.99 191.45
Transactions: 10.69
% Blocks changed per Read: 1.33 Recursive Call %: 97.59
Rollback per transaction %: 5.07 Rows per Sort: 15.85
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 99.72 In-memory Sort %: 100.00
Library Hit %: 96.79 Soft Parse %: 92.22
Execute to Parse %: 72.71 Latch Hit %: 99.77
Parse CPU to Parse Elapsd %: 60.10 % Non-Parse CPU: 78.07
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
db file sequential read 249,234 0 1,537 6 6.5
db file scattered read 61,776 0 769 12 1.6
row cache lock 780,098 10 566 1 20.2
library cache lock 697,849 157 432 1 18.1
latch free 127,926 4,715 387 3 3.3
global cache cr request 370,770 3,091 309 1 9.6
PL/SQL lock timer 59 58 112 1903 0.0
wait for scn from all nodes 303,572 18 103 0 7.9
library cache pin 26,231 2 100 4 0.7
global cache null to x 17,717 716 92 5 0.5
buffer busy waits 5,388 18 74 14 0.1
db file parallel read 5,245 0 69 13 0.1
log file sync 20,407 29 66 3 0.5
enqueue 52,200 70 60 1 1.4
buffer busy global CR 4,845 33 55 11 0.1
CGS wait for IPC msg 412,512 407,106 50 0 10.7
ksxr poll remote instances 1,279,565 483,046 48 0 33.2
log file parallel write 160,040 0 42 0 4.1
library cache load lock 1,491 2 29 20 0.0
global cache open x 19,507 344 28 1 0.5
buffer busy global cache 957 0 22 23 0.0
global cache s to x 16,516 180 20 1 0.4
db file parallel write 11,120 0 12 1 0.3
log file sequential read 618 0 11 18 0.0
DFS lock handle 23,768 0 10 0 0.6
control file sequential read 8,563 0 4 0 0.2
KJC: Wait for msg sends to c 1,549 57 4 3 0.0
lock escalate retry 76 76 4 52 0.0
SQL*Net break/reset to clien 12,546 0 3 0 0.3
SQL*Net more data to client 85,773 0 3 0 2.2
control file parallel write 1,265 0 2 1 0.0
global cache null to s 648 23 1 2 0.0
global cache busy 200 0 1 5 0.0
global cache open s 1,493 28 1 1 0.0
log file switch completion 12 0 1 61 0.0
PX Deq Credit: send blkd 161 70 1 4 0.0
kksfbc child completion 119 118 1 5 0.0
PX Deq: reap credit 5,948 5,456 0 0 0.2
PX Deq: Execute Reply 83 29 0 3 0.0
process startup 8 0 0 25 0.0
LGWR wait for redo copy 992 12 0 0 0.0
IPC send completion sync 450 450 0 0 0.0
PX Deq: Parse Reply 100 28 0 1 0.0
undo segment extension 10,380 10,372 0 0 0.3
PX Deq: Join ACK 146 65 0 1 0.0
buffer deadlock 222 221 0 0 0.0
async disk IO 1,179 0 0 0 0.0
wait list latch free 2 0 0 16 0.0
PX Deq: Msg Fragment 112 28 0 0 0.0
Library Cache Activity for DB: PROD Instance: PROD1 Snaps: 149817 -149837
->"Pct Misses" should be very low
Get Pct Pin Pct Invali-
Namespace Requests Miss Requests Miss Reloads dations
BODY 116,007 1.1 133,347 19.9 24,338 0
CLUSTER 4,224 0.6 5,131 1.0 0 0
INDEX 15,048 24.1 13,798 26.4 2 0
JAVA DATA 82 0.0 692 39.6 136 0
JAVA RESOURCE 66 39.4 206 25.2 12 0
PIPE 1,140 0.5 1,160 0.5 0 0
SQL AREA 1,197,908 12.6 13,517,660 1.5 111,833 73
TABLE/PROCEDURE 3,847,439 0.8 4,230,265 7.9 142,200 0
TRIGGER 8,444 2.4 8,657 18.5 1,274 0
GES Lock GES Pin GES Pin GES Inval GES Invali-
Namespace Requests Requests Releases Requests dations
BODY 1 1,234 1,258 985 0
CLUSTER 3,222 25 25 25 0
INDEX 13,792 3,641 3,631 3,629 0
JAVA DATA 0 0 0 0 0
JAVA RESOURCE 0 26 25 0 0
PIPE 0 0 0 0 0
SQL AREA 0 0 0 0 0
TABLE/PROCEDURE 857,137 13,130 13,264 10,762 0
TRIGGER 0 200 202 200 0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
PROD 21184234 PROD2 2 9.2.0.8.0 YES npi-or-db-p-
12.npi.corp
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 149847 30-Oct-09 14:00:05 493 #########
End Snap: 149857 30-Oct-09 15:00:02 432 #########
Elapsed: 59.95 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 8,192M Std Block Size: 8K
Shared Pool Size: 1,024M Log Buffer: 10,240K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 71,853.44 32,058.65
Logical reads: 273,904.84 122,207.36
Block changes: 889.13 396.70
Physical reads: 40.40 18.03
Physical writes: 20.97 9.35
User calls: 153.74 68.60
Parses: 66.19 29.53
Hard parses: 2.66 1.19
Sorts: 25.70 11.47
Logons: 0.16 0.07
Executes: 726.41 324.10
Transactions: 2.24
% Blocks changed per Read: 0.32 Recursive Call %: 92.41
Rollback per transaction %: 4.84 Rows per Sort: 193.55
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 99.99
Buffer Hit %: 99.99 In-memory Sort %: 100.00
Library Hit %: 99.35 Soft Parse %: 95.97
Execute to Parse %: 90.89 Latch Hit %: 99.99
Parse CPU to Parse Elapsd %: 36.55 % Non-Parse CPU: 98.28
Wait Events for DB: PROD Instance: PROD2 Snaps: 149847 -149857
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
enqueue 65,823 33,667 90,459 1374 8.2
row cache lock 38,996 560 1,795 46 4.8
PX Deq Credit: send blkd 522 499 1,223 2344 0.1
PX Deq: Parse Reply 466 416 987 2117 0.1
db file sequential read 50,130 0 421 8 6.2
library cache lock 78,842 172 210 3 9.8
db file scattered read 6,904 0 152 22 0.9
global cache cr request 84,801 575 113 1 10.5
latch free 8,096 736 65 8 1.0
log file sync 5,676 27 41 7 0.7
wait for scn from all nodes 18,891 10 24 1 2.3
CGS wait for IPC msg 394,678 392,142 21 0 49.0
library cache pin 1,339 0 17 13 0.2
global cache null to x 2,145 48 16 8 0.3
global cache s to x 3,242 32 16 5 0.4
buffer busy waits 366 10 15 40 0.0
ksxr poll remote instances 70,990 31,295 14 0 8.8
db file parallel read 359 0 11 31 0.0
global cache open x 2,708 55 10 4 0.3
async disk IO 3,474 0 8 2 0.4
global cache open s 3,470 10 6 2 0.4
log file parallel write 13,076 0 5 0 1.6
global cache busy 58 40 5 90 0.0
PL/SQL lock timer 1 1 5 4877 0.0
DFS lock handle 3,362 0 5 1 0.4
log file sequential read 412 0 4 10 0.1
db file parallel write 2,774 0 3 1 0.3
library cache load lock 59 0 3 58 0.0
buffer busy global CR 722 0 3 4 0.1
control file sequential read 6,398 0 3 0 0.8
SQL*Net break/reset to clien 16,078 0 2 0 2.0
name-service call wait 26 0 2 67 0.0
control file parallel write 1,248 0 2 1 0.2
process startup 24 0 1 49 0.0
KJC: Wait for msg sends to c 3,491 4 1 0 0.4
SQL*Net more data to client 23,724 0 1 0 2.9
buffer busy global cache 23 0 0 19 0.0
global cache null to s 114 0 0 4 0.0
PX Deq: reap credit 5,646 5,509 0 0 0.7
log file switch completion 4 0 0 58 0.0
lock escalate retry 54 54 0 1 0.0
IPC send completion sync 119 118 0 0 0.0
direct path read 2,820 0 0 0 0.3
direct path read (lob) 3,632 0 0 0 0.5
PX Deq: Join ACK 88 37 0 0 0.0
direct path write 2,470 0 0 0 0.3
kksfbc child completion 6 6 0 6 0.0
buffer deadlock 3 3 0 11 0.0
global cache quiesce wait 4 4 0 8 0.0
Library Cache Activity for DB: PROD Instance: PROD2 Snaps: 149847 -149857
->"Pct Misses" should be very low
Get Pct Pin Pct Invali-
Namespace Requests Miss Requests Miss Reloads dations
BODY 27,353 0.5 28,091 6.5 1,643 0
CLUSTER 203 1.0 269 1.5 0 0
INDEX 526 9.9 271 19.9 0 0
JAVA DATA 18 0.0 120 6.7 4 0
JAVA RESOURCE 20 45.0 56 26.8 3 0
JAVA SOURCE 1 100.0 1 100.0 0 0
PIPE 999 0.4 1,043 0.4 0 0
SQL AREA 131,793 7.6 3,406,577 0.4 7,012 0
TABLE/PROCEDURE 926,987 0.2 1,907,993 1.0 8,845 0
TRIGGER 1,519 0.1 1,532 4.9 69 0
GES Lock GES Pin GES Pin GES Inval GES Invali-
Namespace Requests Requests Releases Requests dations
BODY 1 129 277 117 0
CLUSTER 168 2 2 2 0
INDEX 271 52 56 52 0
JAVA DATA 0 0 0 0 0
JAVA RESOURCE 0 9 6 0 0
JAVA SOURCE 0 1 1 1 0
PIPE 0 0 0 0 0
SQL AREA 0 0 0 0 0
TABLE/PROCEDURE 89,523 764 868 460 0
TRIGGER 0 2 14 2 0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DB Name DB Id Instance Inst Num Release Cluster Host
PROD 21184234 PROD3 3 9.2.0.8.0 YES npi-or-db-p-
13.npi.corp
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 149808 30-Oct-09 14:00:00 31 #########
End Snap: 149809 30-Oct-09 15:00:02 34 11,831.4
Elapsed: 60.03 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 8,192M Std Block Size: 8K
Shared Pool Size: 1,024M Log Buffer: 10,240K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 1,518.14 36,700.35
Logical reads: 1,333.43 32,235.02
Block changes: 5.09 123.01
Physical reads: 54.31 1,312.88
Physical writes: 3.91 94.44
User calls: 1.46 35.40
Parses: 2.24 54.21
Hard parses: 0.04 0.93
Sorts: 0.84 20.28
Logons: 0.06 1.45
Executes: 3.11 75.23
Transactions: 0.04
% Blocks changed per Read: 0.38 Recursive Call %: 94.31
Rollback per transaction %: 45.64 Rows per Sort: 215.97
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.99 Redo NoWait %: 100.00
Buffer Hit %: 96.21 In-memory Sort %: 100.00
Library Hit %: 99.07 Soft Parse %: 98.29
Execute to Parse %: 27.94 Latch Hit %: 99.98
Parse CPU to Parse Elapsd %: 69.88 % Non-Parse CPU: 97.92
Wait Events for DB: PROD Instance: PROD3 Snaps: 149808 -149809
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
enqueue 19,510 7,472 15,509 795 130.9
PX Deq: Parse Reply 1,152 1,071 2,577 2237 7.7
row cache lock 2,202 518 1,579 717 14.8
db file scattered read 31,556 0 354 11 211.8
db file sequential read 17,272 0 67 4 115.9
db file parallel read 1,722 0 34 20 11.6
global cache cr request 53,754 91 32 1 360.8
wait for scn from all nodes 1,897 13 10 5 12.7
CGS wait for IPC msg 403,358 401,478 10 0 2,707.1
DFS lock handle 4,753 0 8 2 31.9
direct path read 1,248 0 6 5 8.4
PX Deq: Execute Reply 110 38 6 51 0.7
global cache open s 160 10 5 31 1.1
control file sequential read 6,442 0 3 0 43.2
name-service call wait 26 0 2 78 0.2
latch free 129 109 2 13 0.9
KJC: Wait for msg sends to c 153 24 1 9 1.0
control file parallel write 1,245 0 1 1 8.4
buffer busy waits 199 0 1 6 1.3
process startup 20 0 1 44 0.1
global cache null to x 74 2 1 9 0.5
global cache null to s 19 0 1 29 0.1
global cache open x 268 1 1 2 1.8
library cache lock 1,150 0 0 0 7.7
PX Deq: Join ACK 129 48 0 3 0.9
log file parallel write 1,157 0 0 0 7.8
async disk IO 219 0 0 1 1.5
direct path write 1,024 0 0 0 6.9
ksxr poll remote instances 6,740 4,595 0 0 45.2
PX Deq: reap credit 6,580 6,511 0 0 44.2
buffer busy global CR 73 0 0 2 0.5
log file sequential read 11 0 0 10 0.1
log file sync 100 0 0 1 0.7
global cache s to x 282 2 0 0 1.9
db file parallel write 95 0 0 1 0.6
library cache pin 142 0 0 0 1.0
SQL*Net break/reset to clien 28 0 0 1 0.2
IPC send completion sync 81 81 0 0 0.5
PX Deq: Signal ACK 32 14 0 1 0.2
PX Deq Credit: send blkd 3 1 0 7 0.0
SQL*Net more data to client 841 0 0 0 5.6
PX Deq: Msg Fragment 37 17 0 0 0.2
log file single write 4 0 0 1 0.0
db file single write 1 0 0 1 0.0
SQL*Net message from client 4,213 0 13,673 3246 28.3
gcs remote message 214,784 75,745 7,016 33 1,441.5
wakeup time manager 233 233 6,812 29237 1.6
PX Idle Wait 2,338 2,294 5,686 2432 15.7
PX Deq: Execution Msg 2,151 1,979 4,796 2229 14.4
Library Cache Activity for DB: PROD Instance: PROD3 Snaps: 149808 -149809
->"Pct Misses" should be very low
Get Pct Pin Pct Invali-
Namespace Requests Miss Requests Miss Reloads dations
BODY 1,290 0.0 1,290 0.0 0 0
CLUSTER 18 0.0 8 0.0 0 0
SQL AREA 4,893 2.0 36,371 0.5 2 0
TABLE/PROCEDURE 1,555 3.9 3,834 4.9 71 0
TRIGGER 286 0.0 286 0.0 0 0
GES Lock GES Pin GES Pin GES Inval GES Invali-
Namespace Requests Requests Releases Requests dations
BODY 1 0 0 0 0
CLUSTER 4 0 0 0 0
SQL AREA 0 0 0 0 0
TABLE/PROCEDURE 863 224 42 42 0
TRIGGER 0 0 0 0 0
------------------------------------------------------------- -
Active session Spike on Oracle RAC 11G R2 on HP UX
Dear Experts,
We need urgent help please, as we are facing very low performance in production database.
We are having oracle 11G RAC on HP Unix environment. Following is the ADDM report. Kindly check and please help me to figure it out the issue and resolve it at earliest.
---------Instance 1---------------
ADDM Report for Task 'TASK_36650'
Analysis Period
AWR snapshot range from 11634 to 11636.
Time period starts at 21-JUL-13 07.00.03 PM
Time period ends at 21-JUL-13 09.00.49 PM
Analysis Target
Database 'MCMSDRAC' with DB ID 2894940361.
Database version 11.2.0.1.0.
ADDM performed an analysis of instance mcmsdrac1, numbered 1 and hosted at
mcmsdbl1.
Activity During the Analysis Period
Total database time was 38466 seconds.
The average number of active sessions was 5.31.
Summary of Findings
Description Active Sessions Recommendations
Percent of Activity
1 CPU Usage 1.44 | 27.08 1
2 Interconnect Latency .07 | 1.33 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Findings and Recommendations
Finding 1: CPU Usage
Impact is 1.44 active sessions, 27.08% of total activity.
Host CPU was a bottleneck and the instance was consuming 99% of the host CPU.
All wait times will be inflated by wait for CPU.
Host CPU consumption was 99%.
Recommendation 1: Host Configuration
Estimated benefit is 1.44 active sessions, 27.08% of total activity.
Action
Consider adding more CPUs to the host or adding instances serving the
database on other hosts.
Action
Session CPU consumption was throttled by the Oracle Resource Manager.
Consider revising the resource plan that was active during the analysis
period.
Finding 2: Interconnect Latency
Impact is .07 active sessions, 1.33% of total activity.
Higher than expected latency of the cluster interconnect was responsible for
significant database time on this instance.
The instance was consuming 110 kilo bits per second of interconnect bandwidth.
20% of this interconnect bandwidth was used for global cache messaging, 21%
for parallel query messaging and 7% for database lock management.
The average latency for 8K interconnect messages was 42153 microseconds.
The instance is using the private interconnect device "lan2" with IP address
172.16.200.71 and source "Oracle Cluster Repository".
The device "lan2" was used for 100% of interconnect traffic and experienced 0
send or receive errors during the analysis period.
Recommendation 1: Host Configuration
Estimated benefit is .07 active sessions, 1.33% of total activity.
Action
Investigate cause of high network interconnect latency between database
instances. Oracle's recommended solution is to use a high speed
dedicated network.
Action
Check the configuration of the cluster interconnect. Check OS setup like
adapter setting, firmware and driver release. Check that the OS's socket
receive buffers are large enough to store an entire multiblock read. The
value of parameter "db_file_multiblock_read_count" may be decreased as a
workaround.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Additional Information
Miscellaneous Information
Wait class "Application" was not consuming significant database time.
Wait class "Cluster" was not consuming significant database time.
Wait class "Commit" was not consuming significant database time.
Wait class "Concurrency" was not consuming significant database time.
Wait class "Configuration" was not consuming significant database time.
Wait class "Network" was not consuming significant database time.
Wait class "User I/O" was not consuming significant database time.
Session connect and disconnect calls were not consuming significant database
time.
Hard parsing of SQL statements was not consuming significant database time.
The database's maintenance windows were active during 100% of the analysis
period.
----------------Instance 2 --------------------
ADDM Report for Task 'TASK_36652'
Analysis Period
AWR snapshot range from 11634 to 11636.
Time period starts at 21-JUL-13 07.00.03 PM
Time period ends at 21-JUL-13 09.00.49 PM
Analysis Target
Database 'MCMSDRAC' with DB ID 2894940361.
Database version 11.2.0.1.0.
ADDM performed an analysis of instance mcmsdrac2, numbered 2 and hosted at
mcmsdbl2.
Activity During the Analysis Period
Total database time was 2898 seconds.
The average number of active sessions was .4.
Summary of Findings
Description Active Sessions Recommendations
Percent of Activity
1 Top SQL Statements .11 | 27.65 5
2 Interconnect Latency .1 | 24.15 1
3 Shared Pool Latches .09 | 22.42 1
4 PL/SQL Execution .06 | 14.39 2
5 Unusual "Other" Wait Event .03 | 8.73 4
6 Unusual "Other" Wait Event .03 | 6.42 3
7 Unusual "Other" Wait Event .03 | 6.29 6
8 Hard Parse .02 | 5.5 0
9 Soft Parse .02 | 3.86 2
10 Unusual "Other" Wait Event .01 | 3.75 4
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Findings and Recommendations
Finding 1: Top SQL Statements
Impact is .11 active sessions, 27.65% of total activity.
SQL statements consuming significant database time were found. These
statements offer a good opportunity for performance improvement.
Recommendation 1: SQL Tuning
Estimated benefit is .05 active sessions, 12.88% of total activity.
Action
Investigate the PL/SQL statement with SQL_ID "d1s02myktu19h" for
possible performance improvements. You can supplement the information
given here with an ASH report for this SQL_ID.
Related Object
SQL statement with SQL_ID d1s02myktu19h.
begin dbms_utility.validate(:1,:2,:3,:4); end;
Rationale
The SQL Tuning Advisor cannot operate on PL/SQL statements.
Rationale
Database time for this SQL was divided as follows: 13% for SQL
execution, 2% for parsing, 85% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "d1s02myktu19h" was executed 48 times and had
an average elapsed time of 7 seconds.
Rationale
Waiting for event "library cache pin" in wait class "Concurrency"
accounted for 70% of the database time spent in processing the SQL
statement with SQL_ID "d1s02myktu19h".
Rationale
Top level calls to execute the PL/SQL statement with SQL_ID
"63wt8yna5umd6" are responsible for 100% of the database time spent on
the PL/SQL statement with SQL_ID "d1s02myktu19h".
Related Object
SQL statement with SQL_ID 63wt8yna5umd6.
begin DBMS_UTILITY.COMPILE_SCHEMA( 'TPAUSER', FALSE ); end;
Recommendation 2: SQL Tuning
Estimated benefit is .02 active sessions, 4.55% of total activity.
Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"fk3bh3t41101x".
Related Object
SQL statement with SQL_ID fk3bh3t41101x.
SELECT MEM.MEMBER_CODE ,MEM.E_NAME,Pol.Policy_no
,pol.date_from,pol.date_to,POL.E_NAME,MEM.SEX,(SYSDATE-MEM.BIRTH_DATE
) AGE,POL.SCHEME_NO FROM TPAUSER.MEMBERS MEM,TPAUSER.POLICY POL WHERE
POL.QUOTATION_NO=MEM.QUOTATION_NO AND POL.BRANCH_CODE=MEM.BRANCH_CODE
and endt_no=(select max(endt_no) from tpauser.members mm where
mm.member_code=mem.member_code AND mm.QUOTATION_NO=MEM.QUOTATION_NO)
and member_code like '%' || nvl(:1,null) ||'%' ORDER BY MEMBER_CODE
Rationale
The SQL spent 92% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "fk3bh3t41101x" was executed 14 times and had
an average elapsed time of 4.9 seconds.
Rationale
At least one execution of the statement ran in parallel.
Recommendation 3: SQL Tuning
Estimated benefit is .02 active sessions, 3.79% of total activity.
Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"7mhjbjg9ntqf5".
Related Object
SQL statement with SQL_ID 7mhjbjg9ntqf5.
SELECT SUM(CNT) FROM (SELECT COUNT(PROC_CODE) CNT FROM
TPAUSER.TORBINY_PROCEDURE WHERE BRANCH_CODE = :B6 AND QUOTATION_NO =
:B5 AND CLASS_NO = :B4 AND OPTION_NO = :B3 AND PR_EFFECTIVE_DATE<=
:B2 AND PROC_CODE = :B1 UNION SELECT COUNT(MED_CODE) CNT FROM
TPAUSER.TORBINY_MEDICINE WHERE BRANCH_CODE = :B6 AND QUOTATION_NO =
:B5 AND CLASS_NO = :B4 AND OPTION_NO = :B3 AND M_EFFECTIVE_DATE<= :B2
AND MED_CODE = :B1 UNION SELECT COUNT(LAB_CODE) CNT FROM
TPAUSER.TORBINY_LAB WHERE BRANCH_CODE = :B6 AND QUOTATION_NO = :B5
AND CLASS_NO = :B4 AND OPTION_NO = :B3 AND L_EFFECTIVE_DATE<= :B2 AND
LAB_CODE = :B1 )
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 0% for SQL execution,
0% for parsing, 100% for PL/SQL execution and 0% for Java execution.
Rationale
SQL statement with SQL_ID "7mhjbjg9ntqf5" was executed 31 times and had
an average elapsed time of 3.4 seconds.
Rationale
Top level calls to execute the SELECT statement with SQL_ID
"a11nzdnd91gsg" are responsible for 100% of the database time spent on
the SELECT statement with SQL_ID "7mhjbjg9ntqf5".
Related Object
SQL statement with SQL_ID a11nzdnd91gsg.
SELECT POLICY_NO,SCHEME_NO FROM TPAUSER.POLICY WHERE QUOTATION_NO
=:B1
Recommendation 4: SQL Tuning
Estimated benefit is .01 active sessions, 3.03% of total activity.
Action
Investigate the SELECT statement with SQL_ID "4uqs4jt7aca5s" for
possible performance improvements. You can supplement the information
given here with an ASH report for this SQL_ID.
Related Object
SQL statement with SQL_ID 4uqs4jt7aca5s.
SELECT DISTINCT USER_ID FROM GV$SESSION, USERS WHERE UPPER (USERNAME)
= UPPER (USER_ID) AND USERS.APPROVAL_CLAIM='VC' AND USER_ID=:B1
Rationale
The SQL spent only 0% of its database time on CPU, I/O and Cluster
waits. Therefore, the SQL Tuning Advisor is not applicable in this case.
Look at performance data for the SQL to find potential improvements.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "4uqs4jt7aca5s" was executed 261 times and had
an average elapsed time of 0.35 seconds.
Rationale
At least one execution of the statement ran in parallel.
Rationale
Top level calls to execute the PL/SQL statement with SQL_ID
"91vt043t78460" are responsible for 100% of the database time spent on
the SELECT statement with SQL_ID "4uqs4jt7aca5s".
Related Object
SQL statement with SQL_ID 91vt043t78460.
begin TPAUSER.RECEIVE_NEW_FAX_APRROVAL(:V00001,:V00002,:V00003,:V0000
4); end;
Recommendation 5: SQL Tuning
Estimated benefit is .01 active sessions, 3.03% of total activity.
Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"7kt28fkc0yn5f".
Related Object
SQL statement with SQL_ID 7kt28fkc0yn5f.
SELECT COUNT(*) FROM TPAUSER.APPROVAL_MASTER WHERE APPROVAL_STATUS IS
NULL AND (UPPER(CODED) = UPPER(:B1 ) OR UPPER(PROCESSED_BY) =
UPPER(:B1 ))
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "7kt28fkc0yn5f" was executed 1034 times and
had an average elapsed time of 0.063 seconds.
Rationale
Top level calls to execute the PL/SQL statement with SQL_ID
"91vt043t78460" are responsible for 100% of the database time spent on
the SELECT statement with SQL_ID "7kt28fkc0yn5f".
Related Object
SQL statement with SQL_ID 91vt043t78460.
begin TPAUSER.RECEIVE_NEW_FAX_APRROVAL(:V00001,:V00002,:V00003,:V0000
4); end;
Finding 2: Interconnect Latency
Impact is .1 active sessions, 24.15% of total activity.
Higher than expected latency of the cluster interconnect was responsible for
significant database time on this instance.
The instance was consuming 128 kilo bits per second of interconnect bandwidth.
17% of this interconnect bandwidth was used for global cache messaging, 6% for
parallel query messaging and 8% for database lock management.
The average latency for 8K interconnect messages was 41863 microseconds.
The instance is using the private interconnect device "lan2" with IP address
172.16.200.72 and source "Oracle Cluster Repository".
The device "lan2" was used for 100% of interconnect traffic and experienced 0
send or receive errors during the analysis period.
Recommendation 1: Host Configuration
Estimated benefit is .1 active sessions, 24.15% of total activity.
Action
Investigate cause of high network interconnect latency between database
instances. Oracle's recommended solution is to use a high speed
dedicated network.
Action
Check the configuration of the cluster interconnect. Check OS setup like
adapter setting, firmware and driver release. Check that the OS's socket
receive buffers are large enough to store an entire multiblock read. The
value of parameter "db_file_multiblock_read_count" may be decreased as a
workaround.
Symptoms That Led to the Finding:
Inter-instance messaging was consuming significant database time on this
instance.
Impact is .06 active sessions, 14.23% of total activity.
Wait class "Cluster" was consuming significant database time.
Impact is .06 active sessions, 14.23% of total activity.
Finding 3: Shared Pool Latches
Impact is .09 active sessions, 22.42% of total activity.
Contention for latches related to the shared pool was consuming significant
database time.
Waits for "library cache lock" amounted to 5% of database time.
Waits for "library cache pin" amounted to 17% of database time.
Recommendation 1: Application Analysis
Estimated benefit is .09 active sessions, 22.42% of total activity.
Action
Investigate the cause for latch contention using the given blocking
sessions or modules.
Rationale
The session with ID 17 and serial number 15595 in instance number 1 was
the blocking session responsible for 34% of this recommendation's
benefit.
Symptoms That Led to the Finding:
Wait class "Concurrency" was consuming significant database time.
Impact is .1 active sessions, 24.96% of total activity.
Finding 4: PL/SQL Execution
Impact is .06 active sessions, 14.39% of total activity.
PL/SQL execution consumed significant database time.
Recommendation 1: SQL Tuning
Estimated benefit is .05 active sessions, 12.5% of total activity.
Action
Tune the entry point PL/SQL "SYS.DBMS_UTILITY.COMPILE_SCHEMA" of type
"PACKAGE" and ID 6019. Refer to the PL/SQL documentation for addition
information.
Rationale
318 seconds spent in executing PL/SQL "SYS.DBMS_UTILITY.VALIDATE#2" of
type "PACKAGE" and ID 6019.
Recommendation 2: SQL Tuning
Estimated benefit is .01 active sessions, 1.89% of total activity.
Action
Tune the entry point PL/SQL
"SYSMAN.EMD_MAINTENANCE.EXECUTE_EM_DBMS_JOB_PROCS" of type "PACKAGE" and
ID 68654. Refer to the PL/SQL documentation for addition information.
Finding 5: Unusual "Other" Wait Event
Impact is .03 active sessions, 8.73% of total activity.
Wait event "DFS lock handle" in wait class "Other" was consuming significant
database time.
Recommendation 1: Application Analysis
Estimated benefit is .03 active sessions, 8.73% of total activity.
Action
Investigate the cause for high "DFS lock handle" waits. Refer to
Oracle's "Database Reference" for the description of this wait event.
Recommendation 2: Application Analysis
Estimated benefit is .03 active sessions, 8.27% of total activity.
Action
Investigate the cause for high "DFS lock handle" waits in Service
"mcmsdrac".
Recommendation 3: Application Analysis
Estimated benefit is .02 active sessions, 5.05% of total activity.
Action
Investigate the cause for high "DFS lock handle" waits in Module "TOAD
9.7.2.5".
Recommendation 4: Application Analysis
Estimated benefit is .01 active sessions, 3.21% of total activity.
Action
Investigate the cause for high "DFS lock handle" waits in Module
"toad.exe".
Symptoms That Led to the Finding:
Wait class "Other" was consuming significant database time.
Impact is .15 active sessions, 38.29% of total activity.
Finding 6: Unusual "Other" Wait Event
Impact is .03 active sessions, 6.42% of total activity.
Wait event "reliable message" in wait class "Other" was consuming significant
database time.
Recommendation 1: Application Analysis
Estimated benefit is .03 active sessions, 6.42% of total activity.
Action
Investigate the cause for high "reliable message" waits. Refer to
Oracle's "Database Reference" for the description of this wait event.
Recommendation 2: Application Analysis
Estimated benefit is .03 active sessions, 6.42% of total activity.
Action
Investigate the cause for high "reliable message" waits in Service
"mcmsdrac".
Recommendation 3: Application Analysis
Estimated benefit is .02 active sessions, 4.13% of total activity.
Action
Investigate the cause for high "reliable message" waits in Module "TOAD
9.7.2.5".
Symptoms That Led to the Finding:
Wait class "Other" was consuming significant database time.
Impact is .15 active sessions, 38.29% of total activity.
Finding 7: Unusual "Other" Wait Event
Impact is .03 active sessions, 6.29% of total activity.
Wait event "enq: PS - contention" in wait class "Other" was consuming
significant database time.
Recommendation 1: Application Analysis
Estimated benefit is .03 active sessions, 6.29% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits. Refer to
Oracle's "Database Reference" for the description of this wait event.
Recommendation 2: Application Analysis
Estimated benefit is .02 active sessions, 6.02% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits in Service
"mcmsdrac".
Recommendation 3: Application Analysis
Estimated benefit is .02 active sessions, 4.93% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits with
P1,P2,P3 ("name|mode, instance, slave ID") values "1347616774", "1" and
"3599" respectively.
Recommendation 4: Application Analysis
Estimated benefit is .01 active sessions, 2.74% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits in Module
"Inbox Reader_92.exe".
Recommendation 5: Application Analysis
Estimated benefit is .01 active sessions, 2.74% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits in Module
"TOAD 9.7.2.5".
Recommendation 6: Application Analysis
Estimated benefit is .01 active sessions, 1.37% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits with
P1,P2,P3 ("name|mode, instance, slave ID") values "1347616774", "1" and
"3598" respectively.
Symptoms That Led to the Finding:
Wait class "Other" was consuming significant database time.
Impact is .15 active sessions, 38.29% of total activity.
Finding 8: Hard Parse
Impact is .02 active sessions, 5.5% of total activity.
Hard parsing of SQL statements was consuming significant database time.
Hard parses due to cursor environment mismatch were not consuming significant
database time.
Hard parsing SQL statements that encountered parse errors was not consuming
significant database time.
Hard parses due to literal usage and cursor invalidation were not consuming
significant database time.
The Oracle instance memory (SGA and PGA) was adequately sized.
No recommendations are available.
Symptoms That Led to the Finding:
Contention for latches related to the shared pool was consuming
significant database time.
Impact is .09 active sessions, 22.42% of total activity.
Wait class "Concurrency" was consuming significant database time.
Impact is .1 active sessions, 24.96% of total activity.
Finding 9: Soft Parse
Impact is .02 active sessions, 3.86% of total activity.
Soft parsing of SQL statements was consuming significant database time.
Recommendation 1: Application Analysis
Estimated benefit is .02 active sessions, 3.86% of total activity.
Action
Investigate application logic to keep open the frequently used cursors.
Note that cursors are closed by both cursor close calls and session
disconnects.
Recommendation 2: Database Configuration
Estimated benefit is .02 active sessions, 3.86% of total activity.
Action
Consider increasing the session cursor cache size by increasing the
value of parameter "session_cached_cursors".
Rationale
The value of parameter "session_cached_cursors" was "100" during the
analysis period.
Symptoms That Led to the Finding:
Contention for latches related to the shared pool was consuming
significant database time.
Impact is .09 active sessions, 22.42% of total activity.
Wait class "Concurrency" was consuming significant database time.
Impact is .1 active sessions, 24.96% of total activity.
Finding 10: Unusual "Other" Wait Event
Impact is .01 active sessions, 3.75% of total activity.
Wait event "IPC send completion sync" in wait class "Other" was consuming
significant database time.
Recommendation 1: Application Analysis
Estimated benefit is .01 active sessions, 3.75% of total activity.
Action
Investigate the cause for high "IPC send completion sync" waits. Refer
to Oracle's "Database Reference" for the description of this wait event.
Recommendation 2: Application Analysis
Estimated benefit is .01 active sessions, 3.75% of total activity.
Action
Investigate the cause for high "IPC send completion sync" waits with P1
("send count") value "1".
Recommendation 3: Application Analysis
Estimated benefit is .01 active sessions, 2.59% of total activity.
Action
Investigate the cause for high "IPC send completion sync" waits in
Service "mcmsdrac".
Recommendation 4: Application Analysis
Estimated benefit is .01 active sessions, 1.73% of total activity.
Action
Investigate the cause for high "IPC send completion sync" waits in
Module "TOAD 9.7.2.5".
Symptoms That Led to the Finding:
Wait class "Other" was consuming significant database time.
Impact is .15 active sessions, 38.29% of total activity.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Additional Information
Miscellaneous Information
Wait class "Application" was not consuming significant database time.
Wait class "Commit" was not consuming significant database time.
Wait class "Configuration" was not consuming significant database time.
CPU was not a bottleneck for the instance.
Wait class "Network" was not consuming significant database time.
Wait class "User I/O" was not consuming significant database time.
Session connect and disconnect calls were not consuming significant database
time.
The database's maintenance windows were active during 100% of the analysis
period.
Please help.Hello experts...
Please do the needful... It's really very urgent.
Thanks,
Syed -
Hi,
I have a Oracle 9.2 DB with 2 cpu's,when i took a statspack report i found that CPU Time is very hight i,e more that 81%....
one of the query is taking near about 51% CPU Usage...
i.e
SELECT /*+ INDEX (OM_COURSE_FEE OCF_CFC_CODE) */DISTINCT CF_COURSE_CODE FROM OM_COURSE_FEE,OT_STUDENT_FEE_COL_HEAD,OT_STUDENT_FEE_COL_DETL WHERE SFCH_SYS_ID = SFCD_SFCH_SYS_ID AND
SFCD_FEE_TYPE_CODE = CF_TYPE_CODE AND CF_COURSE_CODE IN ( 'PE1','PE2','CCT','CPT' ) AND SFCH_TXN_CODE = :b1 AND SFCD_SUBSCRBE_JOURNAL_YN IN ( 'T','R','1','C' ) AND SFCH_APPR_UID IS NOT NULL
AND SFCH_APPR_DT IS NOT NULL AND SFCH_NO = :b2 AND NOT EXISTS (SELECT 'X' FROM OM_STUDENT_HEAD WHERE STUD_SRN = SFCH_STUD_SRN_TEMP_NO AND NVL(STUD_CLO_STATUS,0) != 1 AND
NVL(STUD_REG_STATUS,0) != 23 AND STUD_COURSE_CODE != 'CCT' AND CF_COURSE_CODE= STUD_COURSE_CODE ) ORDER BY 1 DESCExplain Plan for......
SQL> select * from table(dbms_xplan.display());
PLAN_TABLE_OUTPUT
| Id | Operation | Name
| Rows | Bytes | Cost | Pstart| Pstop |
| 0 | SELECT STATEMENT |
PLAN_TABLE_OUTPUT
| 1 | 69 | 45 | | |
| 1 | SORT UNIQUE |
| 1 | 69 | 27 | | |
| 2 | TABLE ACCESS BY GLOBAL INDEX ROWID | OT_STUDENT_FEE_COL_DETL
| 1 | 12 | 2 | ROWID | ROW L |
| 3 | NESTED LOOPS |
| 1 | 69 | 9 | | |
PLAN_TABLE_OUTPUT
| 4 | NESTED LOOPS |
| 1 | 57 | 7 | | |
| 5 | TABLE ACCESS BY GLOBAL INDEX ROWID | OT_STUDENT_FEE_COL_HEAD
| 1 | 48 | 5 | ROWID | ROW L |
| 6 | INDEX SKIP SCAN | OT_STUDENT_FEE_COL_HEAD_UK0
1 | 1 | | 4 | | |
| 7 | INLIST ITERATOR |
| | | | | |
PLAN_TABLE_OUTPUT
| 8 | TABLE ACCESS BY INDEX ROWID | OM_COURSE_FEE
| 1 | 9 | 2 | | |
| 9 | INDEX RANGE SCAN | OCF_CFC_CODE
| 1 | | 1 | | |
| 10 | FILTER |
| | | | | |
| 11 | TABLE ACCESS BY GLOBAL INDEX ROWID| OM_STUDENT_HEAD
PLAN_TABLE_OUTPUT
| 1 | 21 | 4 | ROWID | ROW L |
| 12 | INDEX RANGE SCAN | IDM_STUD_SRN_COURSE
| 1 | | 3 | | |
| 13 | INDEX RANGE SCAN | IDM_SFCD_FEE_TYPE_CODE
| 34600 | | 1 | | |
PLAN_TABLE_OUTPUT
Note: cpu costing is off, PLAN_TABLE' is old version
21 rows selected.
SQL>Statspack report
DB Name DB Id Instance Inst Num Release Cluster Host
ai 1372079993 ai11 1 9.2.0.6.0 YES ai1
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 175 12-Dec-08 13:21:33 ####### .0
End Snap: 176 12-Dec-08 13:56:09 ####### .0
Elapsed: 34.60 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 3,264M Std Block Size: 8K
Shared Pool Size: 608M Log Buffer: 977K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 5,727.62 21,658.54
Logical reads: 16,484.89 62,336.32
Block changes: 32.49 122.88
Physical reads: 200.46 758.03
Physical writes: 5.08 19.23
User calls: 97.43 368.44
Parses: 11.66 44.11
Hard parses: 0.39 1.48
Sorts: 3.22 12.19
Logons: 0.02 0.06
Executes: 36.70 138.77
Transactions: 0.26
% Blocks changed per Read: 0.20 Recursive Call %: 28.65
Rollback per transaction %: 20.95 Rows per Sort: 131.16
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 98.79 In-memory Sort %: 99.99
Library Hit %: 98.92 Soft Parse %: 96.65
Execute to Parse %: 68.21 Latch Hit %: 99.98
Parse CPU to Parse Elapsd %: 60.50 % Non-Parse CPU: 99.48
Shared Pool Statistics Begin End
Memory Usage %: 90.06 89.79
% SQL with executions>1: 72.46 72.46
% Memory for SQL w/exec>1: 69.42 69.51
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
CPU time 3,337 81.43
db file sequential read 60,550 300 7.32
global cache cr request 130,852 177 4.33
db file scattered read 72,915 101 2.46
db file parallel read 3,384 75 1.84
Cluster Statistics for DB: ai Instance: ai11 Snaps: 175 -176
Global Cache Service - Workload Characteristics
Ave global cache get time (ms): 1.3
Ave global cache convert time (ms): 2.1
Ave build time for CR block (ms): 0.1
Ave flush time for CR block (ms): 0.3
Ave send time for CR block (ms): 0.3
Ave time to process CR block request (ms): 0.7
Ave receive time for CR block (ms): 4.9
Ave pin time for current block (ms): 0.0
Ave flush time for current block (ms): 0.0
Ave send time for current block (ms): 0.3
Ave time to process current block request (ms): 0.3
Ave receive time for current block (ms): 2.8
Global cache hit ratio: 1.5
Ratio of current block defers: 0.0
% of messages sent for buffer gets: 1.4
% of remote buffer gets: 0.1
Ratio of I/O for coherence: 1.1
Ratio of local vs remote work: 9.7
Ratio of fusion vs physical writes: 0.1
Global Enqueue Service Statistics
Ave global lock get time (ms): 0.8
Ave global lock convert time (ms): 0.0
Ratio of global lock gets vs global lock releases: 1.1
GCS and GES Messaging statistics
Ave message sent queue time (ms): 0.4
Ave message sent queue time on ksxp (ms): 2.7
Ave message received queue time (ms): 0.2
Ave GCS message process time (ms): 0.1
Ave GES message process time (ms): 0.1
% of direct sent messages: 19.4
% of indirect sent messages: 43.5
% of flow controlled messages: 37.1
GES Statistics for DB: ai Instance: ai11 Snaps: 175 -176
Statistic Total per Second per Trans
dynamically allocated gcs resourc 0 0.0 0.0
dynamically allocated gcs shadows 0 0.0 0.0
flow control messages received 0 0.0 0.0
flow control messages sent 0 0.0 0.0
gcs ast xid 0 0.0 0.0
gcs blocked converts 1,231 0.6 2.2
gcs blocked cr converts 2,432 1.2 4.4
gcs compatible basts 0 0.0 0.0
gcs compatible cr basts (global) 658 0.3 1.2
gcs compatible cr basts (local) 57,822 27.9 105.3
gcs cr basts to PIs 0 0.0 0.0
gcs cr serve without current lock 0 0.0 0.0
gcs error msgs 0 0.0 0.0
gcs flush pi msgs 821 0.4 1.5
gcs forward cr to pinged instance 0 0.0 0.0
gcs immediate (compatible) conver 448 0.2 0.8
gcs immediate (null) converts 1,114 0.5 2.0
gcs immediate cr (compatible) con 42,094 20.3 76.7
gcs immediate cr (null) converts 396,284 190.9 721.8
gcs msgs process time(ms) 42,220 20.3 76.9
gcs msgs received 545,133 262.6 993.0
gcs out-of-order msgs 5 0.0 0.0
gcs pings refused 1 0.0 0.0
gcs queued converts 0 0.0 0.0
gcs recovery claim msgs 0 0.0 0.0
gcs refuse xid 0 0.0 0.0
gcs retry convert request 0 0.0 0.0
gcs side channel msgs actual 2,397 1.2 4.4
gcs side channel msgs logical 232,024 111.8 422.6
gcs write notification msgs 15 0.0 0.0
gcs write request msgs 278 0.1 0.5
gcs writes refused 1 0.0 0.0
ges msgs process time(ms) 4,873 2.3 8.9
ges msgs received 39,769 19.2 72.4
global posts dropped 0 0.0 0.0
global posts queue time 0 0.0 0.0
global posts queued 0 0.0 0.0
global posts requested 0 0.0 0.0
global posts sent 0 0.0 0.0
implicit batch messages received 39,098 18.8 71.2
implicit batch messages sent 33,386 16.1 60.8
lmd msg send time(ms) 635 0.3 1.2
lms(s) msg send time(ms) 2 0.0 0.0
messages flow controlled 196,546 94.7 358.0
messages received actual 182,783 88.0 332.9
messages received logical 584,848 281.7 1,065.3
messages sent directly 102,657 49.4 187.0
messages sent indirectly 230,329 110.9 419.5
msgs causing lmd to send msgs 9,169 4.4 16.7
msgs causing lms(s) to send msgs 3,347 1.6 6.1
msgs received queue time (ms) 142,759 68.8 260.0
msgs received queued 584,818 281.7 1,065.2
msgs sent queue time (ms) 99,300 47.8 180.9
msgs sent queue time on ksxp (ms) 608,239 293.0 1,107.9
msgs sent queued 230,391 111.0 419.7
msgs sent queued on ksxp 229,013 110.3 417.1
GES Statistics for DB: ai Instance: ai11 Snaps: 175 -176
Statistic Total per Second per Trans
process batch messages received 65,059 31.3 118.5
process batch messages sent 50,959 24.5 92.8
Wait Events for DB: ai Instance: ai11 Snaps: 175 -176
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
db file sequential read 60,550 0 300 5 110.3
global cache cr request 130,852 106 177 1 238.3
db file scattered read 72,915 0 101 1 132.8
db file parallel read 3,384 0 75 22 6.2
latch free 7,253 1,587 52 7 13.2
enqueue 44,947 0 16 0 81.9
log file parallel write 2,140 0 6 3 3.9
db file parallel write 341 0 5 14 0.6
global cache open x 1,134 3 4 4 2.1
CGS wait for IPC msg 166,993 164,390 4 0 304.2
library cache lock 3,169 0 3 1 5.8
log file sync 494 0 3 5 0.9
row cache lock 702 0 3 4 1.3
DFS lock handle 6,900 0 2 0 12.6
control file parallel write 689 0 2 3 1.3
control file sequential read 2,785 0 2 1 5.1
wait for master scn 687 0 2 2 1.3
global cache null to x 699 0 2 2 1.3
global cache s to x 778 5 1 2 1.4
direct path write 148 0 0 3 0.3
SQL*Net more data to client 3,621 0 0 0 6.6
global cache open s 149 0 0 2 0.3
library cache pin 78 0 0 2 0.1
ksxr poll remote instances 3,536 2,422 0 0 6.4
LGWR wait for redo copy 12 6 0 9 0.0
buffer busy waits 23 0 0 5 0.0
direct path read 9 0 0 10 0.0
buffer busy global CR 5 0 0 17 0.0
SQL*Net break/reset to clien 172 0 0 0 0.3
global cache quiesce wait 4 1 0 7 0.0
KJC: Wait for msg sends to c 86 0 0 0 0.2
BFILE get length 67 0 0 0 0.1
global cache null to s 9 0 0 1 0.0
BFILE open 6 0 0 0 0.0
BFILE read 60 0 0 0 0.1
BFILE internal seek 60 0 0 0 0.1
BFILE closure 6 0 0 0 0.0
cr request retry 4 4 0 0 0.0
SQL*Net message from client 201,907 0 180,583 894 367.8
gcs remote message 171,236 43,749 3,975 23 311.9
ges remote message 79,324 40,722 2,017 25 144.5
SQL*Net more data from clien 447 0 9 21 0.8
SQL*Net message to client 201,901 0 0 0 367.8
Background Wait Events for DB: ai Instance: ai11 Snaps: 175 -176
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
enqueue 44,666 0 16 0 81.4
latch free 4,895 108 6 1 8.9
log file parallel write 2,140 0 6 3 3.9
db file parallel write 341 0 5 14 0.6
CGS wait for IPC msg 166,969 164,366 4 0 304.1
DFS lock handle 6,900 0 2 0 12.6
control file parallel write 689 0 2 3 1.3
control file sequential read 2,733 0 2 1 5.0
wait for master scn 687 0 2 2 1.3
ksxr poll remote instances 3,528 2,419 0 0 6.4
LGWR wait for redo copy 12 6 0 9 0.0
db file sequential read 7 0 0 9 0.0
global cache null to x 26 0 0 2 0.0
global cache open x 16 0 0 1 0.0
global cache cr request 1 0 0 0 0.0
rdbms ipc message 28,636 20,600 16,937 591 52.2
gcs remote message 171,254 43,746 3,974 23 311.9
pmon timer 708 686 2,022 2856 1.3
ges remote message 79,305 40,719 2,017 25 144.5
smon timer 125 0 1,972 15776 0.2
SQL ordered by Gets for DB: ai Instance: ai11 Snaps: 175 -176
-> End Buffer Gets Threshold: 10000
-> Note that resources reported for PL/SQL includes the resources used by
all SQL statements called within the PL/SQL code. As individual SQL
statements are also reported, it is possible and valid for the summed
total % to exceed 100
CPU Elapsd
Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value
17,503,305 84 208,372.7 51.1 676.25 1218.80 17574787
SELECT /*+ INDEX (OM_COURSE_FEE OCF_CFC_CODE) */DISTINCT CF_COU
RSE_CODE FROM OM_COURSE_FEE,OT_STUDENT_FEE_COL_HEAD,OT_STUDENT
FEECOL_DETL WHERE SFCH_SYS_ID = SFCD_SFCH_SYS_ID AND SFCD_FE
E_TYPE_CODE = CF_TYPE_CODE AND CF_COURSE_CODE IN ( 'PE1','PE2',
'CCT','CPT' ) AND SFCH_TXN_CODE = :b1 AND SFCD_SUBSCRBE_JOURNAuser00726 wrote:
somw of the changes that have been done....
cai11_lmd0_15771.trc.
Thu Oct 2 13:35:48 2008
CKPT: Begin resize of buffer pool 3 (DEFAULT for block size 8192)
CKPT: Current size = 2512 MB, Target size = 3072 MB
CKPT: Resize completed for buffer pool DEFAULT for blocksize 8192
Thu Oct 2 13:35:50 2008
ALTER SYSTEM SET db_cache_size='3072M' SCOPE=BOTH SID='icai11';
Thu Oct 2 14:04:34 2008
ALTER SYSTEM SET sga_max_size='5244772679680' SCOPE=SPFILE SID='*';
ALTER SYSTEM SET sga_max_size='5244772679680' SCOPE=SPFILE SID='*';
Thu Oct 2 15:24:14 2008
CKPT: Begin resize of buffer pool 3 (DEFAULT for block size 8192)
CKPT: Current size = 3072 MB, Target size = 2512 MB
CKPT: Resize completed for buffer pool DEFAULT for blocksize 8192
Thu Oct 2 15:24:20 2008
ALTER SYSTEM SET db_cache_size='2512M' SCOPE=BOTH SID='icai11';
Thu Oct 2 15:32:33 2008
CKPT: Begin resize of buffer pool 3 (DEFAULT for block size 8192)
CKPT: Current size = 2512 MB, Target size = 3072 MB
CKPT: Resize completed for buffer pool DEFAULT for blocksize 8192
Thu Oct 2 15:32:34 2008
ALTER SYSTEM SET db_cache_size='3072M' SCOPE=BOTH SID='icai11';
Thu Oct 2 15:36:46 2008
ALTER SYSTEM SET shared_pool_size='640M' SCOPE=BOTH SID='icai11';
Thu Oct 2 16:33:52 2008
CKPT: Begin resize of buffer pool 3 (DEFAULT for block size 8192)
CKPT: Current size = 3072 MB, Target size = 2512 MB
CKPT: Resize completed for buffer pool DEFAULT for blocksize 8192
Thu Oct 2 16:33:56 2008
ALTER SYSTEM SET db_cache_size='2512M' SCOPE=BOTH SID='icai11';
Thu Oct 2 16:39:30 2008
ALTER SYSTEM SET pga_aggregate_target='750M' SCOPE=BOTH SID='icai11';Just to make certain that I am not missing anything, if the above you set (all scaled to GB just for the sake of comparison):
sga_max_size=4885GB
db_cache_size=3GB
shared_pool_size=0.625GB
pga_aggregate_target=0.733GB
The SQL statement is forcing the use of a specific index, is this the best index?:
SELECT /*+ INDEX (OM_COURSE_FEE OCF_CFC_CODE) */DISTINCT CF_COURSE_CODE FROM OM_COURSE_FEE,OT_STUDENT_FEE_COL_HEAD,OT_STUDENT_FEE_COL_DETL WHERE SFCH_SYS_ID = SFCD_SFCH_SYS_ID AND
SFCD_FEE_TYPE_CODE = CF_TYPE_CODE AND CF_COURSE_CODE IN ( 'PE1','PE2','CCT','CPT' ) AND SFCH_TXN_CODE = :b1 AND SFCD_SUBSCRBE_JOURNAL_YN IN ( 'T','R','1','C' ) AND SFCH_APPR_UID IS NOT NULL
AND SFCH_APPR_DT IS NOT NULL AND SFCH_NO = :b2 AND NOT EXISTS (SELECT 'X' FROM OM_STUDENT_HEAD WHERE STUD_SRN = SFCH_STUD_SRN_TEMP_NO AND NVL(STUD_CLO_STATUS,0) != 1 AND
NVL(STUD_REG_STATUS,0) != 23 AND STUD_COURSE_CODE != 'CCT' AND CF_COURSE_CODE= STUD_COURSE_CODE ) ORDER BY 1 DESC
Unfortunately, explain plans, even with dbms_xplan.display, may not show the actual execution plan - this is more of a problem when the SQL statement includes bind variables (capturing a 10046 trace at level 8 or 12 will help). With the information provided, it looks like the problem is with the number of logical reads performed: 17,503,305 in 84 executions = 208,373 logical reads per execution. Something is causing the SQL statement to execute inefficiently, possibly a join problem between tables, possibly the forced use of an index.
From one of my previous posts related to this same SQL statement:
SELECT /*+ INDEX (OM_COURSE_FEE OCF_CFC_CODE) */ DISTINCT
CF_COURSE_CODE
FROM
OM_COURSE_FEE,
OT_STUDENT_FEE_COL_HEAD,
OT_STUDENT_FEE_COL_DETL
WHERE
SFCH_SYS_ID = SFCD_SFCH_SYS_ID
AND SFCD_FEE_TYPE_CODE = CF_TYPE_CODE
AND CF_COURSE_CODE IN ( 'PE1','PE2','CCT','CPT' )
AND SFCH_TXN_CODE = :b1
AND SFCD_SUBSCRBE_JOURNAL_YN IN ( 'T','R','1','C' )
AND SFCH_APPR_UID IS NOT NULL
AND SFCH_APPR_DT IS NOT NULL
AND SFCH_NO = :b2
AND NOT EXISTS (
SELECT
'X'
FROM
OM_STUDENT_HEAD
WHERE
STUD_SRN = SFCH_STUD_SRN_TEMP_NO
AND NVL(STUD_CLO_STATUS,0) != 1
AND NVL(STUD_REG_STATUS,0) != 23
AND STUD_COURSE_CODE != 'CCT'
AND CF_COURSE_CODE = STUD_COURSE_CODE)
ORDER BY
1 DESC
| Id | Operation | Name | Rows | Bytes | Cost | Pstart| Pstop |
| 0 | SELECT STATEMENT 1 69 34
| 1 | SORT UNIQUE 1 69 22
| 2 | TABLE ACCESS BY GLOBAL INDEX ROWID | OT_STUDENT_FEE_COL_DETL | 1 | 12 | 2 | ROWID | ROW L |
| 3 | NESTED LOOPS 1 69 9
| 4 | NESTED LOOPS 1 57 7
| 5 | TABLE ACCESS BY GLOBAL INDEX ROWID | OT_STUDENT_FEE_COL_HEAD | 1 | 48 | 5 | ROWID | ROW L |
| 6 | INDEX SKIP SCAN OT_STUDENT_FEE_COL_HEAD_UK0| 1 | 1 | 4 |
| 7 | INLIST ITERATOR |
| 8 | TABLE ACCESS BY INDEX ROWID | OM_COURSE_FEE | 1 | 9 | 2 | | |
| 9 | INDEX RANGE SCAN | OCF_CFC_CODE | 1 | | 1 | | |
| 10 | FILTER
| 11 | TABLE ACCESS BY GLOBAL INDEX ROWID| OM_STUDENT_HEAD | 1 | 21 | 4 | ROWID | ROW L |
| 12 | INDEX RANGE SCAN | IDM_STUD_SRN_COURSE | 1 | | 3 | | |
| 13 | INDEX RANGE SCAN | IDM_SFCD_FEE_TYPE_CODE | 34600 | | 1 | | |It appears, based just on the SQL statement, that there is no direct relationship between OM_COURSE_FEE and OT_STUDENT_FEE_COL_HEAD, yet the plan above indicates that the two tables are being joined together, likely as a result of the index hint. There is the possibility of additional predicates being generated by Oracle which will make this possible without introducing a Cartesian join, but those predicates are not displayed with an explain plan (they will appear with a DBMS_XPLAN). I may not be remembering correctly, but if the optimizer goal is set to first rows, a Cartesian join might appear in a plan as a nested loops join - Jonathan will know for certain if that is the case.
Have you investigated the above as a possible cause? If you know that there is a problem with this one SQL statement, a 10046 trace at level 8 or 12 is much more helpful than a Statspack report.
Charles Hooper
IT Manager/Oracle DBA
K&M Machine-Fabricating, Inc. -
Automatic Parallelism causes Merge statement to take longer.
We have a problem in a new project as part of the ETL load into the Oracle datawarehouse we perform a merge statement to update rows in a global temporary table then load
the results into a permanant table, when testing with automatic parallel execution enabled the plan changes and the merge never finishes and consumes vast amounts of resources.
The database version is:-
Database version :11.2.0.3
OS: redhat 64bit
three node rac 20 cores per node
when executing serially the query response is typically similar to the following:
MERGE /*+ gather_plan_statistics no_parallel */ INTO T_GTTCHARGEVALUES USING
(SELECT
CASTACCOUNTID,
CHARGESCHEME,
MAX(CUMULATIVEVALUE) AS MAXMONTHVALUE,
MAX(CUMULATIVECOUNT) AS MAXMONTHCOUNT
FROM
V_CACHARGESALL
WHERE
CHARGEDATE >= TRUNC(TO_DATE(:B1,'YYYY-MM-DD'),'MM')
AND CHARGEDATE < TO_DATE(:B1,'YYYY-MM-DD')
GROUP BY
CASTACCOUNTID,
CHARGESCHEME
HAVING MAX(CUMULATIVECOUNT) IS NOT NULL ) MTOTAL
ON
(T_GTTCHARGEVALUES.CASTACCOUNTID=MTOTAL.CASTACCOUNTID AND
T_GTTCHARGEVALUES.CHARGESCHEME=MTOTAL.CHARGESCHEME)
WHEN MATCHED
THEN UPDATE SET
CUMULATIVEVALUE=CUMULATIVEVALUE+MTOTAL.MAXMONTHVALUE ,
CUMULATIVECOUNT=CUMULATIVECOUNT+MTOTAL.MAXMONTHCOUNT;
1448340 rows merged.
select * from table(dbms_xplan.display_cursor(null,null,'ALLSTATS LAST'));
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem |
| 0 | MERGE STATEMENT | | 1 | | 0 |00:03:08.43 | 2095K| 186K| | | |
| 1 | MERGE | T_GTTCHARGEVALUES | 1 | | 0 |00:03:08.43 | 2095K| 186K| | | |
| 2 | VIEW | | 1 | | 1448K|00:02:53.14 | 619K| 177K| | | |
|* 3 | HASH JOIN | | 1 | 1 | 1448K|00:02:52.70 | 619K| 177K| 812K| 812K| 1218K (0)|
| 4 | VIEW | | 1 | 1 | 203 |00:02:51.26 | 608K| 177K| | | |
|* 5 | FILTER | | 1 | | 203 |00:02:51.26 | 608K| 177K| | | |
| 6 | SORT GROUP BY | | 1 | 1 | 480 |00:02:51.26 | 608K| 177K| 73728 | 73728 | |
|* 7 | FILTER | | 1 | | 21M|00:02:56.04 | 608K| 177K| | | |
| 8 | PARTITION RANGE ITERATOR| | 1 | 392K| 21M|00:02:51.32 | 608K| 177K| | | |
|* 9 | TABLE ACCESS FULL | T_CACHARGES | 24 | 392K| 21M|00:02:47.48 | 608K| 177K| | | |
| 10 | TABLE ACCESS FULL | T_GTTCHARGEVALUES | 1 | 1451K| 1451K|00:00:00.48 | 10980 | 0 | | | |
Predicate Information (identified by operation id):
3 - access("T_GTTCHARGEVALUES"."CASTACCOUNTID"="MTOTAL"."CASTACCOUNTID" AND "T_GTTCHARGEVALUES"."CHARGESCHEME"="MTOTAL"."CHARGESCHEME")
5 - filter(MAX("CUMULATIVECOUNT") IS NOT NULL)
7 - filter(TRUNC(TO_DATE(:B1,'YYYY-MM-DD'),'fmmm')<TO_DATE(:B1,'YYYY-MM-DD'))
9 - filter(("LOGICALLYDELETED"=0 AND "CHARGEDATE">=TRUNC(TO_DATE(:B1,'YYYY-MM-DD'),'fmmm') AND "CHARGEDATE"<TO_DATE(:B1,'YYYY-MM-DD')))removing the no_parallel hint results in the following, (this is pulled from the sql monitoring report and editied to remove the lines relating to individual parallel servers)
I understand that the query is considered for parallel execution due to the estimated length of time it will run for and although the degree of parallleism seems excessive
it is the default maximum for the server configuration, what we are tryig to understand is which statistics could be inacurate or missing and could cause this kind of problem.
In this case we can add the no_parallel hint in the etl package as a workaround but would really liek to identify the root cause to avoid similar problems elsewhere.
SQL Monitoring Report
SQL Text
MERGE INTO T_GTTCHARGEVALUES USING (SELECT CASTACCOUNTID, CHARGESCHEME, MAX(CUMULATIVEVALUE) AS MAXMONTHVALUE,
MAX(CUMULATIVECOUNT) AS MAXMONTHCOUNT FROM V_CACHARGESALL WHERE CHARGEDATE >= TRUNC(TO_DATE(:B1,'YYYY-MM-DD'),'MM')
AND CHARGEDATE < to_date(:B1,'YYYY-MM-DD')
GROUP BY CASTACCOUNTID, CHARGESCHEME HAVING MAX(CUMULATIVECOUNT) IS NOT NULL ) MTOTAL
ON (T_GTTCHARGEVALUES.CASTACCOUNTID=MTOTAL.CASTACCOUNTID AND
T_GTTCHARGEVALUES.CHARGESCHEME=MTOTAL.CHARGESCHEME) WHEN MATCHED THEN UPDATE SET
CUMULATIVEVALUE=CUMULATIVEVALUE+MTOTAL.MAXMONTHVALUE ,
CUMULATIVECOUNT=CUMULATIVECOUNT+MTOTAL.MAXMONTHCOUNT
Error: ORA-1013
ORA-01013: user requested cancel of current operation
Global Information
Status : DONE (ERROR)
Instance ID : 1
Session : XXXX(2815:12369)
SQL ID : 70kzttjbyyspt
SQL Execution ID : 16777216
Execution Started : 04/27/2012 09:43:27
First Refresh Time : 04/27/2012 09:43:27
Last Refresh Time : 04/27/2012 09:48:43
Duration : 316s
Module/Action : SQL*Plus/-
Service : SYS$USERS
Program : sqlplus@XXXX (TNS V1-V3)
Binds
========================================================================================================================
| Name | Position | Type | Value |
========================================================================================================================
| :B1 | 1 | VARCHAR2(32) | 2012-04-25 |
========================================================================================================================
Global Stats
====================================================================================================================
| Elapsed | Queuing | Cpu | IO | Application | Concurrency | Cluster | Other | Buffer | Read | Read |
| Time(s) | Time(s) | Time(s) | Waits(s) | Waits(s) | Waits(s) | Waits(s) | Waits(s) | Gets | Reqs | Bytes |
====================================================================================================================
| 7555 | 0.00 | 4290 | 2812 | 0.08 | 27 | 183 | 243 | 3M | 294K | 7GB |
====================================================================================================================
SQL Plan Monitoring Details (Plan Hash Value=323941584)
==========================================================================================================================================================================================================
| Id | Operation | Name | Rows | Cost | Time | Start | Execs | Rows | Read | Read | Mem | Activity | Activity Detail |
| | | | (Estim) | | Active(s) | Active | | (Actual) | Reqs | Bytes | (Max) | (%) | (# samples) |
==========================================================================================================================================================================================================
| 0 | MERGE STATEMENT | | | | | | 1 | | | | | | |
| 1 | MERGE | T_GTTCHARGEVALUES | | | | | 1 | | | | | | |
| 2 | PX COORDINATOR | | | | 57 | +1 | 481 | 0 | 317 | 5MB | | 4.05 | latch: shared pool (40) |
| | | | | | | | | | | | | | os thread startup (17) |
| | | | | | | | | | | | | | Cpu (7) |
| | | | | | | | | | | | | | DFS lock handle (36) |
| | | | | | | | | | | | | | SGA: allocation forcing component growth (14) |
| | | | | | | | | | | | | | latch: parallel query alloc buffer (200) |
| 3 | PX SEND QC (RANDOM) | :TQ10003 | 1 | 19054 | | | | | | | | | |
| 4 | VIEW | | | | | | | | | | | | |
| 5 | FILTER | | | | | | | | | | | | |
| 6 | SORT GROUP BY | | 1 | 19054 | | | | | | | | | |
| 7 | PX RECEIVE | | 1 | 19054 | | | | | | | | | |
| 8 | PX SEND HASH | :TQ10002 | 1 | 19054 | | | 240 | | | | | | |
| 9 | SORT GROUP BY | | 1 | 19054 | 246 | +70 | 240 | 0 | | | 228M | 49.32 | Cpu (3821) |
| 10 | FILTER | | | | 245 | +71 | 240 | 3G | | | | 0.08 | Cpu (6) |
| 11 | HASH JOIN | | 1 | 19054 | 259 | +57 | 240 | 3G | | | 276M | 4.31 | Cpu (334) |
| 12 | PX RECEIVE | | 1M | 5 | 259 | +57 | 240 | 1M | | | | 0.04 | Cpu (3) |
| 13 | PX SEND HASH | :TQ10000 | 1M | 5 | 6 | +56 | 240 | 1M | | | | 0.01 | Cpu (1) |
| 14 | PX BLOCK ITERATOR | | 1M | 5 | 6 | +56 | 240 | 1M | | | | 0.03 | Cpu (1) |
| | | | | | | | | | | | | | PX Deq: reap credit (1) |
| 15 | TABLE ACCESS FULL | T_GTTCHARGEVALUES | 1M | 5 | 7 | +55 | 5486 | 1M | 5487 | 86MB | | 2.31 | gc cr grant 2-way (3) |
| | | | | | | | | | | | | | gc current block lost (7) |
| | | | | | | | | | | | | | Cpu (7) |
| | | | | | | | | | | | | | db file sequential read (162) |
| 16 | PX RECEIVE | | 78M | 19047 | 255 | +61 | 240 | 801K | | | | 0.03 | IPC send completion sync (2) |
| 17 | PX SEND HASH | :TQ10001 | 78M | 19047 | 250 | +66 | 240 | 3M | | | | 0.06 | Cpu (5) |
| 18 | PX BLOCK ITERATOR | | 78M | 19047 | 250 | +66 | 240 | 4M | | | | | |
| 19 | TABLE ACCESS FULL | T_CACHARGES | 78M | 19047 | 254 | +62 | 1016 | 4M | 288K | 6GB | | 37.69 | gc buffer busy acquire (104) |
| | | | | | | | | | | | | | gc cr block 2-way (1) |
| | | | | | | | | | | | | | gc cr block lost (9) |
| | | | | | | | | | | | | | gc cr grant 2-way (14) |
| | | | | | | | | | | | | | gc cr multi block request (1) |
| | | | | | | | | | | | | | gc current block 2-way (3) |
| | | | | | | | | | | | | | gc current block 3-way (2) |
| | | | | | | | | | | | | | gc current block busy (1) |
| | | | | | | | | | | | | | gc current grant busy (2) |
| | | | | | | | | | | | | | Cpu (58) |
| | | | | | | | | | | | | | latch: gc element (1) |
| | | | | | | | | | | | | | db file parallel read (26) |
| | | | | | | | | | | | | | db file scattered read (207) |
| | | | | | | | | | | | | | db file sequential read (2433) |
| | | | | | | | | | | | | | direct path read (1) |
| | | | | | | | | | | | | | read by other session (57) |
==========================================================================================================================================================================================================
Parallel Execution Details (DOP=240 , Servers Allocated=480)
Instances : 3chris_c wrote:
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem |
|* 9 | TABLE ACCESS FULL | T_CACHARGES | 24 | 392K| 21M|00:02:47.48 | 608K| 177K| | | |
Based on the discrepancy between the estimated number of rows and the actual, and the below posted bind value of 2012-04-25 i'd first be checking if the statistics on T_CACHARGES are up to date.
As a reference
http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:4399338600346902127
So that would be my first avenue of exploration.
Cheers, -
Here's is the contents of v$system_event..
Is this
EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT
enq: TX - index contention 40564851 214701526 5.29
enq: TX - row lock contention 188846 12454614 65.95
enq: SQ - contention 141971 70568 0.5
cause for concern?
EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT
SQL*Net message to client 6015051449 607254 0
SQL*Net message from client 6015048542 178177969892 29.62
gcs remote message 2948555287 2633481757 0.89
CGS wait for IPC msg 1517805027 634397 0
db file sequential read 1500615188 816364485 0.54
ges remote message 1247679701 1407300224 1.13
gc cr multi block request 778432813 9913464 0.01
gc current block 2-way 747852637 38030616 0.05
db file scattered read 709428365 460939295 0.65
rdbms ipc message 708473316 37650068633 53.14
gc buffer busy acquire 671285134 1033621285 1.54
PX Deq: reap credit 667784615 484449 0
gcs log flush sync 592376026 171712257 0.29
gc cr block 2-way 530861847 19607062 0.04
library cache pin 437937120 15126237 0.03
log file sync 379523272 797193932 2.1
DIAG idle wait 359607166 2822108755 7.85
log file parallel write 351225436 259263769 0.74
LNS ASYNC end of log 350170653 1398410516 3.99
LNS wait on SENDREQ 321652621 3209301 0.01
PX qref latch 297396661 94308 0
read by other session 289140108 148440270 0.51
buffer deadlock 163505781 983055 0.01
gc current block busy 119223825 467716658 3.92
PX Deq: Table Q Normal 117332841 23574867 0.2
ksxr poll remote instances 110480324 90333 0
buffer busy waits 106938153 19933900 0.19
direct path read 93429599 108427028 1.16
SQL*Net more data from client 86471785 23026529 0.27
gc current grant busy 84978157 28215346 0.33
control file sequential read 82646297 23694583 0.29
PX Deq Credit: send blkd 78641669 9569299 0.12
latch: cache buffers chains 74218671 690277 0.01
gc current grant 2-way 72557796 1920419 0.03
library cache: mutex X 71106697 75993 0
DFS lock handle 70722498 2716407 0.04
gc cr grant 2-way 64558237 1633004 0.03
PX Deq: Execution Msg 61706261 314222076 5.09
gc cr block busy 61469863 119850802 1.95
library cache lock 52428649 3773354 0.07
PX Deq: Slave Session Stats 48040224 1886805 0.04
db file parallel read 46415188 118467902 2.55
IPC send completion sync 46250594 965101 0.02
enq: TX - index contention 40564851 214701526 5.29
PX Deq: Execute Reply 39689685 17243721 0.43
gc buffer busy release 36976909 242714774 6.56
SQL*Net more data to client 36627952 44167 0
PX Deq: Msg Fragment 30501244 343397 0.01
rdbms ipc reply 29725302 1352370 0.05
RMAN backup & recovery I/O 28824547 37722662 1.31
reliable message 27892263 3082134 0.11
PX Idle Wait 27356097 4651277341 170.03
ASM file metadata operation 25098749 8850323 0.35
gc object scan 22705857 7485 0
db file parallel write 19896252 52152606 2.62
latch: ges resource hash list 19336183 427451 0.02
enq: PS - contention 19143961 707455 0.04
PX Deq: Parse Reply 19093356 895799 0.05
gc cr disk read 17816846 448909 0.03
ASM background timer 16101806 1383957874 85.95
PX Deq: Slave Join Frag 16044789 233149 0.01
wait for unread message on broadcast channel 15056320 1413552546 93.88
cursor: mutex X 13435193 24140 0
KJC: Wait for msg sends to complete 13268497 11397 0
PX Deq: Signal ACK RSG 13214824 101941 0.01
KSV master wait 13206286 4235645 0.32
direct path read temp 12617694 5487608 0.43
PX Deq Credit: need buffer 11675868 879967 0.08
row cache lock 11536185 398216 0.03
PX Deq Credit: Session Stats 9480862 78910 0.01
SQL*Net message to dblink 9312894 1538 0
SQL*Net message from dblink 9312894 6279631 0.67
control file parallel write 7760982 11854435 1.53
pmon timer 7558889 1412576090 186.88
PX Deq: Join ACK 7548816 498931 0.07
gc current multi block request 6035173 155898 0.03
PING 5706961 1413230267 247.63
enq: XR - database force logging 4662671 198813 0.04
class slave wait 4561877 7097429006 1555.81
Streams AQ: waiting for messages in the queue 4495828 1543411682 343.3
SQL*Net more data from dblink 3696582 444575 0.12
LGWR wait for redo copy 3655353 17840 0
log file sequential read 3387305 6610414 1.95
Log archive I/O 2990486 276772 0.09
SQL*Net break/reset to client 2971976 2385935 0.8
direct path write temp 2839390 2522114 0.89
Space Manager: slave idle wait 2827526 1412987186 499.73
latch: shared pool 2808517 298150 0.11
latch: gc element 2421717 24688 0.01
SGA: MMAN sleep for component shrink 2336447 2458094 1.05
latch: enqueue hash chains 2279645 15435 0.01
latch free 2089418 78732 0.04
gc current split 2044784 1864009 0.91
PX Deq: Signal ACK EXT 1976164 19263 0.01
enq: FB - contention 1473469 61036 0.04
cursor: pin S wait on X 1313129 1464789 1.12
SQL*Net more data to dblink 1232891 986 0
Streams AQ: RAC qmn coordinator idle wait 1211300 788 0
enq: HW - contention 1175390 2077008 1.77
latch: session allocation 1167768 21883 0.02
Streams AQ: qmn coordinator idle wait 1144699 1412546634 1233.99
Streams AQ: qmn slave idle wait 1031585 2227183681 2158.99
lock deadlock retry 962937 4698 0
enq: CF - contention 956154 609647 0.64
latch: cache buffers lru chain 902764 37552 0.04
latch: object queue header operation 817911 27717 0.03
global enqueue expand wait 768633 654105 0.85
Data file init write 756191 329758 0.44
latch: gcs resource hash 647021 4147 0.01
local write wait 603007 286191 0.47
latch: row cache objects 599358 6453 0.01
ges lmd/lmses to freeze in rcfg - mrcvr 481759 156345 0.32
shared server idle wait 471190 1413238589 2999.3
enq: RF - DG Broker Current File ID 469833 23209 0.05
smon timer 432383 1411851085 3265.28
SGA: allocation forcing component growth 363333 379008 1.04
gc current retry 341104 1121252 3.29
enq: RF - synch: DG Broker metadata 319143 588290 1.84
enq: PG - contention 313659 14830 0.05
enq: TT - contention 260134 11207172 43.08
enq: KO - fast object checkpoint 236745 820808 3.47
dispatcher timer 236637 1413242481 5972.2
direct path write 231382 191008 0.83
cursor: pin S 229011 394 0
Streams AQ: waiting for time management or cleanup tasks 199981 1413148548 7066.41
enq: TX - row lock contention 188846 12454614 65.95
enq: TX - allocate ITL entry 153703 54252 0.35
enq: SQ - contention 141971 70568 0.5
ksdxexeother 141885 56 0
latch: redo allocation 138912 1858 0.01
recovery area: computing applied logs 126415 45925 0.36
gc current block congested 126318 21768 0.17
resmgr:cpu quantum 123074 151384 1.23
jobq slave wait 120678 35574221 294.79
Datapump dump file I/O 90431 9127 0.1
ges inquiry response 89402 4041 0.05
os thread startup 83809 222586 2.66
cr request retry 80062 71896 0.9
PX Deq: Table Q Sample 79665 133402 1.67
gc cr block congested 79026 14792 0.19
gc cr failure 77521 25019 0.32
enq: WF - contention 73983 825388 11.16
enq: TQ - TM contention 72871 3319 0.05
lock escalate retry 65714 1574 0.02
buffer exterminate 59775 64919 1.09
fbar timer 47136 1413183353 29980.98
log file switch completion 46911 452097 9.64
recovery area: computing obsolete files 45699 8547 0.19
enq: US - contention 40401 8805 0.22
enq: TM - contention 39149 5435032 138.83
library cache load lock 36311 382575 10.54
kjbdrmcvtq lmon drm quiesce: ping completion 31668 47443 1.5
enq: TD - KTF dump entries 31468 1424 0.05
enq: RO - fast object reuse 28422 31772 1.12
parallel recovery slave wait for change 27558 3163 0.11
name-service call wait 23694 181533 7.66
control file single write 22375 1624 0.07
kksfbc child completion 21239 106926 5.03
PX Deq: Table Q qref 19325 245 0.01
enq: TX - contention 18805 113253 6.02
latch: messages 17203 181 0.01
enq: RS - prevent file delete 16913 1013 0.06
enq: RS - prevent aging list update 15682 642 0.04
PX Deq: Table Q Get Keys 14322 42935 3
gc current grant congested 14292 2192 0.15
cursor: mutex S 13285 8 0
log file single write 13164 5371 0.41
latch: undo global data 12649 178 0.01
kksfbc research 11894 12680 1.07
parallel recovery slave idle wait 11193 5872 0.52
wait list latch free 11026 11794 1.07
enq: CT - state 11001 417 0.04
latch: checkpoint queue latch 10526 132 0.01
enq: PE - contention 10506 1139 0.11
ARCH wait on SENDREQ 9957 216480 21.74
gc cr grant congested 9465 1584 0.17
wait for scn ack 9377 3155 0.34
enq: TA - contention 8856 324 0.04
log buffer space 8777 89323 10.18
enq: TK - Auto Task Serialization 8542 343 0.04
enq: DR - contention 7842 323 0.04
process diagnostic dump 7707 2072 0.27
JOX Jit Process Sleep 7612 11286431 1482.72
enq: TC - contention 7357 340817 46.33
ges global resource directory to be frozen 7140 12299 1.72
enq: CO - master slave det 6850 312 0.05
enq: JS - job run lock - synchronize 6704 397 0.06
gcs drm freeze in enter server mode 6542 40742 6.23
enq: TS - contention 5959 89332 14.99
ARCH wait for archivelog lock 5600 36 0.01
PX Nsq: PQ load info query 5377 104798 19.49
db file single write 5373 3452 0.64
gc remaster 5315 50625 9.52
latch: parallel query alloc buffer 4939 1906 0.39
enq: TO - contention 4799 143 0.03
enq: AF - task serialization 4395 161 0.04
enq: PI - contention 4251 163 0.04
ges2 LMON to wake up LMD - mrcvr 4210 28 0.01
enq: DL - contention 3889 239 0.06
kjctssqmg: quick message send wait 3408 22 0.01
LNS wait on DETACH 3275 741 0.23
ksfd: async disk IO 3274 1 0
LNS wait on ATTACH 3273 51940 15.87
ARCH wait on DETACH 3231 714 0.22
ARCH wait on ATTACH 3226 43238 13.4
enq: BR - file shrink 2787 116 0.04
write complete waits 2631 1070 0.41
enq: MD - contention 2596 67 0.03
enq: WL - contention 2198 266518 121.25
single-task message 2098 25896 12.34
enq: OD - Serializing DDLs 2054 66 0.03
resmgr:internal state change 2001 14735 7.36
ARCH wait on c/f tx acquire 2 1751 175230 100.07
enq: WR - contention 1636 69 0.04
latch: cache buffer handles 1610 29 0.02
statement suspended, wait error to be cleared 1497 22626 15.11
Streams AQ: qmn coordinator waiting for slave to start 1214 678966 559.28
enq: PD - contention 1182 33 0.03
JS kgl get object wait 1096 4922 4.49
undo segment extension 1070 10065 9.41
PL/SQL lock timer 949 8739819 9209.5
enq: AE - lock 937 28 0.03
LGWR-LNS wait on channel 832 913 1.1
ges DFS hang analysis phase 2 acks 816 495 0.61
latch: redo writing 729 9 0.01
gc quiesce 665 564 0.85
enq: JS - queue lock 482 2111 4.38
PX Deq: Test for credit 442 13 0.03
enq: SS - contention 386 274 0.71
recovery area: computing dropped files 328 1400 4.27
recovery area: computing backed up files 328 496 1.51
ksdxexeotherwait 279 10592 37.97
log switch/archive 250 137570 550.28
gc domain validation 223 39964 179.21
auto-sqltune: wait graph update 195 96514 494.95
wait for a undo record 170 1214 7.14
parallel recovery coord send blocked 168 4 0.02
enq: JS - wdw op 168 3741 22.27
enq: KT - contention 165 5 0.03
switch logfile command 156 6290 40.32
gcs resource directory to be unfrozen 149 12839 86.17
Data Guard Broker Wait 139 10906 78.46
enq: SK - contention 129 4 0.03
enq: JS - job recov lock 128 4 0.03
gc cr block lost 125 6772 54.17
virtual circuit wait 122 3 0.03
ges LMON to get to FTDONE 100 187 1.87
enq: CU - contention 80 242 3.02
enq: JQ - contention 78 7 0.09
cursor: pin X 73 83 1.14
parallel recovery coord wait for reply 70 510 7.29
PX Deq: Txn Recovery Start 67 3436 51.29
SQL*Net break/reset to dblink 60 0 0
gc current block lost 57 2869 50.33
ges LMD suspend for testing event 51 709 13.89
inactive session 46 4550 98.91
recovery read 45 5 0.11
JS kill job wait 41 3548 86.53
enq: AS - service activation 40 1 0.03
enq: TL - contention 35 2 0.05
enq: UL - contention 34 524 15.42
gcs enter server mode 33 1559 47.24
wait for stopper event to be increased 30 218 7.27
enq: TQ - DDL contention 24 300 12.52
enq: MR - contention 21 1 0.03
ges reconfiguration to start 20 54 2.72
ges enter server mode 20 502 25.08
buffer latch 18 1337 74.26
enq: SR - contention 18 1 0.05
Streams: RAC waiting for inter instance ack 18 3748 208.21
enq: PR - contention 17 46 2.72
kupp process wait 16 166 10.39
checkpoint completed 15 3678 245.19
PX Deque wait 14 68 4.87
enq: BF - allocation contention 14 1 0.08
enq: XL - fault extent map 14 51 3.66
enq: FU - contention 14 17 1.18
enq: TH - metric threshold evaluation 13 114 8.78
enq: MW - contention 12 0 0.04
enq: DD - contention 10 0 0.04
process terminate 8 41 5.08
ges cgs registration 8 151 18.9
buffer resize 7 0 0
ktm: instance recovery 7 698 99.66
LNS wait on LGWR 6 0 0
ASM background starting 6 381 63.43
gc cr block 3-way 5 0 0.08
enq: PV - syncstart 5 9 1.74
Global transaction acquire instance locks 4 4 1.09
enq: RS - read alert level 4 0 0.04
LGWR wait on LNS 3 0 0
gc recovery 3 540 179.85
Streams AQ: enqueue blocked on low memory 3 544 181.2
DBWR range invalidation sync 3 17 5.83
enq: DM - contention 3 0 0.03
enq: RF - FSFO Observer Heartbeat 3 0 0.03
enq: JS - q mem clnup lck 3 0 0
DG Broker configuration file I/O 2 0 0
enq: RC - Result Cache: Contention 2 493 246.6
enq: KM - contention 2 0 0.03
enq: RT - contention 2 0 0.04
instance state change 2 0 0.12
kkdlgon 2 10 5.11
enq: TQ - INI contention 2 292 146.07
enq: JS - contention 2 0 0
ARCH wait for netserver start 1 400 400.02
log file switch (checkpoint incomplete) 1 3 3.44
JS coord start wait 1 50 50.09
ges lmd and pmon to attach 1 1 1.26
wait for tmc2 to complete 1 3 3.03
control file heartbeat 1 400 400.02
enq: SW - contention 1 0 0.04
enq: PW - perwarm status in dbw0 1 0 0.09
enq: FS - contention 1 0 0.04
enq: XR - quiesce database 1 0 0.04
enq: RS - write alert level 1 0 0.02
enq: CN - race with init 1 0 0.03
enq: FE - contention 1 4 3.77
Wait for shrink lock2 1 10 10.03
enq: IA - contention 1 0 0.02
enq: RF - atomicity 1 0 0.05
enq: RF - synchronization: aifo master 1 0 0.02
enq: RF - RF - Database Automatic Disable 1 0 0.06
enq: WP - contention 1 0 0.02
enq: TB - SQL Tuning Base Cache Load 1 0 0.05
enq: JS - evt notify 1 0 0.02Edited by: steffi on Mar 20, 2011 12:21 AM
Edited by: steffi on Mar 20, 2011 8:18 AM
Edited by: steffi on Mar 20, 2011 8:19 AMText can be formatted by tagging the beginning and end of the block of text with the code ta
\Formatted text goes here.
\Example:
This is formatted.When cutting and pasting text such as execution plans, excerpts from AWR reports, etc, it will maintain spacing and formatting, and make for much easier reading.
As to the content, well, dumping the contents of v$system_event is pretty close to useless.
As to the first three events you listed, 'enq: TX - index contention', 'enq: TX - row lock contention', 'enq: SQ - contention', well, all of those are easily tunable.
First, for 'enq: SQ - contention', check your sequences. Do you have any NOCACHE sequences? Or sequences with small caches?
As for 'enq: TX - row lock contention', well that's fairly self-explanatory. You have multiple sessions trying to lock the same row in the same table at the same time.
Last, 'enq: TX - index contention', that's non-row level contention on an index. For example, if you have a unique index, insert a row w/ column value 1, but don't commit, then try to insert that same value from another session.
But, before you do any of that, I think you need to clearly understand where the bottlenecks are. Try taking some AWR snapshots, about 5 minutes apart, when you're having performance problems. Look at the AWR report for that 5 minute period. In particular, look at your Top 5 timed events.
Hope that helps,
-Mark -
Implementing SQL Plan Management on Oracle Database 11.2.0.2
Environment:
Oracle Linux 5 update 10 (UEK)
Oracle GI 11.2.0.2.0 (Oracle ASM 11.2.0.2)
Oracle Database 11.2.0.2.0 Enterprise Edition with RAC option (3 nodes)
No PSU applied/CPU July 2013 applied to RDBMS
Database servicing Siebel CRM 8.1.1.1 Application that uses bind peeking.
Siebel CRM soon to be version 8.1.1.11
There are a few bugs for SQL Plan Management (SPM) on 11.2.0.2 (see below). The slowness and non-peeked binds issues seem very problematic. I've seen a few bloggers recommend to not use SPM unless your at Oracle Database 12c. Several of the bugs seemed to be fixed in 11.2.0.3 but we don't have any plans to move to 11.2.0.3 since we'll most likely be at 12.1.0.x in 7 months. Any recommendation from the community on whether I should capture and use SQL Plan Management with 11.2.0.2? Why, why not? If there's other relevant information needed, let me know.
Bug 9910484 - SQL Plan Management Capture uses excessive space in SYSAUX (Doc ID 9910484.8)
Affects:
Product (Component) Oracle Server (Rdbms)
Range of versions believed to be affected Versions >= 11.1 but BELOW 12.1
Versions confirmed as being affected
•11.2.0.2
•11.2.0.1
•11.1.0.7
Platforms affected Generic (all / most platforms affected)
Fixed:
•12.1.0.1 (Base Release)
•11.2.0.3 (Server Patch Set)
•11.2.0.2 Patch 4 on Windows Platforms
•11.1.0.7 Patch 41 on Windows Platforms
Bug 11719151 - SQL Plan Management capture causes slowness (Doc ID 11719151.8)
Affects:
Product (Component) Oracle Server (Rdbms)
Range of versions believed to be affected Versions >= 11.2.0.2 but BELOW 12.1
Versions confirmed as being affected
•11.2.0.2
Platforms affected Generic (all / most platforms affected)
Fixed:
•12.1.0.1 (Base Release)
•11.2.0.3 (Server Patch Set)
•11.2.0.2 Patch 22 on Windows Platforms
Bug 9942454 - DBMS_SPM.LOAD_PLANS_FROM_SQLSET gets XML parsing error (Doc ID 9942454.8)
Affects:
Product (Component) Oracle Server (Rdbms)
Range of versions believed to be affected Versions BELOW 12.1
Versions confirmed as being affected
•11.2.0.2
•11.2.0.1
•11.1.0.7
Platforms affected Generic (all / most platforms affected)
Fixed:
•12.1.0.1 (Base Release)
•11.2.0.3 (Server Patch Set)
Bug 12732879 - Execution Plan of Query with non-peeked binds is not reproducible (Doc ID 12732879.8)
Affects:
Product (Component) Oracle Server (Rdbms)
Range of versions believed to be affected Versions >= 9.2 but BELOW 12.1
Versions confirmed as being affected
•11.2.0.3
•11.2.0.2
•11.1.0.7
Platforms affected Generic (all / most platforms affected)
Fixed:
•12.1.0.1 (Base Release)
•11.2.0.4 (Future Patch Set)
Bug 11687175 - High DFS lock handle waits in the database with SPM if FIXED_DATE is set (Doc ID 11687175.8)
Affects:
Product (Component) Oracle Server (Rdbms)
Range of versions believed to be affected Versions >= 11 but BELOW 12.1
Versions confirmed as being affected
11.2.0.2
11.2.0.1
11.1.0.7
11.1.0.6
Platforms affected Generic (all / most platforms affected)
Fixed:
12.1.0.1 (Base Release)
11.2.0.3 (Server Patch Set)
Bug 13384234 ORA-29981 with select query with database Change notification
Affects:
Product (Component) Oracle Server (Rdbms)
Range of versions believed to be affected Versions BELOW 12.1
Versions confirmed as being affected
11.2.0.2
Platforms affected Generic (all / most platforms affected)
Fixed:
11.2.0.4 (Future Patch Set)
Thanks so much.See these MOS Docs
11.2.0.2 Patch Set - Availability and Known Issues (Doc ID 1179474.1)
Important Changes to Oracle Database Patch Sets Starting With 11.2.0.2 (Doc ID 1189783.1)
HTH
Srini -
How to set the correct shared pool size and db_buffer_cache using awr
Hi All,
I want to how to set the correct size for shared_pool_size and db_cache_size using shared pool advisory and buffer pool advisory of awr report. I have paste the shared and buffer pool advisory of awr report.
Shared Pool Advisory
* SP: Shared Pool Est LC: Estimated Library Cache Factr: Factor
* Note there is often a 1:Many correlation between a single logical object in the Library Cache, and the physical number of memory objects associated with it. Therefore comparing the number of Lib Cache objects (e.g. in v$librarycache), with the number of Lib Cache Memory Objects is invalid.
Shared Pool Size(M) SP Size Factr Est LC Size (M) Est LC Mem Obj Est LC Time Saved (s) Est LC Time Saved Factr Est LC Load Time (s) Est LC Load Time Factr Est LC Mem Obj Hits (K)
4,096 1.00 471 25,153 184,206 1.00 149 1.00 9,069
4,736 1.16 511 27,328 184,206 1.00 149 1.00 9,766
5,248 1.28 511 27,346 184,206 1.00 149 1.00 9,766
5,760 1.41 511 27,346 184,206 1.00 149 1.00 9,766
6,272 1.53 511 27,346 184,206 1.00 149 1.00 9,766
6,784 1.66 511 27,346 184,206 1.00 149 1.00 9,766
7,296 1.78 511 27,346 184,206 1.00 149 1.00 9,766
7,808 1.91 511 27,346 184,206 1.00 149 1.00 9,766
8,320 2.03 511 27,346 184,206 1.00 149 1.00 9,766
Buffer Pool Advisory
* Only rows with estimated physical reads >0 are displayed
* ordered by Block Size, Buffers For Estimate
P Size for Est (M) Size Factor Buffers (thousands) Est Phys Read Factor Estimated Phys Reads (thousands) Est Phys Read Time Est %DBtime for Rds
D 4,096 0.10 485 1.02 1,002 1 0.00
D 8,192 0.20 970 1.00 987 1 0.00
D 12,288 0.30 1,454 1.00 987 1 0.00
D 16,384 0.40 1,939 1.00 987 1 0.00
D 20,480 0.50 2,424 1.00 987 1 0.00
D 24,576 0.60 2,909 1.00 987 1 0.00
D 28,672 0.70 3,394 1.00 987 1 0.00
D 32,768 0.80 3,878 1.00 987 1 0.00
D 36,864 0.90 4,363 1.00 987 1 0.00
D 40,960 1.00 4,848 1.00 987 1 0.00
D 45,056 1.10 5,333 1.00 987 1 0.00
D 49,152 1.20 5,818 1.00 987 1 0.00
D 53,248 1.30 6,302 1.00 987 1 0.00
D 57,344 1.40 6,787 1.00 987 1 0.00
D 61,440 1.50 7,272 1.00 987 1 0.00
D 65,536 1.60 7,757 1.00 987 1 0.00
D 69,632 1.70 8,242 1.00 987 1 0.00
D 73,728 1.80 8,726 1.00 987 1 0.00
D 77,824 1.90 9,211 1.00 987 1 0.00
D 81,920 2.00 9,696 1.00 987 1 0.00
My shared pool size is 4gb and db_cache_size is 40Gb.
Please help me in configuring the correct size for this.
Thanks and Regards,Hi ,
Actually batch load is taking too much time.
Please find below the 1 hr awr report
Snap Id Snap Time Sessions Cursors/Session
Begin Snap: 6557 27-Nov-11 16:00:06 126 1.3
End Snap: 6558 27-Nov-11 17:00:17 130 1.6
Elapsed: 60.17 (mins)
DB Time: 34.00 (mins)
Report Summary
Cache Sizes
Begin End
Buffer Cache: 40,960M 40,960M Std Block Size: 8K
Shared Pool Size: 4,096M 4,096M Log Buffer: 25,908K
Load Profile
Per Second Per Transaction Per Exec Per Call
DB Time(s): 0.6 1.4 0.00 0.07
DB CPU(s): 0.5 1.2 0.00 0.06
Redo size: 281,296.9 698,483.4
Logical reads: 20,545.6 51,016.4
Block changes: 1,879.5 4,667.0
Physical reads: 123.7 307.2
Physical writes: 66.4 164.8
User calls: 8.2 20.4
Parses: 309.4 768.4
Hard parses: 8.5 21.2
W/A MB processed: 1.7 4.3
Logons: 0.7 1.6
Executes: 1,235.9 3,068.7
Rollbacks: 0.0 0.0
Transactions: 0.4
Instance Efficiency Percentages (Target 100%)
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 99.66 In-memory Sort %: 100.00
Library Hit %: 99.19 Soft Parse %: 97.25
Execute to Parse %: 74.96 Latch Hit %: 99.97
Parse CPU to Parse Elapsd %: 92.41 % Non-Parse CPU: 98.65
Shared Pool Statistics
Begin End
Memory Usage %: 80.33 82.01
% SQL with executions>1: 90.90 86.48
% Memory for SQL w/exec>1: 90.10 86.89
Top 5 Timed Foreground Events
Event Waits Time(s) Avg wait (ms) % DB time Wait Class
DB CPU 1,789 87.72
db file sequential read 27,531 50 2 2.45 User I/O
db file scattered read 26,322 30 1 1.47 User I/O
row cache lock 1,798 20 11 0.96 Concurrency
OJVM: Generic 36 15 421 0.74 Other
Host CPU (CPUs: 24 Cores: 12 Sockets: )
Load Average Begin Load Average End %User %System %WIO %Idle
0.58 1.50 2.8 0.7 0.1 96.6
Instance CPU
%Total CPU %Busy CPU %DB time waiting for CPU (Resource Manager)
2.2 63.6 0.0
Memory Statistics
Begin End
Host Mem (MB): 131,072.0 131,072.0
SGA use (MB): 50,971.4 50,971.4
PGA use (MB): 545.5 1,066.3
% Host Mem used for SGA+PGA: 39.30 39.70
RAC Statistics
Begin End
Number of Instances: 2 2
Global Cache Load Profile
Per Second Per Transaction
Global Cache blocks received: 3.09 7.68
Global Cache blocks served: 1.86 4.62
GCS/GES messages received: 78.64 195.27
GCS/GES messages sent: 53.82 133.65
DBWR Fusion writes: 0.52 1.30
Estd Interconnect traffic (KB) 65.50
Global Cache Efficiency Percentages (Target local+remote 100%)
Buffer access - local cache %: 99.65
Buffer access - remote cache %: 0.02
Buffer access - disk %: 0.34
Global Cache and Enqueue Services - Workload Characteristics
Avg global enqueue get time (ms): 0.0
Avg global cache cr block receive time (ms): 1.7
Avg global cache current block receive time (ms): 1.0
Avg global cache cr block build time (ms): 0.0
Avg global cache cr block send time (ms): 0.0
Global cache log flushes for cr blocks served %: 1.4
Avg global cache cr block flush time (ms): 0.9
Avg global cache current block pin time (ms): 0.0
Avg global cache current block send time (ms): 0.0
Global cache log flushes for current blocks served %: 0.1
Avg global cache current block flush time (ms): 0.0
Global Cache and Enqueue Services - Messaging Statistics
Avg message sent queue time (ms): 0.0
Avg message sent queue time on ksxp (ms): 0.4
Avg message received queue time (ms): 0.5
Avg GCS message process time (ms): 0.0
Avg GES message process time (ms): 0.0
% of direct sent messages: 79.13
% of indirect sent messages: 17.10
% of flow controlled messages: 3.77
Cluster Interconnect
Begin End
Interface IP Address Pub Source IP Pub Src
en9 10.51.10.61 N Oracle Cluster Repository
Main Report
* Report Summary
* Wait Events Statistics
* SQL Statistics
* Instance Activity Statistics
* IO Stats
* Buffer Pool Statistics
* Advisory Statistics
* Wait Statistics
* Undo Statistics
* Latch Statistics
* Segment Statistics
* Dictionary Cache Statistics
* Library Cache Statistics
* Memory Statistics
* Streams Statistics
* Resource Limit Statistics
* Shared Server Statistics
* init.ora Parameters
More RAC Statistics
* RAC Report Summary
* Global Messaging Statistics
* Global CR Served Stats
* Global CURRENT Served Stats
* Global Cache Transfer Stats
* Interconnect Stats
* Dynamic Remastering Statistics
Back to Top
Statistic Name Time (s) % of DB Time
sql execute elapsed time 1,925.20 94.38
DB CPU 1,789.38 87.72
connection management call elapsed time 99.65 4.89
PL/SQL execution elapsed time 89.81 4.40
parse time elapsed 46.32 2.27
hard parse elapsed time 25.01 1.23
Java execution elapsed time 21.24 1.04
PL/SQL compilation elapsed time 11.92 0.58
failed parse elapsed time 9.37 0.46
hard parse (sharing criteria) elapsed time 8.71 0.43
sequence load elapsed time 0.06 0.00
repeated bind elapsed time 0.02 0.00
hard parse (bind mismatch) elapsed time 0.01 0.00
DB time 2,039.77
background elapsed time 122.00
background cpu time 113.42
Statistic Value End Value
NUM_LCPUS 0
NUM_VCPUS 0
AVG_BUSY_TIME 12,339
AVG_IDLE_TIME 348,838
AVG_IOWAIT_TIME 221
AVG_SYS_TIME 2,274
AVG_USER_TIME 9,944
BUSY_TIME 299,090
IDLE_TIME 8,375,051
IOWAIT_TIME 6,820
SYS_TIME 57,512
USER_TIME 241,578
LOAD 1 2
OS_CPU_WAIT_TIME 312,200
PHYSICAL_MEMORY_BYTES 137,438,953,472
NUM_CPUS 24
NUM_CPU_CORES 12
GLOBAL_RECEIVE_SIZE_MAX 1,310,720
GLOBAL_SEND_SIZE_MAX 1,310,720
TCP_RECEIVE_SIZE_DEFAULT 16,384
TCP_RECEIVE_SIZE_MAX 9,223,372,036,854,775,807
TCP_RECEIVE_SIZE_MIN 4,096
TCP_SEND_SIZE_DEFAULT 16,384
TCP_SEND_SIZE_MAX 9,223,372,036,854,775,807
TCP_SEND_SIZE_MIN 4,096
Back to Wait Events Statistics
Back to Top
Operating System Statistics - Detail
Snap Time Load %busy %user %sys %idle %iowait
27-Nov 16:00:06 0.58
27-Nov 17:00:17 1.50 3.45 2.79 0.66 96.55 0.08
Back to Wait Events Statistics
Back to Top
Foreground Wait Class
* s - second, ms - millisecond - 1000th of a second
* ordered by wait time desc, waits desc
* %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
* Captured Time accounts for 95.7% of Total DB time 2,039.77 (s)
* Total FG Wait Time: 163.14 (s) DB CPU time: 1,789.38 (s)
Wait Class Waits %Time -outs Total Wait Time (s) Avg wait (ms) %DB time
DB CPU 1,789 87.72
User I/O 61,229 0 92 1 4.49
Other 102,743 40 31 0 1.50
Concurrency 3,169 10 24 7 1.16
Cluster 58,920 0 11 0 0.52
System I/O 45,407 0 6 0 0.29
Configuration 107 7 1 5 0.03
Commit 383 0 0 1 0.01
Network 15,275 0 0 0 0.00
Application 52 8 0 0 0.00
Back to Wait Events Statistics
Back to Top
Foreground Wait Events
* s - second, ms - millisecond - 1000th of a second
* Only events with Total Wait Time (s) >= .001 are shown
* ordered by wait time desc, waits desc (idle events last)
* %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn % DB time
db file sequential read 27,531 0 50 2 18.93 2.45
db file scattered read 26,322 0 30 1 18.10 1.47
row cache lock 1,798 0 20 11 1.24 0.96
OJVM: Generic 36 42 15 421 0.02 0.74
db file parallel read 394 0 7 19 0.27 0.36
control file sequential read 22,248 0 6 0 15.30 0.28
reliable message 4,439 0 4 1 3.05 0.18
gc current grant busy 7,597 0 3 0 5.22 0.16
PX Deq: Slave Session Stats 2,661 0 3 1 1.83 0.16
DFS lock handle 3,208 0 3 1 2.21 0.16
direct path write temp 4,842 0 3 1 3.33 0.15
library cache load lock 39 0 3 72 0.03 0.14
gc cr multi block request 37,008 0 3 0 25.45 0.14
IPC send completion sync 5,451 0 2 0 3.75 0.10
gc cr block 2-way 4,669 0 2 0 3.21 0.09
enq: PS - contention 3,183 33 1 0 2.19 0.06
gc cr grant 2-way 5,151 0 1 0 3.54 0.06
direct path read temp 1,722 0 1 1 1.18 0.05
gc current block 2-way 1,807 0 1 0 1.24 0.03
os thread startup 6 0 1 108 0.00 0.03
name-service call wait 12 0 1 47 0.01 0.03
PX Deq: Signal ACK RSG 2,046 50 0 0 1.41 0.02
log file switch completion 3 0 0 149 0.00 0.02
rdbms ipc reply 3,610 0 0 0 2.48 0.02
gc current grant 2-way 1,432 0 0 0 0.98 0.02
library cache pin 903 32 0 0 0.62 0.02
PX Deq: reap credit 35,815 100 0 0 24.63 0.01
log file sync 383 0 0 1 0.26 0.01
Disk file operations I/O 405 0 0 0 0.28 0.01
library cache lock 418 3 0 0 0.29 0.01
kfk: async disk IO 23,159 0 0 0 15.93 0.01
gc current block busy 4 0 0 35 0.00 0.01
gc current multi block request 1,206 0 0 0 0.83 0.01
ges message buffer allocation 38,526 0 0 0 26.50 0.00
enq: FB - contention 131 0 0 0 0.09 0.00
undo segment extension 8 100 0 6 0.01 0.00
CSS initialization 8 0 0 6 0.01 0.00
SQL*Net message to client 14,600 0 0 0 10.04 0.00
enq: HW - contention 96 0 0 0 0.07 0.00
CSS operation: action 8 0 0 4 0.01 0.00
gc cr block busy 33 0 0 1 0.02 0.00
latch free 30 0 0 1 0.02 0.00
enq: TM - contention 49 6 0 0 0.03 0.00
enq: JQ - contention 19 100 0 1 0.01 0.00
SQL*Net more data to client 666 0 0 0 0.46 0.00
asynch descriptor resize 3,179 100 0 0 2.19 0.00
latch: shared pool 3 0 0 3 0.00 0.00
CSS operation: query 24 0 0 0 0.02 0.00
PX Deq: Signal ACK EXT 72 0 0 0 0.05 0.00
KJC: Wait for msg sends to complete 269 0 0 0 0.19 0.00
latch: object queue header operation 4 0 0 1 0.00 0.00
gc cr block congested 5 0 0 0 0.00 0.00
utl_file I/O 11 0 0 0 0.01 0.00
enq: TO - contention 3 33 0 0 0.00 0.00
SQL*Net message from client 14,600 0 219,478 15033 10.04
jobq slave wait 7,726 100 3,856 499 5.31
PX Deq: Execution Msg 10,556 19 50 5 7.26
PX Deq: Execute Reply 2,946 31 27 9 2.03
PX Deq: Parse Reply 3,157 35 3 1 2.17
PX Deq: Join ACK 2,976 28 2 1 2.05
PX Deq Credit: send blkd 7 14 0 4 0.00
Back to Wait Events Statistics
Back to Top
Background Wait Events
* ordered by wait time desc, waits desc (idle events last)
* Only events with Total Wait Time (s) >= .001 are shown
* %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn % bg time
os thread startup 140 0 13 90 0.10 10.35
db file parallel write 8,233 0 6 1 5.66 5.08
log file parallel write 3,906 0 6 1 2.69 4.62
log file sequential read 350 0 5 16 0.24 4.49
control file sequential read 13,737 0 5 0 9.45 3.72
DFS lock handle 2,990 27 2 1 2.06 1.43
db file sequential read 921 0 2 2 0.63 1.39
SQL*Net break/reset to client 18 0 1 81 0.01 1.19
control file parallel write 2,455 0 1 1 1.69 1.12
ges lms sync during dynamic remastering and reconfig 24 100 1 50 0.02 0.98
library cache load lock 35 0 1 24 0.02 0.68
ASM file metadata operation 3,483 0 1 0 2.40 0.65
enq: CO - master slave det 1,203 100 1 0 0.83 0.46
kjbdrmcvtq lmon drm quiesce: ping completion 9 0 1 62 0.01 0.46
enq: WF - contention 11 0 0 35 0.01 0.31
CGS wait for IPC msg 32,702 100 0 0 22.49 0.19
gc object scan 28,788 100 0 0 19.80 0.15
row cache lock 535 0 0 0 0.37 0.14
library cache pin 370 55 0 0 0.25 0.12
ksxr poll remote instances 19,119 100 0 0 13.15 0.11
name-service call wait 6 0 0 19 0.00 0.10
gc current block 2-way 304 0 0 0 0.21 0.09
gc cr block 2-way 267 0 0 0 0.18 0.08
gc cr grant 2-way 355 0 0 0 0.24 0.08
ges LMON to get to FTDONE 3 100 0 24 0.00 0.06
enq: CF - contention 145 76 0 0 0.10 0.05
PX Deq: reap credit 8,842 100 0 0 6.08 0.05
reliable message 126 0 0 0 0.09 0.05
db file scattered read 19 0 0 3 0.01 0.05
library cache lock 162 1 0 0 0.11 0.04
latch: shared pool 2 0 0 27 0.00 0.04
Disk file operations I/O 504 0 0 0 0.35 0.04
gc current grant busy 148 0 0 0 0.10 0.04
gcs log flush sync 84 0 0 1 0.06 0.04
ges message buffer allocation 24,934 0 0 0 17.15 0.02
enq: CR - block range reuse ckpt 83 0 0 0 0.06 0.02
latch free 22 0 0 1 0.02 0.02
CSS operation: action 13 0 0 2 0.01 0.02
CSS initialization 4 0 0 6 0.00 0.02
direct path read 1 0 0 21 0.00 0.02
rdbms ipc reply 153 0 0 0 0.11 0.01
db file parallel read 2 0 0 8 0.00 0.01
direct path write 5 0 0 3 0.00 0.01
gc current multi block request 49 0 0 0 0.03 0.01
gc current block busy 5 0 0 2 0.00 0.01
enq: PS - contention 24 50 0 0 0.02 0.01
gc cr multi block request 54 0 0 0 0.04 0.01
ges generic event 1 100 0 10 0.00 0.01
gc current grant 2-way 35 0 0 0 0.02 0.01
kfk: async disk IO 183 0 0 0 0.13 0.01
Log archive I/O 3 0 0 2 0.00 0.01
gc buffer busy acquire 2 0 0 3 0.00 0.00
LGWR wait for redo copy 123 0 0 0 0.08 0.00
IPC send completion sync 18 0 0 0 0.01 0.00
enq: TA - contention 11 0 0 0 0.01 0.00
read by other session 2 0 0 2 0.00 0.00
enq: TM - contention 9 89 0 0 0.01 0.00
latch: ges resource hash list 135 0 0 0 0.09 0.00
PX Deq: Slave Session Stats 12 0 0 0 0.01 0.00
KJC: Wait for msg sends to complete 89 0 0 0 0.06 0.00
enq: TD - KTF dump entries 8 0 0 0 0.01 0.00
enq: US - contention 7 0 0 0 0.00 0.00
CSS operation: query 12 0 0 0 0.01 0.00
enq: TK - Auto Task Serialization 6 100 0 0 0.00 0.00
PX Deq: Signal ACK RSG 24 50 0 0 0.02 0.00
log file single write 6 0 0 0 0.00 0.00
enq: WL - contention 2 100 0 1 0.00 0.00
ADR block file read 13 0 0 0 0.01 0.00
ADR block file write 5 0 0 0 0.00 0.00
latch: object queue header operation 1 0 0 1 0.00 0.00
gc cr block busy 1 0 0 1 0.00 0.00
rdbms ipc message 103,276 67 126,259 1223 71.03
PX Idle Wait 6,467 67 12,719 1967 4.45
wait for unread message on broadcast channel 7,240 100 7,221 997 4.98
gcs remote message 218,809 84 7,213 33 150.49
DIAG idle wait 203,228 95 7,185 35 139.77
shared server idle wait 121 100 3,630 30000 0.08
ASM background timer 3,343 0 3,611 1080 2.30
Space Manager: slave idle wait 723 100 3,610 4993 0.50
heartbeat monitor sleep 722 100 3,610 5000 0.50
ges remote message 73,089 52 3,609 49 50.27
dispatcher timer 66 88 3,608 54660 0.05
pmon timer 1,474 82 3,607 2447 1.01
PING 1,487 19 3,607 2426 1.02
Streams AQ: qmn slave idle wait 125 0 3,594 28754 0.09
Streams AQ: qmn coordinator idle wait 250 50 3,594 14377 0.17
smon timer 18 50 3,505 194740 0.01
JOX Jit Process Sleep 73 100 976 13370 0.05
class slave wait 56 0 605 10806 0.04
KSV master wait 2,215 98 1 0 1.52
SQL*Net message from client 109 0 0 2 0.07
PX Deq: Parse Reply 27 44 0 1 0.02
PX Deq: Join ACK 30 40 0 1 0.02
PX Deq: Execute Reply 20 30 0 0 0.01
Streams AQ: RAC qmn coordinator idle wait 259 100 0 0 0.18
Back to Wait Events Statistics
Back to Top
Wait Event Histogram
* Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
* % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
* % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
* Ordered by Event (idle events last)
% of Waits
Event Total Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
ADR block file read 13 100.0
ADR block file write 5 100.0
ADR file lock 6 100.0
ARCH wait for archivelog lock 3 100.0
ASM file metadata operation 3483 99.6 .1 .1 .2
CGS wait for IPC msg 32.7K 100.0
CSS initialization 12 50.0 50.0
CSS operation: action 21 28.6 9.5 61.9
CSS operation: query 36 86.1 5.6 8.3
DFS lock handle 6198 98.6 1.2 .1 .1
Disk file operations I/O 909 95.7 3.6 .7
IPC send completion sync 5469 99.9 .1 .0 .0
KJC: Wait for msg sends to complete 313 100.0
LGWR wait for redo copy 122 100.0
Log archive I/O 3 66.7 33.3
OJVM: Generic 36 55.6 44.4
PX Deq: Signal ACK EXT 72 98.6 1.4
PX Deq: Signal ACK RSG 2070 99.7 .0 .1 .0 .1
PX Deq: Slave Session Stats 2673 99.7 .2 .1 .0
PX Deq: reap credit 44.7K 100.0
SQL*Net break/reset to client 20 95.0 5.0
SQL*Net message to client 14.7K 100.0
SQL*Net more data from client 32 100.0
SQL*Net more data to client 689 100.0
asynch descriptor resize 3387 100.0
buffer busy waits 2 100.0
control file parallel write 2455 96.6 2.2 .6 .6 .1
control file sequential read 36K 99.4 .3 .1 .1 .1 .1 .0
db file parallel read 397 8.8 .8 5.5 12.6 17.4 46.3 8.6
db file parallel write 8233 85.4 10.3 2.3 1.4 .4 .1
db file scattered read 26.3K 79.2 1.5 8.2 10.5 .6 .1 .0
db file sequential read 28.4K 60.2 3.3 18.0 18.1 .3 .1 .0
db file single write 2 100.0
direct path read 2 50.0 50.0
direct path read temp 1722 95.8 2.8 .1 .5 .8 .1
direct path write 6 83.3 16.7
direct path write temp 4842 96.3 2.7 .5 .2 .0 .0 .2
enq: AF - task serialization 1 100.0
enq: CF - contention 145 99.3 .7
enq: CO - master slave det 1203 98.9 .8 .2
enq: CR - block range reuse ckpt 83 100.0
enq: DR - contention 2 100.0
enq: FB - contention 131 100.0
enq: HW - contention 97 100.0
enq: JQ - contention 19 89.5 10.5
enq: JS - job run lock - synchronize 3 100.0
enq: MD - contention 1 100.0
enq: MW - contention 2 100.0
enq: PS - contention 3207 99.5 .4 .1
enq: TA - contention 11 100.0
enq: TD - KTF dump entries 8 100.0
enq: TK - Auto Task Serialization 6 100.0
enq: TM - contention 58 100.0
enq: TO - contention 3 100.0
enq: TQ - DDL contention 1 100.0
enq: TS - contention 1 100.0
enq: UL - contention 1 100.0
enq: US - contention 7 100.0
enq: WF - contention 11 81.8 18.2
enq: WL - contention 2 50.0 50.0
gc buffer busy acquire 2 50.0 50.0
gc cr block 2-way 4934 99.9 .1 .0 .0
gc cr block busy 35 68.6 31.4
gc cr block congested 6 100.0
gc cr disk read 2 100.0
gc cr grant 2-way 4824 100.0 .0
gc cr grant congested 2 100.0
gc cr multi block request 37.1K 99.8 .2 .0 .0 .0 .0 .0
gc current block 2-way 2134 99.9 .0 .0
gc current block busy 7 14.3 14.3 14.3 28.6 28.6
gc current block congested 2 100.0
gc current grant 2-way 1337 99.9 .1
gc current grant busy 7123 99.2 .2 .2 .0 .0 .3 .1
gc current grant congested 2 100.0
gc current multi block request 1260 99.8 .2
gc object scan 28.8K 100.0
gcs log flush sync 65 95.4 3.1 1.5
ges LMON to get to FTDONE 3 100.0
ges generic event 1 100.0
ges inquiry response 2 100.0
ges lms sync during dynamic remastering and reconfig 24 16.7 29.2 54.2
ges message buffer allocation 63.1K 100.0
kfk: async disk IO 23.3K 100.0 .0 .0
kjbdrmcvtq lmon drm quiesce: ping completion 9 11.1 88.9
ksxr poll remote instances 19.1K 100.0
latch free 52 59.6 40.4
latch: call allocation 2 100.0
latch: gc element 1 100.0
latch: gcs resource hash 1 100.0
latch: ges resource hash list 135 100.0
latch: object queue header operation 5 40.0 40.0 20.0
latch: shared pool 5 40.0 20.0 20.0 20.0
library cache load lock 74 9.5 5.4 8.1 17.6 10.8 13.5 35.1
library cache lock 493 99.2 .4 .4
library cache pin 1186 98.4 .3 1.2 .1
library cache: mutex X 6 100.0
log file parallel write 3897 72.9 1.5 17.1 7.5 .6 .3 .1
log file sequential read 350 4.6 3.1 59.4 30.0 2.9
log file single write 6 100.0
log file switch completion 3 33.3 66.7
log file sync 385 90.4 3.6 4.7 .8 .5
name-service call wait 18 5.6 5.6 5.6 16.7 44.4 22.2
os thread startup 146 100.0
rdbms ipc reply 3763 99.7 .3
read by other session 2 50.0 50.0
reliable message 4565 99.7 .2 .0 .0 .1
row cache lock 2334 99.3 .2 .1 .1 .3
undo segment extension 8 50.0 37.5 12.5
utl_file I/O 11 100.0
ASM background timer 3343 57.0 .3 .1 .1 .1 21.1 21.4
DIAG idle wait 203.2K 3.4 .2 .4 18.0 41.4 14.8 21.8
JOX Jit Process Sleep 73 2.7 97.3
KSV master wait 2213 99.4 .1 .2 .3
PING 1487 81.0 19.0
PX Deq Credit: send blkd 7 57.1 14.3 14.3 14.3
PX Deq: Execute Reply 2966 59.8 .8 9.5 5.6 10.2 2.6 11.4
PX Deq: Execution Msg 10.6K 72.4 12.1 2.6 2.5 .1 5.6 4.6 .0
PX Deq: Join ACK 3006 77.9 22.1 .1
PX Deq: Parse Reply 3184 67.1 31.1 1.6 .2
PX Idle Wait 6466 .2 8.7 4.3 4.8 .3 .1 5.0 76.6
SQL*Net message from client 14.7K 72.4 2.8 .8 .5 .9 .4 2.8 19.3
Space Manager: slave idle wait 722 100.0
Streams AQ: RAC qmn coordinator idle wait 259 100.0
Streams AQ: qmn coordinator idle wait 250 50.0 50.0
Streams AQ: qmn slave idle wait 125 100.0
class slave wait 55 67.3 7.3 1.8 5.5 1.8 7.3 9.1
dispatcher timer 66 6.1 93.9
gcs remote message 218.6K 7.7 1.8 1.2 1.6 1.7 15.7 70.3
ges remote message 72.9K 29.7 5.1 2.7 2.2 1.5 4.0 54.7
heartbeat monitor sleep 722 100.0
jobq slave wait 7725 .1 .0 99.9
pmon timer 1474 18.4 81.6
rdbms ipc message 103.3K 20.7 2.7 1.5 1.3 .9 .7 40.7 31.6
shared server idle wait 121 100.0
smon timer 18 100.0
wait for unread message on broadcast channel 7238 .3 99.7
Back to Wait Events Statistics
Back to Top
Wait Event Histogram Detail (64 msec to 2 sec)
* Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
* Units for % of Total Waits: ms is milliseconds s is 1024 milliseconds (approximately 1 second)
* % of Total Waits: total waits for all wait classes, including Idle
* % of Total Waits: value of .0 indicates value was <.05%; value of null is truly 0
* Ordered by Event (only non-idle events are displayed)
% of Total Waits
Event Waits 64ms to 2s <32ms <64ms <1/8s <1/4s <1/2s <1s <2s >=2s
ASM file metadata operation 6 99.8 .1 .1
DFS lock handle 6 99.9 .1 .0
OJVM: Generic 16 55.6 2.8 41.7
PX Deq: Signal ACK RSG 3 99.9 .0 .1
PX Deq: Slave Session Stats 3 99.9 .0 .0 .0
SQL*Net break/reset to client 1 95.0 5.0
control file sequential read 1 100.0 .0
db file parallel read 34 91.4 8.6
db file scattered read 4 100.0 .0 .0
db file sequential read 6 100.0 .0 .0 .0
direct path write temp 11 99.8 .1 .1 .0
enq: WF - contention 2 81.8 18.2
gc cr block 2-way 1 100.0 .0
gc cr multi block request 1 100.0 .0
gc current block 2-way 1 100.0 .0
gc current block busy 2 71.4 28.6
gc current grant busy 8 99.9 .0 .1
ges lms sync during dynamic remastering and reconfig 13 45.8 20.8 33.3
kjbdrmcvtq lmon drm quiesce: ping completion 8 11.1 11.1 77.8
latch: shared pool 1 80.0 20.0
library cache load lock 26 64.9 14.9 12.2 4.1 4.1
log file parallel write 2 99.9 .0 .0
log file sequential read 10 97.1 2.0 .6 .3
log file switch completion 2 33.3 66.7
name-service call wait 4 77.8 22.2
os thread startup 146 100.0
reliable message 4 99.9 .0 .1
row cache lock 2 99.7 .0 .0 .3
Back to Wait Events Statistics
Back to Top
Wait Event Histogram Detail (4 sec to 2 min)
* Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
* Units for % of Total Waits: s is 1024 milliseconds (approximately 1 second) m is 64*1024 milliseconds (approximately 67 seconds or 1.1 minutes)
* % of Total Waits: total waits for all wait classes, including Idle
* % of Total Waits: value of .0 indicates value was <.05%; value of null is truly 0
* Ordered by Event (only non-idle events are displayed)
% of Total Waits
Event Waits 4s to 2m <2s <4s <8s <16s <32s < 1m < 2m >=2m
row cache lock 6 99.7 .3
Back to Wait Events Statistics
Back to Top
Wait Event Histogram Detail (4 min to 1 hr)
No data exists for this section of the report.
Back to Wait Events Statistics
Back to Top
Service Statistics
* ordered by DB Time
Service Name DB Time (s) DB CPU (s) Physical Reads (K) Logical Reads (K)
ubshost 1,934 1,744 445 73,633
SYS$USERS 105 45 1 404
SYS$BACKGROUND 0 0 1 128
ubshostXDB 0 0 0 0
Back to Wait Events Statistics
Back to Top
Service Wait Class Stats
* Wait Class info for services in the Service Statistics section.
* Total Waits and Time Waited displayed for the following wait classes: User I/O, Concurrency, Administrative, Network
* Time Waited (Wt Time) in seconds
Service Name User I/O Total Wts User I/O Wt Time Concurcy Total Wts Concurcy Wt Time Admin Total Wts Admin Wt Time Network Total Wts Network Wt Time
ubshost 60232 90 2644 4 0 0 13302 0
SYS$USERS 997 2 525 19 0 0 1973 0
SYS$BACKGROUND 1456 2 1258 14 0 0 0 0
I am not able to paste the whole awr report. I have paste some of the sections of awr report.
Please help.
Thanks and Regards,
Maybe you are looking for
-
so when i open an app downloaded from the app store the game opens for about 10 seconds then quits?
-
Problem syncing, deleted itunes, re-downloaded it, and now my music won't show up...what do i do?
Okay, so I connected my ipod to my windows computer after loosing it for a few months and it said something I downloaded was preventing my computer from igknowledging my ipod. I went through and deleted a bunch of stuff and still nothing worked. I do
-
Iphone 6 64gb 2yr contract pricing $899
I recently bought a new iPhone, upgrading after a previous 2yr contract to a new one at AT&T. The phone was bought at an Apple Store and when asked I was told that my contract remain the same, all the discounts are not affected and it would be a same
-
Calibration error message in WCS
I get a calibration error message in WCS when I try to add data points in the RF calibrate models section. Here is the error message: Error in Calibration. MessageMEDIATION-5, WcCalibration!x.x.x.x!xx:xx:xx:xx:xx:xx,row Status,No access The first se
-
How to create Live Soccer/Basketball score templates in Photoshop CS 5.1/64BIT
I have a video about a sport competition. You know, we all see these templates in TV/NET as a info table. I want to do it in Photoshop CS5.1. I heard that we can create the template in Photoshop and put it in Premiere. If yes, how ? Is there any tuto