Enq wait events
hi,
Please could someone point me to some doc explaining enq events like:
enq: AT - contention and how to interpret the name|mode parameter for such events (select event#,name,parameter1 p1, parameter2 p2, parameter3 p3 from v$event_name where name='enq: AT - contention')
Thanks for ur time,
RUben
In addition,
UKJA@ukja102> @lock_type at
UKJA@ukja102> set serveroutput on
UKJA@ukja102>
UKJA@ukja102> exec print_table('select * from v$lock_type where type = upper(''&1'')');
TYPE : AT
NAME : Alter Tablespace
ID1_TAG : 0
ID2_TAG : 0
IS_USER : NO
DESCRIPTION : Serializes 'alter tablespace' operations
-----------------Dion Cho
Similar Messages
-
Enq: TX - row lock contention wait event
Hi,
I would like to find which DML query has not given COMMIT or ROLLBACK after the execution. Because one of the development database have more table locks and developer reported that their session was hanging. I referred AWR report also and more timed waits occurred in the enq: TX - row lock contention. I need to trace which DML query has not commit or rollback.
Please help me to solve the issue.
Database version: 11.2.0.1.0
Foreground Wait Events
Event
Waits
%Time -outs
Total Wait Time (s)
Avg wait (ms)
Waits /txn
% DB time
enq: TX - row lock contention
320
0
72,047
225147
0.20
99.53
log file sync
547
0
14
26
0.35
0.02
library cache lock
13
0
11
843
0.01
0.02
SQL*Net break/reset to client
1,080
0
2
1
0.69
0.00
SQL*Net message to client
659,006
0
1
0
421.63
0.00
direct path sync
3
0
1
299
0.00
0.00
SQL*Net more data from client
5,541
0
1
0
3.55
0.00
db file scattered read
554
0
0
1
0.35
0.00
SQL*Net more data to client
14,975
0
0
0
9.58
0.00
db file sequential read
2,817
0
0
0
1.80
0.00
ADR block file read
4
0
0
43
0.00
0.00
enq: CR - block range reuse ckpt
2
0
0
71
0.00
0.00
asynch descriptor resize
38,073
100
0
0
24.36
0.00
latch: shared pool
61
0
0
1
0.04
0.00
control file sequential read
6,900
0
0
0
4.41
0.00
Disk file operations I/O
550
0
0
0
0.35
0.00
cursor: pin S
1
0
0
8
0.00
0.00
direct path write temp
34
0
0
0
0.02
0.00
library cache: mutex X
5
0
0
1
0.00
0.00
latch: In memory undo latch
2
0
0
1
0.00
0.00
buffer busy waits
14
0
0
0
0.01
0.00
SQL*Net message from client
658,990
0
294,847
447
421.62
jobq slave wait
669
99
333
497
0.43
PL/SQL lock timer
1
100
1
998
0.00Oracle does not and cannot tell you from historical views (e.g. AWR) which DMLs have not COMMITed or ROLLBACKed. A Transaction ends with a COMMIT or ROLLBACK. The transaction could have a million (or more) DML statements with a million (or more) SELECT statements between the first DML and the COMMIT / ROLLBACK.
Even identifying such DMLs in real time is close to impossible. Because the session holding the lock may have issued a dozen or a million subsequent SQL statements while other sessions are waiting for the lock. You can only identify the session that is the lock holder (the BLOCKING_SESSION in V$SESSION).
If you have tracing enabled for all sessions, then you could review the trace file for the BLOCKING_SESSION to identify the DML(s) the session has executed.
Hemant K Chitale -
Enq: TX - row lock contention in TOP 5 wait event
DB version:11.1.0.7.0
I am having enq: TX - row lock contention in top 5 wait event.
AWR analyze period - 9-10(pm). During this time only one sql loader is running to insert the data. No other job are running. So there is no chance of other session blocking this session. is there any chance of row lock contention happen by same session.
SQL> SELECT INDEX_NAME,INDEX_TYPE,UNIQUENESS FROM DBA_INDEXES WHERE TABLE_NAME='DATA_DATA';
INDEX_NAME INDEX_TYPE UNIQUENES
CIDX BITMAP NONUNIQUE
VIDX BITMAP NONUNIQUE
Thanks.SQL> SELECT INDEX_NAME,INDEX_TYPE,UNIQUENESS FROM DBA_INDEXES WHERE TABLE_NAME='DATA_DATA';
INDEX_NAME INDEX_TYPE UNIQUENES
CIDX BITMAP NONUNIQUE
VIDX BITMAP NONUNIQUEYou have bitmap indexes here on a table being inserted into. Bitmap Indexes are another source of lock(and deadlock) in OLTP application. You said that the SQLloader was the unique active program but may be you are also triggering another procedure after the load. Procedure in which you might be using also automomous transactions and so on...
Check first if your table is subject to DML operation in a a multi-user concurrent accesss and in which case you have to get rid of those bitmap indexes
http://hourim.wordpress.com/2011/03/14/deadlock-%e2%80%93-part-1-bitmap-index/
Best regards
Mohamed Houri
www.hourim.wordpress.com -
Wait event enq: JD - contention
Hi All,
In the top-5 wait events of the AWR-report i encounter a enq: JD contention wait event (wait class Other).
I only figured out that JD means Job Queue Date.
Can anybody tell me more about this wait event?
Thanks in advance,
Jeroen van SchaijkHi All,
I have additional information:
This wait event synchronizes dates between job queue coordinator and slave processes.
It appears that this wait event takes more than 10% of the total wait time!
Does anybody know how to deal with this wait event?
Thanks in advance,
Jeroen van Schaijk -
Enq: ST - contention wait event
Hi,
At one of our production database I am getting enq: ST - contention wait event at the Top 5 Timed Event section of the AWR report. I have checked with Metalink (Doc ID:974863.1), where I have been suggested to migrate to LMT. But all the tablespaces at the database are in LMT. It would be really very helpful if someone can explain the occurance of such wait event and the way this can be eliminated in future.
Environment:
Database Version: 10.2.0.3
OS: Solaris 10
Thanks for your valuable time.
Regards,
JoyI suggest reviewing the following document:
http://www.perfvision.com/papers/09_enqueues.ppt
Specifically, look at slide 61 and its notes section.
Take a look at the output of the following SQL statement:
SELECT
SUBSTR(TABLESPACE_NAME,1,20) NAME,
EXTENT_MANAGEMENT,
ALLOCATION_TYPE,
INITIAL_EXTENT,
NEXT_EXTENT
FROM
DBA_TABLESPACES
ORDER BY
EXTENT_MANAGEMENT;Is everything listed as LOCAL? For those tablespaces with a UNIFORM ALLOCATION_TYPE, what is the value of INITIAL_EXTENT?
Charles Hooper
Co-author of "Expert Oracle Practices: Oracle Database Administration from the Oak Table"
http://hoopercharles.wordpress.com/
IT Manager/Oracle DBA
K&M Machine-Fabricating, Inc. -
Enq: PS – contention wait event
Hi Guys,
I am running Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production on RHEL5.
I changed my db parameter PARALLEL_MAX_SERVERS from 10 to 20. Now, most of my queries are hanging due to this enq: PS – contention wait event. I looked at the req_dgree from the V$PX_SESSION and it is too high.
I reduced the PARALLEL_MAX_SERVERS back to 10 and still the queries are still hanging and the req_dgree still too high.
How can I fix this situation?
Thanks in advance....user11979518 wrote:
Hi Guys,
I am running Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production on RHEL5.
I changed my db parameter PARALLEL_MAX_SERVERS from 10 to 20. Now, most of my queries are hanging due to this enq: PS – contention wait event. I looked at the req_dgree from the V$PX_SESSION and it is too high.
I reduced the PARALLEL_MAX_SERVERS back to 10 and still the queries are still hanging and the req_dgree still too high.
How can I fix this situation?do use PARALLEL.
PARALLEL sometimes make SQL slower;.
PARALLEL is NOT magic solution to all/every performance problem.
more is not always better; as you have now learned.
How many cores/CPU exist in this DB Server system? -
Hi: I'm analyzing this STATSPACK report: it is "volume test" on our UAT server, so most input is from 'bind variables'. Our shared pool is well utilized in oracle. Oracle redo logs is not appropriately configured on this server, as in 'Top 5 wait events' there are 2 for redos.
I need to know what else information can be dig-out from 'foreground wait events' & 'background wait events', and what can assist us to better understanding, in combination of 'Top 5 wait event's, that how the server/test went? it could be overwelming No. of wait events, so appreciate any helpful diagnostic or analysis. Database is oracle 11.2.0.4 upgraded from 11.2.0.3, on IBM AIX power system 64bit, level 6.x
STATSPACK report for
Database DB Id Instance Inst Num Startup Time Release RAC
~~~~~~~~ ----------- ------------ -------- --------------- ----------- ---
700000XXX XXX 1 22-Apr-15 12:12 11.2.0.4.0 NO
Host Name Platform CPUs Cores Sockets Memory (G)
~~~~ ---------------- ---------------------- ----- ----- ------- ------------
dXXXX_XXX AIX-Based Systems (64- 2 1 0 16.0
Snapshot Snap Id Snap Time Sessions Curs/Sess Comment
~~~~~~~~ ---------- ------------------ -------- --------- ------------------
Begin Snap: 5635 22-Apr-15 13:00:02 114 4.6
End Snap: 5636 22-Apr-15 14:00:01 128 8.8
Elapsed: 59.98 (mins) Av Act Sess: 0.6
DB time: 35.98 (mins) DB CPU: 19.43 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 2,064M Std Block Size: 8K
Shared Pool: 3,072M Log Buffer: 13,632K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ ------------------ ----------------- ----------- -----------
DB time(s): 0.6 0.0 0.00 0.00
DB CPU(s): 0.3 0.0 0.00 0.00
Redo size: 458,720.6 8,755.7
Logical reads: 12,874.2 245.7
Block changes: 1,356.4 25.9
Physical reads: 6.6 0.1
Physical writes: 61.8 1.2
User calls: 2,033.7 38.8
Parses: 286.5 5.5
Hard parses: 0.5 0.0
W/A MB processed: 1.7 0.0
Logons: 1.2 0.0
Executes: 801.1 15.3
Rollbacks: 6.1 0.1
Transactions: 52.4
Instance Efficiency Indicators
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 99.98 Optimal W/A Exec %: 100.00
Library Hit %: 99.77 Soft Parse %: 99.82
Execute to Parse %: 64.24 Latch Hit %: 99.98
Parse CPU to Parse Elapsd %: 53.15 % Non-Parse CPU: 98.03
Shared Pool Statistics Begin End
Memory Usage %: 10.50 12.79
% SQL with executions>1: 69.98 78.37
% Memory for SQL w/exec>1: 70.22 81.96
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time
CPU time 847 50.2
enq: TX - row lock contention 4,480 434 97 25.8
log file sync 284,169 185 1 11.0
log file parallel write 299,537 164 1 9.7
log file sequential read 698 16 24 1.0
Host CPU (CPUs: 2 Cores: 1 Sockets: 0)
~~~~~~~~ Load Average
Begin End User System Idle WIO WCPU
1.16 1.84 19.28 14.51 66.21 1.20 82.01
Instance CPU
~~~~~~~~~~~~ % Time (seconds)
Host: Total time (s): 7,193.8
Host: Busy CPU time (s): 2,430.7
% of time Host is Busy: 33.8
Instance: Total CPU time (s): 1,203.1
% of Busy CPU used for Instance: 49.5
Instance: Total Database time (s): 2,426.4
%DB time waiting for CPU (Resource Mgr): 0.0
Memory Statistics Begin End
~~~~~~~~~~~~~~~~~ ------------ ------------
Host Mem (MB): 16,384.0 16,384.0
SGA use (MB): 7,136.0 7,136.0
PGA use (MB): 282.5 361.4
% Host Mem used for SGA+PGA: 45.3 45.8
Foreground Wait Events DB/Inst: XXXXXs Snaps: 5635-5636
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by Total Wait Time desc, Waits desc (idle events last)
Avg %Total
%Tim Total Wait wait Waits Call
Event Waits out Time (s) (ms) /txn Time
enq: TX - row lock contentio 4,480 0 434 97 0.0 25.8
log file sync 284,167 0 185 1 1.5 11.0
Disk file operations I/O 8,741 0 4 0 0.0 .2
direct path write 13,247 0 3 0 0.1 .2
db file sequential read 6,058 0 1 0 0.0 .1
buffer busy waits 1,800 0 1 1 0.0 .1
SQL*Net more data to client 29,161 0 1 0 0.2 .1
direct path read 7,696 0 1 0 0.0 .0
db file scattered read 316 0 1 2 0.0 .0
latch: shared pool 144 0 0 2 0.0 .0
CSS initialization 30 0 0 3 0.0 .0
cursor: pin S 10 0 0 9 0.0 .0
row cache lock 41 0 0 2 0.0 .0
latch: row cache objects 19 0 0 3 0.0 .0
log file switch (private str 8 0 0 7 0.0 .0
library cache: mutex X 28 0 0 2 0.0 .0
latch: cache buffers chains 54 0 0 1 0.0 .0
latch free 290 0 0 0 0.0 .0
control file sequential read 1,568 0 0 0 0.0 .0
log file switch (checkpoint 4 0 0 6 0.0 .0
direct path sync 8 0 0 3 0.0 .0
latch: redo allocation 60 0 0 0 0.0 .0
SQL*Net break/reset to clien 34 0 0 1 0.0 .0
latch: enqueue hash chains 45 0 0 0 0.0 .0
latch: cache buffers lru cha 7 0 0 2 0.0 .0
latch: session allocation 5 0 0 1 0.0 .0
latch: object queue header o 6 0 0 1 0.0 .0
ASM file metadata operation 30 0 0 0 0.0 .0
latch: In memory undo latch 15 0 0 0 0.0 .0
latch: undo global data 8 0 0 0 0.0 .0
SQL*Net message from client 6,362,536 0 278,225 44 33.7
jobq slave wait 7,270 100 3,635 500 0.0
SQL*Net more data from clien 7,976 0 15 2 0.0
SQL*Net message to client 6,362,544 0 8 0 33.7
Background Wait Events DB/Inst: XXXXXs Snaps: 5635-5636
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by Total Wait Time desc, Waits desc (idle events last)
Avg %Total
%Tim Total Wait wait Waits Call
Event Waits out Time (s) (ms) /txn Time
log file parallel write 299,537 0 164 1 1.6 9.7
log file sequential read 698 0 16 24 0.0 1.0
db file parallel write 9,556 0 13 1 0.1 .8
os thread startup 146 0 10 70 0.0 .6
control file parallel write 2,037 0 2 1 0.0 .1
Log archive I/O 35 0 1 30 0.0 .1
LGWR wait for redo copy 2,447 0 0 0 0.0 .0
db file async I/O submit 9,556 0 0 0 0.1 .0
db file sequential read 145 0 0 2 0.0 .0
Disk file operations I/O 349 0 0 0 0.0 .0
db file scattered read 30 0 0 4 0.0 .0
control file sequential read 5,837 0 0 0 0.0 .0
ADR block file read 19 0 0 4 0.0 .0
ADR block file write 5 0 0 15 0.0 .0
direct path write 14 0 0 2 0.0 .0
direct path read 3 0 0 7 0.0 .0
latch: shared pool 3 0 0 6 0.0 .0
log file single write 56 0 0 0 0.0 .0
latch: redo allocation 53 0 0 0 0.0 .0
latch: active service list 1 0 0 3 0.0 .0
latch free 11 0 0 0 0.0 .0
rdbms ipc message 314,523 5 57,189 182 1.7
Space Manager: slave idle wa 4,086 88 18,996 4649 0.0
DIAG idle wait 7,185 100 7,186 1000 0.0
Streams AQ: waiting for time 2 50 4,909 ###### 0.0
Streams AQ: qmn slave idle w 129 0 3,612 28002 0.0
Streams AQ: qmn coordinator 258 50 3,612 14001 0.0
smon timer 43 2 3,605 83839 0.0
pmon timer 1,199 99 3,596 2999 0.0
SQL*Net message from client 17,019 0 31 2 0.1
SQL*Net message to client 12,762 0 0 0 0.1
class slave wait 28 0 0 0 0.0
thank you very much!Hi: just know it now: it is a large amount of 'concurrent transaction' designed in this "Volume Test" - to simulate large incoming transaction volme, so I guess wait in eq:TX - row is expected.
The fact: (1) redo logs at uat server is known to not well-tune for configurations (2) volume test slow 5%, however data amount in its test is kept the same by each time import production data, by the team. So why it slowed 5% this year?
The wait histogram is pasted below, any one interest to take a look? any ideas?
Wait Event Histogram DB/Inst: XXXX/XXXX Snaps: 5635-5636
-> Total Waits - units: K is 1000, M is 1000000, G is 1000000000
-> % of Waits - column heading: <=1s is truly <1024ms, >1s is truly >=1024ms
-> % of Waits - value: .0 indicates value was <.05%, null is truly 0
-> Ordered by Event (idle events last)
Total ----------------- % of Waits ------------------
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
ADR block file read 19 26.3 5.3 10.5 57.9
ADR block file write 5 40.0 60.0
ADR file lock 6 100.0
ARCH wait for archivelog l 14 100.0
ASM file metadata operatio 30 100.0
CSS initialization 30 100.0
Disk file operations I/O 9090 97.2 1.4 .6 .4 .2 .1 .1
LGWR wait for redo copy 2447 98.5 .5 .4 .2 .2 .2 .1
Log archive I/O 35 40.0 8.6 25.7 2.9 22.9
SQL*Net break/reset to cli 34 85.3 8.8 5.9
SQL*Net more data to clien 29K 99.9 .0 .0 .0 .0 .0
buffer busy waits 1800 96.8 .7 .7 .6 .3 .4 .5
control file parallel writ 2037 90.7 5.0 2.1 .8 1.0 .3 .1
control file sequential re 7405 100.0 .0
cursor: pin S 10 10.0 90.0
db file async I/O submit 9556 99.9 .0 .0 .0
db file parallel read 1 100.0
db file parallel write 9556 62.0 32.4 1.7 .8 1.5 1.3 .1
db file scattered read 345 72.8 3.8 2.3 11.6 9.0 .6
db file sequential read 6199 97.2 .2 .3 1.6 .7 .0 .0
direct path read 7699 99.1 .4 .2 .1 .1 .0
direct path sync 8 25.0 37.5 12.5 25.0
direct path write 13K 97.8 .9 .5 .4 .3 .1 .0
enq: TX - row lock content 4480 .4 .7 1.3 3.0 6.8 12.3 75.4 .1
latch free 301 98.3 .3 .7 .7
latch: In memory undo latc 15 93.3 6.7
latch: active service list 1 100.0
latch: cache buffers chain 55 94.5 3.6 1.8
latch: cache buffers lru c 9 88.9 11.1
latch: call allocation 6 100.0
latch: checkpoint queue la 3 100.0
latch: enqueue hash chains 45 97.8 2.2
latch: messages 4 100.0
latch: object queue header 7 85.7 14.3
latch: redo allocation 113 97.3 1.8 .9
latch: row cache objects 19 89.5 5.3 5.3
latch: session allocation 5 80.0 20.0
latch: shared pool 147 90.5 1.4 2.7 1.4 .7 1.4 2.0
latch: undo global data 8 100.0
library cache: mutex X 28 89.3 3.6 3.6 3.6
log file parallel write 299K 95.6 2.6 1.0 .4 .3 .2 .0
log file sequential read 698 29.5 .1 4.6 46.8 18.9
log file single write 56 100.0
log file switch (checkpoin 4 25.0 50.0 25.0
log file switch (private s 8 12.5 37.5 50.0
log file sync 284K 93.3 3.7 1.4 .7 .5 .3 .1
os thread startup 146 100.0
row cache lock 41 85.4 9.8 2.4 2.4
DIAG idle wait 7184 100.0
SQL*Net message from clien 6379K 86.6 5.1 2.9 1.3 .7 .3 2.8 .3
SQL*Net message to client 6375K 100.0 .0 .0 .0 .0 .0 .0
Wait Event Histogram DB/Inst: XXXX/xxxx Snaps: 5635-5636
-> Total Waits - units: K is 1000, M is 1000000, G is 1000000000
-> % of Waits - column heading: <=1s is truly <1024ms, >1s is truly >=1024ms
-> % of Waits - value: .0 indicates value was <.05%, null is truly 0
-> Ordered by Event (idle events last)
Total ----------------- % of Waits ------------------
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
SQL*Net more data from cli 7976 99.7 .1 .1 .0 .1
Space Manager: slave idle 4086 .1 .2 .0 .0 .3 3.2 96.1
Streams AQ: qmn coordinato 258 49.2 .8 50.0
Streams AQ: qmn slave idle 129 100.0
Streams AQ: waiting for ti 2 50.0 50.0
class slave wait 28 92.9 3.6 3.6
jobq slave wait 7270 .0 100.0
pmon timer 1199 100.0
rdbms ipc message 314K 10.3 7.3 39.7 15.4 10.6 5.3 8.2 3.3
smon timer 43 100.0 -
Tuning row lock contention wait events
Hello everyone,
Working on 10g/windows
Top 5 events
EVENT TOTAL_WAITS TIME_WAITED AVG_MS PERCENT
CPU 9462339 48
enq: TX - row lock contention 12531 3660728 2921.34 18
control file parallel write 1300731 3088079 23.74 16
log file parallel write 1510503 1264080 8.37 6
log file sync 1072553 968007 9.03 5
Distribution of row lock wait during the last 4 days in the database server
END_INTERVAL_TIME TOTAL_WAITS TIME_WAITED_MICRO AVG_WAIT_MS
2008-04-01 16:00:58 909 2721008230 2993.41
2008-04-01 15:00:27 50 149941140 2998.82
2008-03-31 12:00:42 193 575595397 2982.36
2008-03-29 23:00:13 172 513058700 2982.9
2008-03-29 22:00:37 164 483940046 2950.85
2008-03-27 22:00:35 565 1667120838 2950.66
2008-03-26 18:00:59 348 1042918982 2996.89
My analysis:
It's obvious that the row lock contention wait time is huge, and this direct me to find out SQL stmt, causing this.
all the SQL statement was SELECT ....... FOR UPDATE stmt.
I was also able to find out locked tables.
My tuning idea:
1. I'm thinking to reorganize hot tables as well as their indexes, but by instinct it seems to not give so much value to avoid the huge row lock wait time.
2. I'm also seeing if I can reduce the number of rows per block, by increasing PCTFREE and diminishing PCTUSED, so the contention will spread over many blocks instead of one heavy block.
Question
As SQL stmt related to those locked tables are select ... for update, how could I tune this kind of stmt?
Does someone have other idea to come up with this row lock contention?
Tanks for your effort and helpTaking another look at your suggested function based index, it depends on the data type of the DEV.POS_FOLIO_ID.POS_FOLIO_ID column. If the column is defined as a number, and it is a primary key, there will already be a usable index on that column.
Yesterday, I wrote this: "Once I understood why or how the sessions were trying to insert duplicate primary key values, I would try to determine why the average number of seconds for the wait event is almost 3 seconds (maybe a timeout)."
After fixing the formatting of the top 5 wait events (total duration unknown):
EVENT TOTAL_WAITS TIME_WAITED AVG_MS PERCENT
CPU 94,623.39 48
enq: TX - row lock contention 12,531 36,607.28 2921.34 18
control file parallel write 1,300,731 30,880.79 23.74 16
log file parallel write 1,510,503 12,640.80 8.37 6
log file sync 1,072,553 9,680.07 9.03 512,531 * 3 second time out = 37,593 seconds = 10.44 hours.
What if the reason for the 3 second average wait time is due to a timeout. I performed a little experiment... I changed a row in a test table and then made a pot of coffee.
In session 1:
CREATE TABLE T1 (
C1 NUMBER(10),
C2 NUMBER(10),
PRIMARY KEY (C1));
INSERT INTO T1
SELECT
ROWNUM,
ROWNUM*10
FROM
DUAL
CONNECT BY
LEVEL<=1000000;
COMMIT;I now have a test table with 1,000,000 rows. I start monitoring the changes in the wait events roughly every 60 seconds, and V$SESSION_WAIT and V$LOCK roughly 4 times per second.
Back in session 1:
UPDATE
T1
SET
C1=-C1
WHERE
C1<=100;I have now modified the first 100 rows that were inserted into the table, time to make the pot of coffee.
In session 2, I try to insert a row with a primary key value of -10:
INSERT INTO T1 VALUES (
-10,
10);Session 2 hangs.
If I take the third 60 second snap of the system wide wait events as the zero point, and the 11th snap as the end point. There were 149 waits on ENQ: TX - ROW LOCK CONTENTION, 148 time outs, 446.62 seconds of total time in the wait event, with an average wait time of 2.997450 seconds.
Rolling down to the session level wait events, SID 208 (my session 2) had 149 waits on ENQ: TX - ROW LOCK CONTENTION, for a total time of 446.61 seconds with an average wait time of 2.997383 seconds. All of the 149 waits and the wait time was in this one session that was locked up for the full duration of this time period because session 1 was making a pot of coffee.
Rolling down to V$SESSION_WAIT (sampled roughly 4 times per second): At the start of the third time interval, SID 208 has been in the ENQ: TX - ROW LOCK CONTENTION wait event for 39 seconds and is actively waiting trying to execute SQL with a hash value of 1001532423, the wait object is -1, wait file is 0, wait block is 0, wait row is 0, P1 is 1415053316, P2 is 196646, P3 is 4754.
At the end of the 11th time interval: , SID 208 has been in the ENQ: TX - ROW LOCK CONTENTION wait event for 483 seconds and is actively waiting trying to execute SQL with a hash value of 1001532423, the wait object is -1, wait file is 0, wait block is 0, wait row is 0, P1 is 1415053316, P2 is 196646, P3 is 4754.
Rolling down to V$LOCK (sampled roughly 4 times per second): I see that SID 214 (session 1) is blocking SID 208 (session 2). SID 214 has a TX lock in mode 6 with ID1 of 196646 and ID2 of 4754. SID 208 is requesting a TX lock in mode 4 with ID1 of 196646 and ID2 of 4754.
So, it seems that I need a faster coffee pot rather than an additional index on my table. It could be that the above process would have found that the application associated with SID 214 was abandoned or crashed and for some reason the lock was not released for a long period of time, a little less than 10.44 hours in your case.
Charles Hooper
IT Manager/Oracle DBA
K&M Machine-Fabricating, Inc. -
Hi ! I have the following wait events in my top timed and I don't know who originated them:
Wait Event Wait Time Summary Avg Wait Time (ms)
I# Class Event Waits %Timeouts Total(s) Avg(ms) %DB time Avg Min Max Std Dev Cnt
* DB CPU N/A N/A 59,651.48 N/A 45.87 2
User I/O db file sequential read 4,369,213 0.0 20,831.46 4.8 16.02 4.72 4.29 5.14 0.60 2
Other enq: CF - contention 155,822 3.9 10,390.74 66.7 7.99 68.62 60.31 76.94 11.76 2
System I/O RMAN backup & recovery I/O 87,205 0.0 5,477.09 62.8 9.15 62.81 62.81 62.81 1
Cluster gc current block 2-way 2,914,457 0.0 4,811.61 1.7 3.70 1.67 1.60 1.74 0.10 2
System I/O control file sequential read 3,038,672 0.0 3,762.66 1.2 2.89 1.24 1.22 1.27 0.04 2
Concurrenc os thread startup 2,842 0.0 3,695.14 1300.2 2.84 1311.83 1143.07 1480.59 238.66 2
System I/O log file parallel write 1,341,907 0.0 2,530.17 1.9 1.95 1.88 1.88 1.89 0.01 2
Other reliable message 471,495 0.1 2,388.01 5.1 1.84 5.08 4.12 6.03 1.35 2
Concurrenc row cache lock 3,135,774 0.0 2,224.53 0.7 1.71 0.72 0.68 0.75 0.05 2
1 DB CPU N/A N/A 22,584.30 N/A 37.75
User I/O db file sequential read 2,451,215 0.0
System I/O RMAN backup & recovery I/O 87,205 0.0
Other enq: CF - contention 59,735 5.3
Cluster gc current block 2-way 1,803,542 0.0
System I/O control file sequential read 1,831,180 0.0
Concurrenc os thread startup 1,323 0.0
System I/O log file parallel write 727,883 0.0
Cluster gc cr multi block request 523,744 0.0
Concurrenc row cache lock 1,830,913 0.0
2 DB CPU N/A N/A
User I/O db file sequential read 1,917,998 0.0
Other enq: CF - contention 96,087 3.0
Cluster gc current block 2-way 1,110,915 0.0
Concurrenc os thread startup 1,519 0.0
System I/O control file sequential read 1,207,492 0.0
User I/O direct path read 404,587 0.0
Other reliable message 233,033 0.1
System I/O log file parallel write 614,024 0.0
System I/O control file parallel write 128,905 0.0
Those are the most worrying events:
enq: CF - contention
I/O control file sequential read
Concurrenc os thread startup
I have been investigating and I wonder what is wrong to get Concurrenc os thread startup. According to one blog, os thread should be always related with network issues...
The awr snapshot is from my production window day.
Rac 11.2.0.3 two nodes on Solaris Sparc 10.I have studied the ASH in the problematic period and I have found that there are some full scans:
Summary of All User Input
Format : TEXT
DB Id : 2752323407
Inst num : 1
Begin time : 08-Feb-13 09:30:00
End time : 08-Feb-13 10:20:00
Slot width : Default
Report targets : 0
Report name : ashrpt_1_0208_1020.txt
ASH Report For dbp/dbp1
DB Name DB Id Instance Inst Num Release RAC Host
dbp 2752323407 dbp1 1 11.2.0.3.0 YES host-dbp-1
CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size
16 12,651M (100%) 10,048M (79.4%) 1,921M (15.2%) 32.0M (0.3%)
Analysis Begin Time: 08-Feb-13 09:30:00
Analysis End Time: 08-Feb-13 10:20:00
Elapsed Time: 50.0 (mins)
Begin Data Source: DBA_HIST_ACTIVE_SESS_HISTORY
in AWR snapshot 5100
End Data Source: DBA_HIST_ACTIVE_SESS_HISTORY
in AWR snapshot 5101
+ V$ACTIVE_SESSION_HISTORY
Sample Count: 10,069
Average Active Sessions: 33.56
Avg. Active Session per CPU: 2.10
Report Target: None specified
Top User Events DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
Avg Active
Event Event Class % Event Sessions
library cache lock Concurrency 43.73 14.68
cursor: pin S wait on X Concurrency 18.61 6.25
CPU + Wait for CPU CPU 15.77 5.29
reliable message Other 5.88 1.97
enq: KO - fast object checkpoint Application 3.48 1.17
Top Background Events DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
Avg Active
Event Event Class % Activity Sessions
CPU + Wait for CPU CPU 1.25 0.42
Top Cluster Events DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
Event % Event Remote I % Activity
gc current block 2-way 1.72 2 1.72
gc cr grant 2-way 1.58 N/A 1.07
Top Event P1/P2/P3 Values DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
Event % Event P1 Value, P2 Value, P3 Value % Activity
Parameter 1 Parameter 2 Parameter 3
library cache lock 43.75 "29115227816","29218763456"," 1.22
handle address lock address 100*mode+namespace
"29115227816","28694732944"," 1.20
"29115227816","28812373936"," 1.17
cursor: pin S wait on X 18.61 "1497800770","3934190043136", 1.54
idn value where
"1497800770","7773890805760", 1.15
reliable message 6.07 "30432532808","30354909248"," 0.13
channel context channel handle broadcast message
enq: KO - fast object checkpoi 3.49 "1263468550","65640","1" 0.52
name|mode 2 0
db file sequential read 1.81 "1","25220","1" 0.01
file# block# blocks
Top Service/Module DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
Service Module % Activity Action % Action
dbp_DVEBMGS11 CL_SQL_STATEMENT======== 86.80 383 86.80
dbp_D10_0066 CL_SQL_STATEMENT======== 6.28 383 3.34
104 2.94
dbp_D10_0064 CL_SQL_STATEMENT======== 2.40 383 2.39
SYS$BACKGROUND UNNAMED 1.51 UNNAMED 1.51
Top Client IDs DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
No data exists for this section of the report.
Top SQL Command Types DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
-> 'Distinct SQLIDs' is the count of the distinct number of SQLIDs
with the given SQL Command Type found over all the ASH samples
in the analysis period
Distinct Avg Active
SQL Command Type SQLIDs % Activity Sessions
SELECT 485 94.56 31.74
ALTER TABLE 220 2.89 0.97
Top Phases of Execution DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
Avg Active
Phase of Execution % Activity Sessions
Parse 67.50 22.66
SQL Execution 30.46 10.22
Hard Parse 5.37 1.80
Top Remote Instances DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
Wait Class % Wait Remote I % Activity
Cluster 5.22 2 3.90
N/A 1.27
Top SQL with Top Events DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
Sampled #
SQL ID Planhash of Executions % Activity
Event % Event Top Row Source % RwSrc
350v06jcnd822 N/A 0 18.03
library cache lock 9.41 ** Row Source Not Available ** 9.41
SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
"DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
N/A 0 18.03
cursor: pin S wait on X 8.62 ** Row Source Not Available ** 8.62
SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
"DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
48unmtd983uz6 N/A 0 16.75
library cache lock 12.87 ** Row Source Not Available ** 12.87
SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
N/A 0 16.75
cursor: pin S wait on X 3.88 ** Row Source Not Available ** 3.88
SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
350v06jcnd822 2426825131 0 15.49
library cache lock 9.74 ** Row Source Not Available ** 9.74
SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
"DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
cursor: pin S wait on X 4.14 ** Row Source Not Available ** 4.14
CPU + Wait for CPU 1.61 SELECT STATEMENT 1.58
48unmtd983uz6 3511339786 0 14.98
library cache lock 11.50 ** Row Source Not Available ** 11.50
SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
cursor: pin S wait on X 1.97 ** Row Source Not Available ** 1.97
CPU + Wait for CPU 1.51 SELECT STATEMENT 1.42
07tcvyb6frtkx 2929764020 1 1.87
gc cr grant 2-way 0.80 TABLE ACCESS - BY USER ROWID 0.75
SELECT "D3"."SID_0SHIP_TO" AS "SID" FROM "/BIC/FZ99IC035" "F" JOIN "/BIC/DZ99IC
0352" "D2" ON "F" . "KEY_Z99IC0352" = "D2" . "DIMID" JOIN "/BI0/XMATERIAL" "X9"
ON "D2" . "SID_0MATERIAL" = "X9" . "SID" JOIN "/BIC/DZ99IC0355" "D5" ON "F" .
"KEY_Z99IC0355" = "D5" . "DIMID" JOIN "/BIC/DZ99IC0353" "D3" ON "F" . "KEY_Z99
Top SQL with Top Events DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
Sampled #
SQL ID Planhash of Executions % Activity
Event % Event Top Row Source % RwSrc
Top SQL with Top Row Sources DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
Sampled #
SQL ID PlanHash of Executions % Activity
Row Source % RwSrc Top Event % Event
350v06jcnd822 N/A 0 18.03
** Row Source Not Available ** 18.03 library cache lock 9.41
SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
"DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
48unmtd983uz6 N/A 0 16.75
** Row Source Not Available ** 16.75 library cache lock 12.87
SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
350v06jcnd822 2426825131 0 15.49
** Row Source Not Available ** 13.91 library cache lock 9.74
SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
"DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
SELECT STATEMENT 1.58 CPU + Wait for CPU 1.58
48unmtd983uz6 3511339786 0 14.98
** Row Source Not Available ** 13.56 library cache lock 11.50
SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
SELECT STATEMENT 1.42 CPU + Wait for CPU 1.42
07tcvyb6frtkx 2929764020 1 1.87
TABLE ACCESS - BY USER ROWID 1.59 gc cr grant 2-way 0.75
SELECT "D3"."SID_0SHIP_TO" AS "SID" FROM "/BIC/FZ99IC035" "F" JOIN "/BIC/DZ99IC
0352" "D2" ON "F" . "KEY_Z99IC0352" = "D2" . "DIMID" JOIN "/BI0/XMATERIAL" "X9"
ON "D2" . "SID_0MATERIAL" = "X9" . "SID" JOIN "/BIC/DZ99IC0355" "D5" ON "F" .
"KEY_Z99IC0355" = "D5" . "DIMID" JOIN "/BIC/DZ99IC0353" "D3" ON "F" . "KEY_Z99
Top SQL using literals DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
-> FORCE_MATCHING_SIGNATURE is used to identify SQL statements that are
identical except for their use of literals.
-> Please refer to the Oracle Database Reference to understand how
the FORCE_MATCHING_SIGNATURE is derived.
# of Sampled
FORCE_MATCHING_SIGNATURE % Activity SQL Versions
Example SQL 1
Example SQL 2
1021017294885722791 2.89 218
0htvt0stu1vtq
SELECT COUNT(*) FROM "/BIC/FZ99IC003" WHERE "KEY_Z99IC003P" = :A0
0htvt0stu1vtq
Top Parsing Module/Action DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
Module Action % Activ Event
CL_SQL_STATEMENT============== 383 67.25 library
383 cursor:
383 CPU + Wa
Top Sessions running PQs DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
-> This section aggregates all the work done by the PQ slaves into
the session issuing the parallel query.
Sid,Srl# (Inst) % Activity SQL ID Event % Event
User Program
1506, 19(1) 33.57 350v06jcnd822 library cache lock 19.15
UserID:
SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
"DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
cursor: pin S wait on X 12.76
SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
"DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
CPU + Wait for CPU 1.61
SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
"DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
2255, 1067(1) 31.78 48unmtd983uz6 library cache lock 24.37
UserID:
SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
cursor: pin S wait on X 5.85
SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
CPU + Wait for CPU 1.51
SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
Top DB Objects DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
-> With respect to Application, Cluster, User I/O and buffer busy waits only.
Object ID % Activity Event % Event
Object Name (Type) Tablespace
13661539 2.45 gc buffer busy acquire 0.87
SAPSR3./BIC/EZ99IC013 (TABLE) PSAPSR3SSD
Top DB Files DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
-> With respect to Cluster and User I/O events only.
File ID % Activity Event % Event
File Name Tablespace
53 3.60 gc current block 2-way 0.98
+dbp_DATA/dbp_2/datafile/psapsr3ssd.315.805562113 PSAPSR3SSD
Top Latches DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
No data exists for this section of the report.
Activity Over Time DB/Inst: dbp/dbp1 (Feb 08 09:30 to 10:20)
-> Analysis period is divided into smaller time slots
-> Top 3 events are reported in each of those slots
-> 'Slot Count' shows the number of ASH samples in that slot
-> 'Event Count' shows the number of ASH samples waiting for
that event in that slot
-> '% Event' is 'Event Count' over all ASH samples in the analysis period
Slot Event
Slot Time (Duration) Count Event Count % Event
09:30:00 (5.0 min) 260 gc buffer busy acquire 43 0.43
reliable message 34 0.34
CPU + Wait for CPU 29 0.29
09:35:00 (5.0 min) 303 CPU + Wait for CPU 76 0.75
db file sequential read 40 0.40
gc buffer busy acquire 39 0.39
09:40:00 (5.0 min) 366 CPU + Wait for CPU 209 2.08
db file sequential read 26 0.26
gc current block 2-way 22 0.22
09:45:00 (5.0 min) 511 CPU + Wait for CPU 249 2.47
cursor: pin S wait on X 93 0.92
reliable message 45 0.45
09:50:00 (5.0 min) 2,245 cursor: pin S wait on X 1,442 14.32
library cache lock 407 4.04
reliable message 112 1.11
09:55:00 (5.0 min) 2,037 library cache lock 1,378 13.69
cursor: pin S wait on X 297 2.95
CPU + Wait for CPU 125 1.24
10:00:00 (5.0 min) 1,823 library cache lock 1,371 13.62
CPU + Wait for CPU 263 2.61
reliable message 72 0.72
10:05:00 (5.0 min) 1,273 library cache lock 866 8.60
CPU + Wait for CPU 155 1.54
reliable message 96 0.95
10:10:00 (5.0 min) 798 library cache lock 350 3.48
CPU + Wait for CPU 287 2.85
reliable message 54 0.54
10:15:00 (5.0 min) 436 CPU + Wait for CPU 200 1.99
reliable message 61 0.61
enq: KO - fast object checkpoi 42 0.42
-------------------------------------------------------------Problems are always on instance 1.
The queries are different each day, the top sql with performance problem changes the sql_id and I cant attack them or apply a sql profile or tune them because they only execute during a period.
Any idea?
:( -
Hello SAP Community,
I start by mentioning a few details about the system I'll be talking about in this subject:
- SAP NetWeaver 7.0
- Oracle Database 10.2g
I was reading the following Note: "Note 618868 - FAQ: Oracle performance", in order to try to understand what's causing the oracle database to have slow performance.
While reading section 3 "How can I determine whether the general database performance can be optimized?" I found out that the ratio of "Busy wait time to CPU time" is away above the recommended 60:40 value. I'm getting a 94:6 ratio. This value was calculated using the query:
SELECT
ROUND((STM1.VALUE - STM2.VALUE) / 1000000) "BUSY WAIT TIME (S)",
ROUND(STM2.VALUE / 1000000) "CPU TIME (S)",
ROUND((STM1.VALUE - STM2.VALUE) / STM1.VALUE * 100) || ' : ' ||
ROUND(STM2.VALUE / STM1.VALUE * 100) RATIO
FROM V$SYS_TIME_MODEL STM1, V$SYS_TIME_MODEL STM2
WHERE STM1.STAT_NAME = 'DB time' AND STM2.STAT_NAME = 'DB CPU';
With such high values, SAP recommends to improve system performance doing some "wait event tuning".
Can someone give me some directions about this subject? Some guides specific to this subject would be nice. Any further information about my system you may require, please ask me.
Thanks in advance.
Best regards,
Daniel GarridoHello again,
Before I did any changes to the Oracle's parameters I checked the Note 619188 - FAQ: Oracle wait events, to understand what could be causing such high event wait time.
With the query:
SELECT EVENT, TOTAL_WAITS, TIME_WAITED, AVG_MS,
ROUND(RATIO_TO_REPORT(TIME_WAITED) OVER () * 100) PERCENT
FROM (SELECT SUBSTR(EVENT, 1, 30) EVENT, TOTAL_WAITS, TIME_WAITED,
ROUND(TIME_WAITED_MICRO / TOTAL_WAITS / 1000, 2) AVG_MS
FROM V$SYSTEM_EVENT
WHERE WAIT_CLASS NOT IN ('Idle', 'System I/O')
UNION
SELECT 'CPU' EVENT, NULL, VALUE, NULL
FROM V$SYSSTAT
WHERE STATISTIC# = 12
ORDER BY 3 DESC)
WHERE ROWNUM <=10;
I got the non-idle events that took more time in my system and the result was:
Result of the SELECT statement
EVENT
TOTAL_WAITS
TIME_WAITED
AVG_MS
PERCENT
log file switch (archiving nee
578.686
57.850.863
999.69
80
buffer busy waits
712.163
6.420.932
90.16
9
CPU
0
2.791.238
4
db file sequential read
4.005.546
1.746.442
4.36
2
log file sync
10.176.490
1.577.177
1.55
2
enq: TX - row lock contention
854.451
642.955
7.52
1
db file scattered read
1.055.533
621.332
5.89
1
enq: CF - contention
210.085
246.910
11.75
0
read by other session
561.558
119.910
2.14
0
log file switch completion
10.777
85.843
79.65
0
So most of the TIME_WAITED for wait events was because of the "log file switch (archiving needed)", after reading what could cause such wait event, I understood this was related with a problem I previously had in the server, where the archiving folder was with no space left. (Meanwhile the backup of the archives is being done and so the folder is being cleaned on a daily basis).
Thank you all for your help! -
Top 5 wait events in AWR Repprt
Hi,
The following is top 5 wait event in my AWR reports...
Whenever I take reports this are always top 5 events
Top 5 Timed Events
=============================================================================================================
Event
CPU time
Waits 4,717
% Total Call Time 62.0
log file sync
Waits 64,963
Time(s) 1,362
Avg Wait(ms) 21
% Total Call Time 17.9
Wait Class Commit
log file parallel write
Waits 63,485
Time(s) 1,004
Avg Wait(ms) 16
% Total Call Time 13.2
Wait Class System I/O
enq: TX - row lock contention
Waits 348
Time(s) 984
Avg Wait(ms) 2,828
% Total Call Time 12.9
Wait Class Application
db file parallel write
Waits 29,305
Time(s) 561
Avg Wait(ms) 19
% Total Call Time 7.4
Wait Class System I/O
------------------------------------------------------------------------------------------------------------Start with Performance Tuning Guide
10.2.3 Table of Wait Events and Potential Causes -
Hi all,
We have a 2 node RAC 10.2.0.3 database.From quite sometime we are experiencing "transaction" wait event apart from "enq:TX".
What is this "transaction wait event"? How does it differ from enq:TX row lock contention event. I have searched a lot but couldn't find any proper document to describe it.
Has any one encountered this event before?
Thanks.Yasir Hashmi wrote:
Hi all,
We have a 2 node RAC 10.2.0.3 database.From quite sometime we are experiencing "transaction" wait event apart from "enq:TX".
What is this "transaction wait event"? How does it differ from enq:TX row lock contention event. I have searched a lot but couldn't find any proper document to describe it.
Has any one encountered this event before?
Thanks.what SQL produces this event?
against which object type does this wait occur?
do as below so we can know complete Oracle version & OS name.
Post via COPY & PASTE complete results of
SELECT * from v$version; -
What's wrong from this wait event
Please,
Below are two tables showing respectively database wait event by wait class and session wait event by wait class.
1. Database wait event by wait class
WAIT TOTAL PCT TIME PCT
CLASS WAITS WAITS WAITED_SECS TIME
Application 7427 .08 1572.45 76.29
User I/O 50416 .57 193.24 9.38
Network 8714874 98.66 177.67 8.62
System I/O 48169 .55 85.98 4.17
2. Session wait event by wait class
SID USER WAIT TOTAL TIME_
NAME CLASS WAITS WAITED_SECS
318 PMS1000 Application 110 321.64
259 PMS1000 Application 81 212.8
318 PMS1000 Network 541943 31.8
259 PMS1000 Network 258368 17.76
258 PMS1000 Network 132774 9.34
318 PMS1000 User I/O 1392 7.49
Top Events found:
CPU + WAIT for CPU
ROW lock contention
SQL*Net more data from/to client
Question:
What may cause the application wait_class to be at the top?, event though the row lock contention was found ?
I also think the system may sufering from a network bottleneck, I also thinking to set SDU parameter, but the network is 1Gb speed, and I don't know if this can help.
Does someone give me some clue to pinpoint what is going wrong wiht the above stats?
thanks enoughuser552326
I've used the "code" tags to make your first section of data more readable:
WAIT TOTAL PCT TIME PCT
CLASS WAITS WAITS WAITED_SECS TIME
Application 7427 .08 1572.45 76.29
User I/O 50416 .57 193.24 9.38
Network 8714874 98.66 177.67 8.62
System I/O 48169 .55 85.98 4.17 If you want to know what events belong to each wait class you can query v$event_name:
select wait_class, name
from v$event_name
order by wait_class, nameThe events in class "Application" are:
SQL*Net break/reset to client
SQL*Net break/reset to dblink
Streams capture: filter callback waiting for ruleset
Streams: apply reader waiting for DDL to apply
Wait for Table Lock
enq: KO - fast object checkpoint
enq: PW - flush prewarm buffers
enq: RO - contention
enq: RO - fast object reuse
enq: TM - contention
enq: TX - row lock contention
enq: UL - contention
As you can see, this is consistent with your comment about the top event being "row lock contention". The implication of the name given to this wait class is that it is your application design that is causing the problem. Your biggest problem is that your code allows your users to lock each other out.
Looking at the summary numbers, the time spent on waiting for other users to get out of the way is a very large fraction of your total wait - until you deal with that, problems relating to I/O and network appear to be pretty irrelevant. Having said that, you seem to do a very large number of round-trips on the network - so maybe the amount of time you are losing is not hugely significant compared to the amount of work you are getting done. (You didn't actually tell us how long it took or how many concurrent users, to accumulate this wait time).
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk -
My wait events - can anyone see a problem?
hi,
this is what i have, can anyone see a problem?
thanks
EVENT TOTAL_WAITS PCT_WAITS TIME_WAIT_SEC PCT_TIME_WAITED TOTAL_TIMEOUTS PCT_TIMEOUTS AVERAGE_WAIT_SEC
Streams AQ: qmn slave idle wait 148147 .3 4051461.88 38.04 3478 .07 27.35
Streams AQ: qmn coordinator idle wa 291006 .59 3962890.53 37.21 148370 3.13 13.62
it
Streams AQ: waiting for time manage 948 0 2021434.2 18.98 948 .02 2132.31
ment or cleanup tasks
control file parallel write 1292057 2.64 266839.64 2.51 0 0 .21
log file parallel write 28433394 58.02 134658.55 1.26 0 0 0
db file sequential read 8307195 16.95 69830.07 .66 0 0 .01
free buffer waits 3117839 6.36 43374.04 .41 3106093 65.55 .01
log buffer space 55520 .11 20810.2 .2 20235 .43 .37
db file scattered read 583604 1.19 18169.58 .17 0 0 .03
write complete waits 17946 .04 17536.66 .16 17941 .38 .98
log file sync 282268 .58 10005.35 .09 9369 .2 .04
enq: RO - fast object reuse 26602 .05 6623.44 .06 2171 .05 .25
enq: CF - contention 1839 0 5178.14 .05 1723 .04 2.82
Streams AQ: qmn coordinator waiting 999 0 4311.01 .04 883 .02 4.32
for slave to start
buffer busy waits 32464 .07 3898.51 .04 3950 .08 .12
control file sequential read 2199199 4.49 3558.34 .03 0 0 0
SGA: MMAN sleep for component shrin 234330 .48 2523.65 .02 234216 4.94 .01
k
buffer exterminate 1583 0 1539.72 .01 1573 .03 .97
library cache pin 317 0 927.71 .01 316 .01 2.93
enq: CI - contention 1829 0 570.84 .01 159 0 .31
log file switch completion 1658 0 517.18 0 425 .01 .31
enq: TX - row lock contention 257 0 438.8 0 149 0 1.71
read by other session 27269 .06 355.17 0 52 0 .01
os thread startup 3869 .01 338.67 0 98 0 .09
latch: shared pool 760 0 285.87 0 0 0 .38
latch: row cache objects 664 0 250 0 0 0 .38
Data file init write 16324 .03 231.59 0 0 0 .01
reliable message 19189 .04 218.16 0 170 0 .01
latch: library cache 483 0 172.51 0 0 0 .36
SQL*Net message from dblink 1143086 2.33 128.69 0 0 0 0
latch free 6091 .01 121.1 0 0 0 .02
library cache load lock 90 0 89.48 0 18 0 .99
log file single write 1894 0 69.76 0 0 0 .04
cursor: pin S wait on X 5183 .01 55.87 0 5165 .11 .01
local write wait 6732 .01 42.58 0 2 0 .01
log file switch (checkpoint incompl 95 0 42.11 0 30 0 .44
ete)
row cache lock 119 0 30.96 0 10 0 .26
SQL*Net more data from dblink 17198 .04 25.92 0 0 0 0
log file switch (private strand flu 69 0 17.54 0 5 0 .25
sh incomplete)
enq: HW - contention 180 0 16.53 0 5 0 .09
enq: PR - contention 9 0 14.5 0 2 0 1.61
enq: JS - queue lock 51 0 12.36 0 0 0 .24
SQL*Net more data to client 48311 .1 11.66 0 0 0 0
enq: TM - contention 12 0 10.66 0 3 0 .89
class slave wait 3128 .01 7.03 0 1 0 0
JS coord start wait 68 0 6.42 0 68 0 .09
direct path write 92712 .19 6.06 0 0 0 0
control file heartbeat 1 0 3.91 0 1 0 3.91
PX Deq: Par Recov Execute 100 0 3.8 0 0 0 .04
log file sequential read 1900 0 2.88 0 0 0 0
single-task message 50 0 2.61 0 0 0 .05
enq: TX - contention 11 0 2.38 0 0 0 .22
undo segment extension 1181001 2.41 1.95 0 1180981 24.92 0
db file single write 165 0 1.3 0 0 0 .01
enq: TX - index contention 97 0 1.27 0 0 0 .01
LGWR wait for redo copy 20840 .04 .66 0 0 0 0
JS kgl get object wait 8 0 .63 0 8 0 .08
SQL*Net message to dblink 1143086 2.33 .55 0 0 0 0
kksfbc child completion 14 0 .55 0 11 0 .04
direct path read temp 217237 .44 .41 0 0 0 0
latch: cache buffers chains 2138 0 .37 0 0 0 0
latch: messages 1245 0 .27 0 0 0 0
latch: redo writing 786 0 .15 0 0 0 0
PX Deq: Par Recov Reply 65 0 .09 0 0 0 0
latch: checkpoint queue latch 171 0 .09 0 0 0 0
latch: redo allocation 1029 0 .08 0 0 0 0
latch: cache buffers lru chain 268 0 .07 0 0 0 0
SGA: allocation forcing component g 5 0 .05 0 2 0 .01
rowth
db file parallel read 83 0 .04 0 0 0 0
latch: In memory undo latch 558 0 .04 0 0 0 0
latch: object queue header operatio 338 0 .04 0 0 0 0
n
direct path read 5042 .01 .03 0 0 0 0
PX Deque wait 7 0 .02 0 0 0 0
direct path write temp 4691 .01 .02 0 0 0 0
enq: SQ - contention 1 0 .02 0 0 0 .02
latch: session allocation 190 0 .02 0 0 0 0
PX Deq: Join ACK 15 0 .01 0 0 0 0
cursor: pin S 894 0 .01 0 0 0 0
enq: TX - allocate ITL entry 37 0 .01 0 0 0 0
kkdlgon 15 0 .01 0 0 0 0
latch: enqueue hash chains 37 0 .01 0 0 0 0
library cache lock 1 0 .01 0 0 0 .01
Log archive I/O 1 0 0 0 0 0 0
PX Deq: Par Recov Change Vector 2 0 0 0 0 0 0
PX Deq: Signal ACK 3 0 0 0 0 0 0
PX Deq: Test for msg 1 0 0 0 0 0 0
PX qref latch 1 0 0 0 1 0 0
SQL*Net break/reset to dblink 5 0 0 0 0 0 0
SQL*Net more data to dblink 1 0 0 0 0 0 0
buffer deadlock 27 0 0 0 27 0 0
checkpoint completed 4 0 0 0 0 0 0
cursor: mutex S 3 0 0 0 0 0 0
cursor: mutex X 1 0 0 0 0 0 0
enq: JS - q mem clnup lck 1 0 0 0 0 0 0
enq: PS - contention 2 0 0 0 0 0 0
enq: US - contention 1 0 0 0 0 0 0
instance state change 2 0 0 0 0 0 0
latch: library cache lock 4 0 0 0 0 0 0
latch: library cache pin 1 0 0 0 0 0 0
latch: object queue header heap 8 0 0 0 0 0 0
latch: undo global data 3 0 0 0 0 0 0
recovery read 39 0 0 0 0 0 0Hi,
If its for a week than I won't bother. Probably you should try to get teh same report for these wait events in a much smaller period , like 20-30minutes of time period when your db is fully operational. If still at that time the wait events,these or any other, shoot up to high wait times, things can be investigated more deeply.
HTH
Aman.... -
I have problem with enq: TM – contention wait events.
From ASH report in Top Event P1/P2/P3 I find problematic object – table T1. It has two columns C1 and C2. There are PK and FK constraints on these two columns.Table T1 has one UNIQUE INDEX - index_name(C1, C2).
With this:
SELECT * FROM (
SELECT c.table_name, cc.column_name, cc.position column_position
FROM user_constraints c, user_cons_columns cc
WHERE c.constraint_name = cc.constraint_name
AND c.constraint_type = 'R'
MINUS
SELECT i.table_name, ic.column_name, ic.column_position
FROM user_indexes i, user_ind_columns ic
WHERE i.index_name = ic.index_name
+)+
ORDER BY table_name, column_position;
I find unindexed Foreign Key constraints, and show me that no index on C2 column.
Can I create new index on this C2 column: CREATE INDEX index_name_c2 ON T1(C2) because I have index on C1 and C2 columns ?
In ASH report in Top DB Objects I see another indexes and table with enq: TM – contention event. Whey Oracle show me this objects?Based on what you've written I see nothing that should stop you from creating an index but I would recommend you do so in dev or test before production.
As to why Oracle is showing it to you? Likely because it thinks there is a potential issue there.
Remember always that the best designed database in the world is going to have a top 5 or top 10 list too. The only way to not have events reported is with shutdown abort.
Maybe you are looking for
-
Is there a way to force full video clip buffering before playback begins?
i have 138K download radio link and ALL flash videos on websites that don't download pause every 5 seconds and buffers for 3.... is there a way to buffer the whole thing before playback? Mac OS 10.6.2 Adobe Flash 10,0,32,18
-
How do i get my events to change colors when i add new ones in my calendar
How do I get my events to change colors when I add a new event in my calendar?
-
Java stored proc. malformed values in numeric Datafield ...
The java stored procedure functiones well, if it access tables that are on the same server. Retrieving Data from numeric field of a resultset will result in malformed numeric values , if the table I access it on another server. like sms.tabname@serve
-
Unsupported Content-Type: text/html in JAX-WS
Hi All, I was trying to add new WS client proxy using jdeveloper but it gives me some error: Exception in thread "main" com.sun.xml.ws.server.UnsupportedMediaException: Unsupported Content-Type: text/html Supported ones are: [text/xml] at com.sun.xml
-
Export embedded quicktime movies to html slideshow?
I'm trying to make an html slideshow of a keynote presentation that has embedded quicktime movies. The movies just get saved as single png images. Is there a workaround or something obvious I am overlooking? Since iPhoto lets you do this, I would ass