PX Deq: Execution Msg
Hi,
Some process are damn slow when i did run V$session_wait I got many "PX Deq: Execution Msg" and "PX Deq Credit: send blkd" event it is total 40 events and 137 is "SQL*Net message from client".
Thanks
Look in v$session at the status column for the sessions in question. If the status = 'INACTIVE' and the value of the column last_call_et is large (its in seconds) then these could be dead/runaway sessions.
That is the front-end process may no longer exist in which case you would want to kill the sessions. Also you can see sessions like this for connection pooled applications where the application opens a lot of connections by default but does not have enough of a load to use the connections. The sessions will be idle.
Inactive sesssions normally do not have much if any performance impact on Oracle except that an inactvie session can be holding resouces especially locks for uncommitted table changes. Killing the session frees the locks and resouces (PGA memory for one) held by the sessions.
The statspack/AWR information is more difficult. You have to just review it and get a general feel for the reports then compare good period verse bad period reports to see where you can find what appear to be significant differences. The reports do have a heavy hitter SQL section. The source of the problem is often located within this section.
If the customer has identified any specific process as having an issue then tracing that process and using tkprof on the trace may yield faster results as it is likely an application tuning issue and you are working from a known issue rather looking database wide and trying to locate the problem areas.
HTH -- Mark D Powell --
Similar Messages
-
Linux 2.6.18-128.e15 x86-64
Oracle DB 11.1.0.7
Clusterware 11.1.0.7
ASM 11.1.0.7
Eight (8) node rac cluster
I am stumped and am asking for some guidance before resorting to creating a SR.
The query: " select count(*) from gv$transaction where start_date < :b1 " hangs when more than one node in the cluster is up. The sql statement is executed from within a stored procedure. Checking wait events yields the "PX Deq Credit: send blkd". If I execute the same sql statement standalone or within an anonomous pl/sql block from any node the statement works fine. When invoked from within the application it hangs every time. The package is wrapped so I cannot see what has transpired prior to this statement. The offending statement never completes. I have googled and checked metalink without much success.
Ran the racdiag.sql script in hope I could spot something obvious and in preparation to open up a SR (sigh). Again I am stumped for the moment. Thank you.
RCVR RCVRINST SID SNDR EVENT SNDRINST SNDRSID
A QC 4 2075 P#### PX Deq Credit: send blkd 1
PZ98 4 2089 QC PX Deq: Execution Msg 196609 2140
Username QC/Slave SlaveSet SID Slave INS STATE WAIT_EVENT QC SID QC INS Req. DOP Actual DOP
GENEVA_ADMIN QC 2075 4 WAIT PX Deq Credit: send blkd 2075
SYS QC 2140 4 WAIT PX Deq: Execute Reply 2140
- pz98 (Slave) 1 2058 1 WAIT PX Deq: Execution Msg 2140 4 8 8
- pz98 (Slave) 1 2028 2 WAIT PX Deq: Execution Msg 2140 4 8 8
- pz98 (Slave) 1 2127 3 WAIT PX Deq: Execution Msg 2140 4 8 8
- pz98 (Slave) 1 2096 4 WAIT PX Deq: Execution Msg 2140 4 8 8
- pz98 (Slave) 1 2164 5 WAIT PX Deq: Execution Msg 2140 4 8 8
- pz98 (Slave) 1 2034 6 WAIT PX Deq: Execution Msg 2140 4 8 8
- pz98 (Slave) 1 2110 7 WAIT PX Deq: Execution Msg 2140 4 8 8
- pz98 (Slave) 1 2009 8 WAIT PX Deq: Execution Msg 2140 4 8 8
10 rows selected.Hi,
Even I got the same issue on ORACLE 10gR2. I was told that it is a ORACLE bug . Refer to Metalink note 762412.1 . This might be helpful.
We are still planning to implenment the solution but the patch upgrade si not yet scheduled. Please let me know, if this solves u r issue. -
PX Deq Credit: send blkd is getting hang while refreshing materialized view
Hi All,
When we are refreshing materialized view. It is taking more than 2.30 mins. Initially it was taking 1.40 Mins.
We are using parallel and base tables are partitioned. When i checked the tkprof report i see lots of insert query is mostly waiting for PX Deq Credit: send blkd event. When i check the ASH report I don't find any query related to MV was running but still MV refresh was going on
TKPROF: Release 11.2.0.1.0 - Development on Wed Jun 5 16:27:29 2013
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
Trace file: CHDFCI_p001_43384918_PARALLEL.trc
Sort options: exeela prsela fchela
count = number of times OCI procedure was executed
cpu = cpu time in seconds executing
elapsed = elapsed time in seconds executing
disk = number of physical reads of buffers from disk
query = number of buffers gotten for consistent read
current = number of buffers gotten in current mode (usually for update)
rows = number of rows processed by the fetch or execute call
EXPLAIN PLAN option disabled.
SQL ID: 2x210q5g30m4t
Plan Hash: 2058446196
INSERT /*+ BYPASS_RECURSIVE_CHECK APPEND */ INTO
"APPS"."GL_BAL_MV" SELECT * FROM
GL_BAL_V
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 362.20 9372.04 1158765 0 0 0
Fetch 0 0.00 0.00 0 0 0 0
total 2 362.20 9372.04 1158765 0 0 0
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 175 (recursive depth: 1)
Rows Row Source Operation
0 LOAD AS SELECT (cr=0 pr=0 pw=0 time=0 us)
0 PX COORDINATOR (cr=0 pr=0 pw=0 time=0 us)
0 PX SEND QC (RANDOM) :TQ10003 (cr=0 pr=0 pw=0 time=0 us cost=1041298 size=389555904 card=2028937)
78448967 HASH JOIN BUFFERED (cr=0 pr=1158765 pw=1158765 time=276842112 us cost=1041298 size=389555904 card=2028937)
410944 BUFFER SORT (cr=0 pr=0 pw=0 time=492466 us)
410944 PX RECEIVE (cr=0 pr=0 pw=0 time=34526636 us cost=64715 size=147944250 card=1643825)
0 PX SEND HASH :TQ10001 (cr=0 pr=0 pw=0 time=0 us cost=64715 size=147944250 card=1643825)
0 PARTITION RANGE ALL PARTITION: 1 39 (cr=0 pr=0 pw=0 time=0 us cost=64715 size=147944250 card=1643825)
0 TABLE ACCESS FULL GL_CODE_COMBINATIONS PARTITION: 1 39 (cr=0 pr=0 pw=0 time=0 us cost=64715 size=147944250 card=1643825)
78448967 PX RECEIVE (cr=0 pr=0 pw=0 time=2453949696 us cost=976582 size=395060280 card=3873140)
0 PX SEND HASH :TQ10002 (cr=0 pr=0 pw=0 time=0 us cost=976582 size=395060280 card=3873140)
0 HASH JOIN (cr=0 pr=0 pw=0 time=0 us cost=976582 size=395060280 card=3873140)
0 BUFFER SORT (cr=0 pr=0 pw=0 time=0 us)
0 PX RECEIVE (cr=0 pr=0 pw=0 time=0 us cost=32 size=133920 card=2480)
0 PX SEND BROADCAST :TQ10000 (cr=0 pr=0 pw=0 time=0 us cost=32 size=133920 card=2480)
0 HASH JOIN (cr=0 pr=0 pw=0 time=0 us cost=32 size=133920 card=2480)
0 TABLE ACCESS FULL GL_SETS_OF_BOOKS (cr=0 pr=0 pw=0 time=0 us cost=7 size=108 card=6)
0 TABLE ACCESS FULL GL_PERIODS (cr=0 pr=0 pw=0 time=0 us cost=24 size=44640 card=1240)
0 PX BLOCK ITERATOR PARTITION: 1 39 (cr=0 pr=0 pw=0 time=0 us cost=976550 size=30099548160 card=627073920)
0 TABLE ACCESS FULL GL_BALANCES PARTITION: 1 39 (cr=0 pr=0 pw=0 time=0 us cost=976550 size=30099548160 card=627073920)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
PX Deq: Execution Msg 3 0.16 0.17
PX Deq Credit: send blkd 1061004 1.99 5084.61
PX Deq: Table Q Normal 250856 2.00 2306.87
asynch descriptor resize 1 0.00 0.00
Disk file operations I/O 10 0.23 0.26
direct path write temp 3608 1.20 958.39
latch free 26 0.02 0.19
PX qref latch 7647924 0.05 11.85
direct path read temp 578 0.43 35.19
PX Deq Credit: need buffer 4037 0.08 5.84
OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
call count cpu elapsed disk query current rows
Parse 0 0.00 0.00 0 0 0 0
Execute 0 0.00 0.00 0 0 0 0
Fetch 0 0.00 0.00 0 0 0 0
total 0 0.00 0.00 0 0 0 0
Misses in library cache during parse: 0
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
PX Deq: Execution Msg 3 0.47 0.75
PX Deq: Slave Session Stats 1 0.15 0.15
OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
call count cpu elapsed disk query current rows
Parse 1 0.00 0.00 0 0 0 0
Execute 1 362.20 9372.04 1158765 0 0 0
Fetch 0 0.00 0.00 0 0 0 0
total 2 362.20 9372.04 1158765 0 0 0
Misses in library cache during parse: 0
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
PX Deq: Execution Msg 3 0.16 0.17
PX Deq Credit: send blkd 1061004 1.99 5084.61
PX Deq: Table Q Normal 250856 2.00 2306.87
asynch descriptor resize 1 0.00 0.00
Disk file operations I/O 10 0.23 0.26
direct path write temp 3608 1.20 958.39
latch free 26 0.02 0.19
PX qref latch 7647924 0.05 11.85
direct path read temp 578 0.43 35.19
PX Deq Credit: need buffer 4037 0.08 5.84
1 user SQL statements in session.
0 internal SQL statements in session.
1 SQL statements in session.
0 statements EXPLAINed in this session.
Trace file: CHDFCI_p001_43384918_PARALLEL.trc
Trace file compatibility: 11.1.0.7
Sort options: exeela prsela fchela
1 session in tracefile.
1 user SQL statements in trace file.
0 internal SQL statements in trace file.
1 SQL statements in trace file.
1 unique SQL statements in trace file.
8986825 lines in trace file.
9372 elapsed seconds in trace file.When i checked the ASH report during this time. I don't see anything running related to MV.
I am using parallel degree 8 for GL_BALANCES.
Please suggest.Hi
After enabling the DML also, same plan is getting generated.
MV refresh is taking same time.
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | TQ |IN-OUT| PQ Distrib |
PLAN_TABLE_OUTPUT
| 0 | INSERT STATEMENT | | | | 1027K(100)| | | | | | |
| 1 | LOAD AS SELECT | | | | | | | | | | |
| 2 | PX COORDINATOR | | | | | | | | | | |
| 3 | PX SEND QC (RANDOM) | :TQ10003 | 1998K| 365M| 1027K (1)|999:59:59 | | | Q1,03 | P->S | QC (RAND) |
| 4 | HASH JOIN BUFFERED | | 1998K| 365M| 1027K (1)|999:59:59 | | | Q1,03 | PCWP | |
| 5 | BUFFER SORT | | | | | | | | Q1,03 | PCWC | |
| 6 | PX RECEIVE | | 1642K| 141M| 64715 (0)|999:59:59 | | | Q1,03 | PCWP | |
| 7 | PX SEND HASH | :TQ10001 | 1642K| 141M| 64715 (0)|999:59:59 | | | | S->P | HASH |
| 8 | PARTITION RANGE ALL | | 1642K| 141M| 64715 (0)|999:59:59 | 1 | 39 | | | |
| 9 | TABLE ACCESS FULL | GL_CODE_COMBINATIONS | 1642K| 141M| 64715 (0)|999:59:59 | 1 | 39 | | | |
| 10 | PX RECEIVE | | 3820K| 371M| 963K (1)|999:59:59 | | | Q1,03 | PCWP | |
PLAN_TABLE_OUTPUT
| 11 | PX SEND HASH | :TQ10002 | 3820K| 371M| 963K (1)|999:59:59 | | | Q1,02 | P->P | HASH |
| 12 | HASH JOIN | | 3820K| 371M| 963K (1)|999:59:59 | | | Q1,02 | PCWP | |
| 13 | BUFFER SORT | | | | | | | | Q1,02 | PCWC | |
| 14 | PX RECEIVE | | 2480 | 130K| 32 (4)| 00:40:12 | | | Q1,02 | PCWP | |
| 15 | PX SEND BROADCAST | :TQ10000 | 2480 | 130K| 32 (4)| 00:40:12 | | | | S->P | BROADCAST |
| 16 | HASH JOIN | | 2480 | 130K| 32 (4)| 00:40:12 | | | | | |
| 17 | TABLE ACCESS FULL| GL_SETS_OF_BOOKS | 6 | 108 | 7 (0)| 00:08:48 | | | | | |
| 18 | TABLE ACCESS FULL| GL_PERIODS | 1240 | 44640 | 24 (0)| 00:30:09 | | | | | |
| 19 | PX BLOCK ITERATOR | | 618M| 27G| 963K (1)|999:59:59 | 1 | 39 | Q1,02 | PCWC | |
| 20 | TABLE ACCESS FULL | GL_BALANCES | 618M| 27G| 963K (1)|999:59:59 | 1 | 39 | Q1,02 | PCWP | |
--------------------------------------------------------------------------------------------------------------------------------------------------Please find the completion time for MV refresh.
14:58:47 SQL> alter session enable parallel dml;
Session altered.
Elapsed: 00:00:00.27
14:59:50 SQL> exec dbms_mview.REFRESH ('GL_BAL_MV','C',atomic_refresh=>FALSE);
PL/SQL procedure successfully completed.
Elapsed: 02:30:58.37
Thanks -
Performance issue showing read by other session Event
Hi All,
we are having a severe performance issue in my database when we are running batch jobs.
This was a new database(11.2.0.2) and we are testing the performance by running some batch jobs. These batch jobs included some inserts and updates.
I am seeing read by other session in top 5 timed events and cache buffers chains in Latch Miss Sources section.
Please help me to solve this out.
Inst Num Startup Time Release RAC
1 27-Feb-12 09:03 11.2.0.2.0 NO
Platform CPUs Cores Sockets Memory(GB)
Linux x86 64-bit 8 8 8 48.00
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 5605 29-Feb-12 03:00:27 63 4.5
End Snap: 5614 29-Feb-12 12:00:47 63 4.3
Elapsed: 540.32 (mins)
DB Time: 1,774.23 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 1,952M 1,952M Std Block Size: 16K
Shared Pool Size: 1,024M 1,024M Log Buffer: 18,868K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 3.3 0.8 0.02 0.05
DB CPU(s): 1.1 0.3 0.01 0.02
Redo size: 55,763.8 13,849.3
Logical reads: 23,906.6 5,937.4
Block changes: 325.7 80.9
Physical reads: 665.6 165.3
Physical writes: 40.4 10.0
User calls: 60.7 15.1
Parses: 10.6 2.6
Hard parses: 1.1 0.3
W/A MB processed: 0.6 0.2
Logons: 0.1 0.0
Executes: 151.2 37.6
Rollbacks: 0.0 0.0
Transactions: 4.0
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.94 Redo NoWait %: 100.00
Buffer Hit %: 97.90 In-memory Sort %: 100.00
Library Hit %: 98.06 Soft Parse %: 90.16
Execute to Parse %: 92.96 Latch Hit %: 100.00
Parse CPU to Parse Elapsd %: 76.71 % Non-Parse CPU: 98.57
Shared Pool Statistics Begin End
Memory Usage %: 89.38 87.96
% SQL with executions>1: 97.14 95.15
% Memory for SQL w/exec>1: 96.05 92.46
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
db file sequential read 14,092,706 65,613 5 61.6 User I/O
DB CPU 34,819 32.7
read by other session 308,534 1,260 4 1.2 User I/O
direct path read 97,454 987 10 .9 User I/O
db file scattered read 71,870 910 13 .9 User I/O
Host CPU (CPUs: 8 Cores: 8 Sockets: 8)
~~~~~~~~ Load Average
Begin End %User %System %WIO %Idle
0.43 0.36 13.7 0.6 9.7 85.7
Instance CPU
~~~~~~~~~~~~
% of total CPU for Instance: 13.5
% of busy CPU for Instance: 94.2
%DB time waiting for CPU - Resource Mgr: 0.0
Memory Statistics
~~~~~~~~~~~~~~~~~ Begin End
Host Mem (MB): 49,152.0 49,152.0
SGA use (MB): 3,072.0 3,072.0
PGA use (MB): 506.5 629.1
% Host Mem used for SGA+PGA: 7.28 7.53
Time Model Statistics
-> Total time in database user-calls (DB Time): 106453.8s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 105,531.1 99.1
DB CPU 34,818.8 32.7
parse time elapsed 714.7 .7
hard parse elapsed time 684.8 .6
PL/SQL execution elapsed time 161.9 .2
PL/SQL compilation elapsed time 44.2 .0
connection management call elapsed time 16.9 .0
hard parse (sharing criteria) elapsed time 10.2 .0
hard parse (bind mismatch) elapsed time 9.4 .0
sequence load elapsed time 2.9 .0
repeated bind elapsed time 0.5 .0
failed parse elapsed time 0.0 .0
DB time 106,453.8
background elapsed time 1,753.9
background cpu time 61.7
Operating System Statistics
-> *TIME statistic values are diffed.
All others display actual values. End Value is displayed if different
-> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name
Statistic Value End Value
BUSY_TIME 3,704,415
IDLE_TIME 22,203,740
IOWAIT_TIME 2,517,864
NICE_TIME 3
SYS_TIME 145,696
USER_TIME 3,557,758
LOAD 0 0
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 358,813,045,760
VM_OUT_BYTES 29,514,830,848
PHYSICAL_MEMORY_BYTES 51,539,607,552
NUM_CPUS 8
NUM_CPU_CORES 8
NUM_CPU_SOCKETS 8
GLOBAL_RECEIVE_SIZE_MAX 4,194,304
GLOBAL_SEND_SIZE_MAX 1,048,586
TCP_RECEIVE_SIZE_DEFAULT 87,380
TCP_RECEIVE_SIZE_MAX 4,194,304
TCP_RECEIVE_SIZE_MIN 4,096
TCP_SEND_SIZE_DEFAULT 16,384
TCP_SEND_SIZE_MAX 4,194,304
TCP_SEND_SIZE_MIN 4,096
Operating System Statistics -
Snap Time Load %busy %user %sys %idle %iowait
29-Feb 03:00:27 0.4 N/A N/A N/A N/A N/A
29-Feb 04:00:35 1.4 11.9 11.2 0.6 88.1 14.3
29-Feb 05:00:41 1.7 13.8 13.2 0.6 86.2 15.8
29-Feb 06:00:48 1.5 14.0 13.5 0.6 86.0 12.3
29-Feb 07:01:00 1.8 16.3 15.8 0.5 83.7 10.4
29-Feb 08:00:12 2.6 23.2 22.5 0.6 76.8 12.6
29-Feb 09:00:26 1.3 16.6 16.0 0.5 83.4 5.7
29-Feb 10:00:33 1.2 13.8 13.3 0.5 86.2 2.0
29-Feb 11:00:43 1.3 14.5 14.0 0.5 85.5 3.8
29-Feb 12:00:47 0.4 4.9 4.2 0.7 95.1 10.6
Foreground Wait Class
-> s - second, ms - millisecond - 1000th of a second
-> ordered by wait time desc, waits desc
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
-> Captured Time accounts for 97.9% of Total DB time 106,453.79 (s)
-> Total FG Wait Time: 69,415.64 (s) DB CPU time: 34,818.79 (s)
Avg
%Time Total Wait wait
Wait Class Waits -outs Time (s) (ms) %DB time
User I/O 14,693,843 0 69,222 5 65.0
DB CPU 34,819 32.7
Commit 40,629 0 119 3 0.1
System I/O 26,504 0 57 2 0.1
Network 1,945,010 0 11 0 0.0
Other 125,200 99 4 0 0.0
Application 2,673 0 2 1 0.0
Concurrency 3,059 0 1 0 0.0
Configuration 31 19 0 15 0.0
Foreground Wait Events
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
db file sequential read 14,092,706 0 65,613 5 108.0 61.6
read by other session 308,534 0 1,260 4 2.4 1.2
direct path read 97,454 0 987 10 0.7 .9
db file scattered read 71,870 0 910 13 0.6 .9
db file parallel read 35,001 0 372 11 0.3 .3
log file sync 40,629 0 119 3 0.3 .1
control file sequential re 26,504 0 57 2 0.2 .1
direct path read temp 14,499 0 49 3 0.1 .0
direct path write temp 9,186 0 28 3 0.1 .0
SQL*Net message to client 1,923,973 0 5 0 14.7 .0
SQL*Net message from dblin 1,056 0 5 5 0.0 .0
Disk file operations I/O 8,848 0 2 0 0.1 .0
ASM file metadata operatio 36 0 2 54 0.0 .0
SQL*Net break/reset to cli 2,636 0 1 1 0.0 .0
ADR block file read 472 0 1 1 0.0 .0
os thread startup 8 0 1 74 0.0 .0
SQL*Net more data to clien 17,656 0 1 0 0.1 .0
asynch descriptor resize 123,852 100 0 0 0.9 .0
local write wait 110 0 0 4 0.0 .0
utl_file I/O 55,635 0 0 0 0.4 .0
log file switch (private s 8 0 0 52 0.0 .0
cursor: pin S wait on X 2 0 0 142 0.0 .0
enq: KO - fast object chec 13 0 0 20 0.0 .0
PX Deq: Slave Session Stat 248 0 0 1 0.0 .0
enq: RO - fast object reus 18 0 0 11 0.0 .0
latch: cache buffers chain 2,511 0 0 0 0.0 .0
latch: shared pool 195 0 0 1 0.0 .0
CSS initialization 12 0 0 8 0.0 .0
PX qref latch 54 100 0 2 0.0 .0
SQL*Net more data from cli 995 0 0 0 0.0 .0
SQL*Net more data from dbl 300 0 0 0 0.0 .0
kksfbc child completion 1 100 0 56 0.0 .0
library cache: mutex X 244 0 0 0 0.0 .0
PX Deq: Signal ACK RSG 124 0 0 0 0.0 .0
undo segment extension 6 100 0 7 0.0 .0
PX Deq: Signal ACK EXT 124 0 0 0 0.0 .0
library cache load lock 3 0 0 9 0.0 .0
ADR block file write 45 0 0 1 0.0 .0
CSS operation: action 12 0 0 2 0.0 .0
reliable message 28 0 0 1 0.0 .0
CSS operation: query 72 0 0 0 0.0 .0
latch: row cache objects 14 0 0 1 0.0 .0
enq: SQ - contention 17 0 0 0 0.0 .0
latch free 32 0 0 0 0.0 .0
buffer busy waits 52 0 0 0 0.0 .0
enq: PS - contention 16 0 0 0 0.0 .0
enq: TX - row lock content 6 0 0 1 0.0 .0
SQL*Net message to dblink 1,018 0 0 0 0.0 .0
cursor: pin S 23 0 0 0 0.0 .0
latch: cache buffers lru c 8 0 0 0 0.0 .0
SQL*Net message from clien 1,923,970 0 944,508 491 14.7
jobq slave wait 66,732 100 33,334 500 0.5
Streams AQ: waiting for me 6,481 100 32,412 5001 0.0
wait for unread message on 32,858 98 32,411 986 0.3
PX Deq: Execution Msg 1,448 0 190 131 0.0
PX Deq: Execute Reply 1,196 0 74 62 0.0
HS message to agent 228 0 4 19 0.0
single-task message 42 0 4 97 0.0
PX Deq Credit: send blkd 904 0 2 3 0.0
PX Deq Credit: need buffer 205 0 1 3 0.0
Foreground Wait Events
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
PX Deq: Table Q Normal 4,291 0 1 0 0.0
PX Deq: Join ACK 124 0 0 1 0.0
PX Deq: Parse Reply 124 0 0 0 0.0
KSV master wait 256 0 0 0 0.0
Latch Miss Sources
-> only latches with sleeps are shown
-> ordered by name, sleeps desc
NoWait Waiter
Latch Name Where Misses Sleeps Sleeps
ASM map operation freeli kffmTranslate2 0 2 0
DML lock allocation ktadmc 0 2 0
FOB s.o list latch ksfd_allfob 0 2 2
In memory undo latch ktiFlushMe 0 5 0
In memory undo latch ktichg: child 0 3 0
PC and Classifier lists No latch 0 6 0
Real-time plan statistic keswxAddNewPlanEntry 0 20 20
SQL memory manager worka qesmmIRegisterWorkArea:1 0 1 1
active service list kswslogon: session logout 0 23 12
active service list kswssetsvc: PX session swi 0 6 1
active service list kswsite: service iterator 0 1 0
archive process latch kcrrgpll 0 3 3
cache buffers chains kcbgtcr_2 0 1,746 573
cache buffers chains kcbgtcr: fast path (cr pin 0 1,024 2,126
cache buffers chains kcbgcur_2 0 60 8
cache buffers chains kcbchg1: kslbegin: bufs no 0 16 3
cache buffers chains kcbgtcr: fast path 0 14 20
cache buffers chains kcbzibmlt: multi-block rea 0 10 0
cache buffers chains kcbrls_2 0 9 53
cache buffers chains kcbgtcr: kslbegin shared 0 8 1
cache buffers chains kcbrls_1 0 7 84
cache buffers chains kcbgtcr: kslbegin excl 0 6 14
cache buffers chains kcbnew: new latch again 0 6 0
cache buffers chains kcbzgb: scan from tail. no 0 6 0
cache buffers chains kcbzwb 0 5 8
cache buffers chains kcbgcur: fast path (shr) 0 3 0
cache buffers chains kcbget: pin buffer 0 3 0
cache buffers chains kcbzhngcbk2_1 0 1 0
cache buffers lru chain kcbzgws 0 19 0
cache buffers lru chain kcbo_link_q 0 3 0
call allocation ksuxds 0 14 10
call allocation ksudlp: top call 0 2 3
enqueue hash chains ksqgtl3 0 2 1
enqueue hash chains ksqrcl 0 1 2
enqueues ksqgel: create enqueue 0 1 0
object queue header oper kcbo_unlink_q 0 5 2
object queue header oper kcbo_sw_buf 0 2 0
object queue header oper kcbo_link_q 0 1 2
object queue header oper kcbo_switch_cq 0 1 2
object queue header oper kcbo_switch_mq_bg 0 1 4
parallel query alloc buf kxfpbalo 0 1 1
process allocation ksucrp:1 0 2 0
process queue reference kxfpqrsnd 0 1 0
qmn task queue latch kwqmnmvtsks: delay to read 0 1 0
redo allocation kcrfw_redo_gen: redo alloc 0 17 0
row cache objects kqreqd: reget 0 6 0
row cache objects kqrpre: find obj 0 6 13
row cache objects kqrso 0 2 0
row cache objects kqreqd 0 1 2
row cache objects kqrpre: init complete 0 1 1
shared pool kghalo 0 199 106
shared pool kghupr1 0 39 109
shared pool kghfre 0 18 19
shared pool kghalp 0 7 29
space background task la ktsj_grab_task 0 21 27
Mutex Sleep Summary
-> ordered by number of sleeps desc
Wait
Mutex Type Location Sleeps Time (ms)
Library Cache kglhdgn2 106 338 12
Library Cache kgllkc1 57 259 10
Library Cache kgllkdl1 85 123 21
Cursor Pin kkslce [KKSCHLPIN2] 70 286
Library Cache kglget2 2 31 1
Library Cache kglhdgn1 62 31 2
Library Cache kglpin1 4 26 1
Library Cache kglpnal1 90 18 0
Library Cache kglpndl1 95 15 2
Library Cache kgllldl2 112 6 0
Library Cache kglini1 32 1 0
-------------------------------------------------------------Thanks in advance.Hi,
Thanks for reply.
I provided one hour report.
Inst Num Startup Time Release RAC
1 27-Feb-12 09:03 11.2.0.2.0 NO
Platform CPUs Cores Sockets Memory(GB)
Linux x86 64-bit 8 8 8 48.00
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 5606 29-Feb-12 04:00:35 63 3.7
End Snap: 5607 29-Feb-12 05:00:41 63 3.6
Elapsed: 60.11 (mins)
DB Time: 382.67 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ---------- ----------
Buffer Cache: 1,952M 1,952M Std Block Size: 16K
Shared Pool Size: 1,024M 1,024M Log Buffer: 18,868K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 6.4 0.8 0.03 0.03
DB CPU(s): 1.0 0.1 0.00 0.00
Redo size: 84,539.3 10,425.6
Logical reads: 23,345.6 2,879.1
Block changes: 386.5 47.7
Physical reads: 1,605.0 197.9
Physical writes: 7.1 0.9
User calls: 233.9 28.9
Parses: 4.0 0.5
Hard parses: 0.1 0.0
W/A MB processed: 0.1 0.0
Logons: 0.1 0.0
Executes: 210.9 26.0
Rollbacks: 0.0 0.0
Transactions: 8.1
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.62 Redo NoWait %: 100.00
Buffer Hit %: 95.57 In-memory Sort %: 100.00
Library Hit %: 99.90 Soft Parse %: 98.68
Execute to Parse %: 98.10 Latch Hit %: 99.99
Parse CPU to Parse Elapsd %: 32.08 % Non-Parse CPU: 99.90
Shared Pool Statistics Begin End
Memory Usage %: 89.25 89.45
% SQL with executions>1: 96.79 97.52
% Memory for SQL w/exec>1: 95.67 96.56
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
db file sequential read 3,054,464 17,002 6 74.0 User I/O
DB CPU 3,748 16.3
read by other session 199,603 796 4 3.5 User I/O
direct path read 46,301 439 9 1.9 User I/O
db file scattered read 21,113 269 13 1.2 User I/O
Host CPU (CPUs: 8 Cores: 8 Sockets: 8)
~~~~~~~~ Load Average
Begin End %User %System %WIO %Idle
1.45 1.67 13.2 0.6 15.8 86.2
Instance CPU
~~~~~~~~~~~~
% of total CPU for Instance: 13.0
% of busy CPU for Instance: 94.7
%DB time waiting for CPU - Resource Mgr: 0.0
Memory Statistics
~~~~~~~~~~~~~~~~~ Begin End
Host Mem (MB): 49,152.0 49,152.0
SGA use (MB): 3,072.0 3,072.0
PGA use (MB): 513.5 467.7
% Host Mem used for SGA+PGA: 7.29 7.20
Time Model Statistics
-> Total time in database user-calls (DB Time): 22960.5s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 22,835.9 99.5
DB CPU 3,748.4 16.3
parse time elapsed 15.4 .1
hard parse elapsed time 14.3 .1
PL/SQL execution elapsed time 7.5 .0
PL/SQL compilation elapsed time 6.0 .0
connection management call elapsed time 1.6 .0
sequence load elapsed time 0.4 .0
hard parse (sharing criteria) elapsed time 0.0 .0
repeated bind elapsed time 0.0 .0
failed parse elapsed time 0.0 .0
DB time 22,960.5
background elapsed time 238.1
background cpu time 4.9
Operating System Statistics
-> *TIME statistic values are diffed.
All others display actual values. End Value is displayed if different
-> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name
Statistic Value End Value
BUSY_TIME 396,506
IDLE_TIME 2,483,725
IOWAIT_TIME 455,495
NICE_TIME 0
SYS_TIME 16,163
USER_TIME 380,052
LOAD 1 2
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 95,646,943,232
VM_OUT_BYTES 1,686,059,008
PHYSICAL_MEMORY_BYTES 51,539,607,552
NUM_CPUS 8
NUM_CPU_CORES 8
NUM_CPU_SOCKETS 8
GLOBAL_RECEIVE_SIZE_MAX 4,194,304
GLOBAL_SEND_SIZE_MAX 1,048,586
TCP_RECEIVE_SIZE_DEFAULT 87,380
TCP_RECEIVE_SIZE_MAX 4,194,304
TCP_RECEIVE_SIZE_MIN 4,096
TCP_SEND_SIZE_DEFAULT 16,384
TCP_SEND_SIZE_MAX 4,194,304
TCP_SEND_SIZE_MIN 4,096
Operating System Statistics -
Snap Time Load %busy %user %sys %idle %iowait
29-Feb 04:00:35 1.4 N/A N/A N/A N/A N/A
29-Feb 05:00:41 1.7 13.8 13.2 0.6 86.2 15.8
Foreground Wait Class
-> s - second, ms - millisecond - 1000th of a second
-> ordered by wait time desc, waits desc
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
-> Captured Time accounts for 97.6% of Total DB time 22,960.46 (s)
-> Total FG Wait Time: 18,651.75 (s) DB CPU time: 3,748.35 (s)
Avg
%Time Total Wait wait
Wait Class Waits -outs Time (s) (ms) %DB time
User I/O 3,327,253 0 18,576 6 80.9
DB CPU 3,748 16.3
Commit 23,882 0 69 3 0.3
System I/O 1,035 0 3 3 0.0
Network 842,393 0 2 0 0.0
Other 10,120 99 0 0 0.0
Configuration 3 0 0 58 0.0
Application 264 0 0 1 0.0
Concurrency 1,482 0 0 0 0.0
Foreground Wait Events
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
db file sequential read 3,054,464 0 17,002 6 104.5 74.0
read by other session 199,603 0 796 4 6.8 3.5
direct path read 46,301 0 439 9 1.6 1.9
db file scattered read 21,113 0 269 13 0.7 1.2
log file sync 23,882 0 69 3 0.8 .3
db file parallel read 4,727 0 68 14 0.2 .3
control file sequential re 1,035 0 3 3 0.0 .0
SQL*Net message to client 840,792 0 2 0 28.8 .0
direct path read temp 95 0 2 18 0.0 .0
local write wait 79 0 0 4 0.0 .0
Disk file operations I/O 870 0 0 0 0.0 .0
ASM file metadata operatio 4 0 0 50 0.0 .0
log file switch (private s 3 0 0 58 0.0 .0
ADR block file read 36 0 0 3 0.0 .0
enq: RO - fast object reus 5 0 0 16 0.0 .0
latch: cache buffers chain 1,465 0 0 0 0.1 .0
SQL*Net break/reset to cli 256 0 0 0 0.0 .0
asynch descriptor resize 10,059 100 0 0 0.3 .0
SQL*Net more data to clien 1,510 0 0 0 0.1 .0
enq: KO - fast object chec 3 0 0 8 0.0 .0
SQL*Net more data from cli 91 0 0 0 0.0 .0
latch: shared pool 14 0 0 0 0.0 .0
ADR block file write 5 0 0 1 0.0 .0
reliable message 8 0 0 0 0.0 .0
direct path write temp 1 0 0 2 0.0 .0
SQL*Net message from clien 840,794 0 68,885 82 28.8
jobq slave wait 7,365 100 3,679 499 0.3
Streams AQ: waiting for me 721 100 3,605 5000 0.0
wait for unread message on 3,648 98 3,603 988 0.1
KSV master wait 20 0 0 0 0.0
Background Wait Events
-> ordered by wait time desc, waits desc (idle events last)
-> Only events with Total Wait Time (s) >= .001 are shown
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % bg
Event Waits -outs Time (s) (ms) /txn time
log file parallel write 29,353 0 83 3 1.0 34.8
db file parallel write 5,753 0 17 3 0.2 6.9
db file sequential read 1,638 0 15 9 0.1 6.1
control file sequential re 5,142 0 13 2 0.2 5.4
os thread startup 140 0 8 58 0.0 3.4
control file parallel writ 1,440 0 8 6 0.0 3.4
log file sequential read 304 0 8 26 0.0 3.3
db file scattered read 214 0 2 9 0.0 .8
ASM file metadata operatio 1,199 0 1 1 0.0 .3
direct path write 35 0 0 6 0.0 .1
direct path read 41 0 0 5 0.0 .1
kfk: async disk IO 6 0 0 9 0.0 .0
Disk file operations I/O 1,266 0 0 0 0.0 .0
ADR block file read 16 0 0 2 0.0 .0
read by other session 3 0 0 8 0.0 .0
Log archive I/O 2 0 0 10 0.0 .0
log file sync 3 0 0 5 0.0 .0
asynch descriptor resize 341 100 0 0 0.0 .0
CSS initialization 1 0 0 6 0.0 .0
log file single write 4 0 0 1 0.0 .0
latch: redo allocation 3 0 0 1 0.0 .0
ADR block file write 5 0 0 1 0.0 .0
LGWR wait for redo copy 45 0 0 0 0.0 .0
CSS operation: query 6 0 0 0 0.0 .0
CSS operation: action 1 0 0 1 0.0 .0
SQL*Net message to client 420 0 0 0 0.0 .0
rdbms ipc message 47,816 39 61,046 1277 1.6
DIAG idle wait 7,200 100 7,200 1000 0.2
Space Manager: slave idle 1,146 98 5,674 4951 0.0
class slave wait 284 0 3,983 14026 0.0
dispatcher timer 61 100 3,660 60006 0.0
Streams AQ: qmn coordinato 258 50 3,613 14003 0.0
Streams AQ: qmn slave idle 130 0 3,613 27789 0.0
Streams AQ: waiting for ti 7 71 3,608 515430 0.0
wait for unread message on 3,605 100 3,606 1000 0.1
pmon timer 1,201 100 3,604 3001 0.0
smon timer 15 73 3,603 240207 0.0
ASM background timer 754 0 3,602 4777 0.0
shared server idle wait 120 100 3,601 30006 0.0
SQL*Net message from clien 554 0 4 7 0.0
KSV master wait 101 0 0 2 0.0
Wait Event Histogram
-> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
-> % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
-> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
-> Ordered by Event (idle events last)
% of Waits
Total
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
ADR block file read 52 73.1 1.9 9.6 13.5 1.9
ADR block file write 10 100.0
ADR file lock 12 100.0
ARCH wait for archivelog l 3 100.0
ASM file metadata operatio 1203 97.3 .5 .7 .3 .2 .9
CSS initialization 1 100.0
CSS operation: action 1 100.0
CSS operation: query 6 83.3 16.7
Disk file operations I/O 2118 95.4 4.5 .1
LGWR wait for redo copy 45 100.0
Log archive I/O 2 100.0
SQL*Net break/reset to cli 256 99.6 .4
SQL*Net message to client 839.9 100.0 .0
SQL*Net more data from cli 91 100.0
SQL*Net more data to clien 1503 100.0
asynch descriptor resize 10.4K 100.0
buffer busy waits 2 100.0
control file parallel writ 1440 5.7 35.1 24.0 16.3 12.0 5.5 1.5
control file sequential re 6177 69.4 7.5 5.9 8.1 7.1 1.7 .3
db file parallel read 4727 1.7 3.2 3.2 10.1 46.6 33.3 1.8
db file parallel write 5755 42.3 21.3 18.6 11.2 4.6 1.4 .5
db file scattered read 21.5K 8.4 4.3 11.9 18.9 26.3 25.3 4.9
db file sequential read 3053. 28.7 15.1 11.1 17.9 21.5 5.4 .3 .0
direct path read 46.3K 9.9 8.8 18.5 21.7 22.8 15.7 2.7
direct path read temp 95 9.5 9.5 23.2 49.5 8.4
direct path write 35 11.4 31.4 17.1 22.9 11.4 2.9 2.9
direct path write temp 1 100.0
enq: KO - fast object chec 3 66.7 33.3
enq: RO - fast object reus 5 20.0 20.0 20.0 20.0 20.0
kfk: async disk IO 6 50.0 16.7 16.7 16.7
latch free 3 100.0
latch: cache buffers chain 1465 100.0
latch: cache buffers lru c 1 100.0
latch: object queue header 2 100.0
latch: redo allocation 3 33.3 33.3 33.3
latch: row cache objects 2 100.0
latch: shared pool 15 93.3 6.7
local write wait 79 35.4 34.2 21.5 8.9
log file parallel write 29.4K 47.8 21.7 11.9 9.9 6.8 1.6 .3
log file sequential read 304 6.3 3.0 3.6 10.2 23.4 24.3 29.3
log file single write 4 25.0 75.0
log file switch (private s 3 100.0
log file sync 23.9K 40.9 28.0 12.9 9.7 6.7 1.5 .3
os thread startup 140 100.0
read by other session 199.6 37.1 19.9 12.9 13.1 13.8 3.1 .2
reliable message 8 100.0
ASM background timer 755 2.9 .4 .1 .1 .3 .1 .3 95.8
DIAG idle wait 7196 100.0
KSV master wait 121 88.4 2.5 3.3 2.5 .8 .8 1.7
SQL*Net message from clien 840.1 97.1 1.8 .5 .2 .2 .1 .0 .1
Space Manager: slave idle 1147 .1 .5 99.4
Streams AQ: qmn coordinato 258 49.6 .4 50.0
Streams AQ: qmn slave idle 130 .8 99.2
Streams AQ: waiting for me 721 100.0
Streams AQ: waiting for ti 7 28.6 42.9 28.6
class slave wait 283 39.9 2.5 2.5 3.5 4.9 9.2 15.2 22.3
dispatcher timer 60 100.0
jobq slave wait 7360 .0 .0 .0 99.9
pmon timer 1201 100.0
rdbms ipc message 47.8K 2.7 31.6 17.4 1.1 1.1 .9 20.9 24.3
Wait Event Histogram DB/Inst: I2KPROD/I2KPROD Snaps: 5606-5607
-> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
-> % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
-> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
-> Ordered by Event (idle events last)
% of Waits
Total
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
shared server idle wait 120 100.0
smon timer 16 6.3 93.8
wait for unread message on 7250 .1 99.9
Latch Miss Sources
-> only latches with sleeps are shown
-> ordered by name, sleeps desc
NoWait Waiter
Latch Name Where Misses Sleeps Sleeps
In memory undo latch ktichg: child 0 1 0
active service list kswslogon: session logout 0 2 0
cache buffers chains kcbgtcr_2 0 1,123 483
cache buffers chains kcbgtcr: fast path (cr pin 0 496 1,131
cache buffers chains kcbrls_2 0 5 6
cache buffers chains kcbgcur_2 0 4 0
cache buffers chains kcbgtcr: fast path 0 3 1
cache buffers chains kcbzwb 0 2 4
cache buffers chains kcbchg1: kslbegin: bufs no 0 1 0
cache buffers chains kcbnew: new latch again 0 1 0
cache buffers chains kcbrls_1 0 1 6
cache buffers chains kcbzgb: scan from tail. no 0 1 0
cache buffers lru chain kcbzgws 0 1 0
object queue header oper kcbo_switch_cq 0 1 0
object queue header oper kcbo_switch_mq_bg 0 1 2
redo allocation kcrfw_redo_gen: redo alloc 0 3 0
row cache objects kqrpre: find obj 0 1 1
row cache objects kqrso 0 1 0
shared pool kghalo 0 13 3
shared pool kghupr1 0 4 15
shared pool kghalp 0 1 0
space background task la ktsj_grab_task 0 2 2
------------------------------------------------------------- -
EBS 11i AutoInvoice Performance Very Slow
Hi all,
EBS 11i
Oracle 9.2.0.6 32bit
AutoInvoice Transfer Program has been bugging down the Company's business, a bad start for 2013. Please help for this serious problem :(
Attached is the "STATSPACK" report for the slow performance time.
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
OAPROD 4204192100 OAPROD 1 9.2.0.6.0 NO oel5.prod
e.local
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 551 02-Jan-13 22:00:04 123 59.1
End Snap: 553 03-Jan-13 00:00:02 97 48.7
Elapsed: 119.97 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 288M Std Block Size: 8K
Shared Pool Size: 288M Log Buffer: 10,240K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 54,437.02 65,579.53
Logical reads: 2,053,336.52 2,473,626.15
Block changes: 383.39 461.86
Physical reads: 43,270.67 52,127.57
Physical writes: 46.63 56.18
User calls: 35.83 43.16
Parses: 25.53 30.76
Hard parses: 0.18 0.21
Sorts: 186.25 224.38
Logons: 0.08 0.10
Executes: 108.76 131.02
Transactions: 0.83
% Blocks changed per Read: 0.02 Recursive Call %: 85.81
Rollback per transaction %: 3.18 Rows per Sort: 3.77
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.99 Redo NoWait %: 99.98
Buffer Hit %: 97.90 In-memory Sort %: 100.00
Library Hit %: 99.87 Soft Parse %: 99.31
Execute to Parse %: 76.52 Latch Hit %: 99.96
Parse CPU to Parse Elapsd %: 37.01 % Non-Parse CPU: 99.98
Shared Pool Statistics Begin End
Memory Usage %: 95.13 94.82
% SQL with executions>1: 83.58 95.42
% Memory for SQL w/exec>1: 79.28 92.57
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
CPU time 74,717 84.80
latch free 181,788 8,101 9.19
db file scattered read 47,546,136 2,893 3.28
db file sequential read 23,092,208 2,006 2.28
buffer busy waits 1,812,920 193 .22
^LWait Events for DB: OAPROD Instance: OAPROD Snaps: 551 -553
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
latch free 181,788 180,378 8,101 45 30.4
db file scattered read 47,546,136 0 2,893 0 7,957.5
db file sequential read 23,092,208 0 2,006 0 3,864.8
buffer busy waits 1,812,920 2 193 0 303.4
log file switch (checkpoint 124 51 69 560 0.0
log file parallel write 21,251 0 39 2 3.6
log file sync 4,072 0 21 5 0.7
control file parallel write 3,061 0 12 4 0.5
library cache load lock 5 2 9 1806 0.0
log file switch completion 54 6 8 148 0.0
PX Deq: Execute Reply 7,268 0 6 1 1.2
PX Deq: Signal ACK 297 103 6 21 0.0
PX qref latch 310 279 5 16 0.1
PX Deq: Msg Fragment 4,301 0 4 1 0.7
PX Deq Credit: send blkd 1,256 0 2 2 0.2
PX Deq: Parse Reply 267 0 2 7 0.0
process startup 46 0 2 39 0.0
PX Deq: Table Q Sample 84 0 2 22 0.0
library cache pin 337 0 1 4 0.1
kksfbc child completion 116 76 1 11 0.0
write complete waits 2 1 1 491 0.0
PX Deq: Join ACK 212 0 1 4 0.0
SQL*Net break/reset to clien 2,152 0 1 0 0.4
enqueue 80 0 1 7 0.0
db file parallel read 38 0 0 13 0.0
SQL*Net more data to client 24,272 0 0 0 4.1
log file single write 160 0 0 2 0.0
PX Deq: Table Q qref 62 0 0 4 0.0
control file sequential read 4,336 0 0 0 0.7
direct path read 45,818 0 0 0 7.7
library cache lock 1 0 0 76 0.0
direct path write 22,821 0 0 0 3.8
LGWR wait for redo copy 5,387 0 0 0 0.9
PX Deq Credit: need buffer 206 0 0 0 0.0
undo segment extension 40,035 40,028 0 0 6.7
db file parallel write 3,904 0 0 0 0.7
db file single write 40 0 0 0 0.0
log file sequential read 160 0 0 0 0.0
async disk IO 40 0 0 0 0.0
SQL*Net message from client 269,626 0 337,613 1252 45.1
pipe get 5,313 5,286 21,063 3964 0.9
queue messages 1,466 1,464 14,023 9566 0.2
wakeup time manager 239 238 6,738 28191 0.0
PX Idle Wait 3,475 3,042 6,026 1734 0.6
jobq slave wait 961 960 2,824 2939 0.2
PX Deq: Execution Msg 9,423 1 159 17 1.6
PX Deq: Table Q Normal 1,362 4 8 6 0.2
SQL*Net more data from clien 12,643 0 2 0 2.1
SQL*Net message to client 269,616 0 0 0 45.1
^LWait Events for DB: OAPROD Instance: OAPROD Snaps: 551 -553
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
^LBackground Wait Events for DB: OAPROD Instance: OAPROD Snaps: 551 -553
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
log file parallel write 21,251 0 39 2 3.6
latch free 407 353 13 32 0.1
control file parallel write 2,981 0 12 4 0.5
db file scattered read 3,464 0 5 1 0.6
db file sequential read 1,756 0 2 1 0.3
rdbms ipc reply 137 0 2 16 0.0
log file single write 160 0 0 2 0.0
enqueue 17 0 0 6 0.0
control file sequential read 3,346 0 0 0 0.6
LGWR wait for redo copy 5,387 0 0 0 0.9
db file parallel write 3,904 0 0 0 0.7
direct path read 1,833 0 0 0 0.3
direct path write 1,833 0 0 0 0.3
log file sequential read 160 0 0 0 0.0
buffer busy waits 21 0 0 0 0.0
rdbms ipc message 41,753 9,476 33,069 792 7.0
pmon timer 4,438 2,164 7,021 1582 0.7
smon timer 522 1 6,820 13064 0.1
^LSQL ordered by Gets for DB: OAPROD Instance: OAPROD Snaps: 551 -553
-> End Buffer Gets Threshold: 10000
-> Note that resources reported for PL/SQL includes the resources used by
all SQL statements called within the PL/SQL code. As individual SQL
statements are also reported, it is possible and valid for the summed
total % to exceed 100
CPU Elapsd
Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value
5,636,117,159 28 201,289,898.5 38.1 ######## ######### 835260576
Module: RAXTRX
INSERT INTO RA_INTERFACE_ERRORS (INTERFACE_LINE_ID, MESSAGE_
TEXT, INVALID_VALUE) SELECT INTERFACE_LINE_ID, :b_err_msg6, '
trx_number='||T.TRX_NUMBER||','||'customer_trx_id='||TL.CUSTOMER
_TRX_ID FROM RA_INTERFACE_LINES_GT IL, RA_CUSTOMER_TRX_LINES TL,
RA_CUSTOMER_TRX T WHERE IL.REQUEST_ID = :b1 AND IL.INTERFAC
511,626,129 20 25,581,306.5 3.5 8155.90 15045.56 205460129
Module: JDBC Thin Client
select TRANSACTION_BRANCH , BRANCH_ADDRESS_1 , BRANCH_ADDRESS_2
, BRANCH_PHONES , BRANCH_FAX , BRANCH_TIN , TRX_NUMBER , TRX_NU
MBER_DISP , TRX_DATE , TRX_DATE_DISP , BILL_TO_CUSTOMER_ID , BIL
L_TO_CUSTOMER_NAME , SHIP_DATE_ACTUAL_DISP , SHIP_TO_NAME , SHIP
_TO_ADDRESS1 , SHIP_TO_ADDRESS2 , SHIP_TO_ADDRESS3 , SHIP_TO_ADD
114,013,000 8,544 13,344.2 0.8 2919.47 4527.54 803969757
Module: JDBC Thin Client
SELECT COALESCE(DELV_ADDR.CITY, DELV_ADDR.PROVINCE) FROM APPS.OE
_ORDER_HEADERS_ALL OE_HEADERS , APPS.RA_SITE_USES_ALL DELV_SITE
, APPS.RA_ADDRESSES_ALL DELV_ADDR WHERE TO_CHAR(ORDER_NUMBER) =
:B1 AND DELV_SITE.SITE_USE_ID(+) = OE_HEADERS.DELIVER_TO_ORG_ID
AND DELV_ADDR.ADDRESS_ID(+) = DELV_SITE.ADDRESS_ID
113,972,897 8,550 13,330.2 0.8 2924.20 4550.22 3761220362
Module: JDBC Thin Client
SELECT DELV_ADDR.ADDRESS1 FROM APPS.OE_ORDER_HEADERS_ALL OE_HEAD
ERS , APPS.RA_SITE_USES_ALL DELV_SITE , APPS.RA_ADDRESSES_ALL DE
LV_ADDR WHERE TO_CHAR(ORDER_NUMBER) = :B1 AND DELV_SITE.SITE_USE
_ID(+) = OE_HEADERS.DELIVER_TO_ORG_ID AND DELV_ADDR.ADDRESS_ID(+
) = DELV_SITE.ADDRESS_ID
113,972,850 8,551 13,328.6 0.8 2919.13 4533.69 4273350236
Module: JDBC Thin Client...more... > http://www.datafilehost.com/download-60d3df1e.html
If downloading this file...
Uncheck [ ] Use our download manager and get recommended downloads
Your assistance is highly appreciated.
Thanks a lot,
JanHi hussein,
Is this normal locks for the apps table which is "3". Im selecting the locks even if there is no users in the system.
SELECT o.owner, o.object_name, o.object_type, o.last_ddl_time, o.status, l.session_id,
l.oracle_username, l.locked_mode
FROM dba_objects o, gv$locked_object l
WHERE o.object_id = l.object_id;
OWNER OBJECT_NAME OBJECT_TYPE LAST_DDL STATUS SESSION_ID ORACLE_USERNAME LOCKED_MODE
APPLSYS FND_CONCURRENT_QUEUES TABLE 24-APR-10 VALID 109 APPS 3
APPLSYS FND_CONCURRENT_REQUESTS TABLE 24-APR-10 VALID 25 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_T TABLE 24-APR-10 VALID 143 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_T TABLE 24-APR-10 VALID 139 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_T TABLE 24-APR-10 VALID 138 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_T TABLE 24-APR-10 VALID 131 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_T TABLE 24-APR-10 VALID 115 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_T TABLE 24-APR-10 VALID 103 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_T TABLE 24-APR-10 VALID 87 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_H TABLE 24-APR-10 VALID 143 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_H TABLE 24-APR-10 VALID 139 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_H TABLE 24-APR-10 VALID 138 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_H TABLE 24-APR-10 VALID 131 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_H TABLE 24-APR-10 VALID 115 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_H TABLE 24-APR-10 VALID 103 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_H TABLE 24-APR-10 VALID 87 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_I TABLE 24-APR-10 VALID 143 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_I TABLE 24-APR-10 VALID 139 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_I TABLE 24-APR-10 VALID 138 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_I TABLE 24-APR-10 VALID 131 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_I TABLE 24-APR-10 VALID 115 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_I TABLE 24-APR-10 VALID 103 APPS 3
APPLSYS AQ$_FND_CP_GSM_OPP_AQTBL_I TABLE 24-APR-10 VALID 87 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_T TABLE 24-APR-10 VALID 121 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_T TABLE 24-APR-10 VALID 104 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_T TABLE 24-APR-10 VALID 95 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_T TABLE 24-APR-10 VALID 75 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_T TABLE 24-APR-10 VALID 67 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_T TABLE 24-APR-10 VALID 66 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_T TABLE 24-APR-10 VALID 55 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_T TABLE 24-APR-10 VALID 54 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_T TABLE 24-APR-10 VALID 52 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_H TABLE 24-APR-10 VALID 121 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_H TABLE 24-APR-10 VALID 104 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_H TABLE 24-APR-10 VALID 95 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_H TABLE 24-APR-10 VALID 75 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_H TABLE 24-APR-10 VALID 67 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_H TABLE 24-APR-10 VALID 66 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_H TABLE 24-APR-10 VALID 55 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_H TABLE 24-APR-10 VALID 54 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_H TABLE 24-APR-10 VALID 52 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_I TABLE 24-APR-10 VALID 121 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_I TABLE 24-APR-10 VALID 104 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_I TABLE 24-APR-10 VALID 95 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_I TABLE 24-APR-10 VALID 75 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_I TABLE 24-APR-10 VALID 67 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_I TABLE 24-APR-10 VALID 66 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_I TABLE 24-APR-10 VALID 55 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_I TABLE 24-APR-10 VALID 54 APPS 3
APPLSYS AQ$_FND_CP_GSM_IPC_AQTBL_I TABLE 24-APR-10 VALID 52 APPS 3Thanks,
Edited by: yxes2013 on 21.1.2013 10:18 -
Parallel run of the same function from multiple jobs
Hello, everyone!
I have a function which accepts a date range, reads invoices from a partitioned by date table and writes output to a partitioned by invoice table. Each invoice can have records only with one date, so both tables may have one invoice only in one partition, i.e. partitions do not overlap. Function commits after processing each date. The whole process was running about 6 hrs with 46 million records in source table.
We are expecting source table to grow over 150 million rows, so we decided to split it into 3 parallel jobs and each job will process 1/3 of dates, and, as a result, 1/3 of invoices.
So, we call this function from 3 concurrent UNIX jobs and each job passes its own range of dates.
What we noticed, is that even if we run 3 jobs concurrently, they do not run this way! When 1st job ends after 2 hrs of run, the number of commited rows in the target table is equal to the number of rows inserted by this job. When 2nd job ends after 4 hrs of run, the number of rows in the target table is equal the summary of two jobs. And the 3rd job ends only after 6 hrs.
So, instead of improving a process by splitting it into 3 parallel jobs we ended up having 3 jobs instead of one with the same 6 hrs until target table is loaded.
My question is - How to make it work? It looks like Oracle 11g is smart enough to recognize, that all 3 jobs are calling the same function and execute this function only once at the time. I.e. it looks like only one copy of the function is loaded into the memory at the same even if it called by 3 different sessions.
The function itself has a very complicated logic, does a lot of verifications by joining to another tables and we do not want to maintain 3 copies of the same code under different names. And beside this, the plan is that if with 150 mln rows we will have a performance problem, then split it to more concurrent jobs, for example 6 or 8 jobs. Obviously we do not want to maintain so many copies of the same code by copying this function into another names.
I was monitoring jobs by quering V$SESSION and V$SQLAREA ROWS_PROCESSED and EXECUTIONS and I can see, that each job has its own set of SID's (i.e. runs up to 8 parallel processes), but number of commited rows is always eqal to the number of rows from the 1st job, then 2nd+1st, etc. So, it looks like all processes of 2nd and 3rd jobs are waiting until 1st one is done.
Any ideas?OK, this is my SQL and results (some output columns are ommited as irrelevant)
SELECT
TRIM ( SESS.OSUSER ) "OSUser"
, TRIM ( SESS.USERNAME ) "OraUser"
, NVL(TRIM(SESS.SCHEMANAME),'------') "Schema"
, SESS.AUDSID "AudSID"
, SESS.SID "SID"
, TO_CHAR(SESS.LOGON_TIME,'HH24:MI:SS') "Sess Strt"
, SUBSTR(SQLAREA.FIRST_LOAD_TIME,12) "Tran Strt"
, NUMTODSINTERVAL((SYSDATE-TO_DATE(SQLAREA.FIRST_LOAD_TIME,'yyyy-mm-dd hh24:mi:ss')),'DAY') "Tran Time"
, SQLAREA.EXECUTIONS "Execs"
, TO_CHAR(SQLAREA.ROWS_PROCESSED,'999,999,999') "Rows"
, TO_CHAR(TRAN.USED_UREC,'999,999,999') "Undo Rec"
, TO_CHAR(TRAN.USED_UBLK,'999,999,999') "Undo Blks"
, SQLAREA.SORTS "Sorts"
, SQLAREA.FETCHES "Fetches"
, SQLAREA.LOADS "Loads"
, SQLAREA.PARSE_CALLS "Parse Calls"
, TRIM ( SESS.PROGRAM ) "Program"
, SESS.SERIAL# "Serial#"
, TRAN.STATUS "Status"
, SESS.STATE "State"
, SESS.EVENT "Event"
, SESS.P1TEXT||' '||SESS.P1 "P1"
, SESS.P2TEXT||' '||SESS.P2 "P2"
, SESS.P3TEXT||' '||SESS.P3 "P3"
, SESS.WAIT_CLASS "Wait Class"
, NUMTODSINTERVAL(SESS.WAIT_TIME_MICRO/1000000,'SECOND') "Wait Time"
, NUMTODSINTERVAL(SQLAREA.CONCURRENCY_WAIT_TIME/1000000,'SECOND') "Wait Concurr"
, NUMTODSINTERVAL(SQLAREA.CLUSTER_WAIT_TIME/1000000,'SECOND') "Wait Cluster"
, NUMTODSINTERVAL(SQLAREA.USER_IO_WAIT_TIME/1000000,'SECOND') "Wait I/O"
, SESS.ROW_WAIT_FILE# "Row Wait File"
, SESS.ROW_WAIT_OBJ# "Row Wait Obj"
, SESS.USER# "User#"
, SESS.OWNERID "OwnerID"
, SESS.SCHEMA# "Schema#"
, TRIM ( SESS.PROCESS ) "Process"
, NUMTODSINTERVAL(SQLAREA.CPU_TIME/1000000,'SECOND') "CPU Time"
, NUMTODSINTERVAL(SQLAREA.ELAPSED_TIME/1000000,'SECOND') "Elapsed Time"
, SQLAREA.DISK_READS "Disk Reads"
, SQLAREA.DIRECT_WRITES "Direct Writes"
, SQLAREA.BUFFER_GETS "Buffers"
, SQLAREA.SHARABLE_MEM "Sharable Memory"
, SQLAREA.PERSISTENT_MEM "Persistent Memory"
, SQLAREA.RUNTIME_MEM "RunTime Memory"
, TRIM ( SESS.MACHINE ) "Machine"
, TRIM ( SESS.TERMINAL ) "Terminal"
, TRIM ( SESS.TYPE ) "Type"
, SQLAREA.MODULE "Module"
, SESS.SERVICE_NAME "Service name"
FROM V$SESSION SESS
INNER JOIN V$SQLAREA SQLAREA
ON SESS.SQL_ADDRESS = SQLAREA.ADDRESS
and UPPER(SESS.STATUS) = 'ACTIVE'
LEFT JOIN V$TRANSACTION TRAN
ON TRAN.ADDR = SESS.TADDR
ORDER BY SESS.OSUSER
,SESS.USERNAME
,SESS.AUDSID
,NVL(SESS.SCHEMANAME,' ')
,SESS.SID
AudSID SID Sess Strt Tran Strt Tran Time Execs Rows Undo Rec Undo Blks Sorts Fetches Loads Parse Calls Status State Event P1 P2 P3 Wait Class Wait Time Wait Concurr Wait Cluster Wait I/O Row Wait File Row Wait Obj Process CPU Time Elapsed Time Disk Reads Direct Writes Buffers Sharable Memory Persistent Memory RunTime Memory
409585 272 22:15:36 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITED SHORT TIME PX Deq: Execute Reply sleeptime/senderid 200 passes 2 0 Idle 0 0:0:0.436000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 7 21777 22739 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409585 203 22:30:01 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.9674000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 25 124730 4180 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409585 210 22:30:01 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.11714000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 24 124730 22854 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409585 231 22:30:01 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.4623000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 46 21451 4178 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409585 243 22:30:01 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITED SHORT TIME PX qref latch function 154 sleeptime 13835058061074451432 qref 0 Other 0 0:0:0.4000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 35 21451 3550 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409585 252 22:30:01 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.19815000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 49 21451 22860 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409585 273 22:30:01 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.11621000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 22 124730 4182 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409585 277 22:30:01 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING db file parallel read files 20 blocks 125 requests 125 User I/O 0 0:0:0.242651000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 39 21451 4184 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409585 283 22:30:01 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.2781000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 42 21451 3552 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409585 295 22:30:01 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.24424000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 40 21451 22862 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409585 311 22:30:01 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.15788000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 31 21451 22856 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409586 242 22:15:36 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITED KNOWN TIME PX Deq: Execute Reply sleeptime/senderid 200 passes 1 0 Idle 0 0:0:0.522344000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 28 137723 22736 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409586 192 22:29:20 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.14334000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 31 21462 4202 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409586 222 22:29:20 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.16694000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 37 21462 4194 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409586 233 22:29:20 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.7731000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 44 21462 4198 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409586 253 22:29:20 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING db file parallel read files 21 blocks 125 requests 125 User I/O 0 0:0:0.792518000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 39 21462 4204 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409586 259 22:29:20 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.2961000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 35 21462 4196 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409586 291 22:29:20 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq Credit: send blkd sleeptime/senderid 268566527 passes 1 qref 0 Idle 0 0:0:0.9548000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 35 21462 4200 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409587 236 22:15:36 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq: Table Q Normal sleeptime/senderid 200 passes 2 0 Idle 0 0:0:0.91548000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 25 124870 22831 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409587 207 22:30:30 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq: Execution Msg sleeptime/senderid 268566527 passes 3 0 Idle 0 0:0:0.644662000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 43 21423 4208 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409587 241 22:30:30 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING PX Deq: Execution Msg sleeptime/senderid 268566527 passes 3 0 Idle 0 0:0:0.644594000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 47 21423 4192 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448
409587 297 22:30:30 22:15:36 0 0:14:52.999999999 302 383,521 305 0 1 3598 WAITING db file parallel read files 20 blocks 109 requests 109 User I/O 0 0:0:0.793261000 0 0:0:1.124995000 0 0:0:0.0 0 1:56:15.227863000 12 21316 4206 0 0:25:25.760000000 0 2:17:1.815044000 526959 0 25612732 277567 56344 55448Here I found one interesting query http://www.pythian.com/news/922/recent-spike-report-from-vactive_session_history-ash/
But it does not help me -
Any general tips on getting better performance out of multi table insert?
I have been struggling with coding a multi table insert which is the first time I ever use one and my Oracle skills are pretty poor in general so now that the query is built and works fine I am sad to see its quite slow.
I have checked numerous articles on optimizing but the things I try dont seem to get me much better performance.
First let me describe my scenario to see if you agree that my performance is slow...
its an insert all command, which ends up inserting into 5 separate tables, conditionally (at least 4 inserts, sometimes 5 but the fifth is the smallest table). Some stats on these tables as follows:
Source table: 5.3M rows, ~150 columns wide. Parallel degree 4. everything else default.
Target table 1: 0 rows, 27 columns wide. Parallel 4. everything else default.
Target table 2: 0 rows, 63 columns wide. Parallel 4. default.
Target table 3: 0 rows, 33 columns wide. Parallel 4. default.
Target table 4: 0 rows, 9 columns wide. Parallel 4. default.
Target table 5: 0 rows, 13 columns wide. Parallel 4. default.
The parallelism is just about the only customization I myself have done. Why 4? I dont know it's pretty arbitrary to be honest.
Indexes?
Table 1 has 3 index + PK.
Table 2 has 0 index + FK + PK.
Table 3 has 4 index + FK + PK
Table 4 has 3 index + FK + PK
Table 5 has 4 index + FK + PK
None of the indexes are anything crazy, maybe 3 or 4 of all of them are on multiple columns, 2-3 max. The rest are on single columns.
The query itself looks something like this:
insert /*+ append */ all
when 1=1 then
into table1 (...) values (...)
into table2 (...) values (...)
when a=b then
into table3 (...) values (...)
when a=c then
into table3 (...) values (...)
when p=q then
into table4(...) values (...)
when x=y then
into table5(...) values (...)
select .... from source_table
Hints I tried are with append, without append, and parallel (though adding parallel seemed to make the query behave in serial, according to my session browser).
Now for the performance:
It does about 8,000 rows per minute on table1. So that means it should also have that much in table2, table3 and table4, and then a subset of that in table5.
Does that seem normal or am I expecting too much?
I find articles talking about millions of rows per minute... Obviously i dont think I can achieve that much... but maybe 30k or so on each table is a reasonable goal?
If it seems my performance is slow, what else do you think I should try? Is there any information I may try to get to see if maybe its a poorly configured database for this?
P.S. Is it possible I can run this so that it commits every x rows or something? I had the heartbreaking event of a network issue giving me this sudden "ora-25402: transaction must roll back" after it was running for 3.5 hours. So I lost all the progress it made... have to start over. plus i wonder if the sheer amount of data being queued for commit/rollback is causing some of the problem?
Edited by: trant on Jun 27, 2011 9:29 PMLooks like there are about 54 sessions on my database, 7 of the sessions belong to me (2 taken by TOAD and 4 by my parallel slave sessions and 1 by the master of those 4)
In v$session_event there are 546 rows, if i filter it to the SIDs of my current session and order my micro_wait_time desc:
510 events in waitclass Other 30670 9161 329759 10.75 196 3297590639 1736664284 1893977003 0 Other
512 events in waitclass Other 32428 10920 329728 10.17 196 3297276553 1736664284 1893977003 0 Other
243 events in waitclass Other 21513 5 329594 15.32 196 3295935977 1736664284 1893977003 0 Other
223 events in waitclass Other 21570 52 329590 15.28 196 3295898897 1736664284 1893977003 0 Other
241 row cache lock 1273669 0 42137 0.03 267 421374408 1714089451 3875070507 4 Concurrency
241 events in waitclass Other 614793 0 34266 0.06 12 342660764 1736664284 1893977003 0 Other
241 db file sequential read 13323 0 3948 0.3 13 39475015 2652584166 1740759767 8 User I/O
241 SQL*Net message from client 7 0 1608 229.65 1566 16075283 1421975091 2723168908 6 Idle
241 log file switch completion 83 0 459 5.54 73 4594763 3834950329 3290255840 2 Configuration
241 gc current grant 2-way 5023 0 159 0.03 0 1591377 2685450749 3871361733 11 Cluster
241 os thread startup 4 0 55 13.82 26 552895 86156091 3875070507 4 Concurrency
241 enq: HW - contention 574 0 38 0.07 0 378395 1645217925 3290255840 2 Configuration
512 PX Deq: Execution Msg 3 0 28 9.45 28 283374 98582416 2723168908 6 Idle
243 PX Deq: Execution Msg 3 0 27 9.1 27 272983 98582416 2723168908 6 Idle
223 PX Deq: Execution Msg 3 0 25 8.26 24 247673 98582416 2723168908 6 Idle
510 PX Deq: Execution Msg 3 0 24 7.86 23 235777 98582416 2723168908 6 Idle
243 PX Deq Credit: need buffer 1 0 17 17.2 17 171964 2267953574 2723168908 6 Idle
223 PX Deq Credit: need buffer 1 0 16 15.92 16 159230 2267953574 2723168908 6 Idle
512 PX Deq Credit: need buffer 1 0 16 15.84 16 158420 2267953574 2723168908 6 Idle
510 direct path read 360 0 15 0.04 4 153411 3926164927 1740759767 8 User I/O
243 direct path read 352 0 13 0.04 6 134188 3926164927 1740759767 8 User I/O
223 direct path read 359 0 13 0.04 5 129859 3926164927 1740759767 8 User I/O
241 PX Deq: Execute Reply 6 0 13 2.12 10 127246 2599037852 2723168908 6 Idle
510 PX Deq Credit: need buffer 1 0 12 12.28 12 122777 2267953574 2723168908 6 Idle
512 direct path read 351 0 12 0.03 5 121579 3926164927 1740759767 8 User I/O
241 PX Deq: Parse Reply 7 0 9 1.28 6 89348 4255662421 2723168908 6 Idle
241 SQL*Net break/reset to client 2 0 6 2.91 6 58253 1963888671 4217450380 1 Application
241 log file sync 1 0 5 5.14 5 51417 1328744198 3386400367 5 Commit
510 cursor: pin S wait on X 3 2 2 0.83 1 24922 1729366244 3875070507 4 Concurrency
512 cursor: pin S wait on X 2 2 2 1.07 1 21407 1729366244 3875070507 4 Concurrency
243 cursor: pin S wait on X 2 2 2 1.06 1 21251 1729366244 3875070507 4 Concurrency
241 library cache lock 29 0 1 0.05 0 13228 916468430 3875070507 4 Concurrency
241 PX Deq: Join ACK 4 0 0 0.07 0 2789 4205438796 2723168908 6 Idle
241 SQL*Net more data from client 6 0 0 0.04 0 2474 3530226808 2000153315 7 Network
241 gc current block 2-way 5 0 0 0.04 0 2090 111015833 3871361733 11 Cluster
241 enq: KO - fast object checkpoint 4 0 0 0.04 0 1735 4205197519 4217450380 1 Application
241 gc current grant busy 4 0 0 0.03 0 1337 2277737081 3871361733 11 Cluster
241 gc cr block 2-way 1 0 0 0.06 0 586 737661873 3871361733 11 Cluster
223 db file sequential read 1 0 0 0.05 0 461 2652584166 1740759767 8 User I/O
223 gc current block 2-way 1 0 0 0.05 0 452 111015833 3871361733 11 Cluster
241 latch: row cache objects 2 0 0 0.02 0 434 1117386924 3875070507 4 Concurrency
241 enq: TM - contention 1 0 0 0.04 0 379 668627480 4217450380 1 Application
512 PX Deq: Msg Fragment 4 0 0 0.01 0 269 77145095 2723168908 6 Idle
241 latch: library cache 3 0 0 0.01 0 243 589947255 3875070507 4 Concurrency
510 PX Deq: Msg Fragment 3 0 0 0.01 0 215 77145095 2723168908 6 Idle
223 PX Deq: Msg Fragment 4 0 0 0 0 145 77145095 2723168908 6 Idle
241 buffer busy waits 1 0 0 0.01 0 142 2161531084 3875070507 4 Concurrency
243 PX Deq: Msg Fragment 2 0 0 0 0 84 77145095 2723168908 6 Idle
241 latch: cache buffers chains 4 0 0 0 0 73 2779959231 3875070507 4 Concurrency
241 SQL*Net message to client 7 0 0 0 0 51 2067390145 2000153315 7 Network
(yikes, is there a way to wrap that in equivalent of other forums' tag?)
v$session_wait;
223 835 PX Deq Credit: send blkd sleeptime/senderid 268697599 000000001003FFFF passes 1 0000000000000001 qref 0 00 1893977003 0 Other 0 10 WAITING
241 22819 row cache lock cache id 13 000000000000000D mode 0 00 request 5 0000000000000005 3875070507 4 Concurrency -1 0 WAITED SHORT TIME
243 747 PX Deq Credit: send blkd sleeptime/senderid 268697599 000000001003FFFF passes 1 0000000000000001 qref 0 00 1893977003 0 Other 0 7 WAITING
510 10729 PX Deq Credit: send blkd sleeptime/senderid 268697599 000000001003FFFF passes 1 0000000000000001 qref 0 00 1893977003 0 Other 0 2 WAITING
512 12718 PX Deq Credit: send blkd sleeptime/senderid 268697599 000000001003FFFF passes 1 0000000000000001 qref 0 00 1893977003 0 Other 0 4 WAITING
v$sess_io:
223 0 5779 5741 0 0
241 38773810 2544298 15107 27274891 0
243 0 5702 5688 0 0
510 0 5729 5724 0 0
512 0 5682 5678 0 0 -
Hi Gurus,
we are inserting data into one table and using append and parallel hints...but that query is not completed yet and those sessions are in blocked list.
can you please let me know how to overcome this issue and below is the query:
INSERT /*+ APPEND PARALLEL(emp 8) */ INTO emp_temp
 (empno,empname,dateofjoining,deptno,sal)
 SELECT /*+ PARALLEL(empn 8) */
        empno,empname,dateofjoining,deptno,sum(sal) sal
 FROM emp_V;
is it correct usage or anyother best way for improving performance.Thanks for the response.
Below is the issue details more detailed:
We have created table with parallel degree 8 and I am not able to insert data into that table and the sessions are blocked state.
I am seeing event names in blocked state are px deq: parse reply px deq: execution msg
Temp_table -- parallel degree8
I am selecting the data from multiple tables as per my requirement and trying to insert that data into the temp table.
Below is the query:
Insert into /*+PARALLEL(Temp_table)*/ Temp_table(report_date,quantity,revenue)
select /*+PARALLEL(Temp_table_v)*/ report_date,quantity,revenue from Temp_table_v;
In Temp_table_v we are joined the multiple tables and those tables are having millions of records and in that view also we have added parallel hints.
Please help. -
Here's is the contents of v$system_event..
Is this
EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT
enq: TX - index contention 40564851 214701526 5.29
enq: TX - row lock contention 188846 12454614 65.95
enq: SQ - contention 141971 70568 0.5
cause for concern?
EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT
SQL*Net message to client 6015051449 607254 0
SQL*Net message from client 6015048542 178177969892 29.62
gcs remote message 2948555287 2633481757 0.89
CGS wait for IPC msg 1517805027 634397 0
db file sequential read 1500615188 816364485 0.54
ges remote message 1247679701 1407300224 1.13
gc cr multi block request 778432813 9913464 0.01
gc current block 2-way 747852637 38030616 0.05
db file scattered read 709428365 460939295 0.65
rdbms ipc message 708473316 37650068633 53.14
gc buffer busy acquire 671285134 1033621285 1.54
PX Deq: reap credit 667784615 484449 0
gcs log flush sync 592376026 171712257 0.29
gc cr block 2-way 530861847 19607062 0.04
library cache pin 437937120 15126237 0.03
log file sync 379523272 797193932 2.1
DIAG idle wait 359607166 2822108755 7.85
log file parallel write 351225436 259263769 0.74
LNS ASYNC end of log 350170653 1398410516 3.99
LNS wait on SENDREQ 321652621 3209301 0.01
PX qref latch 297396661 94308 0
read by other session 289140108 148440270 0.51
buffer deadlock 163505781 983055 0.01
gc current block busy 119223825 467716658 3.92
PX Deq: Table Q Normal 117332841 23574867 0.2
ksxr poll remote instances 110480324 90333 0
buffer busy waits 106938153 19933900 0.19
direct path read 93429599 108427028 1.16
SQL*Net more data from client 86471785 23026529 0.27
gc current grant busy 84978157 28215346 0.33
control file sequential read 82646297 23694583 0.29
PX Deq Credit: send blkd 78641669 9569299 0.12
latch: cache buffers chains 74218671 690277 0.01
gc current grant 2-way 72557796 1920419 0.03
library cache: mutex X 71106697 75993 0
DFS lock handle 70722498 2716407 0.04
gc cr grant 2-way 64558237 1633004 0.03
PX Deq: Execution Msg 61706261 314222076 5.09
gc cr block busy 61469863 119850802 1.95
library cache lock 52428649 3773354 0.07
PX Deq: Slave Session Stats 48040224 1886805 0.04
db file parallel read 46415188 118467902 2.55
IPC send completion sync 46250594 965101 0.02
enq: TX - index contention 40564851 214701526 5.29
PX Deq: Execute Reply 39689685 17243721 0.43
gc buffer busy release 36976909 242714774 6.56
SQL*Net more data to client 36627952 44167 0
PX Deq: Msg Fragment 30501244 343397 0.01
rdbms ipc reply 29725302 1352370 0.05
RMAN backup & recovery I/O 28824547 37722662 1.31
reliable message 27892263 3082134 0.11
PX Idle Wait 27356097 4651277341 170.03
ASM file metadata operation 25098749 8850323 0.35
gc object scan 22705857 7485 0
db file parallel write 19896252 52152606 2.62
latch: ges resource hash list 19336183 427451 0.02
enq: PS - contention 19143961 707455 0.04
PX Deq: Parse Reply 19093356 895799 0.05
gc cr disk read 17816846 448909 0.03
ASM background timer 16101806 1383957874 85.95
PX Deq: Slave Join Frag 16044789 233149 0.01
wait for unread message on broadcast channel 15056320 1413552546 93.88
cursor: mutex X 13435193 24140 0
KJC: Wait for msg sends to complete 13268497 11397 0
PX Deq: Signal ACK RSG 13214824 101941 0.01
KSV master wait 13206286 4235645 0.32
direct path read temp 12617694 5487608 0.43
PX Deq Credit: need buffer 11675868 879967 0.08
row cache lock 11536185 398216 0.03
PX Deq Credit: Session Stats 9480862 78910 0.01
SQL*Net message to dblink 9312894 1538 0
SQL*Net message from dblink 9312894 6279631 0.67
control file parallel write 7760982 11854435 1.53
pmon timer 7558889 1412576090 186.88
PX Deq: Join ACK 7548816 498931 0.07
gc current multi block request 6035173 155898 0.03
PING 5706961 1413230267 247.63
enq: XR - database force logging 4662671 198813 0.04
class slave wait 4561877 7097429006 1555.81
Streams AQ: waiting for messages in the queue 4495828 1543411682 343.3
SQL*Net more data from dblink 3696582 444575 0.12
LGWR wait for redo copy 3655353 17840 0
log file sequential read 3387305 6610414 1.95
Log archive I/O 2990486 276772 0.09
SQL*Net break/reset to client 2971976 2385935 0.8
direct path write temp 2839390 2522114 0.89
Space Manager: slave idle wait 2827526 1412987186 499.73
latch: shared pool 2808517 298150 0.11
latch: gc element 2421717 24688 0.01
SGA: MMAN sleep for component shrink 2336447 2458094 1.05
latch: enqueue hash chains 2279645 15435 0.01
latch free 2089418 78732 0.04
gc current split 2044784 1864009 0.91
PX Deq: Signal ACK EXT 1976164 19263 0.01
enq: FB - contention 1473469 61036 0.04
cursor: pin S wait on X 1313129 1464789 1.12
SQL*Net more data to dblink 1232891 986 0
Streams AQ: RAC qmn coordinator idle wait 1211300 788 0
enq: HW - contention 1175390 2077008 1.77
latch: session allocation 1167768 21883 0.02
Streams AQ: qmn coordinator idle wait 1144699 1412546634 1233.99
Streams AQ: qmn slave idle wait 1031585 2227183681 2158.99
lock deadlock retry 962937 4698 0
enq: CF - contention 956154 609647 0.64
latch: cache buffers lru chain 902764 37552 0.04
latch: object queue header operation 817911 27717 0.03
global enqueue expand wait 768633 654105 0.85
Data file init write 756191 329758 0.44
latch: gcs resource hash 647021 4147 0.01
local write wait 603007 286191 0.47
latch: row cache objects 599358 6453 0.01
ges lmd/lmses to freeze in rcfg - mrcvr 481759 156345 0.32
shared server idle wait 471190 1413238589 2999.3
enq: RF - DG Broker Current File ID 469833 23209 0.05
smon timer 432383 1411851085 3265.28
SGA: allocation forcing component growth 363333 379008 1.04
gc current retry 341104 1121252 3.29
enq: RF - synch: DG Broker metadata 319143 588290 1.84
enq: PG - contention 313659 14830 0.05
enq: TT - contention 260134 11207172 43.08
enq: KO - fast object checkpoint 236745 820808 3.47
dispatcher timer 236637 1413242481 5972.2
direct path write 231382 191008 0.83
cursor: pin S 229011 394 0
Streams AQ: waiting for time management or cleanup tasks 199981 1413148548 7066.41
enq: TX - row lock contention 188846 12454614 65.95
enq: TX - allocate ITL entry 153703 54252 0.35
enq: SQ - contention 141971 70568 0.5
ksdxexeother 141885 56 0
latch: redo allocation 138912 1858 0.01
recovery area: computing applied logs 126415 45925 0.36
gc current block congested 126318 21768 0.17
resmgr:cpu quantum 123074 151384 1.23
jobq slave wait 120678 35574221 294.79
Datapump dump file I/O 90431 9127 0.1
ges inquiry response 89402 4041 0.05
os thread startup 83809 222586 2.66
cr request retry 80062 71896 0.9
PX Deq: Table Q Sample 79665 133402 1.67
gc cr block congested 79026 14792 0.19
gc cr failure 77521 25019 0.32
enq: WF - contention 73983 825388 11.16
enq: TQ - TM contention 72871 3319 0.05
lock escalate retry 65714 1574 0.02
buffer exterminate 59775 64919 1.09
fbar timer 47136 1413183353 29980.98
log file switch completion 46911 452097 9.64
recovery area: computing obsolete files 45699 8547 0.19
enq: US - contention 40401 8805 0.22
enq: TM - contention 39149 5435032 138.83
library cache load lock 36311 382575 10.54
kjbdrmcvtq lmon drm quiesce: ping completion 31668 47443 1.5
enq: TD - KTF dump entries 31468 1424 0.05
enq: RO - fast object reuse 28422 31772 1.12
parallel recovery slave wait for change 27558 3163 0.11
name-service call wait 23694 181533 7.66
control file single write 22375 1624 0.07
kksfbc child completion 21239 106926 5.03
PX Deq: Table Q qref 19325 245 0.01
enq: TX - contention 18805 113253 6.02
latch: messages 17203 181 0.01
enq: RS - prevent file delete 16913 1013 0.06
enq: RS - prevent aging list update 15682 642 0.04
PX Deq: Table Q Get Keys 14322 42935 3
gc current grant congested 14292 2192 0.15
cursor: mutex S 13285 8 0
log file single write 13164 5371 0.41
latch: undo global data 12649 178 0.01
kksfbc research 11894 12680 1.07
parallel recovery slave idle wait 11193 5872 0.52
wait list latch free 11026 11794 1.07
enq: CT - state 11001 417 0.04
latch: checkpoint queue latch 10526 132 0.01
enq: PE - contention 10506 1139 0.11
ARCH wait on SENDREQ 9957 216480 21.74
gc cr grant congested 9465 1584 0.17
wait for scn ack 9377 3155 0.34
enq: TA - contention 8856 324 0.04
log buffer space 8777 89323 10.18
enq: TK - Auto Task Serialization 8542 343 0.04
enq: DR - contention 7842 323 0.04
process diagnostic dump 7707 2072 0.27
JOX Jit Process Sleep 7612 11286431 1482.72
enq: TC - contention 7357 340817 46.33
ges global resource directory to be frozen 7140 12299 1.72
enq: CO - master slave det 6850 312 0.05
enq: JS - job run lock - synchronize 6704 397 0.06
gcs drm freeze in enter server mode 6542 40742 6.23
enq: TS - contention 5959 89332 14.99
ARCH wait for archivelog lock 5600 36 0.01
PX Nsq: PQ load info query 5377 104798 19.49
db file single write 5373 3452 0.64
gc remaster 5315 50625 9.52
latch: parallel query alloc buffer 4939 1906 0.39
enq: TO - contention 4799 143 0.03
enq: AF - task serialization 4395 161 0.04
enq: PI - contention 4251 163 0.04
ges2 LMON to wake up LMD - mrcvr 4210 28 0.01
enq: DL - contention 3889 239 0.06
kjctssqmg: quick message send wait 3408 22 0.01
LNS wait on DETACH 3275 741 0.23
ksfd: async disk IO 3274 1 0
LNS wait on ATTACH 3273 51940 15.87
ARCH wait on DETACH 3231 714 0.22
ARCH wait on ATTACH 3226 43238 13.4
enq: BR - file shrink 2787 116 0.04
write complete waits 2631 1070 0.41
enq: MD - contention 2596 67 0.03
enq: WL - contention 2198 266518 121.25
single-task message 2098 25896 12.34
enq: OD - Serializing DDLs 2054 66 0.03
resmgr:internal state change 2001 14735 7.36
ARCH wait on c/f tx acquire 2 1751 175230 100.07
enq: WR - contention 1636 69 0.04
latch: cache buffer handles 1610 29 0.02
statement suspended, wait error to be cleared 1497 22626 15.11
Streams AQ: qmn coordinator waiting for slave to start 1214 678966 559.28
enq: PD - contention 1182 33 0.03
JS kgl get object wait 1096 4922 4.49
undo segment extension 1070 10065 9.41
PL/SQL lock timer 949 8739819 9209.5
enq: AE - lock 937 28 0.03
LGWR-LNS wait on channel 832 913 1.1
ges DFS hang analysis phase 2 acks 816 495 0.61
latch: redo writing 729 9 0.01
gc quiesce 665 564 0.85
enq: JS - queue lock 482 2111 4.38
PX Deq: Test for credit 442 13 0.03
enq: SS - contention 386 274 0.71
recovery area: computing dropped files 328 1400 4.27
recovery area: computing backed up files 328 496 1.51
ksdxexeotherwait 279 10592 37.97
log switch/archive 250 137570 550.28
gc domain validation 223 39964 179.21
auto-sqltune: wait graph update 195 96514 494.95
wait for a undo record 170 1214 7.14
parallel recovery coord send blocked 168 4 0.02
enq: JS - wdw op 168 3741 22.27
enq: KT - contention 165 5 0.03
switch logfile command 156 6290 40.32
gcs resource directory to be unfrozen 149 12839 86.17
Data Guard Broker Wait 139 10906 78.46
enq: SK - contention 129 4 0.03
enq: JS - job recov lock 128 4 0.03
gc cr block lost 125 6772 54.17
virtual circuit wait 122 3 0.03
ges LMON to get to FTDONE 100 187 1.87
enq: CU - contention 80 242 3.02
enq: JQ - contention 78 7 0.09
cursor: pin X 73 83 1.14
parallel recovery coord wait for reply 70 510 7.29
PX Deq: Txn Recovery Start 67 3436 51.29
SQL*Net break/reset to dblink 60 0 0
gc current block lost 57 2869 50.33
ges LMD suspend for testing event 51 709 13.89
inactive session 46 4550 98.91
recovery read 45 5 0.11
JS kill job wait 41 3548 86.53
enq: AS - service activation 40 1 0.03
enq: TL - contention 35 2 0.05
enq: UL - contention 34 524 15.42
gcs enter server mode 33 1559 47.24
wait for stopper event to be increased 30 218 7.27
enq: TQ - DDL contention 24 300 12.52
enq: MR - contention 21 1 0.03
ges reconfiguration to start 20 54 2.72
ges enter server mode 20 502 25.08
buffer latch 18 1337 74.26
enq: SR - contention 18 1 0.05
Streams: RAC waiting for inter instance ack 18 3748 208.21
enq: PR - contention 17 46 2.72
kupp process wait 16 166 10.39
checkpoint completed 15 3678 245.19
PX Deque wait 14 68 4.87
enq: BF - allocation contention 14 1 0.08
enq: XL - fault extent map 14 51 3.66
enq: FU - contention 14 17 1.18
enq: TH - metric threshold evaluation 13 114 8.78
enq: MW - contention 12 0 0.04
enq: DD - contention 10 0 0.04
process terminate 8 41 5.08
ges cgs registration 8 151 18.9
buffer resize 7 0 0
ktm: instance recovery 7 698 99.66
LNS wait on LGWR 6 0 0
ASM background starting 6 381 63.43
gc cr block 3-way 5 0 0.08
enq: PV - syncstart 5 9 1.74
Global transaction acquire instance locks 4 4 1.09
enq: RS - read alert level 4 0 0.04
LGWR wait on LNS 3 0 0
gc recovery 3 540 179.85
Streams AQ: enqueue blocked on low memory 3 544 181.2
DBWR range invalidation sync 3 17 5.83
enq: DM - contention 3 0 0.03
enq: RF - FSFO Observer Heartbeat 3 0 0.03
enq: JS - q mem clnup lck 3 0 0
DG Broker configuration file I/O 2 0 0
enq: RC - Result Cache: Contention 2 493 246.6
enq: KM - contention 2 0 0.03
enq: RT - contention 2 0 0.04
instance state change 2 0 0.12
kkdlgon 2 10 5.11
enq: TQ - INI contention 2 292 146.07
enq: JS - contention 2 0 0
ARCH wait for netserver start 1 400 400.02
log file switch (checkpoint incomplete) 1 3 3.44
JS coord start wait 1 50 50.09
ges lmd and pmon to attach 1 1 1.26
wait for tmc2 to complete 1 3 3.03
control file heartbeat 1 400 400.02
enq: SW - contention 1 0 0.04
enq: PW - perwarm status in dbw0 1 0 0.09
enq: FS - contention 1 0 0.04
enq: XR - quiesce database 1 0 0.04
enq: RS - write alert level 1 0 0.02
enq: CN - race with init 1 0 0.03
enq: FE - contention 1 4 3.77
Wait for shrink lock2 1 10 10.03
enq: IA - contention 1 0 0.02
enq: RF - atomicity 1 0 0.05
enq: RF - synchronization: aifo master 1 0 0.02
enq: RF - RF - Database Automatic Disable 1 0 0.06
enq: WP - contention 1 0 0.02
enq: TB - SQL Tuning Base Cache Load 1 0 0.05
enq: JS - evt notify 1 0 0.02Edited by: steffi on Mar 20, 2011 12:21 AM
Edited by: steffi on Mar 20, 2011 8:18 AM
Edited by: steffi on Mar 20, 2011 8:19 AMText can be formatted by tagging the beginning and end of the block of text with the code ta
\Formatted text goes here.
\Example:
This is formatted.When cutting and pasting text such as execution plans, excerpts from AWR reports, etc, it will maintain spacing and formatting, and make for much easier reading.
As to the content, well, dumping the contents of v$system_event is pretty close to useless.
As to the first three events you listed, 'enq: TX - index contention', 'enq: TX - row lock contention', 'enq: SQ - contention', well, all of those are easily tunable.
First, for 'enq: SQ - contention', check your sequences. Do you have any NOCACHE sequences? Or sequences with small caches?
As for 'enq: TX - row lock contention', well that's fairly self-explanatory. You have multiple sessions trying to lock the same row in the same table at the same time.
Last, 'enq: TX - index contention', that's non-row level contention on an index. For example, if you have a unique index, insert a row w/ column value 1, but don't commit, then try to insert that same value from another session.
But, before you do any of that, I think you need to clearly understand where the bottlenecks are. Try taking some AWR snapshots, about 5 minutes apart, when you're having performance problems. Look at the AWR report for that 5 minute period. In particular, look at your Top 5 timed events.
Hope that helps,
-Mark -
Need clarify some fields in v$session
On 11g2/Redhat 5, my query returnsselect v.inst_id,sid,serial#,wait_class,event,status,last_call_et --,cast(Q.SQL_FULLTEXT as varchar(4000)) cur_sql
from gv$session v join gv$sqlarea Q on v.SQL_ID = Q.SQL_ID and v.INST_ID=Q.INST_ID
INST_ID SID SERIAL# WAIT_CLASS EVENT STATUS LAST_CALL_ET
1 16 854 Other PX Deq: reap credit ACTIVE 0
2 218 152 Idle PX Deq: Execution Msg ACTIVE 0
2 592 217 Idle PX Deq: Execute Reply ACTIVE 0
3 604 47 Other PX Deq: reap credit ACTIVE 0
1 592 163 Idle PX Deq: Execution Msg INACTIVE 6181
3 400 15 Idle SQL*Net message from client INACTIVE 7321
1 598 510 Idle SQL*Net message from client INACTIVE 272
2 16 66 Idle PX Deq: Execution Msg INACTIVE 6181
3 217 100 Idle PX Deq: Execution Msg INACTIVE 713
3 16 52 Idle PX Deq: Execution Msg INACTIVE 3882 I know that these session are all SQL Developer sessions and are currently doing nothing. But if I know mothing about the sessions, how can I tell from this query which session is true INACTIVE or INACTIVE. The first 4 ACTIVE sessions are NOT executing SQL.
My task is not to kill the INACTIVE session but is to look at those true active sessions and figure out which of them are causing performance problem. V$SESSION just return too many sessions, so that I want filter out the true inactive ones.
Thanks.figure out which of them are causing performance problemYou could query V$SESSTAT to determine activity.
Note that V$SESSTAT information for a session is cleared when the session is closed -- e.g. by a logout. Another session may be allocated the same SID when it does a login. Also remember that if the application maintains persistent sessions you might be looking at cumulative statistics over a period of days/weeks/months for such sessions. Therefore, you could write you own routines to poll V$SESSTAT at specific intervals and identify activity from changes in statistic values, adjusting for the fact that sessions may be opening and closing frequently (and may be reusing SIDs).
Hemant K Chitale -
Automatic DOP take more time to execute query
We upgraded database to oracle 11gR2. While testing Automatic DOP feature with our existing query it takes more time than with parallel.
Note: No constrains or Index created on table to gain performance while loading data (5000records / sec)
Os : Sun Solaris 64bit
CPU = 8
RAM = 7456M
Default parameter settings:
parallel_degree_policy string MANUAL
parallel_degree_limit string CPU
parallel_threads_per_cpu integer 2
arallel_degree_limit string CPU
cpu_count integer 8
parallel_threads_per_cpu integer 2
resource_manager_cpu_allocation integer 8
Query:
SELECT COUNT(*)
from (
SELECT
/*+ FIRST_ROWS(50), PARALLEL */
Query gets executed in 22minutes : execution plan
COUNT(*)
9600
Elapsed: 00:22:10.71
Execution Plan
Plan hash value: 3765539975
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop |
| 0 | SELECT STATEMENT | | 1 | 21 | 2164K (1)| 07:12:52 | | |
| 1 | SORT AGGREGATE | | 1 | 21 | | | | |
| 2 | PARTITION RANGE OR| | 89030 | 1825K| 2164K (1)| 07:12:52 |KEY(OR)|KEY(OR)|
|* 3 | TABLE ACCESS FULL| SUBSCRIBER_EVENT | 89030 | 1825K| 2164K (1)| 07:12:52 |KEY(OR)|KEY(OR)|Automatic DOP Query: parameters set
alter session set PARALLEL_DEGREE_POLICY = limited;
alter session force parallel query ;Query:
SELECT COUNT(*)
from (
SELECT /*+ FIRST_ROWS(50), PARALLEL*/
This query takes more than 2hrs to execute
COUNT(*)
9600
Elapsed: 02:07:48.81
Execution Plan
Plan hash value: 127536830
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart|Pstop | TQ |IN-OUT| PQ Distrib |
| 0 | SELECT STATEMENT | | 1 | 21 | 150K (1)| 00:30:01 | | | | | |
| 1 | SORT AGGREGATE | | 1 | 21 | | | | | | | |
| 2 | PX COORDINATOR | | | | | | | | | | |
| 3 | PX SEND QC (RANDOM) | :TQ10000 | 1 | 21 | | | | | Q1,00 | P->S | QC (RAND) |
| 4 | SORT AGGREGATE | | 1 | 21 | | | | | Q1,00 | PCWP | |
| 5 | PX BLOCK ITERATOR | | 89030 | 1825K| 150K (1)| 00:30:01 |KEY(OR)|KEY(OR)| Q1,00 | PCWC | |
|* 6 | TABLE ACCESS FULL| SUBSCRIBER_EVENT | 89030 | 1825K| 150K (1)| 00:30:01 |KEY(OR)|KEY(OR)| Q1,00 | PCWP | |
Note
- automatic DOP: Computed Degree of Parallelism is 16 because of degree limitcan some one help us to find out where we did wrong or any pointer will really helpful to resolve an issue.
Edited by: Sachin B on May 11, 2010 4:05 AMGenerated AWR report for ADOP
Foreground Wait Events DB/Inst: HDB/hdb Snaps: 158-161
-> s - second, ms - millisecond - 1000th of a second
-> Only events with Total Wait Time (s) >= .001 are shown
-> ordered by wait time desc, waits desc (idle events last)
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
direct path read 522,173 0 125,051 239 628.4 99.3
db file sequential read 663 0 156 235 0.8 .1
log file sync 165 0 117 712 0.2 .1
Disk file operations I/O 267 0 63 236 0.3 .1
db file scattered read 251 0 36 145 0.3 .0
control file sequential re 217 0 32 149 0.3 .0
library cache load lock 2 0 10 4797 0.0 .0
cursor: pin S wait on X 3 0 9 3149 0.0 .0
read by other session 5 0 2 429 0.0 .0
kfk: async disk IO 613,170 0 2 0 737.9 .0
sort segment request 1 100 1 1007 0.0 .0
os thread startup 16 0 1 43 0.0 .0
direct path write temp 1 0 1 527 0.0 .0
latch free 51 0 0 2 0.1 .0
kksfbc child completion 1 100 0 59 0.0 .0
latch: cache buffers chain 19 0 0 2 0.0 .0
latch: shared pool 36 0 0 1 0.0 .0
PX Deq: Slave Session Stat 21 0 0 1 0.0 .0
library cache: mutex X 45 0 0 1 0.1 .0
CSS initialization 2 0 0 6 0.0 .0
enq: KO - fast object chec 1 0 0 11 0.0 .0
buffer busy waits 3 0 0 1 0.0 .0
cursor: pin S 9 0 0 0 0.0 .0
CSS operation: action 2 0 0 1 0.0 .0
direct path write 1 0 0 2 0.0 .0
jobq slave wait 17,554 100 8,942 509 21.1
PX Deq: Execute Reply 4,060 95 7,870 1938 4.9
SQL*Net message from clien 96 0 5,756 59962 0.1
PX Deq: Execution Msg 618 56 712 1152 0.7
KSV master wait 11 0 0 2 0.0
PX Deq: Join ACK 16 0 0 1 0.0
PX Deq: Parse Reply 14 0 0 1 0.0
Background Wait Events DB/Inst: HDB/hdb Snaps: 158-161
-> ordered by wait time desc, waits desc (idle events last)
-> Only events with Total Wait Time (s) >= .001 are shown
-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Avg
%Time Total Wait wait Waits % bg
Event Waits -outs Time (s) (ms) /txn time
control file sequential re 6,249 0 2,375 380 7.5 55.6
control file parallel writ 2,003 0 744 371 2.4 17.4
db file parallel write 1,604 0 503 313 1.9 11.8
log file parallel write 861 0 320 371 1.0 7.5
db file sequential read 363 0 151 415 0.4 3.5
db file scattered read 152 0 64 421 0.2 1.5
Disk file operations I/O 276 0 21 77 0.3 .5
os thread startup 316 0 15 48 0.4 .4
ADR block file read 24 0 11 450 0.0 .3
rdbms ipc reply 17 12 7 403 0.0 .2
Data file init write 6 0 6 1016 0.0 .1
direct path write 21 0 6 287 0.0 .1
log file sync 7 0 6 796 0.0 .1
ADR block file write 10 0 4 414 0.0 .1
enq: JS - queue lock 1 0 3 2535 0.0 .1
ASM file metadata operatio 1,801 0 2 1 2.2 .0
db file parallel read 30 0 1 40 0.0 .0
kfk: async disk IO 955 0 1 1 1.1 .0
db file single write 1 0 0 415 0.0 .0
reliable message 10 0 0 23 0.0 .0
latch: shared pool 75 0 0 2 0.1 .0
latch: call allocation 26 0 0 2 0.0 .0
CSS initialization 7 0 0 6 0.0 .0
asynch descriptor resize 352 100 0 0 0.4 .0
undo segment extension 2 100 0 5 0.0 .0
CSS operation: action 9 0 0 1 0.0 .0
CSS operation: query 42 0 0 0 0.1 .0
latch: parallel query allo 4 0 0 0 0.0 .0
rdbms ipc message 37,948 97 104,599 2756 45.7
DIAG idle wait 16,762 100 16,927 1010 20.2
ASM background timer 1,724 0 8,467 4912 2.1
shared server idle wait 282 100 8,465 30019 0.3
pmon timer 3,123 90 8,465 2711 3.8
wait for unread message on 8,381 100 8,465 1010 10.1
dispatcher timer 141 100 8,463 60019 0.2
Streams AQ: qmn coordinato 604 50 8,462 14010 0.7
Streams AQ: qmn slave idle 304 0 8,462 27836 0.4
smon timer 35 71 8,382 239496 0.0
Space Manager: slave idle 1,621 99 8,083 4986 2.0
PX Idle Wait 2,392 99 4,739 1981 2.9
class slave wait 46 0 623 13546 0.1
KSV master wait 2 0 0 27 0.0
SQL*Net message from clien 7 0 0 1 0.0
Wait Event Histogram DB/Inst: HDB/hdb Snaps: 158-161
-> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
-> % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
-> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
-> Ordered by Event (idle events last)
% of Waits
Total
Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
ADR block file read 24 100.0
ADR block file write 10 100.0
ADR file lock 12 100.0
ASM file metadata operatio 1812 99.0 .3 .4 .2 .1
CSS initialization 9 100.0
CSS operation: action 11 90.9 9.1
CSS operation: query 54 100.0
Data file init write 6 16.7 16.7 16.7 50.0
Disk file operations I/O 533 88.7 2.6 .6 1.5 .2 6.4
PX Deq: Signal ACK EXT 4 100.0
PX Deq: Signal ACK RSG 2 100.0
PX Deq: Slave Session Stat 21 42.9 28.6 28.6
SQL*Net break/reset to cli 6 100.0
SQL*Net message to client 102 100.0
SQL*Net more data to clien 4 100.0
asynch descriptor resize 527 100.0
buffer busy waits 4 75.0 25.0
control file parallel writ 2003 9.3 .5 .0 .1 90.0
control file sequential re 6466 10.6 .0 .0 .0 .1 .2 89.0
cursor: pin S 9 100.0
cursor: pin S wait on X 3 33.3 33.3 33.3
db file parallel read 30 6.7 30.0 63.3
db file parallel write 1604 7.4 .1 .6 16.5 75.5
db file scattered read 403 3.7 .2 2.5 13.6 14.9 3.5 61.5
db file sequential read 1017 12.3 .8 2.3 7.3 6.6 2.0 68.8
db file single write 1 100.0
direct path read 522.2 2.2 2.1 .1 .0 1.8 17.9 75.9
direct path write 22 4.5 4.5 90.9
direct path write temp 1 100.0
enq: JS - queue lock 1 100.0
enq: KO - fast object chec 1 100.0
enq: PS - contention 1 100.0
kfk: async disk IO 614.1 100.0 .0
kksfbc child completion 1 100.0
latch free 58 46.6 27.6 15.5 10.3
latch: cache buffers chain 19 36.8 10.5 52.6
latch: call allocation 26 76.9 11.5 7.7 3.8
latch: parallel query allo 4 100.0
latch: shared pool 111 44.1 28.8 27.0
library cache load lock 2 100.0
library cache: mutex X 45 84.4 8.9 4.4 2.2
log file parallel write 861 10.0 .1 .1 89.5 .2
log file sync 172 6.4 90.1 3.5
os thread startup 332 100.0
rdbms ipc reply 18 72.2 11.1 16.7
read by other session 5 100.0
reliable message 11 81.8 9.1 9.1
sort segment request 1 100.0
undo segment extension 2 50.0 50.0
ASM background timer 1724 .8 .6 .1 .6 97.9
DIAG idle wait 16.8K 100.0
KSV master wait 13 7.7 23.1 61.5 7.7
PX Deq: Execute Reply 4060 .4 .0 .0 .1 3.4 96.0
PX Deq: Execution Msg 617 34.7 1.5 2.4 1.5 1.5 .2 .8 57.5
PX Deq: Join ACK 16 93.8 6.3
PX Deq: Parse Reply 14 71.4 7.1 14.3 7.1
PX Idle Wait 2384 .0 .6 99.3
SQL*Net message from clien 103 82.5 1.0 1.9 1.0 13.6
Space Manager: slave idle 1621 .2 99.8
Streams AQ: qmn coordinato 604 50.0 50.0
Wait Event Histogram DB/Inst: HDB/hdb Snaps: 158-161
-> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
-> % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
-> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
-> Ordered by Event (idle events last)Edited by: Sachin B on May 11, 2010 4:52 AM -
I am copying one large non partitioned table (64 GB having ~300 Million rows) into other table. Table contains a PK with index.
To speed up insert process, i have used APPEND PARALLEL hint.
I am on 11.1.0.7 running on RHEL 5.4 with 16 GB RAM (4 GB PGA) and 8 processors.
Here is my Statement that does table copy.
INSERT /*+APPEND PARALLEL (A,16) */ INTO TRANSACTION2 A
SELECT /*+PARALLEL (B,16) */ * FROM TRANSACTION1 B
/Execution Plan
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
| 0 | INSERT STATEMENT | | | | 94584 (100)| | | | |
| 1 | LOAD AS SELECT | | | | | | | | |
| 2 | PX COORDINATOR | | | | | | | | |
| 3 | PX SEND QC (RANDOM) | :TQ10000 | 307M| 56G| 94584 (2)| 00:18:56 | Q1,00 | P->S | QC (RAND) |
| 4 | PX BLOCK ITERATOR | | 307M| 56G| 94584 (2)| 00:18:56 | Q1,00 | PCWC | |
|* 5 | TABLE ACCESS FULL | TRANSACTION1 | 307M| 56G| 94584 (2)| 00:18:56 | Q1,00 | PCWP | |
Predicate Information (identified by operation id):
5 - access(:Z>=:Z AND :Z<=:Z)I wanted to understand how Oracle internally load data into other table. Entire copy process took ~3 hrs. I have monitored entire operation and found below px events one after other.
Username QC/Slav SlaveSet SID QC SID Slave INST STATE WAIT_EVENT QC INST Req. DOP Actual DOP
QC 309 309 1 NOT WAIT
- p001 (Slave) 1 174 1 WAIT PX Deq: Execution Msg 1 16 16
- p000 (Slave) 1 196 1 WAIT PX Deq: Execution Msg 1 16 16
- p006 (Slave) 1 203 1 WAIT PX Deq: Execution Msg 1 16 16
- p002 (Slave) 1 223 1 WAIT PX Deq: Execution Msg 1 16 16
- p007 (Slave) 1 242 1 WAIT PX Deq: Execution Msg 1 16 16
- p010 (Slave) 1 247 1 WAIT PX Deq: Execution Msg 1 16 16
- p005 (Slave) 1 255 1 WAIT PX Deq: Execution Msg 1 16 16
- p013 (Slave) 1 263 1 WAIT PX Deq: Execution Msg 1 16 16
- p011 (Slave) 1 296 1 WAIT PX Deq: Execution Msg 1 16 16
- p014 (Slave) 1 337 1 WAIT PX Deq: Execution Msg 1 16 16
- p008 (Slave) 1 342 1 WAIT PX Deq: Execution Msg 1 16 16
- p012 (Slave) 1 375 1 WAIT PX Deq: Execution Msg 1 16 16
- p015 (Slave) 1 384 1 WAIT PX Deq: Execution Msg 1 16 16
- p009 (Slave) 1 397 1 WAIT PX Deq: Execution Msg 1 16 16
- p003 (Slave) 1 150 1 WAIT PX Deq: Execution Msg 1 16 16
- p004 (Slave) 1 164 1 WAIT PX Deq: Execution Msg 1 16 16In above output, i can see that all 16 px slaves are idle and probably doing nothing. If that is correct,then which process is doing FTS ?
And then i observed that QC is doing Index Maintainable.
Username QC/Slav SlaveSet SID QC SID Slave INST STATE WAIT_EVENT QC INST Req. DOP Actual DOP
QC 309 309 1 WAIT direct path read
- p001 (Slave) 1 174 1 WAIT PX Deq: Execution Msg 1 16 16
- p000 (Slave) 1 196 1 WAIT PX Deq: Execution Msg 1 16 16
- p006 (Slave) 1 203 1 WAIT PX Deq: Execution Msg 1 16 16
- p002 (Slave) 1 223 1 WAIT PX Deq: Execution Msg 1 16 16
- p007 (Slave) 1 242 1 WAIT PX Deq: Execution Msg 1 16 16
- p010 (Slave) 1 247 1 WAIT PX Deq: Execution Msg 1 16 16
- p005 (Slave) 1 255 1 WAIT PX Deq: Execution Msg 1 16 16
- p013 (Slave) 1 263 1 WAIT PX Deq: Execution Msg 1 16 16
- p011 (Slave) 1 296 1 WAIT PX Deq: Execution Msg 1 16 16
- p014 (Slave) 1 337 1 WAIT PX Deq: Execution Msg 1 16 16
- p008 (Slave) 1 342 1 WAIT PX Deq: Execution Msg 1 16 16
- p012 (Slave) 1 375 1 WAIT PX Deq: Execution Msg 1 16 16
- p015 (Slave) 1 384 1 WAIT PX Deq: Execution Msg 1 16 16
- p009 (Slave) 1 397 1 WAIT PX Deq: Execution Msg 1 16 16
- p003 (Slave) 1 150 1 WAIT PX Deq: Execution Msg 1 16 16
- p004 (Slave) 1 164 1 WAIT PX Deq: Execution Msg 1 16 16What triggering me is , why Only QC is doing index maintenance ? I was under impression that all px slaves should do that job. But again, it seems all 16 px are sitting idle.
This is what i got from sort_area_active
SID SQL_HASH_VALUE QCSID OPERATION_ID TYPE POLICY WSIZE_MB EXP_SIZE_MB ACT_SIZE_MB MAX_SIZE_MB PASSES TEMP
309 699687363 0 IDX MAINTENANCE (SOR AUTO 10.06 1.19 .84 20.35 34 10936.53
309 699687363 1 LOAD WRITE BUFFERS AUTO .51 .51 .51 .51 0Can anyone help me to understand such Direct load process ?
Many Thanks In Advance !OraDBA02 wrote:
I am copying one large non partitioned table (64 GB having ~300 Million rows) into other table. Table contains a PK with index.
To speed up insert process, i have used APPEND PARALLEL hint.
I am on 11.1.0.7 running on RHEL 5.4 with 16 GB RAM (4 GB PGA) and 8 processors.
Here is my Statement that does table copy.
INSERT /*+APPEND PARALLEL (A,16) */ INTO TRANSACTION2 A
SELECT /*+PARALLEL (B,16) */ * FROM TRANSACTION1 B
/Execution Plan
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
| 0 | INSERT STATEMENT | | | | 94584 (100)| | | | |
| 1 | LOAD AS SELECT | | | | | | | | |
| 2 | PX COORDINATOR | | | | | | | | |
| 3 | PX SEND QC (RANDOM) | :TQ10000 | 307M| 56G| 94584 (2)| 00:18:56 | Q1,00 | P->S | QC (RAND) |
| 4 | PX BLOCK ITERATOR | | 307M| 56G| 94584 (2)| 00:18:56 | Q1,00 | PCWC | |
|* 5 | TABLE ACCESS FULL | TRANSACTION1 | 307M| 56G| 94584 (2)| 00:18:56 | Q1,00 | PCWP | |
--------------------------------------------------------------------------------------------------------------------------------------Can anyone help me to understand such Direct load process ?
Many Thanks In Advance !If above execution plan is your actual execution plan (generated in the session you've used to execute or even better taken from the shared pool via DBMS_XPLAN.DISPLAY_CURSOR) then you've requested parallel DML via your hints but you forgot to enable it on session level using
ALTER SESSION ENABLE PARALLEL DML;because your plan shows a serial DML operation. So effectively you're scanning the source table in parallel and everything DML related has to be done by the QC. That seems to correspond to what you were observing.
With parallel DML enabled you'll notice PX operations around the LOAD AS SELECT operation in addition to the PX operations for the query.
Regards,
Randolf
Oracle related stuff blog:
http://oracle-randolf.blogspot.com/
Co-author of the "OakTable Expert Oracle Practices" book:
http://www.apress.com/book/view/1430226684
http://www.amazon.com/Expert-Oracle-Practices-Database-Administration/dp/1430226684 -
How to set the correct shared pool size and db_buffer_cache using awr
Hi All,
I want to how to set the correct size for shared_pool_size and db_cache_size using shared pool advisory and buffer pool advisory of awr report. I have paste the shared and buffer pool advisory of awr report.
Shared Pool Advisory
* SP: Shared Pool Est LC: Estimated Library Cache Factr: Factor
* Note there is often a 1:Many correlation between a single logical object in the Library Cache, and the physical number of memory objects associated with it. Therefore comparing the number of Lib Cache objects (e.g. in v$librarycache), with the number of Lib Cache Memory Objects is invalid.
Shared Pool Size(M) SP Size Factr Est LC Size (M) Est LC Mem Obj Est LC Time Saved (s) Est LC Time Saved Factr Est LC Load Time (s) Est LC Load Time Factr Est LC Mem Obj Hits (K)
4,096 1.00 471 25,153 184,206 1.00 149 1.00 9,069
4,736 1.16 511 27,328 184,206 1.00 149 1.00 9,766
5,248 1.28 511 27,346 184,206 1.00 149 1.00 9,766
5,760 1.41 511 27,346 184,206 1.00 149 1.00 9,766
6,272 1.53 511 27,346 184,206 1.00 149 1.00 9,766
6,784 1.66 511 27,346 184,206 1.00 149 1.00 9,766
7,296 1.78 511 27,346 184,206 1.00 149 1.00 9,766
7,808 1.91 511 27,346 184,206 1.00 149 1.00 9,766
8,320 2.03 511 27,346 184,206 1.00 149 1.00 9,766
Buffer Pool Advisory
* Only rows with estimated physical reads >0 are displayed
* ordered by Block Size, Buffers For Estimate
P Size for Est (M) Size Factor Buffers (thousands) Est Phys Read Factor Estimated Phys Reads (thousands) Est Phys Read Time Est %DBtime for Rds
D 4,096 0.10 485 1.02 1,002 1 0.00
D 8,192 0.20 970 1.00 987 1 0.00
D 12,288 0.30 1,454 1.00 987 1 0.00
D 16,384 0.40 1,939 1.00 987 1 0.00
D 20,480 0.50 2,424 1.00 987 1 0.00
D 24,576 0.60 2,909 1.00 987 1 0.00
D 28,672 0.70 3,394 1.00 987 1 0.00
D 32,768 0.80 3,878 1.00 987 1 0.00
D 36,864 0.90 4,363 1.00 987 1 0.00
D 40,960 1.00 4,848 1.00 987 1 0.00
D 45,056 1.10 5,333 1.00 987 1 0.00
D 49,152 1.20 5,818 1.00 987 1 0.00
D 53,248 1.30 6,302 1.00 987 1 0.00
D 57,344 1.40 6,787 1.00 987 1 0.00
D 61,440 1.50 7,272 1.00 987 1 0.00
D 65,536 1.60 7,757 1.00 987 1 0.00
D 69,632 1.70 8,242 1.00 987 1 0.00
D 73,728 1.80 8,726 1.00 987 1 0.00
D 77,824 1.90 9,211 1.00 987 1 0.00
D 81,920 2.00 9,696 1.00 987 1 0.00
My shared pool size is 4gb and db_cache_size is 40Gb.
Please help me in configuring the correct size for this.
Thanks and Regards,Hi ,
Actually batch load is taking too much time.
Please find below the 1 hr awr report
Snap Id Snap Time Sessions Cursors/Session
Begin Snap: 6557 27-Nov-11 16:00:06 126 1.3
End Snap: 6558 27-Nov-11 17:00:17 130 1.6
Elapsed: 60.17 (mins)
DB Time: 34.00 (mins)
Report Summary
Cache Sizes
Begin End
Buffer Cache: 40,960M 40,960M Std Block Size: 8K
Shared Pool Size: 4,096M 4,096M Log Buffer: 25,908K
Load Profile
Per Second Per Transaction Per Exec Per Call
DB Time(s): 0.6 1.4 0.00 0.07
DB CPU(s): 0.5 1.2 0.00 0.06
Redo size: 281,296.9 698,483.4
Logical reads: 20,545.6 51,016.4
Block changes: 1,879.5 4,667.0
Physical reads: 123.7 307.2
Physical writes: 66.4 164.8
User calls: 8.2 20.4
Parses: 309.4 768.4
Hard parses: 8.5 21.2
W/A MB processed: 1.7 4.3
Logons: 0.7 1.6
Executes: 1,235.9 3,068.7
Rollbacks: 0.0 0.0
Transactions: 0.4
Instance Efficiency Percentages (Target 100%)
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 99.66 In-memory Sort %: 100.00
Library Hit %: 99.19 Soft Parse %: 97.25
Execute to Parse %: 74.96 Latch Hit %: 99.97
Parse CPU to Parse Elapsd %: 92.41 % Non-Parse CPU: 98.65
Shared Pool Statistics
Begin End
Memory Usage %: 80.33 82.01
% SQL with executions>1: 90.90 86.48
% Memory for SQL w/exec>1: 90.10 86.89
Top 5 Timed Foreground Events
Event Waits Time(s) Avg wait (ms) % DB time Wait Class
DB CPU 1,789 87.72
db file sequential read 27,531 50 2 2.45 User I/O
db file scattered read 26,322 30 1 1.47 User I/O
row cache lock 1,798 20 11 0.96 Concurrency
OJVM: Generic 36 15 421 0.74 Other
Host CPU (CPUs: 24 Cores: 12 Sockets: )
Load Average Begin Load Average End %User %System %WIO %Idle
0.58 1.50 2.8 0.7 0.1 96.6
Instance CPU
%Total CPU %Busy CPU %DB time waiting for CPU (Resource Manager)
2.2 63.6 0.0
Memory Statistics
Begin End
Host Mem (MB): 131,072.0 131,072.0
SGA use (MB): 50,971.4 50,971.4
PGA use (MB): 545.5 1,066.3
% Host Mem used for SGA+PGA: 39.30 39.70
RAC Statistics
Begin End
Number of Instances: 2 2
Global Cache Load Profile
Per Second Per Transaction
Global Cache blocks received: 3.09 7.68
Global Cache blocks served: 1.86 4.62
GCS/GES messages received: 78.64 195.27
GCS/GES messages sent: 53.82 133.65
DBWR Fusion writes: 0.52 1.30
Estd Interconnect traffic (KB) 65.50
Global Cache Efficiency Percentages (Target local+remote 100%)
Buffer access - local cache %: 99.65
Buffer access - remote cache %: 0.02
Buffer access - disk %: 0.34
Global Cache and Enqueue Services - Workload Characteristics
Avg global enqueue get time (ms): 0.0
Avg global cache cr block receive time (ms): 1.7
Avg global cache current block receive time (ms): 1.0
Avg global cache cr block build time (ms): 0.0
Avg global cache cr block send time (ms): 0.0
Global cache log flushes for cr blocks served %: 1.4
Avg global cache cr block flush time (ms): 0.9
Avg global cache current block pin time (ms): 0.0
Avg global cache current block send time (ms): 0.0
Global cache log flushes for current blocks served %: 0.1
Avg global cache current block flush time (ms): 0.0
Global Cache and Enqueue Services - Messaging Statistics
Avg message sent queue time (ms): 0.0
Avg message sent queue time on ksxp (ms): 0.4
Avg message received queue time (ms): 0.5
Avg GCS message process time (ms): 0.0
Avg GES message process time (ms): 0.0
% of direct sent messages: 79.13
% of indirect sent messages: 17.10
% of flow controlled messages: 3.77
Cluster Interconnect
Begin End
Interface IP Address Pub Source IP Pub Src
en9 10.51.10.61 N Oracle Cluster Repository
Main Report
* Report Summary
* Wait Events Statistics
* SQL Statistics
* Instance Activity Statistics
* IO Stats
* Buffer Pool Statistics
* Advisory Statistics
* Wait Statistics
* Undo Statistics
* Latch Statistics
* Segment Statistics
* Dictionary Cache Statistics
* Library Cache Statistics
* Memory Statistics
* Streams Statistics
* Resource Limit Statistics
* Shared Server Statistics
* init.ora Parameters
More RAC Statistics
* RAC Report Summary
* Global Messaging Statistics
* Global CR Served Stats
* Global CURRENT Served Stats
* Global Cache Transfer Stats
* Interconnect Stats
* Dynamic Remastering Statistics
Back to Top
Statistic Name Time (s) % of DB Time
sql execute elapsed time 1,925.20 94.38
DB CPU 1,789.38 87.72
connection management call elapsed time 99.65 4.89
PL/SQL execution elapsed time 89.81 4.40
parse time elapsed 46.32 2.27
hard parse elapsed time 25.01 1.23
Java execution elapsed time 21.24 1.04
PL/SQL compilation elapsed time 11.92 0.58
failed parse elapsed time 9.37 0.46
hard parse (sharing criteria) elapsed time 8.71 0.43
sequence load elapsed time 0.06 0.00
repeated bind elapsed time 0.02 0.00
hard parse (bind mismatch) elapsed time 0.01 0.00
DB time 2,039.77
background elapsed time 122.00
background cpu time 113.42
Statistic Value End Value
NUM_LCPUS 0
NUM_VCPUS 0
AVG_BUSY_TIME 12,339
AVG_IDLE_TIME 348,838
AVG_IOWAIT_TIME 221
AVG_SYS_TIME 2,274
AVG_USER_TIME 9,944
BUSY_TIME 299,090
IDLE_TIME 8,375,051
IOWAIT_TIME 6,820
SYS_TIME 57,512
USER_TIME 241,578
LOAD 1 2
OS_CPU_WAIT_TIME 312,200
PHYSICAL_MEMORY_BYTES 137,438,953,472
NUM_CPUS 24
NUM_CPU_CORES 12
GLOBAL_RECEIVE_SIZE_MAX 1,310,720
GLOBAL_SEND_SIZE_MAX 1,310,720
TCP_RECEIVE_SIZE_DEFAULT 16,384
TCP_RECEIVE_SIZE_MAX 9,223,372,036,854,775,807
TCP_RECEIVE_SIZE_MIN 4,096
TCP_SEND_SIZE_DEFAULT 16,384
TCP_SEND_SIZE_MAX 9,223,372,036,854,775,807
TCP_SEND_SIZE_MIN 4,096
Back to Wait Events Statistics
Back to Top
Operating System Statistics - Detail
Snap Time Load %busy %user %sys %idle %iowait
27-Nov 16:00:06 0.58
27-Nov 17:00:17 1.50 3.45 2.79 0.66 96.55 0.08
Back to Wait Events Statistics
Back to Top
Foreground Wait Class
* s - second, ms - millisecond - 1000th of a second
* ordered by wait time desc, waits desc
* %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
* Captured Time accounts for 95.7% of Total DB time 2,039.77 (s)
* Total FG Wait Time: 163.14 (s) DB CPU time: 1,789.38 (s)
Wait Class Waits %Time -outs Total Wait Time (s) Avg wait (ms) %DB time
DB CPU 1,789 87.72
User I/O 61,229 0 92 1 4.49
Other 102,743 40 31 0 1.50
Concurrency 3,169 10 24 7 1.16
Cluster 58,920 0 11 0 0.52
System I/O 45,407 0 6 0 0.29
Configuration 107 7 1 5 0.03
Commit 383 0 0 1 0.01
Network 15,275 0 0 0 0.00
Application 52 8 0 0 0.00
Back to Wait Events Statistics
Back to Top
Foreground Wait Events
* s - second, ms - millisecond - 1000th of a second
* Only events with Total Wait Time (s) >= .001 are shown
* ordered by wait time desc, waits desc (idle events last)
* %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn % DB time
db file sequential read 27,531 0 50 2 18.93 2.45
db file scattered read 26,322 0 30 1 18.10 1.47
row cache lock 1,798 0 20 11 1.24 0.96
OJVM: Generic 36 42 15 421 0.02 0.74
db file parallel read 394 0 7 19 0.27 0.36
control file sequential read 22,248 0 6 0 15.30 0.28
reliable message 4,439 0 4 1 3.05 0.18
gc current grant busy 7,597 0 3 0 5.22 0.16
PX Deq: Slave Session Stats 2,661 0 3 1 1.83 0.16
DFS lock handle 3,208 0 3 1 2.21 0.16
direct path write temp 4,842 0 3 1 3.33 0.15
library cache load lock 39 0 3 72 0.03 0.14
gc cr multi block request 37,008 0 3 0 25.45 0.14
IPC send completion sync 5,451 0 2 0 3.75 0.10
gc cr block 2-way 4,669 0 2 0 3.21 0.09
enq: PS - contention 3,183 33 1 0 2.19 0.06
gc cr grant 2-way 5,151 0 1 0 3.54 0.06
direct path read temp 1,722 0 1 1 1.18 0.05
gc current block 2-way 1,807 0 1 0 1.24 0.03
os thread startup 6 0 1 108 0.00 0.03
name-service call wait 12 0 1 47 0.01 0.03
PX Deq: Signal ACK RSG 2,046 50 0 0 1.41 0.02
log file switch completion 3 0 0 149 0.00 0.02
rdbms ipc reply 3,610 0 0 0 2.48 0.02
gc current grant 2-way 1,432 0 0 0 0.98 0.02
library cache pin 903 32 0 0 0.62 0.02
PX Deq: reap credit 35,815 100 0 0 24.63 0.01
log file sync 383 0 0 1 0.26 0.01
Disk file operations I/O 405 0 0 0 0.28 0.01
library cache lock 418 3 0 0 0.29 0.01
kfk: async disk IO 23,159 0 0 0 15.93 0.01
gc current block busy 4 0 0 35 0.00 0.01
gc current multi block request 1,206 0 0 0 0.83 0.01
ges message buffer allocation 38,526 0 0 0 26.50 0.00
enq: FB - contention 131 0 0 0 0.09 0.00
undo segment extension 8 100 0 6 0.01 0.00
CSS initialization 8 0 0 6 0.01 0.00
SQL*Net message to client 14,600 0 0 0 10.04 0.00
enq: HW - contention 96 0 0 0 0.07 0.00
CSS operation: action 8 0 0 4 0.01 0.00
gc cr block busy 33 0 0 1 0.02 0.00
latch free 30 0 0 1 0.02 0.00
enq: TM - contention 49 6 0 0 0.03 0.00
enq: JQ - contention 19 100 0 1 0.01 0.00
SQL*Net more data to client 666 0 0 0 0.46 0.00
asynch descriptor resize 3,179 100 0 0 2.19 0.00
latch: shared pool 3 0 0 3 0.00 0.00
CSS operation: query 24 0 0 0 0.02 0.00
PX Deq: Signal ACK EXT 72 0 0 0 0.05 0.00
KJC: Wait for msg sends to complete 269 0 0 0 0.19 0.00
latch: object queue header operation 4 0 0 1 0.00 0.00
gc cr block congested 5 0 0 0 0.00 0.00
utl_file I/O 11 0 0 0 0.01 0.00
enq: TO - contention 3 33 0 0 0.00 0.00
SQL*Net message from client 14,600 0 219,478 15033 10.04
jobq slave wait 7,726 100 3,856 499 5.31
PX Deq: Execution Msg 10,556 19 50 5 7.26
PX Deq: Execute Reply 2,946 31 27 9 2.03
PX Deq: Parse Reply 3,157 35 3 1 2.17
PX Deq: Join ACK 2,976 28 2 1 2.05
PX Deq Credit: send blkd 7 14 0 4 0.00
Back to Wait Events Statistics
Back to Top
Background Wait Events
* ordered by wait time desc, waits desc (idle events last)
* Only events with Total Wait Time (s) >= .001 are shown
* %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn % bg time
os thread startup 140 0 13 90 0.10 10.35
db file parallel write 8,233 0 6 1 5.66 5.08
log file parallel write 3,906 0 6 1 2.69 4.62
log file sequential read 350 0 5 16 0.24 4.49
control file sequential read 13,737 0 5 0 9.45 3.72
DFS lock handle 2,990 27 2 1 2.06 1.43
db file sequential read 921 0 2 2 0.63 1.39
SQL*Net break/reset to client 18 0 1 81 0.01 1.19
control file parallel write 2,455 0 1 1 1.69 1.12
ges lms sync during dynamic remastering and reconfig 24 100 1 50 0.02 0.98
library cache load lock 35 0 1 24 0.02 0.68
ASM file metadata operation 3,483 0 1 0 2.40 0.65
enq: CO - master slave det 1,203 100 1 0 0.83 0.46
kjbdrmcvtq lmon drm quiesce: ping completion 9 0 1 62 0.01 0.46
enq: WF - contention 11 0 0 35 0.01 0.31
CGS wait for IPC msg 32,702 100 0 0 22.49 0.19
gc object scan 28,788 100 0 0 19.80 0.15
row cache lock 535 0 0 0 0.37 0.14
library cache pin 370 55 0 0 0.25 0.12
ksxr poll remote instances 19,119 100 0 0 13.15 0.11
name-service call wait 6 0 0 19 0.00 0.10
gc current block 2-way 304 0 0 0 0.21 0.09
gc cr block 2-way 267 0 0 0 0.18 0.08
gc cr grant 2-way 355 0 0 0 0.24 0.08
ges LMON to get to FTDONE 3 100 0 24 0.00 0.06
enq: CF - contention 145 76 0 0 0.10 0.05
PX Deq: reap credit 8,842 100 0 0 6.08 0.05
reliable message 126 0 0 0 0.09 0.05
db file scattered read 19 0 0 3 0.01 0.05
library cache lock 162 1 0 0 0.11 0.04
latch: shared pool 2 0 0 27 0.00 0.04
Disk file operations I/O 504 0 0 0 0.35 0.04
gc current grant busy 148 0 0 0 0.10 0.04
gcs log flush sync 84 0 0 1 0.06 0.04
ges message buffer allocation 24,934 0 0 0 17.15 0.02
enq: CR - block range reuse ckpt 83 0 0 0 0.06 0.02
latch free 22 0 0 1 0.02 0.02
CSS operation: action 13 0 0 2 0.01 0.02
CSS initialization 4 0 0 6 0.00 0.02
direct path read 1 0 0 21 0.00 0.02
rdbms ipc reply 153 0 0 0 0.11 0.01
db file parallel read 2 0 0 8 0.00 0.01
direct path write 5 0 0 3 0.00 0.01
gc current multi block request 49 0 0 0 0.03 0.01
gc current block busy 5 0 0 2 0.00 0.01
enq: PS - contention 24 50 0 0 0.02 0.01
gc cr multi block request 54 0 0 0 0.04 0.01
ges generic event 1 100 0 10 0.00 0.01
gc current grant 2-way 35 0 0 0 0.02 0.01
kfk: async disk IO 183 0 0 0 0.13 0.01
Log archive I/O 3 0 0 2 0.00 0.01
gc buffer busy acquire 2 0 0 3 0.00 0.00
LGWR wait for redo copy 123 0 0 0 0.08 0.00
IPC send completion sync 18 0 0 0 0.01 0.00
enq: TA - contention 11 0 0 0 0.01 0.00
read by other session 2 0 0 2 0.00 0.00
enq: TM - contention 9 89 0 0 0.01 0.00
latch: ges resource hash list 135 0 0 0 0.09 0.00
PX Deq: Slave Session Stats 12 0 0 0 0.01 0.00
KJC: Wait for msg sends to complete 89 0 0 0 0.06 0.00
enq: TD - KTF dump entries 8 0 0 0 0.01 0.00
enq: US - contention 7 0 0 0 0.00 0.00
CSS operation: query 12 0 0 0 0.01 0.00
enq: TK - Auto Task Serialization 6 100 0 0 0.00 0.00
PX Deq: Signal ACK RSG 24 50 0 0 0.02 0.00
log file single write 6 0 0 0 0.00 0.00
enq: WL - contention 2 100 0 1 0.00 0.00
ADR block file read 13 0 0 0 0.01 0.00
ADR block file write 5 0 0 0 0.00 0.00
latch: object queue header operation 1 0 0 1 0.00 0.00
gc cr block busy 1 0 0 1 0.00 0.00
rdbms ipc message 103,276 67 126,259 1223 71.03
PX Idle Wait 6,467 67 12,719 1967 4.45
wait for unread message on broadcast channel 7,240 100 7,221 997 4.98
gcs remote message 218,809 84 7,213 33 150.49
DIAG idle wait 203,228 95 7,185 35 139.77
shared server idle wait 121 100 3,630 30000 0.08
ASM background timer 3,343 0 3,611 1080 2.30
Space Manager: slave idle wait 723 100 3,610 4993 0.50
heartbeat monitor sleep 722 100 3,610 5000 0.50
ges remote message 73,089 52 3,609 49 50.27
dispatcher timer 66 88 3,608 54660 0.05
pmon timer 1,474 82 3,607 2447 1.01
PING 1,487 19 3,607 2426 1.02
Streams AQ: qmn slave idle wait 125 0 3,594 28754 0.09
Streams AQ: qmn coordinator idle wait 250 50 3,594 14377 0.17
smon timer 18 50 3,505 194740 0.01
JOX Jit Process Sleep 73 100 976 13370 0.05
class slave wait 56 0 605 10806 0.04
KSV master wait 2,215 98 1 0 1.52
SQL*Net message from client 109 0 0 2 0.07
PX Deq: Parse Reply 27 44 0 1 0.02
PX Deq: Join ACK 30 40 0 1 0.02
PX Deq: Execute Reply 20 30 0 0 0.01
Streams AQ: RAC qmn coordinator idle wait 259 100 0 0 0.18
Back to Wait Events Statistics
Back to Top
Wait Event Histogram
* Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
* % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
* % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
* Ordered by Event (idle events last)
% of Waits
Event Total Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s
ADR block file read 13 100.0
ADR block file write 5 100.0
ADR file lock 6 100.0
ARCH wait for archivelog lock 3 100.0
ASM file metadata operation 3483 99.6 .1 .1 .2
CGS wait for IPC msg 32.7K 100.0
CSS initialization 12 50.0 50.0
CSS operation: action 21 28.6 9.5 61.9
CSS operation: query 36 86.1 5.6 8.3
DFS lock handle 6198 98.6 1.2 .1 .1
Disk file operations I/O 909 95.7 3.6 .7
IPC send completion sync 5469 99.9 .1 .0 .0
KJC: Wait for msg sends to complete 313 100.0
LGWR wait for redo copy 122 100.0
Log archive I/O 3 66.7 33.3
OJVM: Generic 36 55.6 44.4
PX Deq: Signal ACK EXT 72 98.6 1.4
PX Deq: Signal ACK RSG 2070 99.7 .0 .1 .0 .1
PX Deq: Slave Session Stats 2673 99.7 .2 .1 .0
PX Deq: reap credit 44.7K 100.0
SQL*Net break/reset to client 20 95.0 5.0
SQL*Net message to client 14.7K 100.0
SQL*Net more data from client 32 100.0
SQL*Net more data to client 689 100.0
asynch descriptor resize 3387 100.0
buffer busy waits 2 100.0
control file parallel write 2455 96.6 2.2 .6 .6 .1
control file sequential read 36K 99.4 .3 .1 .1 .1 .1 .0
db file parallel read 397 8.8 .8 5.5 12.6 17.4 46.3 8.6
db file parallel write 8233 85.4 10.3 2.3 1.4 .4 .1
db file scattered read 26.3K 79.2 1.5 8.2 10.5 .6 .1 .0
db file sequential read 28.4K 60.2 3.3 18.0 18.1 .3 .1 .0
db file single write 2 100.0
direct path read 2 50.0 50.0
direct path read temp 1722 95.8 2.8 .1 .5 .8 .1
direct path write 6 83.3 16.7
direct path write temp 4842 96.3 2.7 .5 .2 .0 .0 .2
enq: AF - task serialization 1 100.0
enq: CF - contention 145 99.3 .7
enq: CO - master slave det 1203 98.9 .8 .2
enq: CR - block range reuse ckpt 83 100.0
enq: DR - contention 2 100.0
enq: FB - contention 131 100.0
enq: HW - contention 97 100.0
enq: JQ - contention 19 89.5 10.5
enq: JS - job run lock - synchronize 3 100.0
enq: MD - contention 1 100.0
enq: MW - contention 2 100.0
enq: PS - contention 3207 99.5 .4 .1
enq: TA - contention 11 100.0
enq: TD - KTF dump entries 8 100.0
enq: TK - Auto Task Serialization 6 100.0
enq: TM - contention 58 100.0
enq: TO - contention 3 100.0
enq: TQ - DDL contention 1 100.0
enq: TS - contention 1 100.0
enq: UL - contention 1 100.0
enq: US - contention 7 100.0
enq: WF - contention 11 81.8 18.2
enq: WL - contention 2 50.0 50.0
gc buffer busy acquire 2 50.0 50.0
gc cr block 2-way 4934 99.9 .1 .0 .0
gc cr block busy 35 68.6 31.4
gc cr block congested 6 100.0
gc cr disk read 2 100.0
gc cr grant 2-way 4824 100.0 .0
gc cr grant congested 2 100.0
gc cr multi block request 37.1K 99.8 .2 .0 .0 .0 .0 .0
gc current block 2-way 2134 99.9 .0 .0
gc current block busy 7 14.3 14.3 14.3 28.6 28.6
gc current block congested 2 100.0
gc current grant 2-way 1337 99.9 .1
gc current grant busy 7123 99.2 .2 .2 .0 .0 .3 .1
gc current grant congested 2 100.0
gc current multi block request 1260 99.8 .2
gc object scan 28.8K 100.0
gcs log flush sync 65 95.4 3.1 1.5
ges LMON to get to FTDONE 3 100.0
ges generic event 1 100.0
ges inquiry response 2 100.0
ges lms sync during dynamic remastering and reconfig 24 16.7 29.2 54.2
ges message buffer allocation 63.1K 100.0
kfk: async disk IO 23.3K 100.0 .0 .0
kjbdrmcvtq lmon drm quiesce: ping completion 9 11.1 88.9
ksxr poll remote instances 19.1K 100.0
latch free 52 59.6 40.4
latch: call allocation 2 100.0
latch: gc element 1 100.0
latch: gcs resource hash 1 100.0
latch: ges resource hash list 135 100.0
latch: object queue header operation 5 40.0 40.0 20.0
latch: shared pool 5 40.0 20.0 20.0 20.0
library cache load lock 74 9.5 5.4 8.1 17.6 10.8 13.5 35.1
library cache lock 493 99.2 .4 .4
library cache pin 1186 98.4 .3 1.2 .1
library cache: mutex X 6 100.0
log file parallel write 3897 72.9 1.5 17.1 7.5 .6 .3 .1
log file sequential read 350 4.6 3.1 59.4 30.0 2.9
log file single write 6 100.0
log file switch completion 3 33.3 66.7
log file sync 385 90.4 3.6 4.7 .8 .5
name-service call wait 18 5.6 5.6 5.6 16.7 44.4 22.2
os thread startup 146 100.0
rdbms ipc reply 3763 99.7 .3
read by other session 2 50.0 50.0
reliable message 4565 99.7 .2 .0 .0 .1
row cache lock 2334 99.3 .2 .1 .1 .3
undo segment extension 8 50.0 37.5 12.5
utl_file I/O 11 100.0
ASM background timer 3343 57.0 .3 .1 .1 .1 21.1 21.4
DIAG idle wait 203.2K 3.4 .2 .4 18.0 41.4 14.8 21.8
JOX Jit Process Sleep 73 2.7 97.3
KSV master wait 2213 99.4 .1 .2 .3
PING 1487 81.0 19.0
PX Deq Credit: send blkd 7 57.1 14.3 14.3 14.3
PX Deq: Execute Reply 2966 59.8 .8 9.5 5.6 10.2 2.6 11.4
PX Deq: Execution Msg 10.6K 72.4 12.1 2.6 2.5 .1 5.6 4.6 .0
PX Deq: Join ACK 3006 77.9 22.1 .1
PX Deq: Parse Reply 3184 67.1 31.1 1.6 .2
PX Idle Wait 6466 .2 8.7 4.3 4.8 .3 .1 5.0 76.6
SQL*Net message from client 14.7K 72.4 2.8 .8 .5 .9 .4 2.8 19.3
Space Manager: slave idle wait 722 100.0
Streams AQ: RAC qmn coordinator idle wait 259 100.0
Streams AQ: qmn coordinator idle wait 250 50.0 50.0
Streams AQ: qmn slave idle wait 125 100.0
class slave wait 55 67.3 7.3 1.8 5.5 1.8 7.3 9.1
dispatcher timer 66 6.1 93.9
gcs remote message 218.6K 7.7 1.8 1.2 1.6 1.7 15.7 70.3
ges remote message 72.9K 29.7 5.1 2.7 2.2 1.5 4.0 54.7
heartbeat monitor sleep 722 100.0
jobq slave wait 7725 .1 .0 99.9
pmon timer 1474 18.4 81.6
rdbms ipc message 103.3K 20.7 2.7 1.5 1.3 .9 .7 40.7 31.6
shared server idle wait 121 100.0
smon timer 18 100.0
wait for unread message on broadcast channel 7238 .3 99.7
Back to Wait Events Statistics
Back to Top
Wait Event Histogram Detail (64 msec to 2 sec)
* Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
* Units for % of Total Waits: ms is milliseconds s is 1024 milliseconds (approximately 1 second)
* % of Total Waits: total waits for all wait classes, including Idle
* % of Total Waits: value of .0 indicates value was <.05%; value of null is truly 0
* Ordered by Event (only non-idle events are displayed)
% of Total Waits
Event Waits 64ms to 2s <32ms <64ms <1/8s <1/4s <1/2s <1s <2s >=2s
ASM file metadata operation 6 99.8 .1 .1
DFS lock handle 6 99.9 .1 .0
OJVM: Generic 16 55.6 2.8 41.7
PX Deq: Signal ACK RSG 3 99.9 .0 .1
PX Deq: Slave Session Stats 3 99.9 .0 .0 .0
SQL*Net break/reset to client 1 95.0 5.0
control file sequential read 1 100.0 .0
db file parallel read 34 91.4 8.6
db file scattered read 4 100.0 .0 .0
db file sequential read 6 100.0 .0 .0 .0
direct path write temp 11 99.8 .1 .1 .0
enq: WF - contention 2 81.8 18.2
gc cr block 2-way 1 100.0 .0
gc cr multi block request 1 100.0 .0
gc current block 2-way 1 100.0 .0
gc current block busy 2 71.4 28.6
gc current grant busy 8 99.9 .0 .1
ges lms sync during dynamic remastering and reconfig 13 45.8 20.8 33.3
kjbdrmcvtq lmon drm quiesce: ping completion 8 11.1 11.1 77.8
latch: shared pool 1 80.0 20.0
library cache load lock 26 64.9 14.9 12.2 4.1 4.1
log file parallel write 2 99.9 .0 .0
log file sequential read 10 97.1 2.0 .6 .3
log file switch completion 2 33.3 66.7
name-service call wait 4 77.8 22.2
os thread startup 146 100.0
reliable message 4 99.9 .0 .1
row cache lock 2 99.7 .0 .0 .3
Back to Wait Events Statistics
Back to Top
Wait Event Histogram Detail (4 sec to 2 min)
* Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
* Units for % of Total Waits: s is 1024 milliseconds (approximately 1 second) m is 64*1024 milliseconds (approximately 67 seconds or 1.1 minutes)
* % of Total Waits: total waits for all wait classes, including Idle
* % of Total Waits: value of .0 indicates value was <.05%; value of null is truly 0
* Ordered by Event (only non-idle events are displayed)
% of Total Waits
Event Waits 4s to 2m <2s <4s <8s <16s <32s < 1m < 2m >=2m
row cache lock 6 99.7 .3
Back to Wait Events Statistics
Back to Top
Wait Event Histogram Detail (4 min to 1 hr)
No data exists for this section of the report.
Back to Wait Events Statistics
Back to Top
Service Statistics
* ordered by DB Time
Service Name DB Time (s) DB CPU (s) Physical Reads (K) Logical Reads (K)
ubshost 1,934 1,744 445 73,633
SYS$USERS 105 45 1 404
SYS$BACKGROUND 0 0 1 128
ubshostXDB 0 0 0 0
Back to Wait Events Statistics
Back to Top
Service Wait Class Stats
* Wait Class info for services in the Service Statistics section.
* Total Waits and Time Waited displayed for the following wait classes: User I/O, Concurrency, Administrative, Network
* Time Waited (Wt Time) in seconds
Service Name User I/O Total Wts User I/O Wt Time Concurcy Total Wts Concurcy Wt Time Admin Total Wts Admin Wt Time Network Total Wts Network Wt Time
ubshost 60232 90 2644 4 0 0 13302 0
SYS$USERS 997 2 525 19 0 0 1973 0
SYS$BACKGROUND 1456 2 1258 14 0 0 0 0
I am not able to paste the whole awr report. I have paste some of the sections of awr report.
Please help.
Thanks and Regards, -
Find the Wait events for an objects.
Hi Gurus.
I need you help to find out the wait events for a particular table and index when batch export job running from the application. I have done some work around to to find out, but i am not able to figure out the issue or what are the wait events. i am working on linux 4node rac 11gR2.
Below are the steps i have done.
1. find object_id for a table finding the wait events to that object_id.
SYS@ccmintpt1 SQL>select data_object_id from dba_objects where object_name='ACCOUNT_DETAIL_DENORM';
DATA_OBJECT_ID
82646
88518
84184
SYS@ccmintpt1 SQL>select w.sid,w.event,w.wait_class,w.wait_class# from gv$session s,gv$session_wait w where s.row_wait_obj#=82646;
no rows selected
SYS@ccmintpt1 SQL>select w.sid,w.event,w.wait_class,w.wait_class# from gv$session s,gv$session_wait w where s.row_wait_obj#=88518;
no rows selected
SYS@ccmintpt1 SQL>select w.sid,w.event,w.wait_class,w.wait_class# from gv$session s,gv$session_wait w where s.row_wait_obj#=84184;
no rows selected
2. find the object id for the index of that table and finding the wait events to that object_id.
SYS@ccmintpt1 SQL>select data_object_id from dba_objects where object_name='XAK2ACCOUNT_DETAIL_DENORM';
DATA_OBJECT_ID
88655
SID EVENT WAIT_CLASS WAIT_CLASS#
1 rdbms ipc message Idle 6
2 heartbeat monitor sleep Idle 6
3 rdbms ipc message Idle 6
4 wait for unread message on broadcast channel Idle 6
6 SQL*Net message from client Idle 6
7 SQL*Net message from client Idle 6
8 SQL*Net message from client Idle 6
9 SQL*Net message from client Idle 6
10 SQL*Net message from client Idle 6
11 SQL*Net message from client Idle 6
12 SQL*Net message from client Idle 6
13 SQL*Net message from client Idle 6
1876 SQL*Net message from client Idle 6
1877 VKTM Logical Idle Wait Idle 6
1878 rdbms ipc message Idle 6
1879 rdbms ipc message Idle 6
1880 rdbms ipc message Idle 6
1881 SQL*Net message from client Idle 6
3200 SQL*Net message from client Idle 6
3210 SQL*Net message from client Idle 6
3212 db file sequential read User I/O 8
3213 SQL*Net message from client Idle 6
3214 SQL*Net message from client Idle 6
3216 SQL*Net message from client Idle 6
3751 rdbms ipc message Idle 6
3752 gcs remote message Idle 6
3754 rdbms ipc message Idle 6
3756 SQL*Net message from client Idle 6
3757 SQL*Net message from client Idle 6
1326 SQL*Net message from client Idle 6
1876 Streams AQ: waiting for time management or cleanup tasks Idle 6
1877 VKTM Logical Idle Wait Idle 6
1878 rdbms ipc message Idle 6
1879 rdbms ipc message Idle 6
1880 rdbms ipc message Idle 6
1882 SQL*Net message from client Idle 6
3751 rdbms ipc message Idle 6
3752 gcs remote message Idle 6
3754 rdbms ipc message Idle 6
3756 rdbms ipc message Idle 6
3758 SQL*Net message from client Idle 6
3759 PX Deq: Execution Msg Idle 6
3200 SQL*Net message from client Idle 6
3210 SQL*Net message from client Idle 6
3212 db file sequential read User I/O 8
3213 SQL*Net message from client Idle 6
3214 SQL*Net message from client Idle 6
3216 SQL*Net message from client Idle 6
3751 rdbms ipc message Idle 6
3752 gcs remote message Idle 6
3754 rdbms ipc message Idle 6
3756 SQL*Net message from client Idle 6
3757 SQL*Net message from client Idle 6
3786 SQL*Net message from client Idle 6
3787 SQL*Net message from client Idle 6
1 heartbeat monitor sleep Idle 6
2 wait for unread message on broadcast channel Idle 6
314 rdbms ipc message Idle 6
315 Streams AQ: qmn coordinator idle wait Idle 6
627 pmon timer Idle 6
628 rdbms ipc message Idle 6
629 Streams AQ: waiting for messages in the queue Idle 6
940 Streams AQ: waiting for time management or cleanup tasks Idle 6
941 VKTM Logical Idle Wait Idle 6
942 rdbms ipc message Idle 6
1253 rdbms ipc message Idle 6
1254 rdbms ipc message Idle 6
1256 Space Manager: slave idle wait Idle 6
1566 DIAG idle wait Idle 6
1567 rdbms ipc message Idle 6
1568 Streams AQ: qmn slave idle wait Idle 6
1879 rdbms ipc message Idle 6
1880 smon timer Idle 6
1881 rdbms ipc message Idle 6
2192 PING Idle 6
2193 rdbms ipc message Idle 6
2194 rdbms ipc message Idle 6
2505 rdbms ipc message Idle 6
2506 rdbms ipc message Idle 6
2818 rdbms ipc message Idle 6
2819 rdbms ipc message Idle 6
2821 PX Deq: reap credit Other 0
3131 DIAG idle wait Idle 6
3132 jobq slave wait Idle 6
3444 rdbms ipc message Idle 6
3445 jobq slave wait Idle 6
3757 ges remote message Idle 6
3758 rdbms ipc message Idle 6
3759 VKRM Idle Idle 6
4070 gcs remote message Idle 6
4072 rdbms ipc message Idle 6
4383 gcs remote message Idle 6
4385 PX Deq: Execution Msg Idle 6
4696 rdbms ipc message Idle 6
4697 rdbms ipc message Idle 6
1 rdbms ipc message Idle 6
2 heartbeat monitor sleep Idle 6
3 rdbms ipc message Idle 6
4 wait for unread message on broadcast channel Idle 6
6 SQL*Net message from client Idle 6
From this output, what are the waits events i need to take into consideration.
Is this the right approach to follow?
Please let me know.
Thanks in advance.user13162661 wrote:
Hi Gurus.
I need you help to find out the wait events for a particular table and index when batch export job running from the application. I have done some work around to to find out, but i am not able to figure out the issue or what are the wait events. i am working on linux 4node rac 11gR2.
Below are the steps i have done.
1. find object_id for a table finding the wait events to that object_id.
SYS@ccmintpt1 SQL>select data_object_id from dba_objects where object_name='ACCOUNT_DETAIL_DENORM';
DATA_OBJECT_ID
82646
88518
84184
SYS@ccmintpt1 SQL>select w.sid,w.event,w.wait_class,w.wait_class# from gv$session s,gv$session_wait w where s.row_wait_obj#=82646;
no rows selected
SYS@ccmintpt1 SQL>select w.sid,w.event,w.wait_class,w.wait_class# from gv$session s,gv$session_wait w where s.row_wait_obj#=88518;
no rows selected
SYS@ccmintpt1 SQL>select w.sid,w.event,w.wait_class,w.wait_class# from gv$session s,gv$session_wait w where s.row_wait_obj#=84184;
no rows selected
2. find the object id for the index of that table and finding the wait events to that object_id.
SYS@ccmintpt1 SQL>select data_object_id from dba_objects where object_name='XAK2ACCOUNT_DETAIL_DENORM';
DATA_OBJECT_ID
88655
SID EVENT WAIT_CLASS WAIT_CLASS#
1 rdbms ipc message Idle 6
2 heartbeat monitor sleep Idle 6
3 rdbms ipc message Idle 6
4 wait for unread message on broadcast channel Idle 6
6 SQL*Net message from client Idle 6
7 SQL*Net message from client Idle 6
8 SQL*Net message from client Idle 6
9 SQL*Net message from client Idle 6
10 SQL*Net message from client Idle 6
11 SQL*Net message from client Idle 6
12 SQL*Net message from client Idle 6
13 SQL*Net message from client Idle 6
1876 SQL*Net message from client Idle 6
1877 VKTM Logical Idle Wait Idle 6
1878 rdbms ipc message Idle 6
1879 rdbms ipc message Idle 6
1880 rdbms ipc message Idle 6
1881 SQL*Net message from client Idle 6
3200 SQL*Net message from client Idle 6
3210 SQL*Net message from client Idle 6
3212 db file sequential read User I/O 8
3213 SQL*Net message from client Idle 6
3214 SQL*Net message from client Idle 6
3216 SQL*Net message from client Idle 6
3751 rdbms ipc message Idle 6
3752 gcs remote message Idle 6
3754 rdbms ipc message Idle 6
3756 SQL*Net message from client Idle 6
3757 SQL*Net message from client Idle 6
1326 SQL*Net message from client Idle 6
1876 Streams AQ: waiting for time management or cleanup tasks Idle 6
1877 VKTM Logical Idle Wait Idle 6
1878 rdbms ipc message Idle 6
1879 rdbms ipc message Idle 6
1880 rdbms ipc message Idle 6
1882 SQL*Net message from client Idle 6
3751 rdbms ipc message Idle 6
3752 gcs remote message Idle 6
3754 rdbms ipc message Idle 6
3756 rdbms ipc message Idle 6
3758 SQL*Net message from client Idle 6
3759 PX Deq: Execution Msg Idle 6
3200 SQL*Net message from client Idle 6
3210 SQL*Net message from client Idle 6
3212 db file sequential read User I/O 8
3213 SQL*Net message from client Idle 6
3214 SQL*Net message from client Idle 6
3216 SQL*Net message from client Idle 6
3751 rdbms ipc message Idle 6
3752 gcs remote message Idle 6
3754 rdbms ipc message Idle 6
3756 SQL*Net message from client Idle 6
3757 SQL*Net message from client Idle 6
3786 SQL*Net message from client Idle 6
3787 SQL*Net message from client Idle 6
1 heartbeat monitor sleep Idle 6
2 wait for unread message on broadcast channel Idle 6
314 rdbms ipc message Idle 6
315 Streams AQ: qmn coordinator idle wait Idle 6
627 pmon timer Idle 6
628 rdbms ipc message Idle 6
629 Streams AQ: waiting for messages in the queue Idle 6
940 Streams AQ: waiting for time management or cleanup tasks Idle 6
941 VKTM Logical Idle Wait Idle 6
942 rdbms ipc message Idle 6
1253 rdbms ipc message Idle 6
1254 rdbms ipc message Idle 6
1256 Space Manager: slave idle wait Idle 6
1566 DIAG idle wait Idle 6
1567 rdbms ipc message Idle 6
1568 Streams AQ: qmn slave idle wait Idle 6
1879 rdbms ipc message Idle 6
1880 smon timer Idle 6
1881 rdbms ipc message Idle 6
2192 PING Idle 6
2193 rdbms ipc message Idle 6
2194 rdbms ipc message Idle 6
2505 rdbms ipc message Idle 6
2506 rdbms ipc message Idle 6
2818 rdbms ipc message Idle 6
2819 rdbms ipc message Idle 6
2821 PX Deq: reap credit Other 0
3131 DIAG idle wait Idle 6
3132 jobq slave wait Idle 6
3444 rdbms ipc message Idle 6
3445 jobq slave wait Idle 6
3757 ges remote message Idle 6
3758 rdbms ipc message Idle 6
3759 VKRM Idle Idle 6
4070 gcs remote message Idle 6
4072 rdbms ipc message Idle 6
4383 gcs remote message Idle 6
4385 PX Deq: Execution Msg Idle 6
4696 rdbms ipc message Idle 6
4697 rdbms ipc message Idle 6
1 rdbms ipc message Idle 6
2 heartbeat monitor sleep Idle 6
3 rdbms ipc message Idle 6
4 wait for unread message on broadcast channel Idle 6
6 SQL*Net message from client Idle 6
From this output, what are the waits events i need to take into consideration.
Is this the right approach to follow?
Please let me know.
Thanks in advance.what actual problem are you trying to solve?
post SQL & results that have you concerned about wait events -
High Enqueue Waits in Statspack Report
Hi Everybody,
Oracle Version:Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
OS:Solaris 64 bit
Statspack Report is showing High Enqueue Waits
Here is a Snapshot.....
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
XXXXX 434917312 XXXXX 1 9.2.0.7.0 NO INgenius1
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 1064 24-Sep-08 06:00:01 1,333 19.6
End Snap: 1065 24-Sep-08 07:00:01 1,344 19.7
Elapsed: 60.00 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 1,152M Std Block Size: 8K
Shared Pool Size: 752M Log Buffer: 1,536K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 27,771.53 1,196.23
Logical reads: 777.20 33.48
Block changes: 180.58 7.78
Physical reads: 33.25 1.43
Physical writes: 7.51 0.32
User calls: 76.89 3.31
Parses: 23.29 1.00
Hard parses: 0.27 0.01
Sorts: 0.42 0.02
Logons: 0.05 0.00
Executes: 88.69 3.82
Transactions: 23.22
% Blocks changed per Read: 23.23 Recursive Call %: 61.50
Rollback per transaction %: 0.00 Rows per Sort: 738.76
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 96.90 In-memory Sort %: 99.67
Library Hit %: 99.76 Soft Parse %: 98.84
Execute to Parse %: 73.74 Latch Hit %: 99.98
Parse CPU to Parse Elapsd %: 94.38 % Non-Parse CPU: 94.45
Shared Pool Statistics Begin End
Memory Usage %: 86.56 87.54
% SQL with executions>1: 50.57 63.10
% Memory for SQL w/exec>1: 10.02 14.63
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
enqueue 19,553 57,405 98.39
db file sequential read 44,161 384 .66
CPU time 333 .57
log file parallel write 166,602 82 .14
log file sync 67,683 71 .12
Wait Events for DB: XXXXX Instance: XXXXX Snaps: 1064 - 1065
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
enqueue 19,553 19,540 57,405 2936 0.2
db file sequential read 44,161 0 384 9 0.5
log file parallel write 166,602 0 82 0 2.0
log file sync 67,683 0 71 1 0.8
db file scattered read 6,676 0 54 8 0.1
db file parallel write 1,135 0 7 6 0.0
direct path read 1,117 0 3 3 0.0
SQL*Net more data to client 3,932 0 2 0 0.0
control file parallel write 1,200 0 2 1 0.0
control file sequential read 1,389 0 1 1 0.0
PX Deq: Execute Reply 112 0 0 4 0.0
direct path write 752 0 0 1 0.0
db file parallel read 9 0 0 42 0.0
Background Wait Events for DB: XXXXX Instance: XXXXX Snaps: 1064 - 10
-> ordered by wait time desc, waits desc (idle events last)
Avg Wt Wait
Eq Requests Succ Gets Failed Gets Waits Time (ms) Time (s)
TC 25 24 0 4 32.00 0
TX 84,615 84,605 0 3 8.33 0
HW 118 118 0 2 2.00 0
PS 29 25 4 2 1.00 0
Here frm Statspack Report we can see that enqueue type- TX is taking up most of the resources.......
I want to find out which sql statements are causing this high enqueue waits......
Any Help Appreciated....
Regards,
Prosenjit MukherjeeHi All,
Here is the Statspack Report..........
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
XXXXX 434917312 XXXXX 1 9.2.0.7.0 NO XXXXXxxx1
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 1064 24-Sep-08 06:00:01 1,333 19.6
End Snap: 1065 24-Sep-08 07:00:01 1,344 19.7
Elapsed: 60.00 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 1,152M Std Block Size: 8K
Shared Pool Size: 752M Log Buffer: 1,536K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 27,771.53 1,196.23
Logical reads: 777.20 33.48
Block changes: 180.58 7.78
Physical reads: 33.25 1.43
Physical writes: 7.51 0.32
User calls: 76.89 3.31
Parses: 23.29 1.00
Hard parses: 0.27 0.01
Sorts: 0.42 0.02
Logons: 0.05 0.00
Executes: 88.69 3.82
Transactions: 23.22
% Blocks changed per Read: 23.23 Recursive Call %: 61.50
Rollback per transaction %: 0.00 Rows per Sort: 738.76
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 96.90 In-memory Sort %: 99.67
Library Hit %: 99.76 Soft Parse %: 98.84
Execute to Parse %: 73.74 Latch Hit %: 99.98
Parse CPU to Parse Elapsd %: 94.38 % Non-Parse CPU: 94.45
Shared Pool Statistics Begin End
Memory Usage %: 86.56 87.54
% SQL with executions>1: 50.57 63.10
% Memory for SQL w/exec>1: 10.02 14.63
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
enqueue 19,553 57,405 98.39
db file sequential read 44,161 384 .66
CPU time 333 .57
log file parallel write 166,602 82 .14
log file sync 67,683 71 .12
Wait Events for DB: XXXXX Instance: XXXXX Snaps: 1064 - 1065
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
enqueue 19,553 19,540 57,405 2936 0.2
db file sequential read 44,161 0 384 9 0.5
log file parallel write 166,602 0 82 0 2.0
log file sync 67,683 0 71 1 0.8
db file scattered read 6,676 0 54 8 0.1
db file parallel write 1,135 0 7 6 0.0
direct path read 1,117 0 3 3 0.0
SQL*Net more data to client 3,932 0 2 0 0.0
control file parallel write 1,200 0 2 1 0.0
control file sequential read 1,389 0 1 1 0.0
PX Deq: Execute Reply 112 0 0 4 0.0
direct path write 752 0 0 1 0.0
db file parallel read 9 0 0 42 0.0
process startup 6 0 0 44 0.0
SQL*Net break/reset to clien 296 0 0 1 0.0
PX Deq: Signal ACK 3 1 0 33 0.0
latch free 17 4 0 3 0.0
LGWR wait for redo copy 286 0 0 0 0.0
PX Deq: Join ACK 3 0 0 2 0.0
PX Deq: Parse Reply 4 0 0 1 0.0
buffer busy waits 12 0 0 0 0.0
PX Deq Credit: need buffer 4 0 0 0 0.0
PX Deq: Table Q Sample 1 0 0 0 0.0
SQL*Net message from client 204,628 0 4,620,111 22578 2.4
virtual circuit status 120 120 3,516 29297 0.0
PX Idle Wait 753 749 1,470 1953 0.0
SQL*Net more data from clien 20,540 0 3 0 0.2
PX Deq: Execution Msg 128 0 2 13 0.0
SQL*Net message to client 204,628 0 0 0 2.4
Background Wait Events for DB: XXXXX Instance: XXXXX Snaps: 1064 - 10
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
log file parallel write 166,634 0 82 0 2.0
db file parallel write 1,134 0 8 7 0.0
control file parallel write 1,200 0 2 1 0.0
control file sequential read 1,327 0 1 1 0.0
db file scattered read 44 0 0 10 0.0
db file sequential read 25 0 0 3 0.0
direct path read 23 0 0 3 0.0
rdbms ipc reply 67 0 0 0 0.0
LGWR wait for redo copy 286 0 0 0 0.0
direct path write 23 0 0 0 0.0
buffer busy waits 2 0 0 0 0.0
rdbms ipc message 86,933 3,437 20,199 232 1.0
pmon timer 1,194 1,194 3,504 2935 0.0
smon timer 17 8 3,239 ###### 0.0
Enqueue activity for DB: XXXXX Instance: XXXXX Snaps: 1064 - 1065
-> Enqueue stats gathered prior to 9i should not be compared with 9i data
-> ordered by Wait Time desc, Waits desc
Avg Wt Wait
Eq Requests Succ Gets Failed Gets Waits Time (ms) Time (s)
TC 25 24 0 4 32.00 0
TX 84,615 84,605 0 3 8.33 0
HW 118 118 0 2 2.00 0
PS 29 25 4 2 1.00 0
End of Report (this is not the entire report,only posted part of it,getting error page when trying to post the entire report)
Regards,
Prosenjit Mukherjee
Maybe you are looking for
-
Safari 5.1.7 keeps crashing after opening
Hi, safari keeps crashing and closing down after about 30 secs from launch. Has anyone else seen this and what are the best options. Thanks, Steve
-
Hi, I have just downloaded the "Win32 API Version 1.0 " and I am trying to compile it under the Solaris 2.6 and Forte 6.1/C++5.2/. I am facing some strange problems like socklen_t in Files.h not defined and so on. I don't know which version of Solari
-
Maxl on error write to does not create error file
The following MaxL statement functions except the creation of the dimbuild.err file. I have tried numerous combinations of syntax, server, folders without success. I am running this from the MaxL editor on the eas web server. import database XYAFIN.F
-
URGENT: Simple Query Returning Unexpected Info
Using the Oracle XSU for Java and one of the samples, i'm doing a simple select * from table. Everything is executing properly, however, the string output of the XML just returns what is documented in the release notes - that is a XML header and "</R
-
hi i cannot understand why my lightroom stop working