Log file sequential read and RFS ping/write - among Top 5 event
I have situation here to discuss. In a 3-node RAC setup which is Logical standby DB; one node is showing high CPU utilization around 40~50%. The CPU utilization was less than 20% 10 days back but from 9th oldest day it jumped and consistently shows the double figure. I ran AWR reports on all three nodes and found one node with high CPU utilization and shows below tops events-
EVENT WAITS TIME(S) AVG WAIT(MS) %TOTAL CALL TIME WAIT CLASS
CPU time 5,802 34.9
RFS ping 15 5,118 33,671 30.8 Other
Log file sequential read 234,831 5,036 21 30.3 System I/O
Sql*Net more data from
client 24,171 1,087 45 6.5 Network
Db file sequential read 130,939 453 3 2.7 User I/O
Findings:-
On AWR report(file attached) for node= sipd207; we can see that "RFS PING" wait event takes 30% of the waits and "log file sequential read" wait event takes 30% of the waits that occurs in database.
Environment :- (Oracle- 10.2.0.4.0, O/S - AIX .3)
1)other node awr shows "log file sync" - is it due to oversized log buffer?
2)Network wait events can be reduced by tweaking SDU & TDU values based on MDU.
3) Why ARCH processes taking much to archives filled redo logs; is it issue with slow disk I/O?
Regards
WORKLOAD REPOSITORY report for<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<DB Name DB Id Instance Inst Num Release RAC Host
XXXPDB 4123595889 XXX2p2 2 10.2.0.4.0 YES sipd207
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 1053 04-Apr-11 18:00:02 59 7.4
End Snap: 1055 04-Apr-11 20:00:35 56 7.5
Elapsed: 120.55 (mins)
DB Time: 233.08 (mins)
Cache Sizes
~~~~~~~~~~~ Begin End
Buffer Cache: 3,728M 3,728M Std Block Size: 8K
Shared Pool Size: 4,080M 4,080M Log Buffer: 14,332K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 245,392.33 10,042.66
Logical reads: 9,080.80 371.63
Block changes: 1,518.12 62.13
Physical reads: 7.50 0.31
Physical writes: 44.00 1.80
User calls: 36.44 1.49
Parses: 25.84 1.06
Hard parses: 0.59 0.02
Sorts: 12.06 0.49
Logons: 0.05 0.00
Executes: 295.91 12.11
Transactions: 24.43
% Blocks changed per Read: 16.72 Recursive Call %: 94.18
Rollback per transaction %: 4.15 Rows per Sort: 53.31
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.99 Redo NoWait %: 100.00
Buffer Hit %: 99.92 In-memory Sort %: 100.00
Library Hit %: 99.83 Soft Parse %: 97.71
Execute to Parse %: 91.27 Latch Hit %: 99.79
Parse CPU to Parse Elapsd %: 15.69 % Non-Parse CPU: 99.95
Shared Pool Statistics Begin End
Memory Usage %: 83.60 84.67
% SQL with executions>1: 97.49 97.19
% Memory for SQL w/exec>1: 97.10 96.67
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
CPU time 4,503 32.2
RFS ping 168 4,275 25449 30.6 Other
log file sequential read 183,537 4,173 23 29.8 System I/O
SQL*Net more data from client 21,371 1,009 47 7.2 Network
RFS write 25,438 343 13 2.5 System I/O
RAC Statistics DB/Inst: UDAS2PDB/udas2p2 Snaps: 1053-1055
Begin End
Number of Instances: 3 3
Global Cache Load Profile
~~~~~~~~~~~~~~~~~~~~~~~~~ Per Second Per Transaction
Global Cache blocks received: 0.78 0.03
Global Cache blocks served: 1.18 0.05
GCS/GES messages received: 131.69 5.39
GCS/GES messages sent: 139.26 5.70
DBWR Fusion writes: 0.06 0.00
Estd Interconnect traffic (KB) 68.60
Global Cache Efficiency Percentages (Target local+remote 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer access - local cache %: 99.91
Buffer access - remote cache %: 0.01
Buffer access - disk %: 0.08
Global Cache and Enqueue Services - Workload Characteristics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg global enqueue get time (ms): 0.5
Avg global cache cr block receive time (ms): 0.9
Avg global cache current block receive time (ms): 1.0
Avg global cache cr block build time (ms): 0.0
Avg global cache cr block send time (ms): 0.1
Global cache log flushes for cr blocks served %: 2.9
Avg global cache cr block flush time (ms): 4.6
Avg global cache current block pin time (ms): 0.0
Avg global cache current block send time (ms): 0.1
Global cache log flushes for current blocks served %: 0.1
Avg global cache current block flush time (ms): 5.0
Global Cache and Enqueue Services - Messaging Statistics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg message sent queue time (ms): 0.1
Avg message sent queue time on ksxp (ms): 0.6
Avg message received queue time (ms): 0.0
Avg GCS message process time (ms): 0.0
Avg GES message process time (ms): 0.1
% of direct sent messages: 31.57
% of indirect sent messages: 5.17
% of flow controlled messages: 63.26
Time Model Statistics DB/Inst: UDAS2PDB/udas2p2 Snaps: 1053-1055
-> Total time in database user-calls (DB Time): 13984.6s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 7,270.6 52.0
DB CPU 4,503.1 32.2
parse time elapsed 506.7 3.6
hard parse elapsed time 497.8 3.6
sequence load elapsed time 152.4 1.1
failed parse elapsed time 19.5 .1
repeated bind elapsed time 3.4 .0
PL/SQL execution elapsed time 0.7 .0
hard parse (sharing criteria) elapsed time 0.3 .0
connection management call elapsed time 0.3 .0
hard parse (bind mismatch) elapsed time 0.0 .0
DB time 13,984.6 N/A
background elapsed time 869.1 N/A
background cpu time 276.6 N/A
Wait Class DB/Inst: UDAS2PDB/udas2p2 Snaps: 1053-1055
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc
Avg
%Time Total Wait wait Waits
Wait Class Waits -outs Time (s) (ms) /txn
System I/O 529,934 .0 4,980 9 3.0
Other 582,349 37.4 4,611 8 3.3
Network 279,858 .0 1,009 4 1.6
User I/O 54,899 .0 317 6 0.3
Concurrency 136,907 .1 58 0 0.8
Cluster 60,300 .0 41 1 0.3
Commit 80 .0 10 130 0.0
Application 6,707 .0 3 0 0.0
Configuration 17,528 98.5 1 0 0.1
Wait Events DB/Inst: UDAS2PDB/udas2p2 Snaps: 1053-1055
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
RFS ping 168 .0 4,275 25449 0.0
log file sequential read 183,537 .0 4,173 23 1.0
SQL*Net more data from clien 21,371 .0 1,009 47 0.1
RFS write 25,438 .0 343 13 0.1
db file sequential read 54,680 .0 316 6 0.3
DFS lock handle 97,149 .0 214 2 0.5
log file parallel write 104,808 .0 157 2 0.6
db file parallel write 143,905 .0 149 1 0.8
RFS random i/o 25,438 .0 86 3 0.1
RFS dispatch 25,610 .0 56 2 0.1
control file sequential read 39,309 .0 55 1 0.2
row cache lock 130,665 .0 47 0 0.7
gc current grant 2-way 35,498 .0 23 1 0.2
wait for scn ack 50,872 .0 20 0 0.3
enq: WL - contention 6,156 .0 14 2 0.0
gc cr grant 2-way 16,917 .0 11 1 0.1
log file sync 80 .0 10 130 0.0
Log archive I/O 3,986 .0 9 2 0.0
control file parallel write 3,493 .0 8 2 0.0
latch free 2,356 .0 6 2 0.0
ksxr poll remote instances 278,473 49.4 6 0 1.6
enq: XR - database force log 2,890 .0 4 1 0.0
enq: TX - index contention 325 .0 3 11 0.0
buffer busy waits 4,371 .0 3 1 0.0
gc current block 2-way 3,002 .0 3 1 0.0
LGWR wait for redo copy 9,601 .2 2 0 0.1
SQL*Net break/reset to clien 6,438 .0 2 0 0.0
latch: ges resource hash lis 23,223 .0 2 0 0.1
enq: WF - contention 32 6.3 2 62 0.0
enq: FB - contention 660 .0 2 2 0.0
enq: PS - contention 1,088 .0 2 1 0.0
library cache lock 869 .0 1 2 0.0
enq: CF - contention 671 .1 1 2 0.0
gc current grant busy 1,488 .0 1 1 0.0
gc current multi block reque 1,072 .0 1 1 0.0
reliable message 618 .0 1 2 0.0
CGS wait for IPC msg 62,402 100.0 1 0 0.4
gc current block 3-way 998 .0 1 1 0.0
name-service call wait 18 .0 1 57 0.0
cursor: pin S wait on X 78 100.0 1 11 0.0
os thread startup 16 .0 1 53 0.0
enq: RO - fast object reuse 193 .0 1 3 0.0
IPC send completion sync 652 99.2 1 1 0.0
local write wait 194 .0 1 3 0.0
gc cr block 2-way 534 .0 0 1 0.0
log file switch completion 17 .0 0 20 0.0
SQL*Net message to client 258,483 .0 0 0 1.5
undo segment extension 17,282 99.9 0 0 0.1
gc cr block 3-way 286 .7 0 1 0.0
enq: TM - contention 76 .0 0 4 0.0
PX Deq: reap credit 15,246 95.6 0 0 0.1
kksfbc child completion 5 100.0 0 49 0.0
enq: TT - contention 141 .0 0 2 0.0
enq: HW - contention 203 .0 0 1 0.0
RFS create 2 .0 0 115 0.0
rdbms ipc reply 339 .0 0 1 0.0
PX Deq Credit: send blkd 452 20.1 0 0 0.0
gcs log flush sync 128 32.8 0 2 0.0
latch: cache buffers chains 128 .0 0 1 0.0
library cache pin 441 .0 0 0 0.0
Wait Events DB/Inst: UDAS2PDB/udas2p2 Snaps: 1053-1055
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
We only apply on one node in a cluster so I would expect that the node running SQL Apply would have much higher usage and waits. Is this what you are asking?
Larry
Similar Messages
-
Very high log file sequential read and control file sequential read waits?
I have a 10.2.0.4 database and have 5 streams capture processes running to replicate data to another database. However I am seeing very high
log file sequential read and control file sequential read by the capture procesess. This is causing slowness in the database as the databass is wasting so much time on these wait events. From AWR report
Elapsed: 20.12 (mins)
DB Time: 67.04 (mins)
and From top 5 wait events
Event Waits Time(s) Avg Wait(ms) % Total Call Time Wait Class
CPU time 1,712 42.6
log file sequential read 99,909 683 7 17.0 System I/O
log file sync 49,702 426 9 10.6 Commit
control file sequential read262,625 384 1 9.6 System I/O
db file sequential read 41,528 378 9 9.4 User I/O
Oracle support hasn't been of much help, other than wasting my 10 days and telling me to try this and try that.
Do you have streams running in your environment, are you experiencing this wait. Have you done anything to resolve these waits..
ThanksWelcome to the forums.
There is insufficient information in what you have posted to know that your analysis of the situation is correct or anything about your Streams environment.
We don't know what you are replicating. Not size, not volume, not type of capture, not rules, etc.
We don't know the distance over which it is being replicated ... 10 ft. or 10 light years.
We don't have any AWR or ASH data to look at.
etc. etc. etc. If this is what you provided Oracle Support it is no wonder they were unable to help you.
To diagnose this problem, if one exists, requires someone on-site or with a very substantial body of data which you have not provided. The first step is to fill in the answers to all of the obvious first level questions. Then we will likely come back with a second level of questioning.
But when you do ... do not post here. Your questions are not "Database General" they are specific to Streams and there is a Streams forum specifically for them.
Thank you. -
Difference between db file sequential read and scattered read
Hi,
Oracle Version : 10.2.0.1
Operating system: Linux
Can any one please help me what is the difference between db file sequential read and scattered read or please give any best related links .
Thanks & Regards,
Poorna Prasad.>
A sequential read is a single-block read. Single block I/Os are usually the result of using indexes.
A db file scattered read issues a scattered read to read the data into multiple discontinuous memory locations. A scattered read is usually a multiblock read. It can occur for a fast full scan (of an index) in addition to a full table scan.
>
See Performance Tuning Guide:
http://download.oracle.com/docs/cd/B19306_01/server.102/b14211/instance_tune.htm#i20526
http://download.oracle.com/docs/cd/B19306_01/server.102/b14211/instance_tune.htm#i15958
Edited by: P. Forstmann on 20 oct. 2009 09:11 -
Update Statement Simply hanged but doing db file sequential read
Hi,
Last night we had issue with one of the prod server where we updating one of table which contains large number records in millions.Same identical machine completed in1 hour and other box never completed but doing db file sequential read but in the long ops the last statement it was done 20:16 after that nothing is happening but i ran few trace on that user.
/u01/app/oracle/admin/SURV2/udump/surv2_ora_10048.trc
Oracle Database 10g Release 10.2.0.4.0 - Production
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
System name: SunOS
Node name: prdfa001
Release: 5.10
Version: Generic_139556-08
Machine: i86pc
Instance name: SURV2
Redo thread mounted by this instance: 1
Oracle process number: 18
Unix process pid: 10048, image: oracle@prdfa001
*** 2010-09-09 23:37:07.484
*** ACTION NAME:() 2010-09-09 23:37:07.473
*** MODULE NAME:(SQL*Plus) 2010-09-09 23:37:07.473
*** SERVICE NAME:(SURV2) 2010-09-09 23:37:07.473
*** SESSION ID:(289.54) 2010-09-09 23:37:07.473
Received ORADEBUG command 'unlimit' from process Unix process pid: 3983, image:
*** 2010-09-09 23:37:20.315
Received ORADEBUG command 'event 10046 trace name context forever, level 12' from process Unix process pid: 3983, image:
WAIT #7: nam='db file sequential read' ela= 11160 file#=13 block#=2252349 blocks=1 obj#=166421 tim=12499462835161
WAIT #7: nam='db file sequential read' ela= 2857 file#=13 block#=2249751 blocks=1 obj#=166421 tim=12499462838137
WAIT #7: nam='db file sequential read' ela= 3810 file#=13 block#=2251361 blocks=1 obj#=166421 tim=12499462842048
WAIT #7: nam='db file sequential read' ela= 4459 file#=13 block#=2247059 blocks=1 obj#=166421 tim=12499462846564
WAIT #7: nam='db file sequential read' ela= 2841 file#=13 block#=2247507 blocks=1 obj#=166421 tim=12499462849468
WAIT #7: nam='db file sequential read' ela= 427 file#=13 block#=2247568 blocks=1 obj#=166421 tim=12499462850032
WAIT #7: nam='db file sequential read' ela= 1187 file#=13 block#=2248264 blocks=1 obj#=166421 tim=12499462851327
WAIT #7: nam='db file sequential read' ela= 2687 file#=13 block#=2250707 blocks=1 obj#=166421 tim=12499462854178
WAIT #7: nam='db file sequential read' ela= 3657 file#=13 block#=2249697 blocks=1 obj#=166421 tim=12499462857896
WAIT #7: nam='db file sequential read' ela= 4139 file#=13 block#=2247074 blocks=1 obj#=166421 tim=12499462862093
WAIT #7: nam='db file sequential read' ela= 4180 file#=47 block#=3649690 blocks=1 obj#=166421 tim=12499509270445
WAIT #7: nam='db file sequential read' ela= 4802 file#=47 block#=3649309 blocks=1 obj#=166421 tim=12499509275327
WAIT #7: nam='db file sequential read' ela= 2459 file#=47 block#=3652697 blocks=1 obj#=166421 tim=12499509277859
WAIT #7: nam='db file sequential read' ela= 4015 file#=47 block#=3652826 blocks=1 obj#=166421 tim=12499509281948
WAIT #7: nam='db file sequential read' ela= 2248 file#=47 block#=3651610 blocks=1 obj#=166421 tim=12499509284269
WAIT #7: nam='db file sequential read' ela= 4824 file#=47 block#=3654297 blocks=1 obj#=166421 tim=12499509289166
WAIT #7: nam='db file sequential read' ela= 2008 file#=47 block#=3652312 blocks=1 obj#=166421 tim=12499509291248
WAIT #7: nam='db file sequential read' ela= 1925 file#=47 block#=3654490 blocks=1 obj#=166421 tim=12499509293246
WAIT #7: nam='db file sequential read' ela= 2859 file#=47 block#=3648458 blocks=1 obj#=166421 tim=12499509296178
WAIT #7: nam='db file sequential read' ela= 1740 file#=47 block#=3648212 blocks=1 obj#=166421 tim=12499509297991
WAIT #7: nam='db file sequential read' ela= 2566 file#=47 block#=3648411 blocks=1 obj#=166421 tim=12499509300631
WAIT #7: nam='db file sequential read' ela= 50772 file#=5 block#=480749 blocks=1 obj#=166421 tim=12499509351477
WAIT #7: nam='db file sequential read' ela= 12928 file#=5 block#=477177 blocks=1 obj#=166421 tim=12499509364482
WAIT #7: nam='db file sequential read' ela= 11116 file#=5 block#=479412 blocks=1 obj#=166421 tim=12499509375672
WAIT #7: nam='db file sequential read' ela= 4803 file#=5 block#=483440 blocks=1 obj#=166421 tim=12499509380549
WAIT #7: nam='db file sequential read' ela= 6900 file#=5 block#=481454 blocks=1 obj#=166421 tim=12499509387522
Received ORADEBUG command 'event 10046 trace name context off' from process Unix process pid: 3983, image:
/u01/app/oracle/admin/SURV2/udump/surv2_ora_1545.trc
Oracle Database 10g Release 10.2.0.4.0 - Production
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
System name: SunOS
Node name: prdfa001
Release: 5.10
Version: Generic_139556-08
Machine: i86pc
Instance name: SURV2
Redo thread mounted by this instance: 1
Oracle process number: 22
Unix process pid: 1545, image: oracle@prdfa001 (TNS V1-V3)
*** ACTION NAME:() 2010-09-09 23:20:13.485
*** MODULE NAME:(sqlplus@prdfa001 (TNS V1-V3)) 2010-09-09 23:20:13.485
*** SERVICE NAME:(SYS$USERS) 2010-09-09 23:20:13.485
*** SESSION ID:(290.697) 2010-09-09 23:20:13.485
===================================================
SYSTEM STATE
System global information:
processes: base 47819b480, size 300, cleanup 4781a5638
allocation: free sessions 47f1d6148, free calls 0
control alloc errors: 0 (process), 0 (session), 0 (call)
PMON latch cleanup depth: 0
seconds since PMON's last scan for dead processes: 20
system statistics:
1171 logons cumulative
19 logons current
89219 opened cursors cumulative
86 opened cursors current
15095069 user commits
5 user rollbacks
58632904 user calls
44023255 recursive calls
224311 recursive cpu usage
201424173 session logical reads
0 session stored procedure space
901812 CPU used when call started
995437 CPU used by this session
6814196 DB time
0 cluster wait time
22542300822 concurrency wait time
3095 application wait time
16479074661 user I/O wait time
1284052668 session connect time
1284067190 process last non-idle time
189018343568 session uga memory
1249667216 session uga memory max
26059216 messages sent
26059220 messages received
239739 background timeouts
162399896 session pga memory
189662872 session pga memory max
4 enqueue timeouts
901146 enqueue waits
0 enqueue deadlocks
32122711 enqueue requests
17819 enqueue conversions
32122676 enqueue releases
0 global enqueue gets sync
0 global enqueue gets async
0 global enqueue get time
0 global enqueue releases
2865667 physical read total IO requests
262620 physical read total multi block requests
270093476864 physical read total bytes
select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'SERVICE_NAME'), INSTANCE_NUMBER, STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN') from v$instance where INSTANCE_NAME=SYS_CONTEXT('USERENV', 'INSTANCE_NAME')
hash=550c95f3d0cfa8290e60ea8382d3a2ca timestamp=09-09-2010 04:24:19
namespace=CRSR flags=RON/KGHP/TIM/PN0/LRG/KST/DBN/MTX/[100100d1]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=9 hpc=0582 hlc=0582
lwt=47df576e8[47df576e8,47df576e8] ltm=47df576f8[47df576f8,47df576f8]
pwt=47df576b0[47df576b0,47df576b0] ptm=47df576c0[47df576c0,47df576c0]
ref=47df57718[47df57718,47df57718] lnd=47df57730[47df57730,47df57730]
LIBRARY OBJECT: object=471ee1d38
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
CHILDREN: size=16
child# table reference handle
0 471ee1800 471ee1470 47df7dce0
DATA BLOCKS:
data# heap pointer status pins change whr
0 47df7de48 471ee1e50 I/P/A/-/- 0 NONE 00
SO: 473691d60, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=473691d60 handle=47bb22fa0 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=473691de0[4735dbcb8,476cfbf58] htb=476cfbf58 ssga=476cfb6a0
user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x0
LIBRARY OBJECT HANDLE: handle=47bb22fa0 mtx=47bb230d0(0) cdp=0
namespace=CRSR flags=RON/KGHP/PN0/EXP/[10010100]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=fd84 hlc=fd84
lwt=47bb23048[47bb23048,47bb23048] ltm=47bb23058[47bb23058,47bb23058]
pwt=47bb23010[47bb23010,47bb23010] ptm=47bb23020[47bb23020,47bb23020]
ref=47bb23078[472f8de18,472f8de18] lnd=47bb23090[47bb23090,47bb23090]
LIBRARY OBJECT: object=472f8d9d8
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
DEPENDENCIES: count=1 size=16
AUTHORIZATIONS: count=1 size=16 minimum entrysize=16
ACCESSES: count=1 size=16
TRANSLATIONS: count=1 size=16
DATA BLOCKS:
data# heap pointer status pins change whr
0 47bb22ee0 472f8daf0 I/P/A/-/- 0 NONE 00
6 472f8e508 46be86250 I/-/A/-/E 0 NONE 00
SO: 4735dbc38, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=4735dbc38 handle=47bb231c8 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=4735dbcb8[476cfbf58,473691de0] htb=476cfbf58 ssga=476cfb6a0
user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x4c894f8b
LIBRARY OBJECT HANDLE: handle=47bb231c8 mtx=47bb232f8(1) cdp=1
name=select value$ from props$ where name = 'GLOBAL_DB_NAME'
hash=4bb432d65c5a391a42a5c3fa74472c7a timestamp=09-09-2010 04:24:12
namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/KST/DBN/MTX/[120100d0]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=0584 hlc=0584
lwt=47bb23270[47bb23270,47bb23270] ltm=47bb23280[47bb23280,47bb23280]
pwt=47bb23238[47bb23238,47bb23238] ptm=47bb23248[47bb23248,47bb23248]
ref=47bb232a0[47bb232a0,47bb232a0] lnd=47bb232b8[47bb232b8,47bb232b8]
LIBRARY OBJECT: object=472f8e6e0
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
CHILDREN: size=16
child# table reference handle
0 472f8e1a8 472f8de18 47bb22fa0
DATA BLOCKS:
data# heap pointer status pins change whr
0 47bb23108 472f8e7f8 I/P/A/-/- 0 NONE 00
SO: 473644348, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=473644348 handle=47bbde418 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=4736443c8[476cfc0b8,476cfc0b8] htb=476cfc0b8 ssga=476cfb6a0
user=47924e810 session=47924e810 count=1 flags=[0000] savepoint=0x4c894f8b
LIBRARY OBJECT HANDLE: handle=47bbde418 mtx=47bbde548(0) cdp=0
name=ALTER SESSION SET TIME_ZONE='+02:00'
hash=3878dff8839e71e3dd05a2e75fbd6390 timestamp=09-09-2010 04:24:04
namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/DBN/[12010040]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=11 hpc=04e8 hlc=04e8
lwt=47bbde4c0[47bbde4c0,47bbde4c0] ltm=47bbde4d0[47bbde4d0,47bbde4d0]
pwt=47bbde488[47bbde488,47bbde488] ptm=47bbde498[47bbde498,47bbde498]
ref=47bbde4f0[47bbde4f0,47bbde4f0] lnd=47bbde508[47bbde508,47bbde508]
LIBRARY OBJECT: object=472fffc08
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
DATA BLOCKS:
data# heap pointer status pins change whr
0 47bbde320 472fffd20 I/P/A/-/- 0 NONE 00
SO: 47aecf9e8, type: 41, owner: 47924e810, flag: INIT/-/-/0x00
(dummy) nxc=0, nlb=0
SO: 47f290540, type: 11, owner: 4781a7dc0, flag: INIT/-/-/0x00
(broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: 4781a7dc0,
event: 1132, last message event: 1132,
last message waited event: 1132, next message: 0(0), messages read: 0
channel: (47a2df4f8) system events broadcast channel
scope: 2, event: 1132, last mesage event: 18,
publishers/subscribers: 0/17,
messages published: 1
SO: 47826b228, type: 3, owner: 4781a7dc0, flag: INIT/-/-/0x00
(call) sess: cur 47924e810, rec 0, usr 47924e810; depth: 0
SO: 476c52968, type: 16, owner: 4781a7dc0, flag: INIT/-/-/0x00
(osp req holder)
PSEUDO PROCESS for group DEFAULT:
SO: 47a1eb7d0, type: 2, owner: 0, flag: INIT/-/-/0x00
(process) Oracle pid=0, calls cur/top: 0/0, flag: (20) PSEUDO
int error: 0, call error: 0, sess error: 0, txn error 0
(post info) last post received: 0 0 0
last post received-location: No post
last process to post me: none
last post sent: 0 0 0
last post sent-location: No post
last process posted by me: none
(latch info) wait_event=0 bits=0
Process Group: DEFAULT, pseudo proc: 47a1eb7d0
O/S info: user: , term: , ospid: (DEAD)
OSD pid info: Unix process pid: 0, image: PSEUDO
Dump of memory from 0x00000004791BF538 to 0x00000004791BF740
4791BF530 00000000 00000000 [........]
4791BF540 00000000 00000000 00000000 00000000 [................]
Repeat 31 times
NO DETACHED BRANCHES.
NO DETACHED NETWORK CONNECTIONS.
CLEANUP STATE OBJECTS:
SO: 47f0cd038, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: instance enqueue anchor state
latch: 0x380009890
SO: 4782cf080, type: 5, owner: 47f0cd038, flag: INIT/-/-/0x00
(enqueue) TA-00000006-00000001 DID: 0001-000F-0000000B
lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 res_flag: 0x2
res: 0x47a28d020, mode: X, lock_flag: 0x0
own: 0x0, sess: 0x0, prv: 0x47a28d030
SO: 47f0cd098, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: switchable channel handle anch
latch: 0x38000ac98
SO: 47f28f868, type: 11, owner: 47f0cd098, flag: INIT/-/-/0x00
(broadcast handle) flag: (c2) ACTIVE SUBSCRIBER, owner: 0,
event: 1, last message event: 1,
last message waited event: 1, next message: 0(0), messages read: 0
channel: (47a2e4190) KPON channel
scope: 2, event: 1, last mesage event: 0,
publishers/subscribers: 0/1,
messages published: 0
SO: 47f0cd0f8, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: TT shared object cleanup SO
latch: 0x38001c6b8
SO: 47f0cd158, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: SS shared object cleanup SO
latch: 0x38001cd48
END OF SYSTEM STATE
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
db file sequential read 2,347,652 9,215 4 64.5 User I/O
db file scattered read 245,687 4,199 17 29.4 User I/O
CPU time 974 6.8
db file parallel write 50,082 408 8 2.9 System I/O
log file parallel write 6,963 52 7 0.4 System I/O
Time Model Statistics DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> Total time in database user-calls (DB Time): 14286.4s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 14,280.3 100.0
DB CPU 974.5 6.8
PL/SQL execution elapsed time 531.8 3.7
parse time elapsed 30.5 .2
hard parse elapsed time 27.1 .2
connection management call elapsed time 14.9 .1
hard parse (sharing criteria) elapsed time 3.4 .0
hard parse (bind mismatch) elapsed time 3.1 .0
PL/SQL compilation elapsed time 2.4 .0
failed parse elapsed time 0.0 .0
repeated bind elapsed time 0.0 .0
sequence load elapsed time 0.0 .0
DB time 14,286.4 N/A
background elapsed time 670.2 N/A
background cpu time 186.1 N/A
Wait Class DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc
Avg
%Time Total Wait wait Waits
Wait Class Waits -outs Time (s) (ms) /txn
User I/O 2,593,484 .0 13,415 5 150.0
System I/O 87,506 .0 515 6 5.1
Other 839 11.4 6 7 0.0
Commit 3,225 .1 6 2 0.2
Concurrency 1,033 .0 5 5 0.1
Configuration 2,514 99.4 0 0 0.1
Network 47,559 .0 0 0 2.8
Application 7 .0 0 0 0.0
Wait Events DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
db file sequential read 2,347,652 .0 9,215 4 135.8
db file scattered read 245,687 .0 4,199 17 14.2
db file parallel write 50,082 .0 408 8 2.9
log file parallel write 6,963 .0 52 7 0.4
control file parallel write 6,203 .0 44 7 0.4
control file sequential read 24,242 .0 11 0 1.4
log file sync 3,225 .1 6 2 0.2
latch free 84 .0 4 47 0.0
os thread startup 25 .0 3 120 0.0
latch: session allocation 39 .0 1 33 0.0
db file parallel read 12 .0 1 92 0.0
enq: TX - index contention 186 .0 1 3 0.0
latch: shared pool 47 .0 1 11 0.0
LGWR wait for redo copy 319 3.1 0 1 0.0
library cache load lock 2 .0 0 172 0.0
buffer busy waits 590 .0 0 0 0.0
log file switch completion 6 .0 0 29 0.0
SGA: allocation forcing comp 11 54.5 0 14 0.0
latch: library cache lock 50 .0 0 3 0.0
read by other session 38 .0 0 4 0.0
direct path read 42 .0 0 3 0.0
SQL*Net message to client 44,807 .0 0 0 2.6
rdbms ipc reply 207 .0 0 0 0.0
SQL*Net more data from clien 1,014 .0 0 0 0.1
latch: cache buffers chains 24 .0 0 1 0.0
latch: library cache 29 .0 0 1 0.0
log file sequential read 8 .0 0 3 0.0
direct path write 50 .0 0 0 0.0
SQL*Net more data to client 398 .0 0 0 0.0
latch: object queue header o 12 .0 0 1 0.0
latch: In memory undo latch 78 .0 0 0 0.0
undo segment extension 2,507 99.7 0 0 0.1
latch: cache buffers lru cha 4 .0 0 1 0.0
log file single write 8 .0 0 0 0.0
local write wait 3 .0 0 1 0.0
enq: RO - fast object reuse 3 .0 0 1 0.0
buffer deadlock 87 92.0 0 0 0.0
enq: JS - queue lock 1 .0 0 1 0.0
cursor: pin S 70 .0 0 0 0.0
latch: row cache objects 2 .0 0 1 0.0
SQL*Net message to dblink 1,338 .0 0 0 0.1
latch: checkpoint queue latc 2 .0 0 0 0.0
reliable message 3 .0 0 0 0.0
log buffer space 1 .0 0 1 0.0
SQL*Net break/reset to clien 4 .0 0 0 0.0
SQL*Net more data from dblin 2 .0 0 0 0.0
SQL*Net message from client 44,949 .0 155,701 3464 2.6
virtual circuit status 621 100.0 18,156 29237 0.0
Streams AQ: qmn slave idle w 664 .0 18,127 27299 0.0
Streams AQ: qmn coordinator 1,339 50.4 18,099 13517 0.1
Streams AQ: waiting for time 12 100.0 8,741 728394 0.0
jobq slave wait 130 100.0 380 2927 0.0
PL/SQL lock timer 1 100.0 1 978 0.0
SQL*Net message from dblink 1,338 .0 0 0 0.1
single-task message 1 .0 0 38 0.0
class slave wait 11 .0 0 1 0.0
SQL ordered by Elapsed Time DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> Resources reported for PL/SQL code includes the resources used by all SQL
statements called by the code.
-> % Total DB Time is the Elapsed Time of the SQL statement divided
into the Total Database Time multiplied by 100
Elapsed CPU Elap per % Total
Time (s) Time (s) Executions Exec (s) DB Time SQL Id
13,664 906 0 N/A 95.6 gr2cx6athc5j5
Module: SQL*Plus
BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
8,792 195 0 N/A 61.5 986fzxtzr52u5
Module: SQL*Plus
UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
2,524 368 1 2524.1 17.7 c4uf0x6hdgnwq
Module: SQL*Plus
UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
SYS_B_1"
1,414 177 1 1414.4 9.9 cbg09ma34kq8w
Module: SQL*Plus
SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
742 137 1 742.2 5.2 g0sg6v994wssq
Module: SQL*Plus
SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
274 11 1 274.2 1.9 6mcpb06rctk0x
Module: DBMS_SCHEDULER
call dbms_space.auto_space_advisor_job_proc ( )
264 8 27 9.8 1.8 8szmwam7fysa3
Module: DBMS_SCHEDULER
insert into wri$_adv_objspace_trend_data select timepoint, space_usage, space_a
lloc, quality from table(dbms_space.object_growth_trend(:1, :2, :3, :4, NULL, N
ULL, NULL, 'FALSE', :5, 'FALSE'))
99 1 1 99.4 0.7 1z0x41f66nvjr
Module: SQL*Plus
UPDATE TIBEX_INSTRUMENTADMIN SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
SYS_B_1"
21 10 1 21.5 0.2 bbc1ck8594kvj
Module: SQL*Plus
UPDATE TIBEX_INSTRUMENTDAILYHIST SET ADJOPEN=NVL(ADJOPEN,OPEN), ADJHIGH=NVL(ADJH
IGH,HIGH), ADJLOW=NVL(ADJLOW,LOW), ADJMID=NVL(ADJMID,MID), ADJCLOSE=NVL(ADJCLOSE
,CLOSE), ADJVOLUME=NVL(ADJVOLUME,VOLUME), ADJCLOSINGBID=NVL(ADJCLOSINGBID,CLOSIN
GBID), ADJCLOSINGOFFER=NVL(ADJCLOSINGOFFER,CLOSINGOFFER)
12 0 1 12.5 0.1 6xm9p9uy5kaap
Module: SQL*Plus
SELECT count(*) FROM TIBEX_INSTRUMENTSTATE WHERE INSTRUMENTID=:"SYS_B_0"
SQL ordered by CPU Time DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> Resources reported for PL/SQL code includes the resources used by all SQL
statements called by the code.
-> % Total DB Time is the Elapsed Time of the SQL statement divided
into the Total Database Time multiplied by 100
CPU Elapsed CPU per % Total
Time (s) Time (s) Executions Exec (s) DB Time SQL Id
906 13,664 0 N/A 95.6 gr2cx6athc5j5
Module: SQL*Plus
BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
368 2,524 1 367.51 17.7 c4uf0x6hdgnwq
Module: SQL*Plus
UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
SYS_B_1"
195 8,792 0 N/A 61.5 986fzxtzr52u5
Module: SQL*Plus
UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
177 1,414 1 176.93 9.9 cbg09ma34kq8w
Module: SQL*Plus
SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
137 742 1 137.38 5.2 g0sg6v994wssq
Module: SQL*Plus
SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
11 274 1 10.82 1.9 6mcpb06rctk0x
Module: DBMS_SCHEDULER
call dbms_space.auto_space_advisor_job_proc ( )
10 21 1 9.65 0.2 bbc1ck8594kvjEdited by: NM on 10-Sep-2010 07:39Hi,
Last night we had issue with one of the prod server where we updating one of table which contains large number records in millions.Same identical machine completed in1 hour and other box never completed but doing db file sequential read but in the long ops the last statement it was done 20:16 after that nothing is happening but i ran few trace on that user.
/u01/app/oracle/admin/SURV2/udump/surv2_ora_10048.trc
Oracle Database 10g Release 10.2.0.4.0 - Production
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
System name: SunOS
Node name: prdfa001
Release: 5.10
Version: Generic_139556-08
Machine: i86pc
Instance name: SURV2
Redo thread mounted by this instance: 1
Oracle process number: 18
Unix process pid: 10048, image: oracle@prdfa001
*** 2010-09-09 23:37:07.484
*** ACTION NAME:() 2010-09-09 23:37:07.473
*** MODULE NAME:(SQL*Plus) 2010-09-09 23:37:07.473
*** SERVICE NAME:(SURV2) 2010-09-09 23:37:07.473
*** SESSION ID:(289.54) 2010-09-09 23:37:07.473
Received ORADEBUG command 'unlimit' from process Unix process pid: 3983, image:
*** 2010-09-09 23:37:20.315
Received ORADEBUG command 'event 10046 trace name context forever, level 12' from process Unix process pid: 3983, image:
WAIT #7: nam='db file sequential read' ela= 11160 file#=13 block#=2252349 blocks=1 obj#=166421 tim=12499462835161
WAIT #7: nam='db file sequential read' ela= 2857 file#=13 block#=2249751 blocks=1 obj#=166421 tim=12499462838137
WAIT #7: nam='db file sequential read' ela= 3810 file#=13 block#=2251361 blocks=1 obj#=166421 tim=12499462842048
WAIT #7: nam='db file sequential read' ela= 4459 file#=13 block#=2247059 blocks=1 obj#=166421 tim=12499462846564
WAIT #7: nam='db file sequential read' ela= 2841 file#=13 block#=2247507 blocks=1 obj#=166421 tim=12499462849468
WAIT #7: nam='db file sequential read' ela= 427 file#=13 block#=2247568 blocks=1 obj#=166421 tim=12499462850032
WAIT #7: nam='db file sequential read' ela= 1187 file#=13 block#=2248264 blocks=1 obj#=166421 tim=12499462851327
WAIT #7: nam='db file sequential read' ela= 2687 file#=13 block#=2250707 blocks=1 obj#=166421 tim=12499462854178
WAIT #7: nam='db file sequential read' ela= 3657 file#=13 block#=2249697 blocks=1 obj#=166421 tim=12499462857896
WAIT #7: nam='db file sequential read' ela= 4139 file#=13 block#=2247074 blocks=1 obj#=166421 tim=12499462862093
WAIT #7: nam='db file sequential read' ela= 4180 file#=47 block#=3649690 blocks=1 obj#=166421 tim=12499509270445
WAIT #7: nam='db file sequential read' ela= 4802 file#=47 block#=3649309 blocks=1 obj#=166421 tim=12499509275327
WAIT #7: nam='db file sequential read' ela= 2459 file#=47 block#=3652697 blocks=1 obj#=166421 tim=12499509277859
WAIT #7: nam='db file sequential read' ela= 4015 file#=47 block#=3652826 blocks=1 obj#=166421 tim=12499509281948
WAIT #7: nam='db file sequential read' ela= 2248 file#=47 block#=3651610 blocks=1 obj#=166421 tim=12499509284269
WAIT #7: nam='db file sequential read' ela= 4824 file#=47 block#=3654297 blocks=1 obj#=166421 tim=12499509289166
WAIT #7: nam='db file sequential read' ela= 2008 file#=47 block#=3652312 blocks=1 obj#=166421 tim=12499509291248
WAIT #7: nam='db file sequential read' ela= 1925 file#=47 block#=3654490 blocks=1 obj#=166421 tim=12499509293246
WAIT #7: nam='db file sequential read' ela= 2859 file#=47 block#=3648458 blocks=1 obj#=166421 tim=12499509296178
WAIT #7: nam='db file sequential read' ela= 1740 file#=47 block#=3648212 blocks=1 obj#=166421 tim=12499509297991
WAIT #7: nam='db file sequential read' ela= 2566 file#=47 block#=3648411 blocks=1 obj#=166421 tim=12499509300631
WAIT #7: nam='db file sequential read' ela= 50772 file#=5 block#=480749 blocks=1 obj#=166421 tim=12499509351477
WAIT #7: nam='db file sequential read' ela= 12928 file#=5 block#=477177 blocks=1 obj#=166421 tim=12499509364482
WAIT #7: nam='db file sequential read' ela= 11116 file#=5 block#=479412 blocks=1 obj#=166421 tim=12499509375672
WAIT #7: nam='db file sequential read' ela= 4803 file#=5 block#=483440 blocks=1 obj#=166421 tim=12499509380549
WAIT #7: nam='db file sequential read' ela= 6900 file#=5 block#=481454 blocks=1 obj#=166421 tim=12499509387522
Received ORADEBUG command 'event 10046 trace name context off' from process Unix process pid: 3983, image:
/u01/app/oracle/admin/SURV2/udump/surv2_ora_1545.trc
Oracle Database 10g Release 10.2.0.4.0 - Production
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
System name: SunOS
Node name: prdfa001
Release: 5.10
Version: Generic_139556-08
Machine: i86pc
Instance name: SURV2
Redo thread mounted by this instance: 1
Oracle process number: 22
Unix process pid: 1545, image: oracle@prdfa001 (TNS V1-V3)
*** ACTION NAME:() 2010-09-09 23:20:13.485
*** MODULE NAME:(sqlplus@prdfa001 (TNS V1-V3)) 2010-09-09 23:20:13.485
*** SERVICE NAME:(SYS$USERS) 2010-09-09 23:20:13.485
*** SESSION ID:(290.697) 2010-09-09 23:20:13.485
===================================================
SYSTEM STATE
System global information:
processes: base 47819b480, size 300, cleanup 4781a5638
allocation: free sessions 47f1d6148, free calls 0
control alloc errors: 0 (process), 0 (session), 0 (call)
PMON latch cleanup depth: 0
seconds since PMON's last scan for dead processes: 20
system statistics:
1171 logons cumulative
19 logons current
89219 opened cursors cumulative
86 opened cursors current
15095069 user commits
5 user rollbacks
58632904 user calls
44023255 recursive calls
224311 recursive cpu usage
201424173 session logical reads
0 session stored procedure space
901812 CPU used when call started
995437 CPU used by this session
6814196 DB time
0 cluster wait time
22542300822 concurrency wait time
3095 application wait time
16479074661 user I/O wait time
1284052668 session connect time
1284067190 process last non-idle time
189018343568 session uga memory
1249667216 session uga memory max
26059216 messages sent
26059220 messages received
239739 background timeouts
162399896 session pga memory
189662872 session pga memory max
4 enqueue timeouts
901146 enqueue waits
0 enqueue deadlocks
32122711 enqueue requests
17819 enqueue conversions
32122676 enqueue releases
0 global enqueue gets sync
0 global enqueue gets async
0 global enqueue get time
0 global enqueue releases
2865667 physical read total IO requests
262620 physical read total multi block requests
270093476864 physical read total bytes
select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'SERVICE_NAME'), INSTANCE_NUMBER, STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN') from v$instance where INSTANCE_NAME=SYS_CONTEXT('USERENV', 'INSTANCE_NAME')
hash=550c95f3d0cfa8290e60ea8382d3a2ca timestamp=09-09-2010 04:24:19
namespace=CRSR flags=RON/KGHP/TIM/PN0/LRG/KST/DBN/MTX/[100100d1]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=9 hpc=0582 hlc=0582
lwt=47df576e8[47df576e8,47df576e8] ltm=47df576f8[47df576f8,47df576f8]
pwt=47df576b0[47df576b0,47df576b0] ptm=47df576c0[47df576c0,47df576c0]
ref=47df57718[47df57718,47df57718] lnd=47df57730[47df57730,47df57730]
LIBRARY OBJECT: object=471ee1d38
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
CHILDREN: size=16
child# table reference handle
0 471ee1800 471ee1470 47df7dce0
DATA BLOCKS:
data# heap pointer status pins change whr
0 47df7de48 471ee1e50 I/P/A/-/- 0 NONE 00
SO: 473691d60, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=473691d60 handle=47bb22fa0 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=473691de0[4735dbcb8,476cfbf58] htb=476cfbf58 ssga=476cfb6a0
user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x0
LIBRARY OBJECT HANDLE: handle=47bb22fa0 mtx=47bb230d0(0) cdp=0
namespace=CRSR flags=RON/KGHP/PN0/EXP/[10010100]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=fd84 hlc=fd84
lwt=47bb23048[47bb23048,47bb23048] ltm=47bb23058[47bb23058,47bb23058]
pwt=47bb23010[47bb23010,47bb23010] ptm=47bb23020[47bb23020,47bb23020]
ref=47bb23078[472f8de18,472f8de18] lnd=47bb23090[47bb23090,47bb23090]
LIBRARY OBJECT: object=472f8d9d8
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
DEPENDENCIES: count=1 size=16
AUTHORIZATIONS: count=1 size=16 minimum entrysize=16
ACCESSES: count=1 size=16
TRANSLATIONS: count=1 size=16
DATA BLOCKS:
data# heap pointer status pins change whr
0 47bb22ee0 472f8daf0 I/P/A/-/- 0 NONE 00
6 472f8e508 46be86250 I/-/A/-/E 0 NONE 00
SO: 4735dbc38, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=4735dbc38 handle=47bb231c8 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=4735dbcb8[476cfbf58,473691de0] htb=476cfbf58 ssga=476cfb6a0
user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x4c894f8b
LIBRARY OBJECT HANDLE: handle=47bb231c8 mtx=47bb232f8(1) cdp=1
name=select value$ from props$ where name = 'GLOBAL_DB_NAME'
hash=4bb432d65c5a391a42a5c3fa74472c7a timestamp=09-09-2010 04:24:12
namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/KST/DBN/MTX/[120100d0]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=0584 hlc=0584
lwt=47bb23270[47bb23270,47bb23270] ltm=47bb23280[47bb23280,47bb23280]
pwt=47bb23238[47bb23238,47bb23238] ptm=47bb23248[47bb23248,47bb23248]
ref=47bb232a0[47bb232a0,47bb232a0] lnd=47bb232b8[47bb232b8,47bb232b8]
LIBRARY OBJECT: object=472f8e6e0
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
CHILDREN: size=16
child# table reference handle
0 472f8e1a8 472f8de18 47bb22fa0
DATA BLOCKS:
data# heap pointer status pins change whr
0 47bb23108 472f8e7f8 I/P/A/-/- 0 NONE 00
SO: 473644348, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=473644348 handle=47bbde418 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=4736443c8[476cfc0b8,476cfc0b8] htb=476cfc0b8 ssga=476cfb6a0
user=47924e810 session=47924e810 count=1 flags=[0000] savepoint=0x4c894f8b
LIBRARY OBJECT HANDLE: handle=47bbde418 mtx=47bbde548(0) cdp=0
name=ALTER SESSION SET TIME_ZONE='+02:00'
hash=3878dff8839e71e3dd05a2e75fbd6390 timestamp=09-09-2010 04:24:04
namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/DBN/[12010040]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=11 hpc=04e8 hlc=04e8
lwt=47bbde4c0[47bbde4c0,47bbde4c0] ltm=47bbde4d0[47bbde4d0,47bbde4d0]
pwt=47bbde488[47bbde488,47bbde488] ptm=47bbde498[47bbde498,47bbde498]
ref=47bbde4f0[47bbde4f0,47bbde4f0] lnd=47bbde508[47bbde508,47bbde508]
LIBRARY OBJECT: object=472fffc08
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
DATA BLOCKS:
data# heap pointer status pins change whr
0 47bbde320 472fffd20 I/P/A/-/- 0 NONE 00
SO: 47aecf9e8, type: 41, owner: 47924e810, flag: INIT/-/-/0x00
(dummy) nxc=0, nlb=0
SO: 47f290540, type: 11, owner: 4781a7dc0, flag: INIT/-/-/0x00
(broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: 4781a7dc0,
event: 1132, last message event: 1132,
last message waited event: 1132, next message: 0(0), messages read: 0
channel: (47a2df4f8) system events broadcast channel
scope: 2, event: 1132, last mesage event: 18,
publishers/subscribers: 0/17,
messages published: 1
SO: 47826b228, type: 3, owner: 4781a7dc0, flag: INIT/-/-/0x00
(call) sess: cur 47924e810, rec 0, usr 47924e810; depth: 0
SO: 476c52968, type: 16, owner: 4781a7dc0, flag: INIT/-/-/0x00
(osp req holder)
PSEUDO PROCESS for group DEFAULT:
SO: 47a1eb7d0, type: 2, owner: 0, flag: INIT/-/-/0x00
(process) Oracle pid=0, calls cur/top: 0/0, flag: (20) PSEUDO
int error: 0, call error: 0, sess error: 0, txn error 0
(post info) last post received: 0 0 0
last post received-location: No post
last process to post me: none
last post sent: 0 0 0
last post sent-location: No post
last process posted by me: none
(latch info) wait_event=0 bits=0
Process Group: DEFAULT, pseudo proc: 47a1eb7d0
O/S info: user: , term: , ospid: (DEAD)
OSD pid info: Unix process pid: 0, image: PSEUDO
Dump of memory from 0x00000004791BF538 to 0x00000004791BF740
4791BF530 00000000 00000000 [........]
4791BF540 00000000 00000000 00000000 00000000 [................]
Repeat 31 times
NO DETACHED BRANCHES.
NO DETACHED NETWORK CONNECTIONS.
CLEANUP STATE OBJECTS:
SO: 47f0cd038, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: instance enqueue anchor state
latch: 0x380009890
SO: 4782cf080, type: 5, owner: 47f0cd038, flag: INIT/-/-/0x00
(enqueue) TA-00000006-00000001 DID: 0001-000F-0000000B
lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 res_flag: 0x2
res: 0x47a28d020, mode: X, lock_flag: 0x0
own: 0x0, sess: 0x0, prv: 0x47a28d030
SO: 47f0cd098, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: switchable channel handle anch
latch: 0x38000ac98
SO: 47f28f868, type: 11, owner: 47f0cd098, flag: INIT/-/-/0x00
(broadcast handle) flag: (c2) ACTIVE SUBSCRIBER, owner: 0,
event: 1, last message event: 1,
last message waited event: 1, next message: 0(0), messages read: 0
channel: (47a2e4190) KPON channel
scope: 2, event: 1, last mesage event: 0,
publishers/subscribers: 0/1,
messages published: 0
SO: 47f0cd0f8, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: TT shared object cleanup SO
latch: 0x38001c6b8
SO: 47f0cd158, type: 1, owner: 0, flag: INIT/-/-/0x00
(cleanup state object) description: SS shared object cleanup SO
latch: 0x38001cd48
END OF SYSTEM STATE
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
db file sequential read 2,347,652 9,215 4 64.5 User I/O
db file scattered read 245,687 4,199 17 29.4 User I/O
CPU time 974 6.8
db file parallel write 50,082 408 8 2.9 System I/O
log file parallel write 6,963 52 7 0.4 System I/O
Time Model Statistics DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> Total time in database user-calls (DB Time): 14286.4s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 14,280.3 100.0
DB CPU 974.5 6.8
PL/SQL execution elapsed time 531.8 3.7
parse time elapsed 30.5 .2
hard parse elapsed time 27.1 .2
connection management call elapsed time 14.9 .1
hard parse (sharing criteria) elapsed time 3.4 .0
hard parse (bind mismatch) elapsed time 3.1 .0
PL/SQL compilation elapsed time 2.4 .0
failed parse elapsed time 0.0 .0
repeated bind elapsed time 0.0 .0
sequence load elapsed time 0.0 .0
DB time 14,286.4 N/A
background elapsed time 670.2 N/A
background cpu time 186.1 N/A
Wait Class DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc
Avg
%Time Total Wait wait Waits
Wait Class Waits -outs Time (s) (ms) /txn
User I/O 2,593,484 .0 13,415 5 150.0
System I/O 87,506 .0 515 6 5.1
Other 839 11.4 6 7 0.0
Commit 3,225 .1 6 2 0.2
Concurrency 1,033 .0 5 5 0.1
Configuration 2,514 99.4 0 0 0.1
Network 47,559 .0 0 0 2.8
Application 7 .0 0 0 0.0
Wait Events DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
db file sequential read 2,347,652 .0 9,215 4 135.8
db file scattered read 245,687 .0 4,199 17 14.2
db file parallel write 50,082 .0 408 8 2.9
log file parallel write 6,963 .0 52 7 0.4
control file parallel write 6,203 .0 44 7 0.4
control file sequential read 24,242 .0 11 0 1.4
log file sync 3,225 .1 6 2 0.2
latch free 84 .0 4 47 0.0
os thread startup 25 .0 3 120 0.0
latch: session allocation 39 .0 1 33 0.0
db file parallel read 12 .0 1 92 0.0
enq: TX - index contention 186 .0 1 3 0.0
latch: shared pool 47 .0 1 11 0.0
LGWR wait for redo copy 319 3.1 0 1 0.0
library cache load lock 2 .0 0 172 0.0
buffer busy waits 590 .0 0 0 0.0
log file switch completion 6 .0 0 29 0.0
SGA: allocation forcing comp 11 54.5 0 14 0.0
latch: library cache lock 50 .0 0 3 0.0
read by other session 38 .0 0 4 0.0
direct path read 42 .0 0 3 0.0
SQL*Net message to client 44,807 .0 0 0 2.6
rdbms ipc reply 207 .0 0 0 0.0
SQL*Net more data from clien 1,014 .0 0 0 0.1
latch: cache buffers chains 24 .0 0 1 0.0
latch: library cache 29 .0 0 1 0.0
log file sequential read 8 .0 0 3 0.0
direct path write 50 .0 0 0 0.0
SQL*Net more data to client 398 .0 0 0 0.0
latch: object queue header o 12 .0 0 1 0.0
latch: In memory undo latch 78 .0 0 0 0.0
undo segment extension 2,507 99.7 0 0 0.1
latch: cache buffers lru cha 4 .0 0 1 0.0
log file single write 8 .0 0 0 0.0
local write wait 3 .0 0 1 0.0
enq: RO - fast object reuse 3 .0 0 1 0.0
buffer deadlock 87 92.0 0 0 0.0
enq: JS - queue lock 1 .0 0 1 0.0
cursor: pin S 70 .0 0 0 0.0
latch: row cache objects 2 .0 0 1 0.0
SQL*Net message to dblink 1,338 .0 0 0 0.1
latch: checkpoint queue latc 2 .0 0 0 0.0
reliable message 3 .0 0 0 0.0
log buffer space 1 .0 0 1 0.0
SQL*Net break/reset to clien 4 .0 0 0 0.0
SQL*Net more data from dblin 2 .0 0 0 0.0
SQL*Net message from client 44,949 .0 155,701 3464 2.6
virtual circuit status 621 100.0 18,156 29237 0.0
Streams AQ: qmn slave idle w 664 .0 18,127 27299 0.0
Streams AQ: qmn coordinator 1,339 50.4 18,099 13517 0.1
Streams AQ: waiting for time 12 100.0 8,741 728394 0.0
jobq slave wait 130 100.0 380 2927 0.0
PL/SQL lock timer 1 100.0 1 978 0.0
SQL*Net message from dblink 1,338 .0 0 0 0.1
single-task message 1 .0 0 38 0.0
class slave wait 11 .0 0 1 0.0
SQL ordered by Elapsed Time DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> Resources reported for PL/SQL code includes the resources used by all SQL
statements called by the code.
-> % Total DB Time is the Elapsed Time of the SQL statement divided
into the Total Database Time multiplied by 100
Elapsed CPU Elap per % Total
Time (s) Time (s) Executions Exec (s) DB Time SQL Id
13,664 906 0 N/A 95.6 gr2cx6athc5j5
Module: SQL*Plus
BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
8,792 195 0 N/A 61.5 986fzxtzr52u5
Module: SQL*Plus
UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
2,524 368 1 2524.1 17.7 c4uf0x6hdgnwq
Module: SQL*Plus
UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
SYS_B_1"
1,414 177 1 1414.4 9.9 cbg09ma34kq8w
Module: SQL*Plus
SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
742 137 1 742.2 5.2 g0sg6v994wssq
Module: SQL*Plus
SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
274 11 1 274.2 1.9 6mcpb06rctk0x
Module: DBMS_SCHEDULER
call dbms_space.auto_space_advisor_job_proc ( )
264 8 27 9.8 1.8 8szmwam7fysa3
Module: DBMS_SCHEDULER
insert into wri$_adv_objspace_trend_data select timepoint, space_usage, space_a
lloc, quality from table(dbms_space.object_growth_trend(:1, :2, :3, :4, NULL, N
ULL, NULL, 'FALSE', :5, 'FALSE'))
99 1 1 99.4 0.7 1z0x41f66nvjr
Module: SQL*Plus
UPDATE TIBEX_INSTRUMENTADMIN SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
SYS_B_1"
21 10 1 21.5 0.2 bbc1ck8594kvj
Module: SQL*Plus
UPDATE TIBEX_INSTRUMENTDAILYHIST SET ADJOPEN=NVL(ADJOPEN,OPEN), ADJHIGH=NVL(ADJH
IGH,HIGH), ADJLOW=NVL(ADJLOW,LOW), ADJMID=NVL(ADJMID,MID), ADJCLOSE=NVL(ADJCLOSE
,CLOSE), ADJVOLUME=NVL(ADJVOLUME,VOLUME), ADJCLOSINGBID=NVL(ADJCLOSINGBID,CLOSIN
GBID), ADJCLOSINGOFFER=NVL(ADJCLOSINGOFFER,CLOSINGOFFER)
12 0 1 12.5 0.1 6xm9p9uy5kaap
Module: SQL*Plus
SELECT count(*) FROM TIBEX_INSTRUMENTSTATE WHERE INSTRUMENTID=:"SYS_B_0"
SQL ordered by CPU Time DB/Inst: SURV2/SURV2 Snaps: 19172-19178
-> Resources reported for PL/SQL code includes the resources used by all SQL
statements called by the code.
-> % Total DB Time is the Elapsed Time of the SQL statement divided
into the Total Database Time multiplied by 100
CPU Elapsed CPU per % Total
Time (s) Time (s) Executions Exec (s) DB Time SQL Id
906 13,664 0 N/A 95.6 gr2cx6athc5j5
Module: SQL*Plus
BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
368 2,524 1 367.51 17.7 c4uf0x6hdgnwq
Module: SQL*Plus
UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
SYS_B_1"
195 8,792 0 N/A 61.5 986fzxtzr52u5
Module: SQL*Plus
UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
177 1,414 1 176.93 9.9 cbg09ma34kq8w
Module: SQL*Plus
SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
137 742 1 137.38 5.2 g0sg6v994wssq
Module: SQL*Plus
SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
11 274 1 10.82 1.9 6mcpb06rctk0x
Module: DBMS_SCHEDULER
call dbms_space.auto_space_advisor_job_proc ( )
10 21 1 9.65 0.2 bbc1ck8594kvjEdited by: NM on 10-Sep-2010 07:39 -
Statspack: High log file sync timeouts and waits
Hi all,
Please see an extract from our statpack report:
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
log file sync 349,713 215,674 74.13
db file sequential read 16,955,622 31,342 10.77
CPU time 21,787 7.49
direct path read (lob) 92,762 8,910 3.06
db file scattered read 4,335,034 4,439 1.53
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
log file sync 349,713 150,785 215,674 617 1.8
db file sequential read 16,955,622 0 31,342 2 85.9
I hope the above is readable. I'm concerned with the very high number of Waits and Timeouts, particulary around the log file sync event. From reading around I suspect that the disk our redo log sits on isn't fast enough.
1) Is this conclusion correct, are these timeouts excessively high (70% seems high...)?
2) I see high waits on almost every other event (but not timeouts), is this pointing towards an incorrect database database setup (give our very high loads of 160 executes second?
Any help would be much appreciated.
JonathanTop 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
log file sync 349,713 215,674 74.13
db file sequential read 16,955,622 31,342 10.77
CPU time 21,787 7.49
direct path read (lob) 92,762 8,910 3.06
db file scattered read 4,335,034 4,439 1.53
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
log file sync 349,713 150,785 215,674 617 1.8
db file sequential read 16,955,622 0 31,342 2 85.9What's the time frame of this report on?
It looks like your disk storage can't keep up with the volume of I/O requests from your database.
The first few thing need to look at, what're IO intensive SQLs in your database. Are these SQLs doing unnecessary full table scan?
Find out the hot blocks and the objects they belong.
Check v$session_wait view.
Is there any other suspicious activity going on in your Server ? Like other program other than Oracle doing high IO activities? Are there any core dump going on? -
Control File Sequential Read in 11.2
Hi to all,
i'm in 11.2 database and i'm looking an awr report.
Time:
Begin Snap: 9642 31-Gen-12 16:00:26 51 2.5
End Snap: 9643 31-Gen-12 16:40:27 52 2.6 This is my load profile:
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ --------------- --------------- ---------- ----------
DB Time(s): 0.2 0.3 0.01 0.02
DB CPU(s): 0.1 0.1 0.00 0.01
Redo size: 2,989.8 4,490.0
Logical reads: 1,823.2 2,738.1and this is the wait:
Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
DB CPU 120 32.5
direct path read 110,371 40 0 11.0 User I/O
db file sequential read 10,026 20 2 5.5 User I/O
control file sequential read 27,409 11 0 3.1 System I/OWhy i've too high control file sequential read?
I've read note 941761.1 and i've traced a session that create snapshot to see if there is
a table that generate to many wait.
This is the end of the trace:
OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
call count cpu elapsed disk query current rows
Parse 420 0.07 0.34 0 0 8 0
Execute 852 1.19 5.70 128 2446 4602 5870
Fetch 1537 0.28 1.18 31 6939 4 1378
total 2809 1.54 7.23 159 9385 4614 7248
Misses in library cache during parse: 215
Misses in library cache during execute: 209
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
Disk file operations I/O 25 0.00 0.00
db file sequential read 178 0.32 2.00
control file sequential read 169 0.02 0.13
asynch descriptor resize 73 0.00 0.00
CSS initialization 2 0.01 0.01
CSS operation: action 2 0.01 0.02
CSS operation: query 6 0.00 0.00
kfk: async disk IO 5 0.00 0.00
direct path write 5 0.00 0.00Any ideas?Just to expand on the previous answer - your database isn't doing anything.
This is an AWR report for a 40 minute period.
You've accumulated a few seconds worth of work.
Nothing happening.
There will always be a top 5. -
Query on dba_free_space ends in wait by event db file sequential read
Hello All,
Env: 10gR2 on WinNT
I gave the query
select tablespace_name,sum(bytes)/1024/1024 from dba_free_space group by tablespace_name and its waiting for ever.
I checked the wait event from v$session and its "db file sequential read".
I put a trace on the session before the running the above query:
OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
call count cpu elapsed disk query current rows
Parse 1 0.06 0.06 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 0 0.00 0.00 0 0 0 0
total 2 0.06 0.06 0 0 0 0
Misses in library cache during parse: 1
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
db file sequential read 13677 0.16 151.34
SQL*Net message to client 1 0.00 0.00
db file scattered read 281 0.01 0.53
latch: cache buffers lru chain 2 0.00 0.00
OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
call count cpu elapsed disk query current rows
Parse 13703 0.31 0.32 0 0 0 0
Execute 14009 0.75 0.83 0 0 0 0
Fetch 14139 0.48 0.74 26 56091 0 15496
total 41851 1.54 1.89 26 56091 0 15496
Misses in library cache during parse: 16
Misses in library cache during execute: 16
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
db file sequential read 26 0.00 0.12
1 user SQL statements in session.
14010 internal SQL statements in session.
14011 SQL statements in session.I took the AWR Report (for 1 hr time period) and the top 5 events came out as,
Event Waits Time (s) (ms) Time Wait Class
db file sequential read 1,134,643 7,580 7 56.8 User I/O
db file scattered read 940,880 5,025 5 37.7 User I/O
CPU time 967 7.2
control file sequential read 4,987 3 1 0.0 System I/O
control file parallel write 2,408 1 1 0.0 System I/O The PHYRDS(from dba_hist_filestatxs) on my system01.dbf is 161,028,980 for the latest snap.
Could someone throw some light into what is happening here ?
TIA,
JJUnder some circumstances querying the dictionary can be slow, usually because of problems with bad execution plans related to bad statistics, try to gather statistics using dbms_stats.gather_fixed_objects_stats(); it has worked for me before.
You can also read Note 414256.1 Poor Performance For Tablespace Page Display In Grid Control Console which in addition points to a possible problem with the recycle bin.
HTH
Enrique -
GG extract - what is excessive 'control file sequential reads'?
Hi,
base SR - 3-2192225691, GG - 10.4.0.19, database - 11.2.0.1.0, running on Linux x86 64-bit
customer opened SR based on high number of control file sequential read
Top 5 Timed Foreground Events (over 13+ hrs)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg
wait % DB
Event Waits Time(s) (ms) time Wait Class
db file sequential read 13,843,266 80,952 6 48.3 User I/O
DB CPU 44,897 26.8
control file sequential read 12,793,822 10,978 1 6.6 System I/O
enq: TX - row lock contention 30,000 10,066 336 6.0 Applicatio
log file sync 959,602 9,276 10 5.5 Commit
However, extract traces do not appear to show any waits associated with control files. A 10046 trace has been requested but not done as of yet.
Are there any known issues we can test for, or is this expected behavior? Any further tests that could be run?
Thanks,
SteveSteve,
There is an Oracle internal list you can use. I just sent you email on how to join.
Here's a shot in the dark: I've seen something similar with log sync waits on systems not moving a lot of data (usually a test system) and we're a little eager to get the next piece of data when at the logical end of file. That can be overcome using THREADOPTIONS EOFDELAYMS. Otherwise you'll need to start using the OGG trace commands.
Good luck,
-joe -
Query occasionally causes table scans (db file sequential read)
Dear all,
we periodically issue a query on a huge table via an oracle job.
Whenever I invoke the query manually, the response time is good. When I start the periodic job, initially the response times are good as well. After some days, however, the query suddenly takes almost forever.
My vague guess is that for some reason the query suddenly changes the execution plan from using the primary key index to a full table scan (or huge index range scan). Maybe because of some problems with the primary key index (fragmentation? Other?).
- Could it be the case that the execution plan for a query changes (automatically) like this? For what reasons?
- Do you have any hints where to look for further information for analysis? (logs, special event types, ...)?
- Of course, the query was designed having involved indexes in mind. Also, I studied the execution plan and did not find hints for problematic table/range scans.
- It is not a lock contention problem
- When the query "takes forever", there is a "db file sequential read" event in v$session_event for the query with an ever increasing wait time. That's why I guess a (unreasonable) table scan is happening.
Some charachteristics of the table in question:
- ~ 30 Mio rows
- There are only insertions to the table, as well as updates on a single, indexed field. No deletes.
- There is an integer primary key field with an B-tree index.
Charachteristics of the query:
The main structure of the query is very simple and as follows: I select a range of about 100 rows via primary key "id", like:
Select * from TheTable where id>11222300 and id <= 11222400
There are several joins with rather small tables, which make the overall query more complicated.
However, the few (100) rows of the huge table in question should always be fetched using the primary key index, shouldn't it?
Please let me know if some relevant information about the problem is missing.
Thanks!
Best regards,
Nang.user2560749 wrote:
Dear all,
we periodically issue a query on a huge table via an oracle job.
Whenever I invoke the query manually, the response time is good. When I start the periodic job, initially the response times are good as well. After some days, however, the query suddenly takes almost forever.
My vague guess is that for some reason the query suddenly changes the execution plan from using the primary key index to a full table scan (or huge index range scan). Maybe because of some problems with the primary key index (fragmentation? Other?).
- Could it be the case that the execution plan for a query changes (automatically) like this? For what reasons?Yes possible, One reason is stats of the table has been changed i.e somebody issued dbms_stats. If you are worried about execution plan getting changed then two option 1) Lock the stats 2) Use hint in the query.
- Do you have any hints where to look for further information for analysis? (logs, special event types, ...)?Have a Ora-10053 trace enabled whenever query plan changes and analysis it.
- Of course, the query was designed having involved indexes in mind. Also, I studied the execution plan and did not find hints for problematic table/range scans.
- It is not a lock contention problem
- When the query "takes forever", there is a "db file sequential read" event in v$session_event for the query with an ever increasing wait time. That's why I guess a (unreasonable) table scan is happening.
If it db file sequential read then i see two things 1) It is doing index range scan(Not table full scan) or 2) It is scanning undo tablespaces.
Some charachteristics of the table in question:
- ~ 30 Mio rows
- There are only insertions to the table, as well as updates on a single, indexed field. No deletes.
- There is an integer primary key field with an B-tree index.
Charachteristics of the query:
The main structure of the query is very simple and as follows: I select a range of about 100 rows via primary key "id", like:
Select * from TheTable where id>11222300 and id <= 11222400
There are several joins with rather small tables, which make the overall query more complicated.
However, the few (100) rows of the huge table in question should always be fetched using the primary key index, shouldn't it?
Yes theoreitically it should practically we can only say by looking at run time explain plan(through 10053,10046 trace).
Please let me know if some relevant information about the problem is missing.
Thanks!
Best regards,
Nang.I am still not sure in which direction you are looking for solution.
Is your query performing bad once in a fortnight and next day it is all same again.
I suggest to
1) Check if the data is scanning undo tablespace. I see you mentioned there is lot of inserts, could be that Oracle would be scanning undo tablespace because of delayed clean out.
2) Check if that particular day the number of records are high compared to other day.
Or once it starts performing bad then for next couple of days there is no change in response time.
1) Check if explain plan has been changed?
And what action you take to bring back the response time to normal?
Regards
Anurag -
Log file in session and call transaction methods
Hi Experts,
How to see log file in session and call transaction methods?hi
If its a session then goto SM35, select ur session and then u can see log button over there at the top... that will give the log
If its a call transaction then in ur program u need to declare internal table IT_MSGS like this
DATA: IT_MSGS LIKE BDCMSGCOLL OCCURS 0 WITH HEADER LINE.
LOOP AT IT_SALES.
PERFORM POPULATE_BDC.
CALL TRANSACTION 'VA01' USING IT_BDCDATA
MODE MODE
UPDATE UPDATE
MESSAGES INTO IT_MSGS.
IF NOT IT_MSGS[] IS INITIAL.
LOOP AT IT_MSGS.
CALL FUNCTION 'FORMAT_MESSAGE'
EXPORTING
ID = IT_MSGS-MSGID
LANG = 'EN'
NO = IT_MSGS-MSGNR
V1 = IT_MSGS-MSGV1
V2 = IT_MSGS-MSGV2
V3 = IT_MSGS-MSGV3
V4 = IT_MSGS-MSGV4
IMPORTING
MSG = V_MSG
EXCEPTIONS
NOT_FOUND = 1
OTHERS = 2.
IF SY-SUBRC <> 0.
MESSAGE ID SY-MSGID TYPE SY-MSGTY NUMBER SY-MSGNO
WITH SY-MSGV1 SY-MSGV2 SY-MSGV3 SY-MSGV4.
ENDIF.
WRITE:/ V_MSG.
ENDLOOP.
ENDIF.
ENDLOOP.
This will create a log for you, dont forget to award points if found helpful -
Respected Sir
Its very serious because log files in bump and udump are created frequently and they take too much space.
The problem is that i am not able to understand or solve these problem.I am pasting few log here.
BDUMP->
Dump file h:\oracle\product\10.2.0\admin\orcl2\bdump\orcl2_lgwr_1364.trc
Thu Jul 30 09:15:58 2009
ORACLE V10.2.0.1.0 - Production vsnsta=0
vsnsql=14 vsnxtr=3
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
Windows Server 2003 Version V5.2 Service Pack 1
CPU : 4 - type 586, 1 Physical Cores
Process Affinity : 0x00000000
Memory (Avail/Total): Ph:3481M/4094M, Ph+PgF:5024M/5975M, VA:1290M/2047M
Instance name: orcl2
Redo thread mounted by this instance: 1
Oracle process number: 6
Windows thread id: 1364, image: ORACLE.EXE (LGWR)
*** 2009-07-30 09:15:58.687
*** SERVICE NAME:() 2009-07-30 09:15:58.671
*** SESSION ID:(166.1) 2009-07-30 09:15:58.671
Media recovery not enabled or manual archival only 0x10000
Maximum redo generation record size = 156160 bytes
Maximum redo generation change vector size = 150672 bytes
*** 2009-07-30 10:34:04.875
Media recovery not enabled or manual archival only 0x10000
*** 2009-07-30 10:34:28.312
Media recovery not enabled or manual archival only 0x10000
*** 2009-07-30 11:23:36.000
Media recovery not enabled or manual archival only 0x10000
*** 2009-07-30 18:06:53.718
Media recovery not enabled or manual archival only 0x10000
*** 2009-07-30 22:02:26.734
Media recovery not enabled or manual archival only 0x10000
*** 2009-07-31 04:55:48.312
Media recovery not enabled or manual archival only 0x10000
UDUMP
Dump file h:\oracle\product\10.2.0\admin\orcl2\udump\orcl2_ora_192.trc
Wed Jul 29 12:34:17 2009
ORACLE V10.2.0.1.0 - Production vsnsta=0
vsnsql=14 vsnxtr=3
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
Windows Server 2003 Version V5.2 Service Pack 1
CPU : 4 - type 586, 1 Physical Cores
Process Affinity : 0x00000000
Memory (Avail/Total): Ph:3653M/4094M, Ph+PgF:5205M/5975M, VA:1325M/2047M
Instance name: orcl2
Redo thread mounted by this instance: 0 <none>
Oracle process number: 149
Windows thread id: 192, image: ORACLE.EXE (SHAD)
*** SERVICE NAME:() 2009-07-29 12:34:17.703
*** SESSION ID:(159.1) 2009-07-29 12:34:17.703
kccsga_update_ckpt: num_1 = 8, num_2 = 0, num_3 = 0, lbn_2 = 0, lbn_3 = 0
Successfully allocated 3 recovery slaves
Using 364 overflow buffers per recovery slave
Thread 1 checkpoint: logseq 317, block 2, scn 2070858
cache-low rba: logseq 317, block 3
on-disk rba: logseq 317, block 248, scn 2071389
start recovery at logseq 317, block 3, scn 0
----- Redo read statistics for thread 1 -----
Read rate (ASYNC): 122Kb in 0.35s => 0.34 Mb/sec
Total physical reads: 4096Kb
Longest record: 8Kb, moves: 0/201 (0%)
Longest LWN: 10Kb, moves: 0/63 (0%), moved: 0Mb
Last redo scn: 0x0000.001f9b5c (2071388)
----- Recovery Hash Table Statistics ---------
Hash table buckets = 32768
Longest hash chain = 1
Average hash chain = 92/92 = 1.0
Max compares per lookup = 1
Avg compares per lookup = 598/690 = 0.9
*** 2009-07-29 12:34:24.015
KCRA: start recovery claims for 92 data blocks
*** 2009-07-29 12:34:24.343
KCRA: blocks processed = 92/92, claimed = 92, eliminated = 0
*** 2009-07-29 12:34:24.390
Recovery of Online Redo Log: Thread 1 Group 1 Seq 317 Reading mem 0
----- Recovery Hash Table Statistics ---------
Hash table buckets = 32768
Longest hash chain = 1
Average hash chain = 92/92 = 1.0
Max compares per lookup = 1
Avg compares per lookup = 690/690 = 1.0
ThanksYou will still get lots of messages. Which messages depends on many things, some of which are changes you can make in the init files. On others, Oracle only knows to get rid of them if people log support calls. Whether they decide to get rid of them is mysterious, but probably not likely on a version soon to stop development.
Don't get mad at Hemant, he's just an experienced guy trying to help you, for free. You can get mad at me though, because my cares are elsewhere. You can get mad at Oracle too, because this is evidence of sloppiness on their part. But be nice to the support people, it's not their fault, they're being paid to help you. Though you can get mad at them if they don't help you after you've paid them, and you've jumped through hoops to get their attention. -
Need fix for db file sequential read
Hi all,
My db is 10.2.0.3.0 on windows. One of our query is very slow. When I checked for the reason from v$session the wait event is db file sequential read. Generated ash report and checked. Report listed one file id(p1), block id(p2), and number of blocks -1(p3). And other objects in the top objects. So how u can fix this now. Does I need to pin the block in the buffer cache..? Or does I need to to pin that complete object. Please suggest. I need to fix this.
Thanks a lot..962013 wrote:
Yes it is not issue, but yesterday query was running fine and today it is getting late. then what could be the reason to getting slow.
We was having standby database and we performed switch over. and now on the primary database (previously standby) query is running and getting slow. I guess this is reason.
So is there any action that we can take to minimize this issue. Please suggest...
Thanks
HOW To Make TUNING request
https://forums.oracle.com/forums/thread.jspa?threadID=2174552#9360003 -
Open file for reading and allow access by others
How can I open a file for reading and still allow others to access (read and write) the file? I plan to keep this file open for the duration of my program and close it at the end. However while my program is running another program will need access to the file so it can write to it. I may be wrong, but I dont think that setting the appropriate permission will solve the problem... Thank you for your time.
Cheers!
CLA, CLED, CTD,CPI, LabVIEW Champion
Platinum Alliance Partner
Senior Engineer
Using LV 2013, 2012
Don't forget Kudos for Good Answers, and Mark a solution if your problem is solved.hi jmcbee,
I think as long as "readers" open a file as "ReadOnly", then the OS (at least Windows) will allow another process to open and write to the same file without a problem. It's been a while since I've done this - maybe 7 years - and I may have opened the "writer" first, though doubt it makes a difference.
Have you tried?
Cheers!
"Inside every large program is a small program struggling to get out." (attributed to Tony Hoare) -
Optimize long execution time due to 'db file sequential read'
Hi to all,
I have got a query that takes long execution time. Most of the time is due to 'db file sequential read'. The query is:
SELECT * FROM Table_Name
WHERE col1 = :some_value
AND col2 BETWEEN :some_range_about_2_Million
| Id | Operation | Name | Rows | Bytes | Cost |
| 0 | SELECT STATEMENT | | 21 | 504 | 26125 |
| 1 | TABLE ACCESS BY INDEX ROWID| Table_Name | 21 | 504 | 26125 |
| 2 | INDEX RANGE SCAN | Index Name | 1705K| | 4100 |
The table is not partitioned having around 0.2 billion records. Record set for column 'col1' is around 1700K.
Another index is available for the col2 and col3 is not getting used.
Any suggestions to optimize it..
Regards.Perhaps a combined index (col2, col1) would work...or try "parallel" hint.
:p -
Query running very slow with db file sequential read waits which all indexes would be recommended
My query is doing lot of db file sequential reads I want to create few indexes and extended stats please help me with your suggestions.
Can anybody suggest I am unable to paste my query here -don't know why it's not getting pasted--is it not permitted ?
thank you.The simplest way to find it out is using Oracle Enterprise Manager. Just locate the SQL ID for the SQL and run "SQL tuning advisor". This will exactly pinpoint where do you need to create indexes etc. Another option is manually running explain plan for the SQL and finding out which predicate is causing full scans with high i/o
Regards
Tushar
Maybe you are looking for
-
Problem with 'Guides' position
Hi, I've just tried the new Adobe Dreamweaver CS5, but I've got a problem. I am editing static html pages so it's important for me to know the exact position of an object. I am using 'Guides' for this (the greeen lines, you can pull from the rulers).
-
The effect of repeated concatenation of Strings can cause a lot of unwanted objects in memory. If I am creating a query statement as follows: String query = "Select * from"+tablename+"where something "+x+"something"+y.......(All in a single statement
-
Reservation with Process_order API
HI When Passing the Reserved_quantity in Process_order API, is there any way to make reservation to particular Warehouse, for example i need to reseve quantity in 'WHS'(Subinventory) not on any others, i am able to do it using Reservation API, but ca
-
My outlook email apear to be stuck, with 2/9 sent. Error messages
-
I have just installed the above software for my 7D. When I plug in my camera it comes up with a notice that has all my camera details and says it will send a report to Apple. then nothing else. I installed the software on a friends Windows computer a