Cluster multi-block requests were consuming significant database time

Hi,
DB : 10.2.0.4 RAC ASM
OS : AIX 5.2 64-bit
We are facing too much performance issues and CPU idle time becoming 20%.Based on the AWR report , the top 5 events are showing that problem is in cluster side.I placed 1st node AWR report here for your suggestions.
WORKLOAD REPOSITORY report for
DB Name DB Id Instance Inst Num Release RAC Host
PROD 1251728398 PROD1 1 10.2.0.4.0 YES msprod1
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 26177 26-Jul-11 14:29:02 142 37.7
End Snap: 26178 26-Jul-11 15:29:11 159 49.1
Elapsed: 60.15 (mins)
DB Time: 915.85 (mins)
Cache Sizes
~~~~~~~~~~~ Begin End
Buffer Cache: 23,504M 23,504M Std Block Size: 8K
Shared Pool Size: 27,584M 27,584M Log Buffer: 14,248K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 28,126.82 2,675.18
Logical reads: 526,807.26 50,105.44
Block changes: 3,080.07 292.95
Physical reads: 962.90 91.58
Physical writes: 157.66 15.00
User calls: 1,392.75 132.47
Parses: 246.05 23.40
Hard parses: 11.03 1.05
Sorts: 42.07 4.00
Logons: 0.68 0.07
Executes: 930.74 88.52
Transactions: 10.51
% Blocks changed per Read: 0.58 Recursive Call %: 32.31
Rollback per transaction %: 9.68 Rows per Sort: 4276.06
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.87 Redo NoWait %: 100.00
Buffer Hit %: 99.84 In-memory Sort %: 99.99
Library Hit %: 98.25 Soft Parse %: 95.52
Execute to Parse %: 73.56 Latch Hit %: 99.51
Parse CPU to Parse Elapsd %: 9.22 % Non-Parse CPU: 99.94
Shared Pool Statistics Begin End
Memory Usage %: 68.11 71.55
% SQL with executions>1: 94.54 92.31
% Memory for SQL w/exec>1: 98.79 98.74
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
CPU time 18,798 34.2
gc cr multi block request 46,184,663 18,075 0 32.9 Cluster
gc buffer busy 2,468,308 6,897 3 12.6 Cluster
gc current block 2-way 1,826,433 4,422 2 8.0 Cluster
db file sequential read 142,632 366 3 0.7 User I/O
RAC Statistics DB/Inst: PROD/PROD1 Snaps: 26177-26178
Begin End
Number of Instances: 2 2
Global Cache Load Profile
~~~~~~~~~~~~~~~~~~~~~~~~~ Per Second Per Transaction
Global Cache blocks received: 14,112.50 1,342.26
Global Cache blocks served: 619.72 58.94
GCS/GES messages received: 2,099.38 199.68
GCS/GES messages sent: 23,341.11 2,220.01
DBWR Fusion writes: 3.43 0.33
Estd Interconnect traffic (KB) 122,826.57
Global Cache Efficiency Percentages (Target local+remote 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer access - local cache %: 97.16
Buffer access - remote cache %: 2.68
Buffer access - disk %: 0.16
Global Cache and Enqueue Services - Workload Characteristics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg global enqueue get time (ms): 0.6
Avg global cache cr block receive time (ms): 2.8
Avg global cache current block receive time (ms): 3.0
Avg global cache cr block build time (ms): 0.0
Avg global cache cr block send time (ms): 0.0
Global cache log flushes for cr blocks served %: 11.3
Avg global cache cr block flush time (ms): 1.7
Avg global cache current block pin time (ms): 0.0
Avg global cache current block send time (ms): 0.0
Global cache log flushes for current blocks served %: 0.0
Avg global cache current block flush time (ms): 4.1
Global Cache and Enqueue Services - Messaging Statistics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Avg message sent queue time (ms): 0.1
Avg message sent queue time on ksxp (ms): 2.4
Avg message received queue time (ms): 0.0
Avg GCS message process time (ms): 0.0
Avg GES message process time (ms): 0.0
% of direct sent messages: 6.27
% of indirect sent messages: 93.48
% of flow controlled messages: 0.25
Time Model Statistics DB/Inst: PROD/PROD1 Snaps: 26177-26178
-> Total time in database user-calls (DB Time): 54951s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 54,618.2 99.4
DB CPU 18,798.1 34.2
parse time elapsed 494.3 .9
hard parse elapsed time 397.4 .7
PL/SQL execution elapsed time 38.6 .1
hard parse (sharing criteria) elapsed time 27.3 .0
sequence load elapsed time 5.0 .0
failed parse elapsed time 3.3 .0
PL/SQL compilation elapsed time 2.1 .0
inbound PL/SQL rpc elapsed time 1.2 .0
repeated bind elapsed time 0.8 .0
connection management call elapsed time 0.6 .0
hard parse (bind mismatch) elapsed time 0.3 .0
DB time 54,951.0 N/A
background elapsed time 1,027.9 N/A
background cpu time 518.1 N/A
Wait Class DB/Inst: PROD/PROD1 Snaps: 26177-26178
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc
Avg
%Time Total Wait wait Waits
Wait Class Waits -outs Time (s) (ms) /txn
Cluster 50,666,311 .0 30,236 1 1,335.4
User I/O 419,542 .0 811 2 11.1
Network 4,824,383 .0 242 0 127.2
Other 797,753 88.5 208 0 21.0
Concurrency 212,350 .1 121 1 5.6
Commit 16,215 .0 53 3 0.4
System I/O 60,831 .0 29 0 1.6
Application 6,069 .0 6 1 0.2
Configuration 763 97.0 0 0 0.0
Second node top 5 events are as below,
Top 5 Timed Events
Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
CPU time 25,959 42.2
db file sequential read 2,288,168 5,587 2 9.1 User I/O
gc current block 2-way 822,985 2,232 3 3.6 Cluster
read by other session 345,338 1,166 3 1.9 User I/O
gc cr multi block request 991,270 831 1 1.4 Cluster
My RAM is 95GB each node and SGA is 51 GB and PGA is 14 GB.
Any inputs from your side are greatly helpful to me ,please.
Thanks,
Sunand

Hi Forstmann,
Thanks for your update.
Even i have collected ADDM report, extract of Node1 report as below
FINDING 1: 40% impact (22193 seconds)
Cluster multi-block requests were consuming significant database time.
RECOMMENDATION 1: SQL Tuning, 6% benefit (3313 seconds)
ACTION: Run SQL Tuning Advisor on the SQL statement with SQL_ID
"59qd3x0jg40h1". Look for an alternative plan that does not use
object scans.
SYMPTOMS THAT LED TO THE FINDING:
SYMPTOM: Inter-instance messaging was consuming significant database
time on this instance. (55% impact [30269 seconds])
SYMPTOM: Wait class "Cluster" was consuming significant database
time. (55% impact [30271 seconds])
FINDING 3: 13% impact (7008 seconds)
Read and write contention on database blocks was consuming significant
database time.
NO RECOMMENDATIONS AVAILABLE
SYMPTOMS THAT LED TO THE FINDING:
SYMPTOM: Inter-instance messaging was consuming significant database
time on this instance. (55% impact [30269 seconds])
SYMPTOM: Wait class "Cluster" was consuming significant database
time. (55% impact [30271 seconds])
Any help from your side , please?
Thanks,
Sunand

Similar Messages

JAVA execution consumed significant database time

Dear all,
DB 11. on solaris..
I have a time consuming package.. this package unloads data in the current db and loads data from another remote db in the same network. below is the recommendation from Oracle ADDM report.
This session is a work flow session. What can I do for JAVA execution consumed significant database time.?
JAVA execution consumed significant database time.
   Recommendation 1: SQL Tuning
   Estimated benefit is .5 active sessions, 43.98% of total activity.
   Action
      Tune the PL/SQL block with SQL_ID "6bd4fvsx8n42v". Refer to the "Tuning
      PL/SQL Applications" chapter of Oracle's "PL/SQL User's Guide and
      Reference".
      Related Object
         SQL statement with SQL_ID 6bd4fvsx8n42v.
         DECLARE job BINARY_INTEGER := :job; next_date DATE := :mydate;
         broken BOOLEAN := FALSE; BEGIN OWF_USER.START_METS_REFRESH(SYSDATE );
         :mydate := next_date; IF broken THEN :b := 1; ELSE :b := 0; END IF;
         END;Please advise
Kai

Hi Forstmann,
Thanks for your update.
Even i have collected ADDM report, extract of Node1 report as below
FINDING 1: 40% impact (22193 seconds)
Cluster multi-block requests were consuming significant database time.
RECOMMENDATION 1: SQL Tuning, 6% benefit (3313 seconds)
ACTION: Run SQL Tuning Advisor on the SQL statement with SQL_ID
"59qd3x0jg40h1". Look for an alternative plan that does not use
object scans.
SYMPTOMS THAT LED TO THE FINDING:
SYMPTOM: Inter-instance messaging was consuming significant database
time on this instance. (55% impact [30269 seconds])
SYMPTOM: Wait class "Cluster" was consuming significant database
time. (55% impact [30271 seconds])
FINDING 3: 13% impact (7008 seconds)
Read and write contention on database blocks was consuming significant
database time.
NO RECOMMENDATIONS AVAILABLE
SYMPTOMS THAT LED TO THE FINDING:
SYMPTOM: Inter-instance messaging was consuming significant database
time on this instance. (55% impact [30269 seconds])
SYMPTOM: Wait class "Cluster" was consuming significant database
time. (55% impact [30271 seconds])
Any help from your side , please?
Thanks,
Sunand

Wait class 'commit' consuming significant database time.

Hi
My awr report was showing that the log file sync as the top wait event.I can also see additional message saying that wait class 'commit' consuming significant database time.Can any one suggest me what are the tuning things i need to consider for this wait class 'LOG FILE SYNC'
Thanks

Be very careful about this.
Follow this only if you can afford to lose some data in case of instance failure
(eg death of the instance from a bug, server panic/reboot, power failure etc).
Oracle's normal behaviour is to guarantee that every committed transaction
IS available by ensuring that it is in the redologs and reapplying it if necessary
in case of an instance failure and recovery or media recovery.
A Commit NOWAIT means that there is a possibility, however slight, that
the last few transaction(s) might not have gotten into the redo logs at the time
of instance failure.
Your application / analysts must be able to identify transactions that are 'lost'
and reapply them after you restart a crashed instance.

Sql statement consuming significant database time

This sql statement consuming significant elapsed time(2,446 seconds). also also cause significant user i/0 , is this because of using temp TABLE IN THIS QUERY ,
or is there any alternative for this query ?
"WITH TEMP AS ( SELECT ROOT, CHILD, LEVEL LEV FROM catagory V START WITH CHILD=:B1 CONNECT BY PRIOR ROOT=CHILD ) SELECT CHILD FROM TEMP WHERE LEV=(SELECT MAX(LEV) FROM TEMP) "
Thanks
Renjith

Click on the FAQ stick posting (at the top of this forum). Find the "+when my query is slow+" question and read the FAQ response to it.

Contention on index block splits consuming significant database time

Hi Guys,
can anybody suggest on how to remove Contention on index block splits,this is giving so many issues on my production DB,the CPU usage shots up and application hangs for few minutes.
DB is 10.2.0.3 and OS is IBM AIX 5.3

I found this.. it might be useful
One possibility is that this is caused by shared CBC latching peculiarities:
1) during normal selects your index root block can be examined under a
shared cache buffers chains latch.
So as long as everybody is only reading the index root block, everybody can
do it concurrently (without pinning the block). The "current holder count"
in the CBC latch structure is just increased by one for every read only
latch get and decreased by one on every release. 0 value means that nobody
has this latch taken currently.
Nobody has to wait for others for reading index root block in all read only
case. That greatly helps to combat hot index root issues.
2) Now if a branch block split happens a level below the root block, the
root block has to be pinned in exclusive mode for reflecting this change in
it. In order to pin a block you need to get the corresponding CBC latch in
exclusive mode.
If there are already a bunch of readers on the latch, then the exclusive
latch getter will just flip a bit in the CBC latch structure - stating it's
interest for exclusive get.
Every read only latch get will check for this bit, if it's set, then the
getters will just spin instead, waiting this bit to be cleared (they may
yield or sleep immediately as well, I haven't checked). Now the exclusive
getter has to spin/wait until all the shared getters have released the latch
and the "current holder count" drops to zero. Once it's zero (and the getter
manager to get on to CPU) it can get the latch, do its work and release the
latch.
During all that time starting from when the "exclusive interest" bit was
set, nobody could access this indexes root block except the processes which
already had the latch in shared mode. Depending on latch spin/sleep strategy
for this particular case and OSD implementation, this could mean that all
those "4000 readers per second" start just spinning on that latch, causing
heavy spike in CPU usage and they all queue up.
How do diagnose that:
You could sample v$latch_misses to see whether the number of "kcbgtcr:
kslbegin shared" nowaitfails/sleeps counter takes an exceptional jump up
once you observe this hiccup.
How to fix that once diagnosed:
The usual stuff, like partitioning if possible or creating a single table
hash cluster instead.
If you see that the problem comes from excessive spinning, think about
reducing the spinning overhead (by reducing spincount for example). This
could affect your other database functions though..
If you can't do the above - then if you have off-peak time, then analyse
indexes (using treedump for start) and if you see a block split coming in a
branch below root block, then force the branch block to split during
off-peak time by inserting carefully picked values into the index tree,
which go exactly in the range which cause the proper block to split. Then
you can just roll back your transaction - the block splits are not rolled
back nor coalesced somehow, as this is done in a separate recursive
transaction.
And this
With indexes, the story is more complicated since you can't just insert a
row into any free block available like with tables. Multiple freelists with
tables help us to spread up inserts to different datablocks, since every
freelist has its distinct set of datablocks in it. With indexes, the
inserted key has to go exactly to the block where the structure of b?tree
index dictates, so multiple freelists can't help to spread contention here.
When any of the index blocks has to split, a new block has to be allocated
from the freelist (and possibly unlinked from previous location in index),
causing an update to freelist entry in segment header block. Now if you had
defined multiple freelists for your segment, they'd still remain in the
single segment header block and if you'd have several simultaneous block
splits, the segment header would become the bottleneck.
You could relieve this by having multiple freelist groups (spreading up
freelists into multiple blocks after segment header), but this approach has
it's problems as well - like a server process which maps to freelist group 1
doesn't see free blocks in freelist group 2, thus possibly wasting space in
some cases...
So, if you have huge contention on regular index blocks, then you should
rethink the design (avoid right hand indexes for example), or physical
design (partition the index), increasing freelists won't help here.
But if you have contention on index segment's header block because of block
splits/freelist operations, then either partition the index or have multiple
freelist groups, adding freelists again won't help here. Note that adding
freelist groups require segment rebuild.

Strange 'gc cr multi block request' during dictionary queries

Hi,
here goes the case :
I've got 4 node 10.2.0.3 RAC , DDL intensive (CTAS, DROP/TRUNCATE) database
It's like 20TB of data, milions of partitions and objects .
When I'm doing queries agains dictionary tables like select * from
dba_segments where owner = 'A' and segment_name = 'B'
I'm observing 'gc cr multi block request' , as far as I know its kind of
scattered reads but using interconnect to gather data .
Whats bothering me is profile of that 'gc cr multi block requests' , here
goes some lines from 10046 trace:
WAIT #6: nam='gc cr multi block request' ela= 861 file#=1 block#$34788
class#=1 obj#8 tim18733123446958
WAIT #6: nam='gc cr multi block request' ela= 69 file#=1 block#$34788
class#=1 obj#8 tim18733123447083
WAIT #6: nam='gc cr multi block request' ela= 60 file#=1 block#$34788
class#=1 obj#8 tim18733123447220
WAIT #6: nam='gc cr multi block request' ela= 99 file#=1 block#$34788
class#=1 obj#8 tim18733123447347
WAIT #6: nam='gc cr multi block request' ela= 111 file#=1 block#$34788
class#=1 obj#8 tim18733123447482
WAIT #6: nam='gc cr multi block request' ela= 193 file#=1 block#$34788
class#=1 obj#8 tim18733123447704
WAIT #6: nam='gc cr multi block request' ela= 84 file#=1 block#$34788
class#=1 obj#8 tim18733123447820
WAIT #6: nam='gc cr multi block request' ela= 81 file#=1 block#$34788
class#=1 obj#8 tim18733123447931
WAIT #6: nam='gc cr multi block request' ela= 108 file#=1 block#$34788
class#=1 obj#8 tim18733123448065
WAIT #6: nam='gc cr multi block request' ela= 111 file#=1 block#$34788
class#=1 obj#8 tim18733123448199
WAIT #6: nam='gc cr multi block request' ela= 105 file#=1 block#$34788
class#=1 obj#8 tim18733123448328
WAIT #6: nam='gc cr multi block request' ela= 100 file#=1 block#$34788
class#=1 obj#8 tim18733123448458
WAIT #6: nam='gc cr multi block request' ela= 151 file#=1 block#$34788
class#=1 obj#8 tim18733123448639
WAIT #6: nam='gc cr multi block request' ela= 84 file#=1 block#$34788
class#=1 obj#8 tim18733123448750
WAIT #6: nam='gc cr multi block request' ela= 90 file#=1 block#$34788
class#=1 obj#8 tim18733123448867
WAIT #6: nam='gc cr multi block request' ela= 98 file#=1 block#$34788
class#=1 obj#8 tim18733123448994
and that pattern repeats with different block#
Question is why Oracle is requesting the same block , over and over (16
times) , I thinks 16 comes from MBRC which is 16 and I've got 16kb blocksize
Looks like a bug :) isnt it ?
Generally all queries agains dictionary tables are slow (minutes) , I'm not
observing any lost block issues so its probably bad plans issue .
Any comments ?
Regards
GregG

It looks like this:
select *
from
dba_segments where owner = 'INS' and segment_name = 'T'
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.10       0.10          0          6          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        2     11.05      79.71      74418      96116          0           1
total        4     11.16      79.82      74418      96122          0           1
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 842
Rows     Row Source Operation
      1 VIEW SYS_DBA_SEGS (cr=96116 pr=74418 pw=0 time=3397685 us)
      1   UNION-ALL (cr=96116 pr=74418 pw=0 time=3397673 us)
      1    FILTER (cr=91749 pr=74315 pw=0 time=3397634 us)
     11     HASH JOIN RIGHT OUTER (cr=91749 pr=74315 pw=0 time=3397620 us)
   1317      TABLE ACCESS FULL USER$ (cr=34 pr=0 pw=0 time=6738 us)
     11      HASH JOIN (cr=91715 pr=74315 pw=0 time=3392292 us)
    262       TABLE ACCESS FULL FILE$ (cr=3 pr=0 pw=0 time=1087 us)
     11       HASH JOIN (cr=91712 pr=74315 pw=0 time=3389434 us)
     75        TABLE ACCESS FULL TS$ (cr=82 pr=0 pw=0 time=2693 us)
     11        NESTED LOOPS (cr=91630 pr=74315 pw=0 time=3386237 us)
     12         HASH JOIN (cr=91592 pr=74313 pw=0 time=23765169 us)
     20          TABLE ACCESS BY INDEX ROWID OBJ$ (cr=424 pr=21 pw=0 time=261735 us)
     20           INDEX SKIP SCAN I_OBJ2 (cr=406 pr=21 pw=0 time=261140 us)(object id 37)
966949          VIEW SYS_OBJECTS (cr=91168 pr=74292 pw=0 time=52219753 us)
966949           UNION-ALL (cr=91168 pr=74292 pw=0 time=50285852 us)
113050            TABLE ACCESS FULL TAB$ (cr=20294 pr=16672 pw=0 time=18883720 us)
627116            TABLE ACCESS FULL TABPART$ (cr=7188 pr=5576 pw=0 time=5652638 us)
     25            TABLE ACCESS FULL CLU$ (cr=20293 pr=16520 pw=0 time=1019 us)
13835            TABLE ACCESS FULL IND$ (cr=20322 pr=16550 pw=0 time=11400216 us)
195932            TABLE ACCESS FULL INDPART$ (cr=2523 pr=2421 pw=0 time=1967714 us)
   1213            TABLE ACCESS FULL LOB$ (cr=20350 pr=16553 pw=0 time=22044772 us)
   8363            TABLE ACCESS FULL TABSUBPART$ (cr=122 pr=0 pw=0 time=9995 us)
   7160            TABLE ACCESS FULL INDSUBPART$ (cr=72 pr=0 pw=0 time=9572 us)
    255            TABLE ACCESS FULL LOBFRAG$ (cr=4 pr=0 pw=0 time=2060 us)
     11         TABLE ACCESS CLUSTER SEG$ (cr=38 pr=2 pw=0 time=32200 us)
     11          INDEX UNIQUE SCAN I_FILE#_BLOCK# (cr=26 pr=0 pw=0 time=11331 us)(object id 9)
      0    NESTED LOOPS (cr=1 pr=1 pw=0 time=8349 us)
      0     NESTED LOOPS (cr=1 pr=1 pw=0 time=8344 us)
      0      FILTER (cr=1 pr=1 pw=0 time=8340 us)
      0       NESTED LOOPS OUTER (cr=1 pr=1 pw=0 time=8334 us)
      0        NESTED LOOPS (cr=1 pr=1 pw=0 time=8327 us)
      0         TABLE ACCESS BY INDEX ROWID UNDO$ (cr=1 pr=1 pw=0 time=8322 us)
      0          INDEX RANGE SCAN I_UNDO2 (cr=1 pr=1 pw=0 time=8314 us)(object id 35)
      0         TABLE ACCESS CLUSTER SEG$ (cr=0 pr=0 pw=0 time=0 us)
      0          INDEX UNIQUE SCAN I_FILE#_BLOCK# (cr=0 pr=0 pw=0 time=0 us)(object id 9)
      0        TABLE ACCESS CLUSTER USER$ (cr=0 pr=0 pw=0 time=0 us)
      0         INDEX UNIQUE SCAN I_USER# (cr=0 pr=0 pw=0 time=0 us)(object id 11)
      0      TABLE ACCESS BY INDEX ROWID FILE$ (cr=0 pr=0 pw=0 time=0 us)
      0       INDEX UNIQUE SCAN I_FILE2 (cr=0 pr=0 pw=0 time=0 us)(object id 42)
      0     TABLE ACCESS CLUSTER TS$ (cr=0 pr=0 pw=0 time=0 us)
      0      INDEX UNIQUE SCAN I_TS# (cr=0 pr=0 pw=0 time=0 us)(object id 7)
      0    FILTER (cr=4366 pr=102 pw=0 time=4366839 us)
      0     HASH JOIN RIGHT OUTER (cr=4366 pr=102 pw=0 time=4366833 us)
   1317      TABLE ACCESS FULL USER$ (cr=60 pr=0 pw=0 time=1397 us)
      0      HASH JOIN (cr=4306 pr=102 pw=0 time=4361061 us)
     75       TABLE ACCESS FULL TS$ (cr=82 pr=0 pw=0 time=745 us)
      0       NESTED LOOPS (cr=4224 pr=102 pw=0 time=4357971 us)
    262        TABLE ACCESS FULL FILE$ (cr=3 pr=0 pw=0 time=1738 us)
      0        TABLE ACCESS CLUSTER SEG$ (cr=4221 pr=102 pw=0 time=4355035 us)
      0         INDEX RANGE SCAN I_FILE#_BLOCK# (cr=4221 pr=102 pw=0 time=4351804 us)(object id 9)
Elapsed times include waiting on following events:
Event waited on                             Times   Max. Wait Total Waited
----------------------------------------   Waited ---------- ------------
library cache lock                             18        0.00          0.00
SQL*Net message to client                       2        0.00          0.00
gc current block 2-way                       2299        0.00          1.16
gc current block 3-way                       1597        0.00          1.28
gc cr block busy                                4        0.01          0.03
gc cr grant 2-way                             313        0.00          0.11
db file sequential read                       627        0.06          2.94
gc cr block 2-way                              37        0.00          0.02
gc cr block 3-way                              27        0.00          0.02
gc current grant 2-way                          1        0.00          0.00
gc cr multi block request                   23073        0.01          7.07
db file scattered read                       1756        0.10         13.70
db file parallel read                        4708        0.19         44.89
latch: KCL gc element parent latch             33        0.00          0.00
SQL*Net message from client                     2     1492.36       1492.36
latch free                                      4        0.00          0.00
latch: gcs resource hash                        2        0.00          0.00
latch: cache buffers chains                     5        0.00          0.00
gc buffer busy                                  1        0.00          0.00
latch: cache buffers lru chain                  2        0.00          0.00
gc cr disk read                                 2        0.00          0.00
and in raw trace:
WAIT #4: nam='gc cr multi block request' ela= 18 file#=1 block#=3396 class#=1 obj#=4 tim=1318904416456439
WAIT #4: nam='gc cr multi block request' ela= 320 file#=1 block#=3396 class#=1 obj#=4 tim=1318904416456780
WAIT #4: nam='gc cr multi block request' ela= 146 file#=1 block#=3396 class#=1 obj#=4 tim=1318904416456949
WAIT #4: nam='gc cr multi block request' ela= 14 file#=1 block#=3396 class#=1 obj#=4 tim=1318904416456979
WAIT #4: nam='gc cr multi block request' ela= 6 file#=1 block#=3396 class#=1 obj#=4 tim=1318904416457000
WAIT #4: nam='gc cr multi block request' ela= 107 file#=1 block#=3396 class#=1 obj#=4 tim=1318904416457119
WAIT #4: nam='gc cr multi block request' ela= 171 file#=1 block#=3396 class#=1 obj#=4 tim=1318904416457594
WAIT #4: nam='gc cr multi block request' ela= 1119 file#=1 block#=3387 class#=1 obj#=4 tim=1318904416458819
WAIT #4: nam='db file parallel read' ela= 9411 files=1 blocks=10 requests=10 obj#=4 tim=1318904416468429
WAIT #4: nam='gc cr multi block request' ela= 592 file#=1 block#=4244 class#=1 obj#=4 tim=1318904416470099
WAIT #4: nam='gc cr multi block request' ela= 7 file#=1 block#=4244 class#=1 obj#=4 tim=1318904416470136
WAIT #4: nam='gc cr multi block request' ela= 203 file#=1 block#=4244 class#=1 obj#=4 tim=1318904416470353
WAIT #4: nam='gc cr multi block request' ela= 69 file#=1 block#=4244 class#=1 obj#=4 tim=1318904416470487
WAIT #4: nam='gc cr multi block request' ela= 962 file#=1 block#=4244 class#=1 obj#=4 tim=1318904416471499
WAIT #4: nam='gc cr multi block request' ela= 49 file#=1 block#=4244 class#=1 obj#=4 tim=1318904416471650Any ideas ?:)
Regards
GregG

Gc cr multi block request.

Hi ,
I am sufferiung from a 40 hour long running job which normally completes in few hrs, that jus creates a table (actually create table as select,joining many tables using parallel query withe 16 sessions).
There is only 2 session that are active at this time, one of them is the parent waiting on PX:DEQ EXECUTE REPLY.
The other sesion is running from 24 hrs, it is actually not waiitng upon any event for a signifcant time.Most of the time it faces GC CR MULTI BLOCK REQUESTS wait event. Is this due to interconnect latency?.I guessed it because the protocol used by the interconnect is UDP which doesnot wait for an acknowledgement. Can you please suggest me how to check for the latency on the interconnect?? Is my doubt on the interconnect is reasonable???????????
I checked the trace files,awr reports,ash reports and every thing but nothing seems to be working fine. All I see is that single sesion using up 98% of the CPU and the TOP wait event in the awr,ash reports being the CPU Time.Pleas help me.Thanks.

You are difficult to help as you don't post
- a four digit database version
- information on your OS
- information, including configuration info, of your interconnect
RAC is a chatty protocol, and sends large packages, of even 64k and higher. This means if you don't use an 1 Gb network card, and don't have 'jumbo frames' enabled, performance will suffer.
As a result, at least in 9i, you will see this very wait event show up all the time, at least on AIX.
Sybrand Bakker
Senior Oracle DBA

[질문]WAS 이용시 gc cr multi block request 발생

OS 버젼 과 Word size : AIX 5.3 64BIT
DB버젼과 Word size : DB VERSION은 10.2.0.3 64BIT
RAC 환경이며 node1과 node2는 동일한 환경으로 구성되어 있습니다.
업무상 주로 이용하는 인스턴스는 node1 이구요.
파티션으로 구성되어 있는 테이블을 full scan하는 쿼리를 수행하는데
WAS에서 해당 쿼리를 날리면 gc cr multi block request 이벤트가 아주 오래 점유하고 있습니다.
동일 쿼리를 pl/sql 로 접속해서 2-tier로 쿼리하면 그런 이벤트 없이 바로 떨어지구요.
2-tier로 쿼리하면 15초,
WAS에서 쿼리하면 4~5분 정도가 걸리죠.
도대체 왜 그런지 알수가 없는데, 가이드가 될 만한 힌트라도 좋으니 의견 부탁드립니다.
감사합니다.

"액셈이 만들어가는 오라클 백과사전"에서 내용이 잘나와있습니다.
이 내용부터 확인하세요.
http://wiki.ex-em.com/index.php/Gc_cr_multi_block_request

Long waits on 'gc cr multi block request' elapsed tim anomaly - tkprof inc

Hi,
in my 4-node RAC 10.2.0.3 when I run query (thats part os user_segments view)
select o.name,
o.subname,
so.object_type, s.type#,
ts.ts#, ts.name, ts.blocksize,
s.file#, s.block#,
s.blocks * ts.blocksize, s.blocks, s.extents,
s.iniexts * ts.blocksize,
decode(bitand(ts.flags, 3), 1, to_number(NULL),
s.extsize * ts.blocksize),
s.minexts, s.maxexts,
decode(bitand(ts.flags, 3), 1, to_number(NULL),
s.extpct),
decode(bitand(ts.flags, 32), 32, to_number(NULL),
decode(s.lists, 0, 1, s.lists)),
decode(bitand(ts.flags, 32), 32, to_number(NULL),
decode(s.groups, 0, 1, s.groups)),
s.cachehint, NVL(s.spare1, 0), s.hwmincr
from sys.obj$ o, sys.ts$ ts, sys.sys_objects so, sys.seg$ s
where s.file# = so.header_file
and s.block# = so.header_block
and s.ts# = so.ts_number
and s.ts# = ts.ts#
and o.obj# = so.object_id
and o.owner# = userenv('SCHEMAID')
and s.type# = so.segment_type_id
and o.type# = so.object_type_id
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.03       0.05          2          5          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        5     11.59     529.83       9580     115021          0        1957
total        7     11.62     529.88       9582     115026          0        1957
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: SYS
Elapsed times include waiting on following events:
Event waited on                             Times   Max. Wait Total Waited
----------------------------------------   Waited ---------- ------------
library cache lock                             13        0.00          0.00
SQL*Net message to client                       5        0.00          0.00
gc cr multi block request                   63408        1.22        507.30
gc cr grant 2-way                              75        0.00          0.02
db file sequential read                       983        0.04          6.13
gc current block 3-way                         97        0.00          0.07
gc current block 2-way                        111        0.00          0.06
db file scattered read                        522        0.17          4.28
db file parallel read                         249        0.07          2.19
cr request retry                               81        0.00          0.02
latch: cache buffers lru chain                  5        0.00          0.00
latch: gcs resource hash                        5        0.00          0.00
latch: KCL gc element parent latch             22        0.00          0.00
latch free                                      3        0.00          0.00
gc cr block 3-way                              10        0.00          0.00
gc cr block 2-way                             101        0.00          0.05
gc cr disk read                                64        0.00          0.02
SQL*Net message from client                     4        0.04          0.14
SQL*Net more data to client                    50        0.00          0.00
********************************************************************************In raw trace file I can see:
WAIT #1: nam='gc cr multi block request' ela= 81 file#=1 block#=223460 class#=1 obj#=4 tim=1273765932709496
WAIT #1: nam='gc cr multi block request' ela= 1221367 file#=1 block#=223456 class#=1 obj#=4 tim=1273765933930894
WAIT #1: nam='gc cr multi block request' ela= 1221273 file#=1 block#=223456 class#=1 obj#=4 tim=1273765935152287
WAIT #1: nam='gc cr multi block request' ela= 1223855 file#=1 block#=223456 class#=1 obj#=4 tim=1273765936376232
WAIT #1: nam='gc cr multi block request' ela= 1180460 file#=1 block#=223456 class#=1 obj#=4 tim=1273765937556773
WAIT #1: nam='gc cr multi block request' ela= 406 file#=1 block#=223476 class#=1 obj#=4 tim=1273765937566092
WAIT #1: nam='gc cr multi block request' ela= 8039 file#=1 block#=263972 class#=1 obj#=4 tim=1273765937742971
WAIT #1: nam='gc cr multi block request' ela= 1221983 file#=1 block#=263972 class#=1 obj#=4 tim=1273765938965148
WAIT #1: nam='gc cr multi block request' ela= 1221642 file#=1 block#=263972 class#=1 obj#=4 tim=1273765940186879
WAIT #1: nam='gc cr multi block request' ela= 1221656 file#=1 block#=263972 class#=1 obj#=4 tim=1273765941408618
WAIT #1: nam='gc cr multi block request' ela= 1221654 file#=1 block#=263972 class#=1 obj#=4 tim=1273765942630357
WAIT #1: nam='gc cr multi block request' ela= 1221657 file#=1 block#=263972 class#=1 obj#=4 tim=1273765943852096
WAIT #1: nam='gc cr multi block request' ela= 578770 file#=1 block#=263972 class#=1 obj#=4 tim=1273765944430948
WAIT #1: nam='gc cr multi block request' ela= 101 file#=1 block#=351188 class#=1 obj#=4 tim=1273765944898180
WAIT #1: nam='gc cr multi block request' ela= 2287 file#=1 block#=351188 class#=1 obj#=4 tim=1273765944900500
WAIT #1: nam='gc cr multi block request' ela= 1221984 file#=1 block#=351188 class#=1 obj#=4 tim=1273765946122713
WAIT #1: nam='gc cr multi block request' ela= 1221641 file#=1 block#=351188 class#=1 obj#=4 tim=1273765947344453
WAIT #1: nam='gc cr multi block request' ela= 1221670 file#=1 block#=351188 class#=1 obj#=4 tim=1273765948566201
WAIT #1: nam='gc cr multi block request' ela= 1221663 file#=1 block#=351188 class#=1 obj#=4 tim=1273765949787950
WAIT #1: nam='gc cr multi block request' ela= 1221670 file#=1 block#=351188 class#=1 obj#=4 tim=1273765951009697
WAIT #1: nam='gc cr multi block request' ela= 182726 file#=1 block#=351188 class#=1 obj#=4 tim=1273765951192501
WAIT #1: nam='gc cr multi block request' ela= 558 file#=1 block#=351188 class#=1 obj#=4 tim=1273765951194110
same for obj#5
WAIT #1: nam='gc cr multi block request' ela= 3103 file#=1 block#=35284 class#=1 obj#=5 tim=1273766101728096
WAIT #1: nam='gc cr multi block request' ela= 1221789 file#=1 block#=35281 class#=1 obj#=5 tim=1273766102950203
WAIT #1: nam='gc cr multi block request' ela= 1221668 file#=1 block#=35281 class#=1 obj#=5 tim=1273766104171949
WAIT #1: nam='gc cr multi block request' ela= 1221639 file#=1 block#=35281 class#=1 obj#=5 tim=1273766105393674
WAIT #1: nam='gc cr multi block request' ela= 1221612 file#=1 block#=35281 class#=1 obj#=5 tim=1273766106615404
WAIT #1: nam='gc cr multi block request' ela= 838378 file#=1 block#=35281 class#=1 obj#=5 tim=1273766107453869
WAIT #1: nam='gc cr multi block request' ela= 695 file#=1 block#=35281 class#=1 obj#=5 tim=1273766107455175Any ideas ?
Regards
GregG

Hi Greg;
You may hit a bug?What is your OS? If linux than please see:
Bug 6268172: INCONSISTENT QUERY PERFORMANCE IN 4 NODE RAC
https://support.oracle.com/CSP/main/article?cmd=show&type=BUG&id=6268172&productFamily=Oracle
Also see:
Bug 9838876: WAITING FOR 'GC CR MULTI BLOCK REQUEST'
https://support.oracle.com/CSP/main/article?cmd=show&type=BUG&id=9838876&productFamily=Oracle
Regard
Helios

Contention for latches related to the shared pool was consuming significant

We are having performance issue on our database. When I look at the AWR, I see that there is a contention for latches. Below is the AWR Report.
ADDM Report for Task 'ADDM:1775307360_12808'
Analysis Period
AWR snapshot range from 12807 to 12808.
Time period starts at 10-MAY-11 01.00.15 PM
Time period ends at 10-MAY-11 02.00.23 PM
Analysis Target
Database 'ADVFDWP' with DB ID 1775307360.
Database version 11.1.0.7.0.
ADDM performed an analysis of all instances.
Activity During the Analysis Period
Total database time was 27827 seconds.
The average number of active sessions was 7.71.
Summary of Findings
Description Active Sessions Recommendations
Percent of Activity
1 Shared Pool Latches 6.43 | 83.42 0
2 Top SQL by DB Time 2.41 | 31.24 3
3 "Concurrency" Wait Class 2.18 | 28.22 0
4 PL/SQL Execution 1.53 | 19.86 1
5 "User I/O" wait Class 1.33 | 17.24 0
6 Hard Parse 1.24 | 16.14 0
7 Undersized Buffer Cache .83 | 10.73 0
8 CPU Usage .7 | 9.02 0
9 Top SQL By I/O .31 | 4.04 1
10 Top Segments by I/O .24 | 3.12 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Findings and Recommendations
Finding 1: Shared Pool Latches
Impact is 6.43 active sessions, 83.42% of total activity.
Contention for latches related to the shared pool was consuming significant
database time in some instances.
Instances that were significantly affected by this finding:
Number Name Percent Impact ADDM Task Name
1 ADVFDWP1 99.31 ADDM:1775307360_1_12808
Check the ADDM analysis of affected instances for recommendations.
Finding 2: Top SQL by DB Time
Impact is 2.41 active sessions, 31.24% of total activity.
SQL statements consuming significant database time were found.
Recommendation 1: SQL Tuning
Estimated benefit is 1.07 active sessions, 13.82% of total activity.
Action
Run SQL Tuning Advisor on the SQL statement with SQL_ID "fdk73nhpt93a5".
Related Object
SQL statement with SQL_ID fdk73nhpt93a5.
INSERT INTO SFCDM.F_LOAN_PTFL_MOL_SNPSHT SELECT * FROM
F_LOAN_PTFL_MOL_SNPSHT_STG
Recommendation 2: SQL Tuning
Estimated benefit is 1 active sessions, 12.96% of total activity.
Action
Tune the PL/SQL block with SQL_ID "7nvgzsgy9ydn9". Refer to the "Tuning
PL/SQL Applications" chapter of Oracle's "PL/SQL User's Guide and
Reference".
Related Object
SQL statement with SQL_ID 7nvgzsgy9ydn9.
begin
insert into SFCDM.F_LOAN_PTFL_MOL_SNPSHT select * from
F_LOAN_PTFL_MOL_SNPSHT_STG;
end;
Recommendation 3: SQL Tuning
Estimated benefit is .4 active sessions, 5.2% of total activity.
Action
Investigate the SQL statement with SQL_ID "fcvfq2gzmxu0t" for possible
performance improvements.
Related Object
SQL statement with SQL_ID fcvfq2gzmxu0t.
select
a11.DT_YR_MO DT_YR_MO,
a11.IND_SCRTZD IND_SCRTZD,
a13.CD_LNSTAT CD_LNSTAT_INTGRTD,
sum(a11.CNT_LOAN) WJXBFS1,
sum(a11.AMT_PART_EOP_UPB) WJXBFS2,
sum(a11.AMT_LST_VLD_PART_UPB) WJXBFS3
from
SFCDM.F_LOAN_PTFL_MOL_SNPSHT
a11
join
SFCDM.D_DETD_LNSTAT_CURR
a12
on
(a11.ID_CYCL_CLOS_DETD_LNSTAT_SRGT = a12.ID_DETD_LNSTAT_SRGT)
join
SFCDM.D_LNSTAT_CD
a13
on
(a12.ID_LNSTAT_CD_SRGT = a13.ID_LNSTAT_CD_SRGT)
join
SFCDM.D_LOAN_CHARTC_CURR_MINI
a14
on
(a11.ID_LOAN_CHARTC_SRGT = a14.ID_LOAN_CHARTC_SRGT)
where
(a11.DT_YR_MO in (201103)
and a14.CD_SFCRM_LOAN_BUS_LI not in ('L', 'T', 'W')
and a13.CD_LNSTAT in (14)
and not exists
(select * from SFCDM.F_LOAN_PTFL_MOL_SNPSHT s
where s.id_loan_syst_gend = a11.id_loan_syst_gend
and s.dt_yr_mo

It is worth checking the actual size of the shared pool e.g.
select pool,sum(bytes)/1024/1024/1024 from v$sgastat group by pool;
the parameters you ahve posted suggest you have set a minimum but no maximum, so it could very large.
Next up is looking for unhared SQL i.e.
select column1 from some_table where column2='A_VALUE';
select column1 from some_table where column2='Another_Value';
where the code should be using binds instead of literals for security and performance reasons, a simple way to find this is to look in v$sql for sql having the same plan_hash_value but different sql_Ids and compare the sql_fulltext of each statement.
Also a possibility is sql with many child cursors, this is trickier as the cause may vary and may not be easy to fix. check th econtents of v$sql for sql that have high values in the child_number column anmd investigate the contents of v$sql_shared_cursor for the reason there are multiple child cursors.
Chris

Global cache average current block request

Hi all experts,
I am getting this error frequently through 12C EM. Please tell me how I can suppress this alert so that I don't get it in future.
Name=<sid>_<instancename>
Type=Database Instance
Host=<servername>.<domainname>
Metric=Global Cache Average Current Block Request Time (centi-seconds)
Timestamp xxxxxxxxxxxxxxxxxxxxxxxx
Severity=Critical
Message=Metrics "Global Cache Average Current Get Time 62" is at 1.46154
Rule Name=XXProd
Rule Owner=SYS

this metric needs to be disabled in EM at the target level for each instance. you can disable "Global Cache Statistics" metrics in two different ways:
Navigate to the instance, then click the Oracle Database drop-down -> Monitoring -> All Metrics -> (expand) Global Cache Statistics
-> Global Cache Average CR Block Request Time (centi-seconds)
-> Global Cache Average Current Block Request Time (centi-seconds)
-> Global Cache Blocks Corrupt
-> Global Cache Blocks Lost
a. At the top level (Global Cache Statistics), click the [Modify] button , then the (*) Disable radio button to disable the entire group
b. If only disabling the individual metric (Global Cache Average CR Block Request Time (centi-seconds)), click on Modify Thresholds button to delete the values. Note that empty Thresholds will disable alerts for that metric.
Same can be done via navigation to the Cluster database instance -> Oracle Database drop-down -> Monitoring -> Metric and Collection Settings -> Global Cache Statistics

Multi User request in Access Enforcer

Is anyone aware of a user limit in an access enforcer multi user request?
We get errors when we submit a multi user access enforcer request with more than 25 users.
Thanks

Hi
There is no standard limit even though we advice to keep the user to max of 20 .
The limit depends upon the email content you have configured .
In case in your email notifications you have taken the argument USERID then mulitple user creation request causes issue and the limit gets set to anything between 20-25 , again depending on content of the email .
Thanks

How to find the First block....in a multi block form

hi
How i can find which is the first block in a multi block form....
( there are n number of forms with multi blocks...so
i need to generalise the code to find the first block in all of
the forms)
regards
Kris

If you searched in the on-line help for "First", you would find that the Get_Form_Property built-in provides two: First_Block and First_Navigation_Block.

How to execute each block in a multi-block canvas while select the tab?

Hi All,
How to execute each block in a multi-block canvas by selecting a tab? I mean to say when i select a particular tab in a tab canvas the records should execute.How can i set this?
Arif

Hi Arif
Good Example Manu offered i wish it works if not pls try the following...
Pls try in the when-tab-page-changed trigger in the form level
DECLARE
    tp_name varchar2(30);
BEGIN
-- Retrieve the NAME of the top most tab page using the built-in GET_CANVAS_PROPERTY function.
--Pass in the name of the Canvas your tabs are in and the system variable topmost_tab_page which stores a property number
tp_name := GET_CANVAS_PROPERTY('CANVAS11',topmost_tab_page);
-- Perform specific tasks based on the name of the top most tab
IF tp_name = 'PAGE12' then
GO_BLOCK('block_name');
   GO_ITEM('DATA_BLOCK_1.FIELD_1');
EXECUTE_QUERY;
ELSIF tp_name = 'PAGE38' then
GO_BLOCK('block_name');
   GO_ITEM('DATA_BLOCK_2.FIELD_1');
EXECUTE_QUERY;
ELSIF tp_name = 'PAGE47' then
GO_BLOCK('block_name');
   GO_ITEM('DATA_BLOCK_3.FIELD_1');
EXECUTE_QUERY;
END IF;
END;
Hope it helps...
Note: i do agree with Craig
Regards,
Abdetu...
Edited by: Abdetu on Feb 2, 2011 7:41 AM

Can tablespace block size different from the database block size

I have a 10.2.0.3 database in Unix system.
I created a database used default block size of 8k. However, the client application requires 16k block size database. Can I work around to create a tablespace that has 16k block size instead of drop the database to recreate the database.
Thanks a lot!

As Steven pointed out, you certainly can.
I would generally question, though, whether you should.
- Why does the application require 16k block sizes? If this is a custom application, it almost certainly doesn't really require 16k blocks. If this is a packaged application, it probably doesn't really require 16k blocks. If 16k blocks are a requirement for support, I would wager that having the application's objects in 16k block size tablespaces in a database with 8k blocks would not be supported.
- Mixing block sizes increases the management complexity of your database, potentially substantially. You need to specify a completely separate buffer cache for the 16k blocks, a buffer cache that would not be integrated with Oracle's automatic SGA management functionality. Figuring out how to split up the buffer cache between 8k and 16k blocks tends to be rather hard (particularly if the mix changes over time), which means that DBAs are going to be spending substantially more time managing the SGA in this sort of system than in a vanilla 10.2.0.3 system. And that DBAs will have many more opportunities to set things up incorrectly.
Justin

Cluster multi-block requests were consuming significant database time

Similar Messages

Maybe you are looking for