RAC gc buffer busy waits !!

Hello All,
We are running on 6 node 10.2.0.3 windows 64 bit rac environment.
JDE is the erp layer.
There is 1 table named f00950.
This table is accessed(only selects) by user that logs into the JDE for the first time. It also refers to the table as and when required since it is JDE security related.
On this table we are witnessing very high GC buffer busy waits.
So today i re-organised the table with higher PCTFREE (40) (ASSM tablespace) thinking that this would give me benefit and i would see a decrease in this wait.
But the numbers are still the same.
About 20% of non-idle waits are on this event.
I know partitioning can help but since this is a system table, JDE mgmt is not giving me the go-ahead.
What else can i do to reduce this wait.
Application partitioning is not possible.
There is no major performance issue but will soon be as more and more users are added as part of JDE.
thanks for any suggestions..
Sat

very high GC buffer busy waits. find hot block by using ASH
http://www.ardentperf.com/2007/09/12/gc-buffer-busy-waits-in-rac-finding-hot-blocks/
When you find that table... ??? so,
My idea:
- increase initrans on tables + indexes ... it should reserver on table + index
example:ALTER TABLE TABLE_NAME INITRANS 10
- make tables + indexes with partitioning
- sequence based index ??? if yes ... and then probably think about reverse key index...
- increase SGA
I know partitioning can help but since this is a system table, JDE mgmt is not giving me the go-ahead.You should... ;)
- On-line Table Reorganization and Redefinition
http://www.oracle.com/technology/products/oracle9i/daily/nov05.html
- Or export , make new tables+indexes (partitions) and then import only data
- Or make new tables+indexes with Create table AS Select
Application partitioning is not possible. - increase initrans on tables + indexes ... it should reserver on table + index
example:ALTER TABLE TABLE_NAME INITRANS 10
- sequence based index ??? if yes ... and then probably think about reverse key index...
- increase SGA
OR check sql statements ... tune them anyway

Similar Messages

  • Gc buffer busy wait in rac

    HI,
    I am facing a problem of gc buffer busy waits and log file switch completion on my live rac database of two nodes.
    Thank you for your cooperation.
    Regards,
    Adnan Hamdus Salam

    Aman.... wrote:
    I am sorry sir. I gave that link for the gc buffer busy wait only and overlooked the log file switch part.
    No need to apologise, or call me sir (the English and American use of the words have very different connotations). I was just adding an extra detail to go with a useful reference.
    Regards
    Jonathan Lewis
    http://jonathanlewis.wordpress.com
    http://www.jlcomp.demon.co.uk
    A general reminder about "Forum Etiquette / Reward Points": http://forums.oracle.com/forums/ann.jspa?annID=718
    If you never mark your questions as answered people will eventually decide that it's not worth trying to answer you because they will never know whether or not their answer has been of any use, or whether you even bothered to read it.
    It is also important to mark answers that you thought helpful - again it lets other people know that you appreciate their help, but it also acts as a pointer for other people when they are researching the same question, moreover it means that when you mark a bad or wrong answer as helpful someone may be prompted to tell you (and the rest of the forum) what's so bad or wrong about the answer you found helpful.

  • How to tune buffer busy waits in RAC

    Hi,
    In our environment we se lot of buffer busy waits.
    How can I tune that.
    we are using 9i RAC on HACMP.
    Regards
    MMU

    There could be several reasons for buffer busy waits, however the common reason is high logical reads..
    Normally they relate to bad SQL, you can try to identify those bad queries using a 10046 trace at level 8 or 12 and then tkrprof the trace file and look at the output.
    In a RAC environment you will see global buffer busy waits which again relates to buffer busy waits at the instance level.

  • Buffer Busy Waits on Primary Keys - RKI

    11.2.03 - HPUX
    Hi All.
    We're seeing significant Buffer Busy Waits (around 15% of non idle waits during bad times) and most of the contention is on the following;
    SYS.SEQ$, and two primary keys..
    The primary key one seems easier to fix in the first instance, with most of the expert thinking seeming to be either Reverse Key Indexes, or hash partitions for index. Thinking this through, wouldn't moving from normal key order to reverse key order cause a lot more I/O - if the assumption is that the rows being queried are likely to be the newer rows being inserted?? If we scatter them evenly among the index segment - won't that have a deleterious effect on I/O for that segment? In effect we're swapping Buffer Busy Waits for much worse (?) I/O?

    >
    We're seeing significant Buffer Busy Waits (around 15% of non idle waits during bad times) and most of the contention is on the following;
    SYS.SEQ$, and two primary keys..
    The primary key one seems easier to fix in the first instance, with most of the expert thinking seeming to be either Reverse Key Indexes, or hash partitions for index. Thinking this through, wouldn't moving from normal key order to reverse key order cause a lot more I/O - if the assumption is that the rows being queried are likely to be the newer rows being inserted?? If we scatter them evenly among the index segment - won't that have a deleterious effect on I/O for that segment? In effect we're swapping Buffer Busy Waits for much worse (?) I/O?
    >
    If you have heavy INSERTs, especially in RAC, you need to solve your current problem. But you need to make sure you know what that problem is.
    You have likely identified it but you should take a quick look at your sequence and the cache value being used. Contention on SEQ$ could mean that you are using NOCACHE or a low value for CACHE.
    If your primary key accesses typically access just one row it won't really matter if those rows are 'scattered'. It sounds like you are thinking that queries of 'newer' rows will make it likely that those queries will need the same blocks because the rows will be sharing a block.
    But why would users be accessing data using a non-meaningful sequence number? I wouldn't expect users to even know what the primary key value was to be able to query it. I would expect them to be querying based on a business value, company_code or date.
    Can you explain why you are worried about that?
    You might be interested in this two part article about reverse key indexes by Richard Foote
    http://richardfoote.wordpress.com/2008/01/14/introduction-to-reverse-key-indexes-part-i/
    http://richardfoote.wordpress.com/2008/01/16/introduction-to-reverse-key-indexes-part-ii-another-myth-bites-the-dust/
    >
    Reverse Key Indexes are designed to resolve a specific issue, that being index block contention. Many indexes in busy database environments with lots of concurrent inserts (and in some scenarios updates and deletes as well) can suffer from index block contention (as highlighted by high levels of “buffer busy waits” and “read by other session” wait events for the index segments). Monotonically increasing indexes, such as Primary Keys generated by a sequence, are especially prone to contention as all inserts need to access the maximum “right-most” leaf block. This is of particular concern in RAC environments, where this “hot” index block needs to be accessed by all the instances and is being bounced around the various SGAs causing expensive block transfers between instances.
    A solution is make the index a Reverse Key Index.

  • High Buffer Busy Wait due to Concurrent INSERTS

    Hi All,
    One of my OLTP database is running on 11.1.0.7 (11.1.0.7.0 - 64bit Production) with RHEL 5.4.
    On frequent basis, i am observing 'BUFFER BUSY WAITS' and last time i tried to capture some dictionary information to dig the waits.
    1. Session Watis :
              Oracle                                                  Sec                                     Hash
    Sid,Serial User     OS User  Svr-Pgm    Wait Event      State-Seq   Wt Module                  Cmnd       Value          P1          P2   P3
    633,40830 OLTP_USE fateadm  21646-orac buffer busy wai Wtng-9999    1 ORDERS             ISRT  3932487748         384     1863905    1
    647, 1761 OLTP_USE fateadm  22715-orac buffer busy wai Wtng-3837    0 ORDERS             ISRT  3932487748         384     1863905    1
    872, 5001 OLTP_USE fateadm  21836-orac buffer busy wai Wtng-9999    1 ORDERS             ISRT  3932487748         384     1863905    1
    702, 1353 OLTP_USE fateadm  21984-orac buffer busy wai Wtng-9999    1 ORDERS             ISRT  3932487748         384     1863905    1
    337,10307 OLTP_USE fateadm  21173-orac buffer busy wai Wtng-9999    1 ORDERS             ISRT  3932487748         384     1863905    1
    751,43016 OLTP_USE fateadm  21619-orac buffer busy wai Wtng-9999    1 ORDERS             ISRT  3932487748         384     1863905    1
    820,17959 OLTP_USE fateadm  21648-orac buffer busy wai Wtng-9999    0 ORDERS             ISRT  3932487748         384     1863905    1
    287,63359 OLTP_USE fateadm  27053-orac buffer busy wai Wtng-9999    0 ORDERS             ISRT  3932487748         384     1863905    1
    629, 1653 OLTP_USE fateadm  22468-orac buffer busy wai Wtng-9999    1 ORDERS             ISRT  3932487748         384     1863905    1
    788,14160 OLTP_USE fateadm  22421-orac buffer busy wai Wtng-9999    0 ORDERS             ISRT  3932487748         384     1863905    1
    615, 4580 OLTP_USE fateadm  21185-orac buffer busy wai Wtng-9999    0 ORDERS             ISRT  3932487748         384     1863905    1
    525,46068 OLTP_USE fateadm  27043-orac buffer busy wai Wtng-9034    1 ORDERS             ISRT  3932487748         384     1863905    1
    919,23243 OLTP_USE fateadm  21428-orac buffer busy wai Wtng-6340    1 ORDERS             ISRT  3932487748         384     1863906    1
    610,34557 OLTP_USE fateadm  21679-orac buffer busy wai Wtng-6422    1 ORDERS             ISRT  3932487748         384     1863906    1
    803, 1583 OLTP_USE fateadm  21580-orac buffer busy wai Wtng-6656    1 ORDERS             ISRT  3932487748         384     1863906    1
    781, 1523 OLTP_USE fateadm  21781-orac buffer busy wai Wtng-9999    0 ORDERS             ISRT  3932487748         384     1863906    1
    369,11005 OLTP_USE fateadm  21718-orac buffer busy wai Wtng-9999    0 ORDERS             ISRT  3932487748         384     1863906    1
    823,35800 OLTP_USE fateadm  21148-orac buffer busy wai Wtng-9999    1 ORDERS             ISRT  3932487748         384     1863906    1
    817, 1537 OLTP_USE fateadm  22505-orac buffer busy wai Wtng-9999    1 ORDERS             ISRT  3932487748         384     1863906    1
    579,54959 OLTP_USE fateadm  22517-orac buffer busy wai Wtng-9999    0 ORDERS             ISRT  3932487748         384     1863906    1
    591,33597 OLTP_USE fateadm  27027-orac buffer busy wai Wtng-9999    1 ORDERS             ISRT  3932487748         384     1863906    1
    481, 3031 OLTP_USE fateadm  21191-orac buffer busy wai Wtng-3502    1 ORDERS             ISRT  3932487748         384     1863906    1
    473,24985 OLTP_USE fateadm  22629-orac buffer busy wai Wtng-9999    0 ORDERS             ISRT  3932487748         384     1863906    1
    868, 3984 OLTP_USE fateadm  27191-orac buffer busy wai Wtng-9999    0 ORDERS             ISRT  3932487748         384     1863906    1
    select owner,segment_name,segment_type from dba_extents where    file_id = 384 and   1863905 between block_id and block_id + blocks -1;
    OWNER                          SEGMENT_NAME                                                                      SEGMENT_TYPE
    ORDER                          ORDER_DETAILS                                                                      TABLE
    select TABLE_NAME,PARTITIONED,ini_trans ,degree,compression,FREELISTS from dba_TABLES WHERE TABLE_NAME='ORDER_DETAILS';
    TABLE_NAME                     PAR  INI_TRANS DEGREE                         COMPRESS  FREELISTS
    ORDER_DETAILS                   NO           1          1                     ENABLED           1
    Tablespace is not ASSM managed !
      select
       object_name,
       statistic_name,
       value
    from
       V$SEGMENT_STATISTICS
    where
       object_name = 'ORDER_DETAILS';
    OBJECT_NAME              STATISTIC_NAME                                                        VALUE
    ORDER_DETAILS             logical reads                                                     487741104
    ORDER_DETAILS             buffer busy waits                                                   4715174
    ORDER_DETAILS             db block changes                                                  200858896
    ORDER_DETAILS             physical reads                                                    143642724
    ORDER_DETAILS             physical writes                                                    20581330
    ORDER_DETAILS             physical reads direct                                              55239903
    ORDER_DETAILS             physical writes direct                                             19500551
    ORDER_DETAILS             space allocated                                                  1.6603E+11
    ORDER_DETAILS             segment scans                                                          9727
    ORDER_DETAILS table is ~ 153 GB non-partitioned table.
    It seems its not a READ BY OTHER SESSIONS wait but BUFFER BUSY due to write-wirte contention inside same block. I have never observed Cache Buffer Chain/ ITL-Wait/ High wait time on dbfile sequential/scattered reads.Table contains one PK (composite index on 3 columns) which seems to be highly fragmented.This non partitioned global Index has 3182037735 rows and blevel is 4.
    BHAVIK_DBA.FATE1NA>select index_name,status,num_rows,blevel,pct_free,ini_trans,clustering_factor from dba_indexes where index_name='IDX_ORDERS';
    INDEX_NAME                     STATUS     NUM_ROWS     BLEVEL   PCT_FREE  INI_TRANS CLUSTERING_FACTOR
    IDX_ORDERS                      VALID    3182037735          4          2          2        2529462377
    1 row selected.
    One of the index column value is being populated by sequence. (Monotonically increasing value)
    SEGMENT_NAME                                                                              MB
    IDX_ORDERS                                                             170590.438
    Index size is greater than table size !Tuning goal here is to reduce buffer busy waits and thus commit latencies.
    I think, i need to increase FREELISTS and PCT_FREE to address this issue, but not much confident whether it is going to solve the issue or not?
    Can i ask for any help here ?

    Hi Johnathan;
    Many thanks for your detailed write-up. I was expecting you !
    Your post here gave lot of information and wisdom that made me think last couple of hrs that is the reason for the delay in reply.
    I did visited your index explosion posts couple of times and that scenario only gave me insight that concurrent DML (INSERT) is cause of index fragmentation in my case.
    Let me also pick the opportunity to ask you to shed more light on some of the information you have highlighted.
    if you can work out the number of concurrent inserts that are really likely to happen at any one instant then a value of freelists that in the range of
    concurrency/4 to concurrency/2 is probably appropriate.May i ask you how did you derive this formula ? I dont want to miss learning opportunity here !
    Note - with multiple freelists, you may find that you now get buffer busy waits on the segment header block.I did not quite get this point ? Can you shed more light please? What piece in segment header block is going to result contention(BBW on SEGMENT HEADER) on all concurrent inserts ?
    The solution to this is to increase the number of freelist groups (making sure that
    freelists and freelist groups have no common factors).My prod db NON-RAC environment. Can i use FREELIST GROUPS here ? My little knowledge did not get, What "common factors" you are referring here ?
    The reads could be related to leaf block splits, but there are several possible scenarios that could lead to that pattern of activity - so the next step is to find out which blocks are being
    read. Capture a sample of the waits, then query dba_extents for the extent_id, file_id, and block_id (don't run that awful query with the "block_id + blocks" predicate) and cross-check the
    list of blocks to see if they are typically the first couple of blocks of an extent or randomly scattered throughout extents. If the former the problem is probably related to ASSM, if the
    latter it may be related to failed probes on index leaf block reuse (i.e. after large scale deletes).I have 10046 trace file with me (giving you some sample below) that can give some information. However, since the issue was critical, i killed the insert process and rebuilt both the indexes. Since, index is rebuilt, i am not able to find any information in dba_extents.
    select SEGMENT_NAME,SEGMENT_TYPE,EXTENT_ID from dba_extents where file_id=42 and block_id=1109331;
    no rows selected
    select SEGMENT_NAME,SEGMENT_TYPE,EXTENT_ID from dba_extents where file_id=42 and block_id=1109395 ;
    no rows selected
    select SEGMENT_NAME,SEGMENT_TYPE,EXTENT_ID from dba_extents where file_id=42 and block_id=1109459;
    no rows selected
    select SEGMENT_NAME,SEGMENT_TYPE,EXTENT_ID from dba_extents where file_id=10 and block_id=1107475;
    no rows selected
    select SEGMENT_NAME,SEGMENT_TYPE,EXTENT_ID from dba_extents where file_id=10 and block_id=1107539;
    no rows selected
    select object_name,object_Type from dba_objects where object_id=17599;
    no rows selected
    WAIT #4: nam='db file sequential read' ela= 49 file#=42 block#=1109331 blocks=1 obj#=17599 tim=1245687162307379
    WAIT #4: nam='db file sequential read' ela= 59 file#=42 block#=1109395 blocks=1 obj#=17599 tim=1245687162307462
    WAIT #4: nam='db file sequential read' ela= 51 file#=42 block#=1109459 blocks=1 obj#=17599 tim=1245687162307538
    WAIT #4: nam='db file sequential read' ela= 49 file#=10 block#=1107475 blocks=1 obj#=17599 tim=1245687162307612
    WAIT #4: nam='db file sequential read' ela= 49 file#=10 block#=1107539 blocks=1 obj#=17599 tim=1245687162307684
    WAIT #4: nam='db file sequential read' ela= 198 file#=10 block#=1107603 blocks=1 obj#=17599 tim=1245687162307905
    WAIT #4: nam='db file sequential read' ela= 88 file#=10 block#=1107667 blocks=1 obj#=17599 tim=1245687162308016
    WAIT #4: nam='db file sequential read' ela= 51 file#=10 block#=1107731 blocks=1 obj#=17599 tim=1245687162308092
    WAIT #4: nam='db file sequential read' ela= 49 file#=10 block#=1107795 blocks=1 obj#=17599 tim=1245687162308166
    WAIT #4: nam='db file sequential read' ela= 49 file#=10 block#=1107859 blocks=1 obj#=17599 tim=1245687162308240
    WAIT #4: nam='db file sequential read' ela= 52 file#=10 block#=1107923 blocks=1 obj#=17599 tim=1245687162308314
    WAIT #4: nam='db file sequential read' ela= 57 file#=42 block#=1109012 blocks=1 obj#=17599 tim=1245687162308395
    WAIT #4: nam='db file sequential read' ela= 52 file#=42 block#=1109076 blocks=1 obj#=17599 tim=1245687162308470
    WAIT #4: nam='db file sequential read' ela= 98 file#=42 block#=1109140 blocks=1 obj#=17599 tim=1245687162308594
    WAIT #4: nam='db file sequential read' ela= 67 file#=42 block#=1109204 blocks=1 obj#=17599 tim=1245687162308686
    WAIT #4: nam='db file sequential read' ela= 53 file#=42 block#=1109268 blocks=1 obj#=17599 tim=1245687162308762
    WAIT #4: nam='db file sequential read' ela= 54 file#=42 block#=1109332 blocks=1 obj#=17599 tim=1245687162308841
    WAIT #4: nam='db file sequential read' ela= 55 file#=42 block#=1109396 blocks=1 obj#=17599 tim=1245687162308920
    WAIT #4: nam='db file sequential read' ela= 54 file#=42 block#=1109460 blocks=1 obj#=17599 tim=1245687162308999
    WAIT #4: nam='db file sequential read' ela= 52 file#=10 block#=1107476 blocks=1 obj#=17599 tim=1245687162309074
    WAIT #4: nam='db file sequential read' ela= 89 file#=10 block#=1107540 blocks=1 obj#=17599 tim=1245687162309187
    WAIT #4: nam='db file sequential read' ela= 407 file#=10 block#=1107604 blocks=1 obj#=17599 tim=1245687162309618TKPROF for above trace
    INSERT into
                     order_rev
                     (aggregated_revenue_id,
                      legal_entity_id,
                      gl_product_group,
                      revenue_category,
                      warehouse_id,
                      tax_region,
                      gl_product_subgroup,
                      total_shipments,
                      total_units_shipped,
                      aggregated_revenue_amount,
                      aggregated_tax_amount,
                      base_currency_code,
                      exchange_rate,
                      accounting_date,
                      inventory_owner_type_id,
                      fin_commission_structure_id,
                      seller_of_record_vendor_id,
                      organizational_unit_id,
                      merchant_id,
                      last_updated_date,
                      revenue_owner_type_id,
                      sales_channel,
                      location)
                     VALUES
                     (order_rev.nextval,:p1,:p2,:p3,:p4,:p5,:p6,:p7,:p8,:p9,:p10,:p11,:p12,to_date(:p13, 'dd-MON-yyyy'),:p14,:p15,:p16,:p17,:p18,sysdate,:p19,:p20,:p21)
    call     count       cpu    elapsed       disk      query    current        rows
    Parse        0      0.00       0.00          0          0          0           0
    Execute    613      5.50      40.32      96672     247585     306916         613
    Fetch        0      0.00       0.00          0          0          0           0
    total      613      5.50      40.32      96672     247585     306916         613
    Misses in library cache during parse: 0
    Optimizer mode: ALL_ROWS
    Parsing user id: 446
    Elapsed times include waiting on following events:
      Event waited on                             Times   Max. Wait  Total Waited
      ----------------------------------------   Waited  ----------  ------------
      db file sequential read                    164224        0.04         62.33
      SQL*Net message to client                     613        0.00          0.00
      SQL*Net message from client                   613        0.03          0.90
      latch: cache buffers chains                     8        0.00          0.00
      latch: object queue header operation            2        0.00          0.00Is there any other way to find out culprit amongst the two you have listed (ASSM / failed probes on index leaf block reuse ) ?

  • Buffer busy wait ratio

    Hi
    Working in oracle 9i AIX 5.3
    I would like to calculate Buffer busy wait ratio
    I knew the formula as
    buffer busy waits/ logical reads
    what would be the query agains this formula.
    thanks
    ranga

    Hi;
    Please see:
    Question 1) i got warning about Parse to Execute Ratio,24,0397,2, (High parse to execute ratio), what is the cause of this? how can i solve?Troubleshooting and Diagnosing ORA-4031 Error [Video] [ID 396940.1]
    http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:1594740500346667363
    Question 2) which document should i read for understanding these informations?See below googling, you can see many good links on it
    http://www.google.com.tr/#hl=tr&biw=1259&bih=793&q=preformance+tuning%2Boracle&aq=f&aqi=&aql=&oq=&fp=59bee11737e76a38
    Question 3) i got 2 nodes RAC 11gr2. "sessions" parameter value is 1500, cursor is 400, processes 500... how can i optimize these values?Those are system parameter and you can modifiy them by observing your system by AWR,ASH etc..(also users error message)
    http://oracleandunix.blogspot.com/2009/10/oracle-11gr2-database-parameters.html
    Question 4) I didnt create any server pool. But my database server is working well. Why do i need server pool? What is the advantages of server pool technology?RAC Assurance Support Team: RAC and Oracle Clusterware Starter Kit and Best Practices (Generic) [ID 810394.1]
    RAC: Frequently Asked Questions [ID 220970.1]
    Regard
    Helios

  • Critical --Buffer Busy Wait

    We have 2 node RAC(9i) ..one query hangs on single node but working on 1st node...
    We trace(10046) for sesion and fire query ...then in trace it continues showing Buffer Busy Wait...wait event....
    that table not in use and there are no locks......
    may i know the quick solution to fix this

    You have not answered the queries from Hemant in your other thread.
    Indexes not in use
    This seems to be related as the instance configuration appears to be different.

  • Buffer busy waits

    Hi,
    Version : Oracle 9i
    I am getting buffer busy waits on some tables. Will increase in inittrans & pctfree of those tables reduce buffer busy waits?
    Tablespace is having segment space mgmt auto & extent management local.
    cursor_sharing is similar.
    Users are not experiencing any problem.Is there any problem other than this in statspack report?
    STATSPACK report for
    DB Name DB Id Instance Inst Num Release Cluster Host
    AHD 3712247982 ahd 1 9.2.0.1.0 NO SBGSDPRI
    Snap Id Snap Time Sessions Curs/Sess Comment
    Begin Snap: 20 13-Feb-07 14:48:35 33 9.9
    End Snap: 21 13-Feb-07 15:12:19 34 10.4
    Elapsed: 23.73 (mins)
    Cache Sizes (end)
    ~~~~~~~~~~~~~~~~~
    Buffer Cache: 656M Std Block Size: 8K
    Shared Pool Size: 152M Log Buffer: 768K
    Load Profile
    ~~~~~~~~~~~~ Per Second Per Transaction
    Redo size: 5,960.83 2,761.29
    Logical reads: 2,376.85 1,101.05
    Block changes: 35.48 16.44
    Physical reads: 97.56 45.20
    Physical writes: 1.15 0.53
    User calls: 92.63 42.91
    Parses: 20.00 9.27
    Hard parses: 0.29 0.13
    Sorts: 4.80 2.22
    Logons: 0.01 0.00
    Executes: 23.14 10.72
    Transactions: 2.16
    % Blocks changed per Read: 1.49 Recursive Call %: 14.69
    Rollback per transaction %: 0.00 Rows per Sort: 472.64
    Instance Efficiency Percentages (Target 100%)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Buffer Nowait %: 99.65 Redo NoWait %: 100.00
    Buffer Hit %: 95.90 In-memory Sort %: 100.00
    Library Hit %: 99.15 Soft Parse %: 98.55
    Execute to Parse %: 13.57 Latch Hit %: 99.70
    Parse CPU to Parse Elapsd %: 90.83 % Non-Parse CPU: 96.58
    Shared Pool Statistics Begin End
    Memory Usage %: 84.68 84.76
    % SQL with executions>1: 77.32 79.22
    % Memory for SQL w/exec>1: 90.74 92.81
    Top 5 Timed Events
    ~~~~~~~~~~~~~~~~~~ % Total
    Event Waits Time (s) Ela Time
    CPU time 125 54.23
    db file sequential read 83,110 69 30.14
    db file scattered read 23,196 27 11.75
    buffer busy waits 11,760 6 2.42
    log file sync 3,078 1 .45
    Wait Events for DB: AHD Instance: ahd Snaps: 20 -21
    -> s - second
    -> cs - centisecond - 100th of a second
    -> ms - millisecond - 1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc (idle events last)
    Avg
    Total Wait wait Waits
    Event Waits Timeouts Time (s) (ms) /txn
    db file sequential read 83,110 0 69 1 27.0
    db file scattered read 23,196 0 27 1 7.5
    buffer busy waits 11,760 0 6 0 3.8
    log file sync 3,078 0 1 0 1.0
    log file parallel write 5,216 4,841 1 0 1.7
    control file sequential read 1,390 0 1 1 0.5
    control file parallel write 462 0 0 1 0.2
    db file parallel write 672 336 0 0 0.2
    latch free 54 24 0 2 0.0
    SQL*Net more data to client 1,026 0 0 0 0.3
    LGWR wait for redo copy 12 0 0 0 0.0
    SQL*Net message from client 131,863 0 22,857 173 42.9
    virtual circuit status 48 48 1,497 31188 0.0
    wakeup time manager 45 45 1,446 32123 0.0
    SQL*Net message to client 131,864 0 0 0 42.9
    SQL*Net more data from clien 27 0 0 0 0.0
    Background Wait Events for DB: AHD Instance: ahd Snaps: 20 -21
    -> ordered by wait time desc, waits desc (idle events last)
    Avg
    Total Wait wait Waits
    Event Waits Timeouts Time (s) (ms) /txn
    log file parallel write 5,216 4,841 1 0 1.7
    control file parallel write 462 0 0 1 0.2
    control file sequential read 184 0 0 2 0.1
    db file parallel write 672 336 0 0 0.2
    log file sync 24 0 0 0 0.0
    db file sequential read 1 0 0 8 0.0
    LGWR wait for redo copy 12 0 0 0 0.0
    rdbms ipc message 12,386 7,345 10,752 868 4.0
    SQL*Net message from client 384 0 1,498 3901 0.1
    smon timer 4 4 1,229 ###### 0.0
    SQL*Net message to client 384 0 0 0 0.1
    SQL ordered by Gets for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Buffer Gets Threshold: 10000
    -> Note that resources reported for PL/SQL includes the resources used by
    all SQL statements called within the PL/SQL code. As individual SQL
    statements are also reported, it is possible and valid for the summed
    total % to exceed 100
    CPU Elapsd
    Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value
    1,269,773 52 24,418.7 37.5 27.03 76.26 3370382957
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1" ) AND ( (
    381,394 44 8,668.0 11.3 21.30 22.94 3653016280
    SELECT count(*) FROM call_req, ctct, loc, site, z_zo, z_lho WHER
    E ( call_req.customer = ctct.id AND ctct.c_l_id = loc.id AND l
    oc.l_si_id = site.id AND site.z_si_zo_id = z_zo.id AND z_zo.zo
    lhoid = z_lho.id AND z_lho.lho_name LIKE :"SYS_B_0" AND cal
    l_req.active_flag = :"SYS_B_1" ) AND ( ( call_req.group_id != :"
    239,582 10 23,958.2 7.1 5.95 17.44 1650906216
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_0" ) AND ( ( call_req.group_id != :"SYS_B_1" ) and
    146,016 9 16,224.0 4.3 2.58 12.01 977739309
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.group_id IN ( SELECT id FROM ctct WHERE id = :"SYS_B_00" O
    R id = :"SYS_B_01" OR id = :"SYS_B_02" OR id = :"SYS_B_03" OR id
    = :"SYS_B_04" OR id = :"SYS_B_05" OR id = :"SYS_B_06" OR id = :
    "SYS_B_07" OR id = :"SYS_B_08" OR id = :"SYS_B_09" OR id = :"SYS
    117,569 7 16,795.6 3.5 0.52 0.52 1972089848
    SELECT call_req.open_date, call_req.id FROM call_req, ctct WHERE
    ( call_req.status = :"SYS_B_00" AND call_req.group_id = ctct.id
    AND ctct.c_last_name LIKE :"SYS_B_01" AND ( call_req.assigne
    e IS NULL ) ) AND ( call_req.group_id IN ( SELECT id FROM ctct W
    HERE id = :"SYS_B_02" OR id = :"SYS_B_03" OR id = :"SYS_B_04" OR
    100,276 4 25,069.0 3.0 2.77 6.95 771782876
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho, ctct cn01 WHERE ( call_req.customer = ctct.i
    d AND ctct.c_l_id = loc.id AND loc.l_si_id = site.id AND site
    .z_si_zo_id = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.
    lho_name LIKE :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1"
    95,832 4 23,958.0 2.8 2.13 3.80 1755292198
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_00" AND call_req.active_flag = :"SYS_B_01" AND ( ca
    86,680 10 8,668.0 2.6 7.69 8.21 3407388950
    SQL ordered by Gets for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Buffer Gets Threshold: 10000
    -> Note that resources reported for PL/SQL includes the resources used by
    all SQL statements called within the PL/SQL code. As individual SQL
    statements are also reported, it is possible and valid for the summed
    total % to exceed 100
    CPU Elapsd
    Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value
    SELECT count(*) FROM call_req, ctct, loc, site, z_zo, z_lho WHER
    E ( call_req.customer = ctct.id AND ctct.c_l_id = loc.id AND l
    oc.l_si_id = site.id AND site.z_si_zo_id = z_zo.id AND z_zo.zo
    lhoid = z_lho.id AND z_lho.lho_name LIKE :"SYS_B_0" ) AND (
    ( call_req.group_id != :"SYS_B_1" ) and ( call_req.group_id !=
    71,839 3 23,946.3 2.1 2.73 6.07 1599404397
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo WHERE ( call_req.active_flag = :"SYS_B_0" AND call_r
    eq.customer = ctct.id AND ctct.c_l_id = loc.id AND loc.l_si_id
    = site.id AND site.z_si_zo_id = z_zo.id AND z_zo.zo_name LIK
    E :"SYS_B_1" ) AND ( ( call_req.group_id != :"SYS_B_2" ) and (
    60,507 9 6,723.0 1.8 1.47 1.49 632450130
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho, ctct cn01 WHERE ( call_req.customer = ctct.i
    d AND ctct.c_l_id = loc.id AND loc.l_si_id = site.id AND site
    .z_si_zo_id = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.
    lho_name LIKE :"SYS_B_0" AND call_req.status = :"SYS_B_1" AND
    57,682 191 302.0 1.7 3.48 3.52 484128938
    SELECT cnote.posted_date, cnote.text FROM cnote WHERE ( ( cnote.
    loc_id = :"SYS_B_0" ) OR cnote.loc_id IS NULL ) AND ( cnote.inte
    rnal IS NULL OR cnote.internal != :"SYS_B_1" ) ORDER BY cnote.p
    osted_date DESC
    52,146 3 17,382.0 1.5 1.22 3.60 930247717
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.group_id IN ( SELECT id FROM ctct WHERE id = :"SYS_B_00" O
    R id = :"SYS_B_01" OR id = :"SYS_B_02" OR id = :"SYS_B_03" OR id
    = :"SYS_B_04" OR id = :"SYS_B_05" OR id = :"SYS_B_06" OR id = :
    "SYS_B_07" OR id = :"SYS_B_08" OR id = :"SYS_B_09" OR id = :"SYS
    43,534 4 10,883.5 1.3 2.05 2.10 2363733805
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho, prob_ctg, ctct cn01 WHERE ( call_req.custome
    r = ctct.id AND ctct.c_l_id = loc.id AND loc.l_si_id = site.id
    AND site.z_si_zo_id = z_zo.id AND z_zo.zo_lho_id = z_lho.id A
    ND z_lho.lho_name LIKE :"SYS_B_00" AND call_req.active_flag =
    SQL ordered by Reads for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Disk Reads Threshold: 1000
    CPU Elapsd
    Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
    81,653 52 1,570.3 58.8 27.03 76.26 3370382957
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1" ) AND ( (
    15,402 10 1,540.2 11.1 5.95 17.44 1650906216
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_0" ) AND ( ( call_req.group_id != :"SYS_B_1" ) and
    13,371 9 1,485.7 9.6 2.58 12.01 977739309
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.group_id IN ( SELECT id FROM ctct WHERE id = :"SYS_B_00" O
    R id = :"SYS_B_01" OR id = :"SYS_B_02" OR id = :"SYS_B_03" OR id
    = :"SYS_B_04" OR id = :"SYS_B_05" OR id = :"SYS_B_06" OR id = :
    "SYS_B_07" OR id = :"SYS_B_08" OR id = :"SYS_B_09" OR id = :"SYS
    6,157 4 1,539.3 4.4 2.77 6.95 771782876
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho, ctct cn01 WHERE ( call_req.customer = ctct.i
    d AND ctct.c_l_id = loc.id AND loc.l_si_id = site.id AND site
    .z_si_zo_id = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.
    lho_name LIKE :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1"
    6,152 4 1,538.0 4.4 2.13 3.80 1755292198
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_00" AND call_req.active_flag = :"SYS_B_01" AND ( ca
    4,622 3 1,540.7 3.3 2.73 6.07 1599404397
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo WHERE ( call_req.active_flag = :"SYS_B_0" AND call_r
    eq.customer = ctct.id AND ctct.c_l_id = loc.id AND loc.l_si_id
    = site.id AND site.z_si_zo_id = z_zo.id AND z_zo.zo_name LIK
    E :"SYS_B_1" ) AND ( ( call_req.group_id != :"SYS_B_2" ) and (
    2,982 3 994.0 2.1 1.22 3.60 930247717
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.group_id IN ( SELECT id FROM ctct WHERE id = :"SYS_B_00" O
    R id = :"SYS_B_01" OR id = :"SYS_B_02" OR id = :"SYS_B_03" OR id
    = :"SYS_B_04" OR id = :"SYS_B_05" OR id = :"SYS_B_06" OR id = :
    "SYS_B_07" OR id = :"SYS_B_08" OR id = :"SYS_B_09" OR id = :"SYS
    1,566 44 35.6 1.1 21.30 22.94 3653016280
    SELECT count(*) FROM call_req, ctct, loc, site, z_zo, z_lho WHER
    E ( call_req.customer = ctct.id AND ctct.c_l_id = loc.id AND l
    oc.l_si_id = site.id AND site.z_si_zo_id = z_zo.id AND z_zo.zo
    _lho_id = z_lho.id AND z_lho.lho_name LIKE :"SYS_B_0" AND cal
    SQL ordered by Reads for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Disk Reads Threshold: 1000
    CPU Elapsd
    Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
    l_req.active_flag = :"SYS_B_1" ) AND ( ( call_req.group_id != :"
    1,540 1 1,540.0 1.1 0.56 1.64 2582352638
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1" AND ( call
    1,106 2 553.0 0.8 1.25 3.01 548248759
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( ( c
    all_req.assignee IS NOT NULL OR call_req.group_id IS NOT NULL )
    AND ( call_req.type = :"SYS_B_00" OR call_req.type = :"SYS_B_01"
    OR call_req.type IS NULL ) AND call_req.active_flag = :"SYS_B_0
    2" ) AND ( call_req.group_id IN ( SELECT id FROM ctct WHERE id =
    875 2 437.5 0.6 0.94 2.95 1195215130
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( ( c
    all_req.assignee IS NULL AND call_req.group_id IS NULL ) AND ( c
    all_req.type = :"SYS_B_00" OR call_req.type = :"SYS_B_01" OR cal
    l_req.type IS NULL ) AND call_req.active_flag = :"SYS_B_02" ) AN
    D ( call_req.group_id IN ( SELECT id FROM ctct WHERE id = :"SYS_
    473 1 473.0 0.3 1.80 5.57 3376831664
    BEGIN statspack.snap; END;
    357 10 35.7 0.3 7.69 8.21 3407388950
    SELECT count(*) FROM call_req, ctct, loc, site, z_zo, z_lho WHER
    E ( call_req.customer = ctct.id AND ctct.c_l_id = loc.id AND l
    oc.l_si_id = site.id AND site.z_si_zo_id = z_zo.id AND z_zo.zo
    _lho_id = z_lho.id AND z_lho.lho_name LIKE :"SYS_B_0" ) AND (
    ( call_req.group_id != :"SYS_B_1" ) and ( call_req.group_id !=
    177 5 35.4 0.1 1.81 2.08 920690862
    SELECT ctct.c_last_name, ctct.c_first_name, ctct.c_middle_name,
    ctct.c_public_phone, ctct.c_contact_num, ctct.c_org_id, ctct.c_l
    _id, ctct.id FROM ctct, ct_ty WHERE ( ctct.c_ctp_id = ct_ty.id A
    SQL ordered by Executions for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Executions Threshold: 100
    CPU per Elap per
    Executions Rows Processed Rows per Exec Exec (s) Exec (s) Hash Value
    7,741 7,738 1.0 0.00 0.00 1060224445
    SELECT grpmem.group_id , grpmem.manager_flag , grpmem.member , g
    rpmem.notify_flag FROM grpmem WHERE grpmem.id = :"SYS_B_0"
    2,459 2,459 1.0 0.00 0.00 3026674282
    SELECT act_log.action_desc , act_log.analyst , act_log.call_req_
    id , act_log.description , act_log.internal , act_log.knowledge_
    session , act_log.knowledge_tool , act_log.last_mod_dt , act_log
    .persid , act_log.system_time , act_log.time_spent , act_log.tim
    e_stamp , act_log.type FROM act_log WHERE act_log.id = :"SYS_B_0
    1,449 1,449 1.0 0.00 0.00 3299996875
    SELECT att_evt.cancel_time , att_evt.event_tmpl , att_evt.fire_t
    ime , att_evt.first_fire_time , att_evt.group_name , att_evt.las
    t_mod_dt , att_evt.num_fire , att_evt.obj_id , att_evt.persid ,
    att_evt.start_time , att_evt.status_flag , att_evt.user_smag , a
    tt_evt.wait_time FROM att_evt WHERE att_evt.id = :"SYS_B_0"
    1,336 1,336 1.0 0.00 0.00 3034229510
    SELECT cr_prp.description , cr_prp.label , cr_prp.last_mod_by ,
    cr_prp.last_mod_dt , cr_prp.owning_cr , cr_prp.persid , cr_prp.r
    equired , cr_prp.sample , cr_prp.sequence , cr_prp.value FROM cr
    prp WHERE crprp.id = :"SYS_B_0"
    968 968 1.0 0.00 0.00 3460529092
    select t.name, (select owner_instance from sys.aq$_queue_table_
    affinities where table_objno = t.objno) from system.aq$_queue
    _tables t where t.name = :1 and t.schema = :2 for update skip lo
    cked
    808 808 1.0 0.00 0.00 3346182257
    SELECT call_req.active_flag , call_req.affected_rc , call_req.as
    signee , call_req.call_back_date , call_req.call_back_flag , cal
    l_req.category , call_req.change , call_req.charge_back_id , cal
    l_req.close_date , call_req.created_via , call_req.customer , ca
    ll_req.description , call_req.event_token , call_req.extern_ref
    720 720 1.0 0.00 0.00 140137628
    Module: Spotlight On Oracle, classic
    SELECT DECODE(:b1,'BL','Buffer hash table instance lock','CF','C
    ontrol file schema global enqueue lock','CI','Cross-instance fun
    ction invocation instance lock','CS','Control file schema global
    enqueue lock','CU','Cursor bind lock','DF','Data file instance
    lock','DL','Direct loader parallel index create','DM','Mount/sta
    718 718 1.0 0.00 0.00 4078915446
    SELECT options.app_name, options.sym, options.id FROM options WH
    ERE ( options.sym = :"SYS_B_0" ) AND ( options.del = :"SYS_B_1"
    ) ORDER BY options.app_name
    634 634 1.0 0.00 0.00 1199698393
    SELECT loc.alias , loc.del , loc.l_addr1 , loc.l_addr2 , loc.l_a
    ddr3 , loc.l_addr4 , loc.l_addr5 , loc.l_addr6 , loc.l_details ,
    loc.l_name , loc.l_si_id , loc.last_mod , loc.persid , loc.z_cb
    SQL ordered by Executions for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Executions Threshold: 100
    CPU per Elap per
    Executions Rows Processed Rows per Exec Exec (s) Exec (s) Hash Value
    l1 , loc.zcb_l2 , loc.z_cb_l3 , loc.z_l_code , loc.z_ro_code ,
    loc.z_zo_code FROM loc WHERE loc.id = :"SYS_B_0"
    531 208 0.4 0.00 0.00 800192270
    SELECT lrel.l_persid, lrel.l_attr, lrel.l_sql, lrel.r_persid, lr
    el.r_attr, lrel.r_sql, lrel.id FROM lrel WHERE lrel.l_persid = :
    "SYS_B_0" and lrel.l_attr = :"SYS_B_1" ORDER BY lrel.l_persid ,
    lrel.l_attr , lrel.l_sql
    438 438 1.0 0.00 0.00 1317334374
    Select PROPERTY_NAME,PROPERTY_VALUE,PROPERTY_TYPE from CI_PROPER
    TIES where PROPERTY_NAME=:"SYS_B_0"
    429 8,151 19.0 0.00 0.00 1976028604
    SELECT cr_stat.sym, cr_stat.code FROM cr_stat WHERE cr_stat.del
    = :"SYS_B_0" ORDER BY cr_stat.sym
    383 383 1.0 0.00 0.00 2599265718
    DELETE FROM anima WHERE id = :"SYS_B_0"
    359 359 1.0 0.00 0.00 1719939797
    DELETE FROM att_evt WHERE id = :"SYS_B_0"
    337 337 1.0 0.00 0.00 3069423312
    SELECT anima.a_act , anima.a_delta , anima.a_lock , anima.a_name
    , anima.a_org , anima.a_string , anima.a_time , anima.t_method
    , anima.t_persid , anima.t_type FROM anima WHERE anima.id = :"SY
    S_B_0"
    332 331 1.0 0.00 0.00 1549656119
    SELECT crsq.id FROM crsq WHERE crsq.code = :"SYS_B_0"
    315 315 1.0 0.00 0.00 1734736338
    UPDATE cr_prp SET last_mod_by = :"SYS_B_0" , last_mod_dt = :"SYS
    _B_1" WHERE id = :"SYS_B_2"
    308 1,580 5.1 0.00 0.00 618252548
    SELECT cr_prp.sequence, cr_prp.id FROM cr_prp WHERE cr_prp.ownin
    g_cr = :"SYS_B_0" ORDER BY cr_prp.sequence
    279 1,716 6.2 0.00 0.00 749386807
    SELECT call_req.open_date, call_req.id FROM call_req WHERE call_
    req.customer = :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1"
    ORDER BY call_req.open_date DESC
    277 277 1.0 0.00 0.00 321149819
    INSERT INTO anima ( a_act, a_delta, a_lock, a_name, a_org, a_str
    ing, a_time, t_method, t_persid, t_type, id ) VALUES ( :"SYS_B_
    0" , :"SYS_B_1" , :"SYS_B_2" , :"SYS_B_3" , :"SYS_B_4" , nu
    ll , :"SYS_B_5" , :"SYS_B_6" , :"SYS_B_7" , :"SYS_B_8" , :"
    SQL ordered by Parse Calls for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Parse Calls Threshold: 1000
    % Total
    Parse Calls Executions Parses Hash Value
    7,733 7,741 27.15 1060224445
    SELECT grpmem.group_id , grpmem.manager_flag , grpmem.member , g
    rpmem.notify_flag FROM grpmem WHERE grpmem.id = :"SYS_B_0"
    2,459 2,459 8.63 3026674282
    SELECT act_log.action_desc , act_log.analyst , act_log.call_req_
    id , act_log.description , act_log.internal , act_log.knowledge_
    session , act_log.knowledge_tool , act_log.last_mod_dt , act_log
    .persid , act_log.system_time , act_log.time_spent , act_log.tim
    e_stamp , act_log.type FROM act_log WHERE act_log.id = :"SYS_B_0
    1,449 1,449 5.09 3299996875
    SELECT att_evt.cancel_time , att_evt.event_tmpl , att_evt.fire_t
    ime , att_evt.first_fire_time , att_evt.group_name , att_evt.las
    t_mod_dt , att_evt.num_fire , att_evt.obj_id , att_evt.persid ,
    att_evt.start_time , att_evt.status_flag , att_evt.user_smag , a
    tt_evt.wait_time FROM att_evt WHERE att_evt.id = :"SYS_B_0"
    1,336 1,336 4.69 3034229510
    SELECT cr_prp.description , cr_prp.label , cr_prp.last_mod_by ,
    cr_prp.last_mod_dt , cr_prp.owning_cr , cr_prp.persid , cr_prp.r
    equired , cr_prp.sample , cr_prp.sequence , cr_prp.value FROM cr
    prp WHERE crprp.id = :"SYS_B_0"
    808 808 2.84 3346182257
    SELECT call_req.active_flag , call_req.affected_rc , call_req.as
    signee , call_req.call_back_date , call_req.call_back_flag , cal
    l_req.category , call_req.change , call_req.charge_back_id , cal
    l_req.close_date , call_req.created_via , call_req.customer , ca
    ll_req.description , call_req.event_token , call_req.extern_ref
    718 718 2.52 4078915446
    SELECT options.app_name, options.sym, options.id FROM options WH
    ERE ( options.sym = :"SYS_B_0" ) AND ( options.del = :"SYS_B_1"
    ) ORDER BY options.app_name
    634 634 2.23 1199698393
    SELECT loc.alias , loc.del , loc.l_addr1 , loc.l_addr2 , loc.l_a
    ddr3 , loc.l_addr4 , loc.l_addr5 , loc.l_addr6 , loc.l_details ,
    loc.l_name , loc.l_si_id , loc.last_mod , loc.persid , loc.z_cb
    l1 , loc.zcb_l2 , loc.z_cb_l3 , loc.z_l_code , loc.z_ro_code ,
    loc.z_zo_code FROM loc WHERE loc.id = :"SYS_B_0"
    531 531 1.86 800192270
    SELECT lrel.l_persid, lrel.l_attr, lrel.l_sql, lrel.r_persid, lr
    el.r_attr, lrel.r_sql, lrel.id FROM lrel WHERE lrel.l_persid = :
    "SYS_B_0" and lrel.l_attr = :"SYS_B_1" ORDER BY lrel.l_persid ,
    lrel.l_attr , lrel.l_sql
    438 438 1.54 1317334374
    Select PROPERTY_NAME,PROPERTY_VALUE,PROPERTY_TYPE from CI_PROPER
    TIES where PROPERTY_NAME=:"SYS_B_0"
    429 429 1.51 1976028604
    SQL ordered by Parse Calls for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Parse Calls Threshold: 1000
    % Total
    Parse Calls Executions Parses Hash Value
    SELECT cr_stat.sym, cr_stat.code FROM cr_stat WHERE cr_stat.del
    = :"SYS_B_0" ORDER BY cr_stat.sym
    383 383 1.34 2599265718
    DELETE FROM anima WHERE id = :"SYS_B_0"
    359 359 1.26 1719939797
    DELETE FROM att_evt WHERE id = :"SYS_B_0"
    337 337 1.18 3069423312
    SELECT anima.a_act , anima.a_delta , anima.a_lock , anima.a_name
    , anima.a_org , anima.a_string , anima.a_time , anima.t_method
    , anima.t_persid , anima.t_type FROM anima WHERE anima.id = :"SY
    S_B_0"
    330 332 1.16 1549656119
    SELECT crsq.id FROM crsq WHERE crsq.code = :"SYS_B_0"
    315 315 1.11 1734736338
    UPDATE cr_prp SET last_mod_by = :"SYS_B_0" , last_mod_dt = :"SYS
    _B_1" WHERE id = :"SYS_B_2"
    308 308 1.08 618252548
    SELECT cr_prp.sequence, cr_prp.id FROM cr_prp WHERE cr_prp.ownin
    g_cr = :"SYS_B_0" ORDER BY cr_prp.sequence
    277 277 0.97 321149819
    INSERT INTO anima ( a_act, a_delta, a_lock, a_name, a_org, a_str
    ing, a_time, t_method, t_persid, t_type, id ) VALUES ( :"SYS_B_
    0" , :"SYS_B_1" , :"SYS_B_2" , :"SYS_B_3" , :"SYS_B_4" , nu
    ll , :"SYS_B_5" , :"SYS_B_6" , :"SYS_B_7" , :"SYS_B_8" , :"
    SYS_B_9" )
    277 279 0.97 749386807
    SELECT call_req.open_date, call_req.id FROM call_req WHERE call_
    req.customer = :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1"
    ORDER BY call_req.open_date DESC
    275 275 0.97 2816620377
    INSERT INTO att_evt ( cancel_time, event_tmpl, fire_time, first_
    fire_time, group_name, last_mod_dt, num_fire, obj_id, persid, st
    art_time, status_flag, user_smag, wait_time, id ) VALUES ( null
    , :"SYS_B_00" , :"SYS_B_01" , :"SYS_B_02" , :"SYS_B_03" ,
    :"SYS_B_04" , :"SYS_B_05" , :"SYS_B_06" , :"SYS_B_07" , :"SY
    269 269 0.94 3605948696
    SELECT slatpl.del , slatpl.elapsed , slatpl.event , slatpl.last_
    mod_by , slatpl.last_mod_dt , slatpl.object_type , slatpl.persid
    , slatpl.service_type , slatpl.sym FROM slatpl WHERE slatpl.id
    SQL ordered by Sharable Memory for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Sharable Memory Threshold: 1048576
    Sharable Mem (b) Executions % Total Hash Value
    23,912,520 231 13.6 139964375
    SELECT anima.a_name, anima.t_persid, anima.t_method, anima.id FR
    OM anima WHERE anima.t_persid LIKE :"SYS_B_0" ORDER BY anima.
    a_name
    18,314,292 26 10.4 380755726
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.ref_num LIKE :"SYS_B_00" ) AND ( call_req.group_id IN (
    SELECT id FROM ctct WHERE id = :"SYS_B_01" OR id = :"SYS_B_02" O
    R id = :"SYS_B_03" OR id = :"SYS_B_04" OR id = :"SYS_B_05" OR id
    = :"SYS_B_06" OR id = :"SYS_B_07" OR id = :"SYS_B_08" OR id = :
    12,365,844 107 7.0 1877135209
    SELECT chg.open_date, chg.chg_ref_num, chg.id FROM chg WHERE ( c
    hg.affected_contact = :"SYS_B_0" and chg.active_flag = :"SYS_B_1
    " ) AND ( chg.affected_contact = :"SYS_B_2" ) ORDER BY chg.open
    _date DESC
    2,692,852 17 1.5 4181730075
    SELECT ctct.c_last_name, ctct.c_first_name, ctct.c_middle_name,
    ctct.c_public_phone, ctct.c_contact_num, ctct.c_org_id, ctct.c_l
    _id, ctct.id FROM ctct, ct_ty WHERE ( ctct.c_last_name LIKE :"
    SYS_B_0" AND ctct.c_ctp_id = ct_ty.id AND ct_ty.id = :"SYS_B_1"
    AND ctct.del = :"SYS_B_2" AND ctct.id IN ( SELECT member FROM g
    2,048,083 10 1.2 153455816
    SELECT ctct.c_last_name, ctct.c_first_name, ctct.c_middle_name,
    ctct.c_public_phone, ctct.c_contact_num, ctct.c_org_id, ctct.c_l
    _id, ctct.id FROM ctct WHERE ( ctct.c_last_name LIKE :"SYS_B_0
    " ) AND ( ( ctct.del = :"SYS_B_1" ) AND ( ctct.c_ctp_id = :"SYS_
    B_2" AND ctct.alias = -:"SYS_B_3" ) ) ORDER BY ctct.c_last_name
    1,653,628 3 0.9 1096419296
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.ref_num LIKE :"SYS_B_0" ) AND ( ( call_req.group_id IN (
    SELECT group_id FROM grpmem WHERE member = :"SYS_B_1" ) ) or ca
    ll_req.assignee = :"SYS_B_2" or call_req.customer = :"SYS_B_3" )
    ORDER BY call_req.open_date DESC
    SQL ordered by Version Count for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Version Count Threshold: 20
    Version
    Count Executions Hash Value
    349 231 139964375
    SELECT anima.a_name, anima.t_persid, anima.t_method, anima.id FR
    OM anima WHERE anima.t_persid LIKE :"SYS_B_0" ORDER BY anima.
    a_name
    196 107 1877135209
    SELECT chg.open_date, chg.chg_ref_num, chg.id FROM chg WHERE ( c
    hg.affected_contact = :"SYS_B_0" and chg.active_flag = :"SYS_B_1
    " ) AND ( chg.affected_contact = :"SYS_B_2" ) ORDER BY chg.open
    _date DESC
    127 26 380755726
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.ref_num LIKE :"SYS_B_00" ) AND ( call_req.group_id IN (
    SELECT id FROM ctct WHERE id = :"SYS_B_01" OR id = :"SYS_B_02" O
    R id = :"SYS_B_03" OR id = :"SYS_B_04" OR id = :"SYS_B_05" OR id
    = :"SYS_B_06" OR id = :"SYS_B_07" OR id = :"SYS_B_08" OR id = :
    36 17 4181730075
    SELECT ctct.c_last_name, ctct.c_first_name, ctct.c_middle_name,
    ctct.c_public_phone, ctct.c_contact_num, ctct.c_org_id, ctct.c_l
    _id, ctct.id FROM ctct, ct_ty WHERE ( ctct.c_last_name LIKE :"
    SYS_B_0" AND ctct.c_ctp_id = ct_ty.id AND ct_ty.id = :"SYS_B_1"
    AND ctct.del = :"SYS_B_2" AND ctct.id IN ( SELECT member FROM g
    33 10 153455816
    SELECT ctct.c_last_name, ctct.c_first_name, ctct.c_middle_name,
    ctct.c_public_phone, ctct.c_contact_num, ctct.c_org_id, ctct.c_l
    _id, ctct.id FROM ctct WHERE ( ctct.c_last_name LIKE :"SYS_B_0
    " ) AND ( ( ctct.del = :"SYS_B_1" ) AND ( ctct.c_ctp_id = :"SYS_
    B_2" AND ctct.alias = -:"SYS_B_3" ) ) ORDER BY ctct.c_last_name
    26 3 1096419296
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.ref_num LIKE :"SYS_B_0" ) AND ( ( call_req.group_id IN (
    SELECT group_id FROM grpmem WHERE member = :"SYS_B_1" ) ) or ca
    ll_req.assignee = :"SYS_B_2" or call_req.customer = :"SYS_B_3" )
    ORDER BY call_req.open_date DESC
    Instance Activity Stats for DB: AHD Instance: ahd Snaps: 20 -21
    Statistic Total per Second per Trans
    CPU used by this session 12,450 8.7 4.1
    CPU used when call started 12,515 8.8 4.1
    CR blocks created 53 0.0 0.0
    DBWR buffers scanned 0 0.0 0.0
    DBWR checkpoint buffers written 1,644 1.2 0.5
    DBWR checkpoints 0 0.0 0.0
    DBWR free buffers found 0 0.0 0.0
    DBWR lru scans 0 0.0 0.0
    DBWR make free requests 0 0.0 0.0
    DBWR summed scan depth 0 0.0 0.0
    DBWR transaction table writes 10 0.0 0.0
    DBWR undo block writes 238 0.2 0.1
    SQL*Net roundtrips to/from client 131,833 92.6 42.9
    active txn count during cleanout 130 0.1 0.0
    background checkpoints completed 0 0.0 0.0
    background checkpoints started 0 0.0 0.0
    background timeouts 2,161 1.5 0.7
    branch node splits 0 0.0 0.0
    buffer is not pinned count 3,147,925 2,210.6 1,024.1
    buffer is pinned count 638,155 448.1 207.6
    bytes received via SQL*Net from c 20,116,711 14,126.9 6,544.2
    bytes sent via SQL*Net to client 33,961,169 23,849.1 11,047.9
    calls to get snapshot scn: kcmgss 76,324 53.6 24.8
    calls to kcmgas 6,266 4.4 2.0
    calls to kcmgcs 110 0.1 0.0
    change write time 25 0.0 0.0
    cleanout - number of ktugct calls 145 0.1 0.1
    cleanouts and rollbacks - consist 0 0.0 0.0
    cleanouts only - consistent read 0 0.0 0.0
    cluster key scan block gets 1,361 1.0 0.4
    cluster key scans 1,146 0.8 0.4
    commit cleanout failures: buffer 0 0.0 0.0
    commit cleanout failures: callbac 3 0.0 0.0
    commit cleanout failures: cannot 0 0.0 0.0
    commit cleanouts 14,837 10.4 4.8
    commit cleanouts successfully com 14,834 10.4 4.8
    commit txn count during cleanout 106 0.1 0.0
    consistent changes 2,123 1.5 0.7
    consistent gets 3,336,864 2,343.3 1,085.5
    consistent gets - examination 197,061 138.4 64.1
    cursor authentications 71 0.1 0.0
    data blocks consistent reads - un 2,123 1.5 0.7
    db block changes 50,525 35.5 16.4
    db block gets 47,774 33.6 15.5
    deferred (CURRENT) block cleanout 7,940 5.6 2.6
    dirty buffers inspected 0 0.0 0.0
    enqueue conversions 29 0.0 0.0
    enqueue releases 14,210 10.0 4.6
    enqueue requests 14,210 10.0 4.6
    enqueue waits 0 0.0 0.0
    execute count 32,955 23.1 10.7
    free buffer inspected 16 0.0 0.0
    free buffer requested 140,283 98.5 45.6
    hot buffers moved to head of LRU 950 0.7 0.3
    immediate (CR) block cleanout app 0 0.0 0.0
    immediate (CURRENT) block cleanou 2,804 2.0 0.9
    Instance Activity Stats for DB: AHD Instance: ahd Snaps: 20 -21
    Statistic Total per Second per Trans
    index fast full scans (full) 157 0.1 0.1
    index fetch by key 70,378 49.4 22.9
    index scans kdiixs1 28,181 19.8 9.2
    leaf node 90-10 splits 10 0.0 0.0
    leaf node splits 76 0.1 0.0
    logons cumulative 11 0.0 0.0
    messages received 5,452 3.8 1.8
    messages sent 5,452 3.8 1.8
    no buffer to keep pinned count 0 0.0 0.0
    no work - consistent read gets 3,085,481 2,166.8 1,003.7
    opened cursors cumulative 4,561 3.2 1.5
    parse count (failures) 0 0.0 0.0
    parse count (hard) 412 0.3 0.1
    parse count (total) 28,484 20.0 9.3
    parse time cpu 426 0.3 0.1
    parse time elapsed 469 0.3 0.2
    physical reads 138,930 97.6 45.2
    physical reads direct 0 0.0 0.0
    physical writes 1,644 1.2 0.5
    physical writes direct 0 0.0 0.0
    physical writes non checkpoint 232 0.2 0.1
    pinned buffers inspected 7 0.0 0.0
    prefetched blocks 32,732 23.0 10.7
    process last non-idle time 12,884,949,552 9,048,419.6 4,191,590.6
    recursive calls 22,718 16.0 7.4
    recursive cpu usage 226 0.2 0.1
    redo blocks written 19,178 13.5 6.2
    redo buffer allocation retries 0 0.0 0.0
    redo entries 27,265 19.2 8.9
    redo log space requests 0 0.0 0.0
    redo log space wait time 0 0.0 0.0
    redo size 8,488,216 5,960.8 2,761.3
    redo synch time 74 0.1 0.0
    redo synch writes 3,078 2.2 1.0
    redo wastage 1,040,788 730.9 338.6
    redo write time 75 0.1 0.0
    redo writer latching time 0 0.0 0.0
    redo writes 5,216 3.7 1.7
    rollback changes - undo records a 6 0.0 0.0
    rollbacks only - consistent read 233 0.2 0.1
    rows fetched via callback 54,581 38.3 17.8
    session connect time 12,884,949,552 9,048,419.6 4,191,590.6
    session logical reads 3,384,638 2,376.9 1,101.1
    session pga memory max 6,168,536 4,331.8 2,006.7
    session uga memory 599,984 421.3 195.2
    session uga memory max 9,592,864 6,736.6 3,120.7
    shared hash latch upgrades - no w 27,737 19.5 9.0
    shared hash latch upgrades - wait 84 0.1 0.0
    sorts (disk) 0 0.0 0.0
    sorts (memory) 6,834 4.8 2.2
    sorts (rows) 3,229,994 2,268.3 1,050.8
    summed dirty queue length 0 0.0 0.0
    switch current to new buffer 990 0.7 0.3
    table fetch by rowid 474,673 333.3 154.4
    table fetch continued row 8 0.0 0.0
    table scan blocks gotten 2,751,375 1,932.2 895.1
    Instance Activity Stats for DB: AHD Instance: ahd Snaps: 20 -21
    Statistic Total per Second per Trans
    table scan rows gotten 55,928,200 39,275.4 18,194.0
    table scans (long tables) 245 0.2 0.1
    table scans (short tables) 3,383 2.4 1.1
    transaction rollbacks 3 0.0 0.0
    transaction tables consistent rea 0 0.0 0.0
    transaction tables consistent rea 0 0.0 0.0
    user calls 131,904 92.6 42.9
    user commits 3,074 2.2 1.0
    user rollbacks 0 0.0 0.0
    workarea executions - onepass 0 0.0 0.0
    workarea executions - optimal 8,438 5.9 2.7
    write clones created in backgroun 0 0.0 0.0
    write clones created in foregroun 0 0.0 0.0
    Tablespace IO Stats for DB: AHD Instance: ahd Snaps: 20 -21
    ->ordered by IOs (Reads + Writes) desc
    Tablespace
    Av Av Av Av Buffer Av Buf
    Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)
    AHD1_DATA
    105,869 74 0.9 1.3 828 1 11,740 0.5
    AHD1_IDX
    38 0 7.4 1.0 563 0 0 0.0
    PERFSTAT
    372 0 3.6 1.0 0 0 0 0.0
    UNDOTBS1
    0 0 0.0 248 0 0 0.0
    SYSTEM
    6 0 6.7 1.0 5 0 0 0.0
    File IO Stats for DB: AHD Instance: ahd Snaps: 20 -21
    ->ordered by Tablespace, File
    Tablespace Filename
    Av Av Av Av Buffer Av Buf
    Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)
    AHD1_DATA E:\ORACLE\ORADATA\AHD\AHD1_DATA.ORA
    53,454 38 0.9 1.3 432 0 5,949 0.5
    E:\ORACLE\ORADATA\AHD\AHD2_DATA.ORA
    52,415 37 0.9 1.3 396 0 5,791 0.5
    AHD1_IDX E:\ORACLE\ORADATA\AHD\AHD1_IDX.ORA
    38 0 7.4 1.0 563 0 0
    PERFSTAT E:\ORACLE\ORADATA\AHD\PERFSTAT.ORA
    372 0 3.6 1.0 0 0 0
    SYSTEM E:\ORACLE\ORADATA\AHD\SYSTEM01.DBF
    6 0 6.7 1.0 5 0 0
    UNDOTBS1 E:\ORACLE\ORADATA\AHD\UNDOTBS01.DBF
    0 0 248 0 0
    Buffer Pool Statistics for DB: AHD Instance: ahd Snaps: 20 -21
    -> Standard block size Pools D: default, K: keep, R: recycle
    -> Default Pools for other block sizes: 2k, 4k, 8k, 16k, 32k
    Free Write Buffer
    Number of Cache Buffer Physical Physical Buffer Complete Busy
    P Buffers Hit % Gets Reads Writes Waits Waits Waits
    D 82,082 97.8 6,327,007 138,971 1,644 0 0 11,760
    Instance Recovery Stats for DB: AHD Instance: ahd Snaps: 20 -21
    -> B: Begin snapshot, E: End snapshot
    Targt Estd Log File Log Ckpt Log Ckpt
    MTTR MTTR Recovery Actual Target Size Timeout Interval
    (s) (s) Estd IOs Redo Blks Redo Blks Redo Blks Redo Blks Redo Blks
    B 75 26 2354 18057 17632 184320 17632
    E 75 27 2967 23569 22952 184320 22952
    Buffer Pool Advisory for DB: AHD Instance: ahd End Snap: 21
    -> Only rows with estimated physical reads >0 are displayed
    -> ordered by Block Size, Buffers For Estimate
    Size for Size Buffers for Est Physical Estimated
    P Estimate (M) Factr Estimate Read Factor Physical Reads
    D 64 .1 8,008 261.38 4,357,231,706
    D 128 .2 16,016 207.44 3,458,029,385
    D 192 .3 24,024 143.22 2,387,570,894
    D 256 .4 32,032 2.29 38,243,018
    D 320 .5 40,040 1.89 31,541,321
    D 384 .6 48,048 1.74 29,023,767
    D 448 .7 56,056 1.69 28,232,064
    D 512 .8 64,064 1.20 19,951,481
    D 576 .9 72,072 1.11 18,529,925
    D 640 1.0 80,080 1.04 17,367,752
    D 656 1.0 82,082 1.00 16,670,129
    D 704 1.1 88,088 0.97 16,124,256
    D 768 1.2 96,096 0.91 15,155,822
    D 832 1.3 104,104 0.90 15,055,099
    D 896 1.4 112,112 0.89 14,839,567
    D 960 1.5 120,120 0.88 14,668,682
    D 1,024 1.6 128,128 0.87 14,479,726
    D 1,088 1.7 136,136 0.84 13,988,866
    D 1,152 1.8 144,144 0.70 11,723,518
    D 1,216 1.9 152,152 0.61 10,156,857
    D 1,280 2.0 160,160 0.20 3,281,883
    Buffer wait Statistics for DB: AHD Instance: ahd Snaps: 20 -21
    -> ordered by wait time desc, waits desc
    Tot Wait Avg
    Class Waits Time (s) Time (ms)
    data block 11,754 6 0
    PGA Aggr Target Stats for DB: AHD Instance: ahd Snaps: 20 -21
    -> B: Begin snap E: End snap (rows dentified with B or E contain data
    which is absolute i.e. not diffed over the interval)
    -> PGA cache hit % - percentage of W/A (WorkArea) data processed only in-memory
    -> Auto PGA Target - actual workarea memory target
    -> W/A PGA Used - amount of memory used for all Workareas (manual + auto)
    -> %PGA W/A Mem - percentage of PGA memory allocated to workareas
    -> %Auto W/A Mem - percentage of workarea memory controlled by Auto Mem Mgmt
    -> %Man W/A Mem - percentage of workarea memory under manual control
    PGA Cache Hit % W/A MB Processed Extra W/A MB Read/Written
    100.0 1,169 0
    %PGA %Auto %Man
    PGA Aggr Auto PGA PGA Mem W/A PGA W/A W/A W/A Global Mem
    Target(M) Target(M) Alloc(M) Used(M) Mem Mem Mem Bound(K)
    B 350 293 37.6 0.0 .0 .0 .0 17,920
    E 350 293 37.5 0.2 .6 100.0 .0 17,920
    PGA Aggr Target Histogram for DB: AHD Instance: ahd Snaps: 20 -21
    -> Optimal Executions are purely in-memory operations
    Low High
    Optimal Optimal Total Execs Optimal Execs 1-Pass Execs M-Pass Execs
    8K 16K 6,809 6,809 0 0
    16K 32K 148 148 0 0
    32K 64K 90 90 0 0
    64K 128K 154 154 0 0
    128K 256K 73 73 0 0
    256K 512K 308 308 0 0
    512K 1024K 374 374 0 0
    1M 2M 171 171 0 0
    2M 4M 217 217 0 0
    4M 8M 10 10 0 0
    PGA Memory Advisory for DB: AHD Instance: ahd End Snap: 21
    -> When using Auto Memory Mgmt, minimally choose a pga_aggregate_target value
    where Estd PGA Overalloc Count is 0
    Estd Extra Estd PGA Estd PGA
    PGA Target Size W/A MB W/A MB Read/ Cache Overalloc
    Est (MB) Factr Processed Written to Disk Hit % Count
    44 0.1 180,060.5 42,218.7 81.0 4
    88 0.3 180,060.5 23,194.7 89.0 0
    175 0.5 180,060.5 9,436.8 95.0 0
    263 0.8 180,060.5 9,356.7 95.0 0
    350 1.0 180,060.5 9,274.8 95.0 0
    420 1.2 180,060.5 9,169.9 95.0 0
    490 1.4 180,060.5 9,148.0 95.0 0
    560 1.6 180,060.5 9,148.0 95.0 0
    630 1.8 180,060.5 9,148.0 95.0 0
    700 2.0 180,060.5 9,148.0 95.0 0
    1,050 3.0 180,060.5 9,148.0 95.0 0
    1,400 4.0 180,060.5 9,148.0 95.0 0
    2,100 6.0 180,060.5 3,983.3 98.0 0
    2,800 8.0 180,060.5 3,983.3 98.0 0
    Rollback Segment Stats for DB: AHD Instance: ahd Snaps: 20 -21
    ->A high value for "Pct Waits" suggests more rollback segments may be required
    ->RBS stats may not be accurate between begin and end snaps when using Auto Undo
    managment, as RBS may be dynamically created and dropped as needed
    Trans Table Pct Undo Bytes
    RBS No Gets Waits Written Wraps Shrinks Extends
    0 29.0 0.00 0 0 0 0
    1 975.0 0.00 122,796 0 0 0
    2 1,244.0 0.00 1,094,706 10 0 5
    3 816.0 0.00 118,596 0 0 0
    4 1,430.0 0.00 212,754 2 0 0
    5 1,716.0 0.00 291,940 2 0 0
    6 1,287.0 0.00 197,900 0 0 0
    7 1,674.0 0.00 279,160 0 0 0
    8 1,031.0 0.00 148,216 0 0 0
    9 947.0 0.00 141,870 0 0 0
    10 834.0 0.00 117,422 0 0 0
    Rollback Segment Storage for DB: AHD Instance: ahd Snaps: 20 -21
    ->Optimal Size should be larger than Avg Active
    RBS No Segment Size Avg Active Optimal Size Maximum Size
    0 385,024 0 385,024
    1 2,220,032 455,412 2,220,032
    2 2,088,960 333,026 2,220,032
    3 2,220,032 456,101 2,220,032
    4 2,220,032 474,584 3,268,608
    5 2,220,032 480,865 3,268,608
    6 2,220,032 513,967 3,268,608
    7 2,220,032 480,785 2,220,032
    8 2,220,032 496,182 2,220,032
    9 2,220,032 486,763 2,220,032
    10 2,220,032 430,016 6,414,336
    Undo Segment Summary for DB: AHD Instance: ahd Snaps: 20 -21
    -> Undo segment block stats:
    -> uS - unexpired Stolen, uR - unexpired Released, uU - unexpired reUsed
    -> eS - expired Stolen, eR - expired Released, eU - expired reUsed
    Undo Undo Num Max Qry Max Tx Snapshot Out of uS/uR/uU/
    TS# Blocks Trans Len (s) Concurcy Too Old Space eS/eR/eU
    1 395 2,900,725 5 1 0 0 0/0/0/0/0/0
    Undo Segment Stats for DB: AHD Instance: ahd Snaps: 20 -21
    -> ordered by Time desc
    Undo Num Max Qry Max Tx Snap Out of uS/uR/uU/
    End Time Blocks Trans Len (s) Concy Too Old Space eS/eR/eU
    13-Feb 15:04 96 ######## 4 1 0 0 0/0/0/0/0/0
    13-Feb 14:54 299 ######## 5 1 0 0 0/0/0/0/0/0
    Latch Activity for DB: AHD Instance: ahd Snaps: 20 -21
    ->"Get Requests", "Pct Get Miss" and "Avg Slps/Miss" are statistics for
    willing-to-wait latch get requests
    ->"NoWait Requests", "Pct NoWait Miss" are for no-wait latch get requests
    ->"Pct Misses" for both should be very close to 0.0
    Pct Avg Wait Pct
    Get Get Slps Time NoWait NoWait
    Latch Requests Miss /Miss (s) Requests Miss
    Consistent RBA 5,216 0.0 0 0
    FOB s.o list latch 34 0.0 0 0
    SQL memory manager latch 1 0.0 0 462 0.0
    SQL memory manager worka 40,347 0.0 0 0
    active checkpoint queue 1,261 0.0 0 0
    archive control 163 0.0 0 0
    archive process latch 29 0.0 0 0
    cache buffer handles 378 0.0 0 0
    cache buffers chains 6,836,244 0.4 0.0 0 266,617 0.0
    cache buffers lru chain 244,157 0.0 0.0 0 140,432 0.0
    channel handle pool latc 21 0.0 0 0
    channel operations paren 960 0.0 0 0
    checkpoint queue latch 86,982 0.0 0 2,337 0.0
    child cursor hash table 6,464 0.0 0.0 0 0
    dml lock allocation 15,005 0.0 0 0
    dummy allocation 21 0.0 0 0
    enqueue hash chains 28,447 0.0 0 0
    enqueues 8,689 0.0 0 0
    event group latch 11 0.0 0 0
    file number translation 4,079 0.0 0 0
    hash table column usage 38 0.0 0 187,596 0.0
    hash table modification 1 0.0 0 0
    job_queue_processes para 23 0.0 0 0
    ktm global data 4 0.0 0 0
    kwqit: protect wakeup ti 45 0.0 0 0
    lgwr LWN SCN 5,328 0.4 0.0 0 0
    library cache 342,865 0.2 0.0 0 342 0.6
    library cache load lock 452 0.0 0 0
    library cache pin 197,662 0.0 0.0 0 0
    library cache pin alloca 124,035 0.0 0.0 0 0
    list of block allocation 55 0.0 0 0
    messages 30,779 0.0 0.0 0 0
    mostly latch-free SCN 5,459 1.8 0.0 0 0
    multiblock read objects 194,822 0.0 0.0 0 0
    ncodef allocation latch 23 0.0 0 0
    object stats modificatio 618 0.0 0 0
    post/wait queue 10,441 0.0 0 3,078 0.0
    process allocation 11 0.0 0 11 0.0
    process group creation 21 0.0 0 0
    redo allocation 37,773 0.0 0.0 0 0
    redo copy 0 0 27,274 0.0
    redo writing 17,880 0.0 0 0
    row cache enqueue latch 169,423 0.0 0.0 0 0
    row cache objects 169,795 0.0 0 3 0.0
    sequence cache 38 0.0 0 0
    session allocation 15,580 0.0 0 0
    session idle bit 269,419 0.0 0.0 0 0
    session switching 23 0.0 0 0
    session timer 478 0.0 0 0
    shared pool 104,427 0.1 0.0 0 0
    Latch Activity for DB: AHD Instance: ahd Snaps: 20 -21
    ->"Get Requests", "Pct Get Miss" and "Avg Slps/Miss" are statistics for
    willing-to-wait latch get requests
    ->"NoWait Requests", "Pct NoWait Miss" are for no-wait latch get requests
    ->"Pct Misses" for both should be very close to 0.0
    Pct Avg Wait Pct
    Get Get Slps Time NoWait NoWait
    Latch Requests Miss /Miss (s) Requests Miss
    sim partition latch 0 0 32 0.0
    simulator hash latch 217,119 0.0 0.0 0 0
    simulator lru latch 16,247 0.0 0 902 0.4
    sort extent pool 29 0.0 0 0
    transaction allocation 36 0.0 0 0
    transaction branch alloc 23 0.0 0 0
    undo global data 19,973 0.0 0 0
    user lock 42 0.0 0 0
    Latch Sleep breakdown for DB: AHD Instance: ahd Snaps: 20 -21
    -> ordered by misses desc
    Get Spin &
    Latch Name Requests Misses Sleeps Sleeps 1->4
    cache buffers chains 6,836,244 26,201 46 0/0/0/0/0
    library cache 342,865 778 5 773/5/0/0/0
    shared pool 104,427 125 3 122/3/0/0/0
    Latch Miss Sources for DB: AHD Instance: ahd Snaps: 20 -21
    -> only latches with sleeps are shown
    -> ordered by name, sleeps desc
    NoWait Waiter
    Latch Name Where Misses Sleeps Sleeps
    cache buffers chains kcbgtcr: kslbegin excl 0 32 30
    cache buffers chains kcbrls: kslbegin 0 7 13
    cache buffers chains kcbzwb 0 4 3
    cache buffers chains kcbgtcr: fast path 0 3 0
    library cache kglic 0 2 0
    library cache kglobpn: child: 0 2 0
    library cache kgllkdl: child: cleanup 0 1 0
    shared pool kghalo 0 2 0
    shared pool kghalp 0 1 0
    Child Latch Statistics DB: AHD Instance: ahd Snaps: 20 -21
    -> only latches with sleeps/gets > 1/100000 are shown
    -> ordered by name, gets desc
    Child Get Spin &
    Latch Name Num Requests Misses Sleeps Sleeps 1->4
    cache buffers chains 439 28,269 1,276 1 0/0/0/0/0
    cache buffers chains 269 26,297 842 1 0/0/0/0/0
    cache buffers chains 1010 17,482 49 2 0/0/0/0/0
    cache buffers chains 260 11,141 20 1 0/0/0/0/0
    cache buffers chains 324 9,454 29 1 0/0/0/0/0
    cache buffers chains 840 7,235 20 1 0/0/0/0/0
    cache buffers chains 46 6,868 25 1 0/0/0/0/0
    cache buffers chains 835 6,799 26 2 0/0/0/0/0
    cache buffers chains 202 6,768 17 1 0/0/0/0/0
    cache buffers chains 740 6,573 38 2 0/0/0/0/0
    cache buffers chains 592 6,508 30 1 0/0/0/0/0
    cache buffers chains 436 6,485 25 2 0/0/0/0/0
    cache buffers chains 513 6,443 16 1 0/0/0/0/0
    cache buffers chains 844 6,436 28 1 0/0/0/0/0
    cache buffers chains 117 6,423 25 1 0/0/0/0/0
    cache buffers chains 389 6,381 25 1 0/0/0/0/0
    cache buffers chains 116 6,349 29 1 0/0/0/0/0
    cache buffers chains 51 6,340 34 1 0/0/0/0/0
    cache buffers chains 914 6,259 31 1 0/0/0/0/0
    cache buffers chains 713 6,249 24 1 0/0/0/0/0
    cache buffers chains 465 6,198 27 2 0/0/0/0/0
    cache buffers chains 416 6,193 27 1 0/0/0/0/0
    cache buffers chains 432 6,155 34 1 0/0/0/0/0
    cache buffers chains 583 6,152 23 2 0/0/0/0/0
    cache buffers chains 126 6,147 35 1 0/0/0/0/0
    cache buffers chains 879 6,043 21 1 0/0/0/0/0
    cache buffers chains 110 6,010 25 1 0/0/0/0/0
    cache buffers chains 138 6,010 25 1 0/0/0/0/0
    cache buffers chains 472 6,002 31 1 0/0/0/0/0
    cache buffers chains 908 5,964 20 1 0/0/0/0/0
    cache buffers chains 860 5,950 23 1 0/0/0/0/0
    cache buffers chains 71 5,945 29 3 0/0/0/0/0
    cache buffers chains 20 5,780 28 1 0/0/0/0/0
    cache buffers chains 932 5,759 25 1 0/0/0/0/0
    cache buffers chains 866 5,610 22 1 0/0/0/0/0
    cache buffers chains 989 5,454 34 2 0/0/0/0/0
    cache buffers chains 1005 5,434 40 1 0/0/0/0/0
    library cache 6 47,067 52 3 49/3/0/0/0
    shared pool 1 99,771 124 3 121/3/0/0/0
    Top 5 Logical Reads per Segment for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Segment Logical Reads Threshold: 10000
    Subobject Obj. Logical
    Owner Tablespace Object Name Name Type Reads %Total
    AHD AHD1_DATA CALL_REQ TABLE 1,714,928 51.19
    AHD AHD1_DATA CTCT TABLE 1,169,360 34.90
    AHD AHD1_IDX SYS_C003707 INDEX 89,152 2.66
    AHD AHD1_DATA CNOTE TABLE 66,272 1.98
    AHD AHD1_IDX CALL_REQ_X5 INDEX 61,360 1.83
    Top 5 Physical Reads per Segment for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Segment Physical Reads Threshold: 1000
    Subobject Obj. Physical
    Owner Tablespace Object Name Name Type Reads %Total
    AHD AHD1_DATA CALL_REQ TABLE 132,989 95.95
    AHD AHD1_DATA CTCT TABLE 5,325 3.84
    AHD AHD1_DATA CI_AUDIT_TRAILS_GU_I INDEX 43 .03
    AHD AHD1_DATA ACT_LOG TABLE 38 .03
    AHD AHD1_DATA CI_EXT_CALLS_GUID INDEX 36 .03
    Top 5 Buf. Busy Waits per Segment for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Segment Buffer Busy Waits Threshold: 100
    Buffer
    Subobject Obj. Busy
    Owner Tablespace Object Name Name Type Waits %Total
    AHD AHD1_DATA CALL_REQ TABLE 11,751 99.95
    AHD AHD1_DATA CTCT TABLE 6 .05
    Dictionary Cache Stats for DB: AHD Instance: ahd Snaps: 20 -21
    ->"Pct Misses" should be very low (< 2% in most cases)
    ->"Cache Usage" is the number of cache entries being used
    ->"Pct SGA" is the ratio of usage to allocated size for that cache
    Get Pct Scan Pct Mod Final
    Cache Requests Miss Reqs Miss Reqs Usage
    dc_files 30 0.0 0 0 15
    dc_histogram_defs 3,022 3.9 0 0 1,919
    dc_object_ids 22,961 0.1 0 0 1,181
    dc_objects 1,092 9.2 0 0 1,026
    dc_profiles 11 0.0 0 0 1
    dc_rollback_segments 168 0.0 0 0 22
    dc_segments 5,519 0.1 0 0 1,334
    dc_sequences 1 0.0 0 1 2
    dc_tablespace_quotas 3 0.0 0 3 2
    dc_tablespaces 25,902 0.0 0 0 16
    dc_user_grants 127 0.0 0 0 22
    dc_usernames 110 0.0 0 0 18
    dc_users 26,077 0.0 0 0 30
    Library Cache Activity for DB: AHD Instance: ahd Snaps: 20 -21
    ->"Pct Misses" should be very low
    Get Pct Pin Pct Invali-
    Namespace Requests Miss Requests Miss Reloads dations
    CLUSTER 19 0.0 16 0.0 0 0
    INDEX 315 0.0 315 0.0 0 0
    SQL AREA 27,908 0.0 94,300 0.5 38 0
    TABLE/PROCEDURE 3,793 2.6 6,017 6.5 55 0
    TRIGGER 20 0.0 20 0.0 0 0
    Shared Pool Advisory for DB: AHD Instance: ahd End Snap: 21
    -> Note there is often a 1:Many correlation between a single logical object
    in the Library Cache, and the physical number of memory objects associated
    with it. Therefore comparing the number of Lib Cache objects (e.g. in
    v$librarycache), with the number of Lib Cache Memory Objects is invalid
    Estd
    Shared Pool SP Estd Estd Estd Lib LC Time
    Size for Size Lib Cache Lib Cache Cache Time Saved Estd Lib Cache
    Estim (M) Factr Size (M) Mem Obj Saved (s) Factr Mem Obj Hits
    88 .6 81 11,169 59,229 1.0 6,202,663
    104 .7 96 13,308 59,237 1.0 6,207,373
    120 .8 112 15,603 59,277 1.0 6,228,405
    136 .9 127 18,086 59,348 1.0 6,265,370
    152 1.0 142 19,501 59,379 1.0 6,295,279
    168 1.1 157 21,035 59,426 1.0 6,314,861
    184 1.2 172 22,038 59,455 1.0 6,325,903
    200 1.3 187 23,807 59,459 1.0 6,328,446
    216 1.4 202 25,911 59,460 1.0 6,329,386
    232 1.5 217 28,194 59,461 1.0 6,330,245
    248 1.6 232 29,884 59,462 1.0 6,330,914
    264 1.7 248 31,127 59,462 1.0 6,331,222
    280 1.8 263 32,878 59,463 1.0 6,331,563
    296 1.9 278 34,121 59,463 1.0 6,331,898
    312 2.1 295 36,139 59,463 1.0 6,332,102
    SGA Memory Summary for DB: AHD Instance: ahd Snaps: 20 -21
    SGA regions Size in Bytes
    Database Buffers 687,865,856
    Fixed Size 455,196
    Redo Buffers 929,792
    Variable Size 293,601,280
    sum 982,852,124
    SGA breakdown difference for DB: AHD Instance: ahd Snaps: 20 -21
    Pool Name Begin value End value % Diff
    java free memory 75,497,472 75,497,472 0.00
    large free memory 41,943,040 41,943,040 0.00
    shared 1M buffer 2,098,176 2,098,176 0.00
    shared Checkpoint queue 846,912 846,912 0.00
    shared FileOpenBlock 695,504 695,504 0.00
    shared KGK heap 3,756 3,756 0.00
    shared KGLS heap 1,230,944 1,438,740 16.88
    shared KQR L PO 2,064 2,064 0.00
    shared KQR M PO 2,480,924 2,514,220 1.34
    shared KQR S PO 383,036 383,036 0.00
    shared KQR S SO 5,636 5,636 0.00
    shared KSXR pending messages que 841,036 841,036 0.00
    shared KSXR receive buffers 1,033,000 1,033,000 0.00
    shared MTTR advisory 97,412 97,412 0.00
    shared PL/SQL DIANA 624,112 624,112 0.00
    shared PL/SQL MPCODE 422,640 422,640 0.00
    shared PLS non-lib hp 2,068 2,068 0.00
    shared character set object 323,724 323,724 0.00
    shared dictionary cache 1,610,880 1,610,880 0.00
    shared errors 35,964 35,964 0.00
    shared event statistics per sess 1,718,360 1,718,360 0.00
    shared fixed allocation callback 300 300 0.00
    shared free memory 26,982,004 26,841,956 -0.52
    shared joxs heap init 4,220 4,220 0.00
    shared kgl simulator 3,980,240 3,996,976 0.42
    shared library cache 54,425,164 53,999,624 -0.78
    shared message pool freequeue 834,752 834,752 0.00
    shared miscellaneous 8,126,704 8,177,516 0.63
    shared parameters 1,632 1,632 0.00
    shared sessions 410,720 410,720 0.00
    shared sim memory hea 377,656 377,656 0.00
    shared sql area 66,513,080 66,768,476 0.38
    shared subheap 45,216 45,216 0.00
    shared table definiti 1,200 2,752 129.33
    shared trigger defini 340 340 0.00
    shared trigger inform 1,292 1,292 0.00
    shared trigger source 100 100 0.00
    buffer_cache 687,865,856 687,865,856 0.00
    fixed_sga 455,196 455,196 0.00
    log_buffer 918,528 918,528 0.00
    init.ora Parameters for DB: AHD Instance: ahd Snaps: 20 -21
    End value
    Parameter Name Begin value (if different)
    aq_tm_processes 1
    background_dump_dest E:\oracle\admin\ahd\bdump
    compatible 9.2.0.0.0
    control_files E:\oracle\oradata\ahd\CONTROL01.C
    core_dump_dest E:\oracle\admin\ahd\cdump
    cursor_sharing SIMILAR
    db_block_size 8192
    db_cache_size 687865856
    db_domain
    db_file_multiblock_read_count 8
    db_name ahd
    db_writer_processes 2
    dispatchers (PROTOCOL=TCP) (SERVICE=ahdXDB)
    fast_start_mttr_target 300
    hash_join_enabled TRUE
    instance_name ahd
    java_pool_size 75497472
    job_queue_processes 10
    large_pool_size 41943040
    log_archive_dest_1 location=c:\archive
    log_archive_format arc%d_%t_%s.arc
    log_archive_start TRUE
    open_cursors 300
    pga_aggregate_target 367001600
    processes 150
    query_rewrite_enabled FALSE
    remote_login_passwordfile EXCLUSIVE
    shared_pool_size 159383552
    sort_area_size 10485760
    star_transformation_enabled FALSE
    timed_statistics TRUE
    undo_management AUTO
    undo_retention 3600
    undo_tablespace UNDOTBS1
    user_dump_dest E:\oracle\admin\ahd\udump
    End of Report

    I am getting buffer busy waits on some tables.
    Users are not experiencing any problem.Looks like you got bit by the CTD troll while sleeping.
    Note also that (if I'm reading the report alright) out of 23 mins you have 6 seconds accounted to buffer busy waits.
    Read the sample chapter here.

  • Buffer busy wait, 1st level bmp

    Hi All !
    OS: Linux redhat 5
    DB: 11gr2
    Block size: 8K
    In an application we use I can see high buffer busy waits over a various periods.
    I collect some info during this event.
    SQL_HASH_VALUE FILE# BLOCK# REASON
    769132182 6 17512 8
    3983195767 6 17512 8
    769132182 6 17512 8
    3240261994 6 17512 8
    3240261994 6 17512 8
    3240261994 6 17512 8
    769132182 6 17512 8
    ... I have total 35 sessions
    File6 / block 17512 =
    TABLESPACE_NAME SEGMENT_TYPE OWNER SEGMENT_NAME
    GBSLOB LOBSEGMENT GBSASP SYS_LOB0000017961C00006$$
    The sql are both inserts and updates to the same large table, blobs are involved (insert/update)
    blobs using securefile
    AWR reports this for a short period
    Buffer busy waits is the top wait event
    Buffer Wait Statistics DB/Inst: GGBISP01/ggbisp01 Snaps: 20925-20926
    -> ordered by wait time desc, waits desc
    Class Waits Total Wait Time (s) Avg Time (ms)
    1st level bmb 574,636 17,118 30
    free list 20,538 70 3
    undo header 41,150 7 0
    data block 263 1 3
    undo block 18 0 0
    I'm trying to find more details about this wait event, I believe it is related to usage of ASSM.
    Can anyone explain more when 1st level bmp is seen ?
    Thank you !
    Best regards
    Magnus Johansson

    MaJo wrote:
    SQL_HASH_VALUE      FILE#     BLOCK#     REASON
    769132182          6      17512          8
    3983195767          6      17512          8
    769132182          6      17512          8
    3240261994          6      17512          8
    3240261994          6      17512          8
    3240261994          6      17512          8
    769132182          6      17512          8
    ... I have total 35 sessions
    File6 / block 17512 =
    TABLESPACE_NAME                SEGMENT_TYPE       OWNER                          SEGMENT_NAME
    GBSLOB                         LOBSEGMENT         GBSASP                         SYS_LOB0000017961C00006$$The sql are both inserts and updates to the same large table, blobs are involved (insert/update)
    blobs using securefile
    AWR reports this for a short period
    Buffer busy waits is the top wait event
    Buffer Wait Statistics          DB/Inst: GGBISP01/ggbisp01  Snaps: 20925-20926
    -> ordered by wait time desc, waits desc
    Class                    Waits Total Wait Time (s)  Avg Time (ms)
    1st level bmb          574,636              17,118             30
    free list               20,538                  70              3
    undo header             41,150                   7              0
    data block                 263                   1              3
    undo block                  18                   0              0
    -------------------------------------------------------------I'm trying to find more details about this wait event, I believe it is related to usage of ASSM.
    Can anyone explain more when 1st level bmp is seen ?
    Your AWR shows an interesting mix of ASSM and freelist group blocks - are you running ASSM ?
    1st level bitmap blocks (bmb) are the blocks in a segment (usually the first one or two of each extent) that show the availability of free space in the other data blocks. Each bitmap block can identify up to 256 other blocks (the last time I checked), although you have to have a fairly large data segment before you reach this level of mapping.
    If you have a high rate of concurrent inserts and updates on a LOB column then you may be running into code that frequently updates bitmap blocks to show that data blocks have changed from empty to full. It's also possible that you've run into one of the many bugs that appeared when you mixed ASSM with LOB segments - you haven't given the exact version of 11.2, but you might want to check the latest versions and any bug reports for patches to your version.
    Regards
    Jonathan Lewis

  • High waits on scatter read and buffer busy wait ..

    one of my system is undergoing some serious problem ~
    i checked when some sql querying few large tables ( 30 millions rows ,partioned )
    system waits became significant .. the ones stand out are scatter read and buffer busy wait ...
    on the longop , you can see those sessions waitting to scan the large table ..
    buffer get ratio dropped from 90% to like 30% ,average buffer hit ratio is like 55%
    mu questions are
    1.what's your view of throwing the problem away ?
    add physical memory, then increase data buffer ?
    it's getting scanned very often , but it's too large to pin it in the keep pool .
    2. why buffer busy wait ?
    i don't fully understand why this happened ?
    becase what they are doing are just querying ,nobody is modifing any data ?
    will increase freelist help this ??
    thanks your talneted ideas
    br/ricky

    Have these SQL statements been added recently, or have they been around for a long time. If the latter, then it seems likely that they've changed their execution plan to start doing tablescans (hence db file scattered read waits) or index fast full scans.
    One of the causes of buffer busy waits is concurrent tablescans/index fast full scans - in 10g this particular cause has been split and reported as "read by other session".
    Your first action should be to see if there is a more efficient execution plan for these queries. If the problem is due to a change in execution plan, common causes are: changes in statistics on the objects, failure to change statistics on the objects, changes in input value on the queries, changes in optimizer parameters, unlucky changes in data volumes.
    Regards
    Jonathan Lewis
    http://jonathanlewis.wordpress.com
    http://www.jlcomp.demon.co.uk

  • What is the difference between buffer busy waits and free buffer waits

    what is the difference between buffer busy waits events and free buffer waits in Oracle database?
    select *
    from
       v$system_event
    where
       event like ‘%wait%’;
    EVENT                       TOTAL_WAITS TOTAL_TIMEOUTS TIME_WAITED AVERAGE_WAIT
    buffer busy waits                636528           1557      549700   .863591232
    write complete waits               1193              0       14799   12.4048617
    free buffer waits                  1601              0         622   .388507183

    jetq wrote:
    Buffer busy waits occur when an Oracle session needs to access a block in the buffer cache, but cannot because the buffer copy of the data block is locked. This buffer busy wait condition can happen for either of the following reasons:
    * The block is being read into the buffer by another session, so the waiting session must wait for the block read to complete. If the OP is running 10g, that would be recorded as "read by other session" not "buffer busy waits" - and unfortunately he didn't tell us the version.
    * Another session has the buffer block locked in a mode that is incompatible with the waiting session's request.
    The Free Buffer Waits Oracle metric wait event indicates that a server process was unable to find a free buffer and has posted the database writer to make free buffers by writing out dirty buffers.
    There is another possibility - if the OP is using a keep and recycle pool: see http://jonathanlewis.wordpress.com/2006/11/21/free-buffer-waits/
    Regards
    Jonathan Lewis
    http://jonathanlewis.wordpress.com
    http://www.jlcomp.demon.co.uk
    To post code, statspack/AWR report, execution plans or trace files, start and end the section with the tag {noformat}{noformat} (lowercase, curly brackets, no spaces) so that the text appears in fixed format.
    "For every expert there is an equal and opposite expert"
    Arthur C. Clarke                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           

  • Problem identifying db object for "buffer busy waits" event.

    10.2.0.3
    AIX 64
    SELECT username, a.p1text, a.p1, a.p2text, a.p2, a.p3text, a.p3, event FROM v$session a WHERE
    a.status='ACTIVE'
    AND a.event = 'buffer busy waits'
    Query reports about 40 active sessions with this information:
    file# 3746
    block# 2
    class# 13
    select
    owner,
    segment_name,
    segment_type
    from
    dba_extents
    where
    file_id = 3746
    and
    2 between block_id and block_id + blocks -1;
    no rows returned
    SELECT MAX(a.file#) FROM sys.file$ a
    3535
    This was only a temporary situation when after couple of minutes(7) wait event "buffer busy waits" dissapeared completely.
    Any ideas?
    Thank you,
    Daniel.

    http://perfvision.com/papers/06_buffer_cache.ppt
    Slide 80-81 points at increasing the size of the initial and next extent for File Header Block buffer busy waits
    Side 85 points at high extent allocation for File Header Block buffer busy waits
    http://perfvision.com/ftp/hotsos/aas.ppt
    Side 55 points at extent allocation too small/too many extents being allocated for File Header Block buffer busy waits
    A couple hints from the documentation:
    http://download.oracle.com/docs/cd/B19306_01/server.102/b14211/instance_tune.htm
    "To determine the possible causes [of buffer busy waits], first query V$SESSION to identify the value of ROW_WAIT_OBJ# when the session waits for buffer busy waits."
    "To identify the object and object type contended for, query DBA_OBJECTS using the value for ROW_WAIT_OBJ# that is returned from V$SESSION."
    "V$SEGMENT_STATISTICS - This is a user-friendly view of statistic values. In addition to all the columns of V$SEGSTAT, it has information about such things as the segment owner and table space name. It makes the statistics easy to understand, but it is more costly."
    You may want to query DBA_TEMP_FILES for the specific FILE_ID identified by the V$SESSION. Taking a look at V$SEGMENT_STATISTICS might also be helpful.
    Are you using dictionary managed tablespaces, locally managed tablespaces with manual extent size management, ASSM with manual extent size management, or ASSM with automatic extent size management?
    Charles Hooper
    IT Manager/Oracle DBA
    K&M Machine-Fabricating, Inc.

  • Difference between wait events 'buffer busy waits' & 'latch: CBC'

    Hi ,
    Can some one explain me the Difference Between 'Buffer Busy Waits' and 'Latch: Cache Buffers Chains"? What was explained in metalink note Doc ID: 833303.1
    , is incomprehensible to me.
    Looking for easy and better explanation.
    Thanks
    Naveen

    Naveen Sangam wrote:
    Can some one explain me the Difference Between 'Buffer Busy Waits' and 'Latch: Cache Buffers Chains"? What was explained in metalink note Doc ID: 833303.1
    , is incomprehensible to me.
    That note is awful and should be witdrawn - amongst other things it makes it sound as if there is only one reason for buffer busy waits. There is an option to comment on such notes - I suggest you use it.
    Going back a little further from Aman's description - and ignoring some of the internal details:
    The buffer cache is split into a large number of small groups of buffered blocks; these groups are usually called "hash buckets" or "hash chains". If you need to find a block and it is buffered, Oracle can do some hashing arithmetic to work out which chain the block will be in.
    The chains are protected by latches (typically a single latch protects 64 - 128 chains). You cannot examine a chain unless you are able to get the latch that protects it. So the commonest action you go through to view or modify a buffered block is:
    <ul>
    work out which chain
    get the latch for the chain xxx
    search along the chain to see if the block is in memory
    attach a "pin" to the buffered block block if you find it (to make sure it can't be kicked out of memory)
    release the latch
    do what you want to do with the buffered block ***
    get the latch for the chain xxx
    remove the pin from the buffered block
    release the latch
    </ul>
    The two points marked xxx are where you might end up sleeping in a wait for the "cache buffers chains latch". If too many people try to get the latch at the same time some of them will eventually fail and go to sleep for a short period.
    The point marked *** is where you get the buffer busy wait. When you attach the pin to the buffered block it's possible that what you want to do to the bllock is incompatible with what other people are already doing - so you have to attach your pin to the "waiters" list (the people currently using the block will have their pins attached to the "users" list". Eventually your wait will time out, or the users will get out of the way and you can move your pin to the users list and do whatever it was you wanted to do.
    Regards
    Jonathan Lewis
    http://jonathanlewis.wordpress.com
    http://www.jlcomp.demon.co.uk
    To post code, statspack/AWR report, execution plans or trace files, start and end the section with the tag {noformat}{noformat} (lowercase, curly brackets, no spaces) so that the text appears in fixed format.
    "Science is more than a body of knowledge; it is a way of thinking"
    Carl Sagan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

  • Id column in gc buffer busy waits

    Hello,
    What's the id column show for gc buffer busy or gc current request waits.. I could not find any description for "gc current request". However, Here is what I found in oracle doc .. It's not so clear..
    Parameter Description
    file# See "file#" on page C-3
    block# See "block#" on page C-2
    id The buffer busy wait event is called from different places in the sessionThanks

    Any one??

  • Statspack interpreting help - buffer busy waits

    Hi,
    I've got statspack report from 9.2.0.8 DB, cpu_count = 12 , there is 'buffer busy waits' in top 5 .
    Is there a problem ?
    DB Name         DB Id    Instance     Inst Num Release     Cluster Host
    XXXX         138180125   XXXX            1 9.2.0.8.0   NO      X1
                  Snap Id     Snap Time      Sessions Curs/Sess Comment
    Begin Snap:     35980 14-Jul-10 01:00:02       17       8.8
      End Snap:     35984 14-Jul-10 05:00:01       17       8.8
       Elapsed:              239.98 (mins)
    Cache Sizes (end)
    ~~~~~~~~~~~~~~~~~
                   Buffer Cache:     3,072M      Std Block Size:          8K
               Shared Pool Size:       512M          Log Buffer:      4,096K
    Load Profile
    ~~~~~~~~~~~~                            Per Second       Per Transaction
                      Redo size:            317,746.23              4,498.59
                  Logical reads:             11,150.77                157.87
                  Block changes:              2,134.89                 30.23
                 Physical reads:                466.05                  6.60
                Physical writes:                133.62                  1.89
                     User calls:                 82.42                  1.17
                         Parses:                 67.92                  0.96
                    Hard parses:                  0.02                  0.00
                          Sorts:                106.77                  1.51
                         Logons:                  0.03                  0.00
                       Executes:                516.58                  7.31
                   Transactions:                 70.63
      % Blocks changed per Read:   19.15    Recursive Call %:     95.00
    Rollback per transaction %:    0.00       Rows per Sort:      4.34
    Instance Efficiency Percentages (Target 100%)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Buffer Nowait %:   99.03       Redo NoWait %:    100.00
                Buffer  Hit   %:   95.91    In-memory Sort %:    100.00
                Library Hit   %:  100.00        Soft Parse %:     99.98
             Execute to Parse %:   86.85         Latch Hit %:     99.65
    Parse CPU to Parse Elapsd %:    7.82     % Non-Parse CPU:     99.91
    Shared Pool Statistics        Begin   End
                 Memory Usage %:   43.53   43.92
        % SQL with executions>1:   64.89   70.00
      % Memory for SQL w/exec>1:   55.95   61.64
    Top 5 Timed Events
    ~~~~~~~~~~~~~~~~~~                                                     % Total
    Event                                               Waits    Time (s) Ela Time
    SQL*Net message from dblink                       797,760      44,575    41.69
    PL/SQL lock timer                                   1,207      34,992    32.73
    db file sequential read                         3,297,249      17,047    15.94
    buffer busy waits                               1,558,995       3,987     3.73
    CPU time                                                        3,204     3.00
    Statistic                                      Total     per Second    per Trans
    enqueue timeouts                                 299            0.0          0.0
    enqueue waits                                    425            0.0          0.0
    exchange deadlocks                                41            0.0          0.0
    execute count                              7,438,297          516.6          7.3
    failed probes on index block recl                 13            0.0          0.0
    free buffer inspected                        107,385            7.5          0.1
    free buffer requested                      7,344,870          510.1          7.2
    hot buffers moved to head of LRU           2,332,802          162.0          2.3
    immediate (CR) block cleanout app            356,492           24.8          0.4
    immediate (CURRENT) block cleanou          1,751,731          121.7          1.7
    index crx upgrade (found)                          7            0.0          0.0
    index crx upgrade (positioned)                25,604            1.8          0.0
    index fast full scans (full)                     888            0.1          0.0
    index fetch by key                         6,008,269          417.3          5.9
    index scans kdiixs1                        2,343,163          162.7          2.3
    leaf node 90-10 splits                           330            0.0          0.0
    leaf node splits                              13,681            1.0          0.0
    logons cumulative                                447            0.0          0.0
    messages received                          2,760,503          191.7          2.7
    messages sent                              2,760,503          191.7          2.7
    no buffer to keep pinned count                     0            0.0          0.0
    no work - consistent read gets            89,143,249        6,190.9         87.7
    opened cursors cumulative                    978,462           68.0          1.0
    parse count (failures)                             0            0.0          0.0
    parse count (hard)                               243            0.0          0.0
    parse count (total)                          977,939           67.9          1.0
    parse time cpu                                   276            0.0          0.0
    parse time elapsed                             3,531            0.3          0.0
    physical reads                             6,710,684          466.1          6.6
    physical reads direct                        140,520            9.8          0.1
    physical writes                            1,924,011          133.6          1.9
    physical writes direct                       149,434           10.4          0.2
    physical writes non checkpoint             1,160,293           80.6          1.1
    pinned buffers inspected                      88,165            6.1          0.1
    prefetched blocks                          2,965,135          205.9          2.9
    prefetched blocks aged out before              1,485            0.1          0.0
    process last non-idle time                    14,401            1.0          0.0
    recovery blocks read                               0            0.0          0.0
    recursive calls                           22,566,381        1,567.2         22.2
    recursive cpu usage                          314,662           21.9          0.3
    redo blocks written                        9,712,190          674.5          9.6
    redo buffer allocation retries                   483            0.0          0.0
    redo entries                              17,147,344        1,190.9         16.9
    redo log space requests                          760            0.1          0.0
    redo log space wait time                       1,255            0.1          0.0
    redo ordering marks                               21            0.0          0.0
    redo size                              4,575,228,028      317,746.2      4,498.6
    redo synch time                               73,190            5.1          0.1
    redo synch writes                            333,440           23.2          0.3
    redo wastage                             240,517,096       16,703.7        236.5
    redo write time                              136,628            9.5          0.1
    redo writer latching time                         56            0.0          0.0
    redo writes                                  865,653           60.1          0.9
    rollback changes - undo records a             60,510            4.2          0.1
    rows fetched via callback                  3,948,006          274.2          3.9
    session connect time                               0            0.0          0.0
    Statistic                                      Total     per Second    per Trans
    session logical reads                    160,559,938       11,150.8        157.9
    session pga memory                       223,020,424       15,488.6        219.3
    session pga memory max                   841,058,240       58,410.9        827.0
    session uga memory                   682,912,005,944   47,427,738.5    671,472.1
    session uga memory max                   505,627,192       35,115.4        497.2
    shared hash latch upgrades - no w          1,661,152          115.4          1.6
    shared hash latch upgrades - wait                101            0.0          0.0
    sorts (disk)                                       2            0.0          0.0
    sorts (memory)                             1,537,403          106.8          1.5
    sorts (rows)                               6,669,072          463.2          6.6
    summed dirty queue length                     71,613            5.0          0.1
    switch current to new buffer                  80,971            5.6          0.1
    table fetch by rowid                      79,047,167        5,489.8         77.7
    table fetch continued row                  5,013,545          348.2          4.9
    table scan blocks gotten                  10,328,271          717.3         10.2
    table scan rows gotten                   381,848,913       26,519.1        375.5
    table scans (long tables)                         82            0.0          0.0
    table scans (short tables)                 1,117,114           77.6          1.1
    transaction rollbacks                         32,437            2.3          0.0
    transaction tables consistent rea                 39            0.0          0.0
    transaction tables consistent rea             82,904            5.8          0.1
    user calls                                 1,186,828           82.4          1.2
    user commits                               1,017,037           70.6          1.0
    user rollbacks                                     0            0.0          0.0
    workarea executions - onepass                      7            0.0          0.0
    workarea executions - optimal              2,291,005          159.1          2.3
    write clones created in backgroun                  3            0.0          0.0
    write clones created in foregroun                711            0.1          0.0
    Class                    Waits   Time (s) Time (ms)
    data block           1,549,301      4,015         3
    segment header             253          1         2
    undo block               2,574          0         0
    undo header              2,209          0         0
    extent map                   2          0         5
    Eq     Requests    Succ Gets Failed Gets       Waits   Time (ms)     Time (s)
    TX    1,749,961    1,749,961           0         202          6.47            1
    HW       20,789       20,789           0         223           .40            0
                                               Pct    Avg   Wait                 Pct
                                  Get          Get   Slps   Time       NoWait NoWait
    Latch                       Requests      Miss  /Miss    (s)     Requests   Miss
    Consistent RBA                  866,126    0.0    0.0      0            0
    FIB s.o chain latch                 594    0.0             0            0
    FOB s.o list latch                2,891    0.0    0.0      0            0
    SQL memory manager latch              4    0.0             0        4,793    0.0
    SQL memory manager worka      3,266,221    0.0    0.0      0            0
    active checkpoint queue       1,905,423    0.5    0.0      0            0
    archive control                      28    0.0             0            0
    cache buffer handles             12,070    0.0             0            0
    cache buffers chains        387,508,854    0.6    0.0    134   11,212,815    0.1
    cache buffers lru chain       1,946,036    0.0    0.0      0   14,681,284    0.2
    channel handle pool latc            668    0.0             0            0
    channel operations paren         12,460    0.0    0.0      0            0
    checkpoint queue latch      209,244,852    0.0    0.0      0    1,775,314    0.0
    child cursor hash table           3,240    0.7    0.0      0            0
    commit callback allocati             16    0.0             0            0
    dictionary lookup                    13    0.0             0            0
    dml lock allocation           5,900,238    0.3    0.0      0            0
    dummy allocation                    894    0.2    0.0      0            0
    enqueue hash chains          19,444,854    0.1    0.0      0            0
    enqueues                      9,380,299    0.7    0.0      0            0
    event group latch                   288    0.0             0            0
    event range base latch                3    0.0             0            0
    global tx hash mapping        4,988,645    0.0    0.0      0            0
    hash table column usage              82    0.0             0        1,783    0.0
    job workq parent latch                1  100.0    0.0      0          242   15.3
    job_queue_processes para            287    0.0             0            0
    ktm global data                     265    0.0             0            0
    lgwr LWN SCN                    868,053    0.1    0.0      0            0
    library cache                29,540,874    0.1    0.0      0           37    0.0
    library cache load lock             102    0.0             0            0
    library cache pin            24,145,848    0.0    0.0      0            0
    library cache pin alloca      3,988,997    0.0    0.0      0            0
    list of block allocation        363,684    0.0    0.0      0            0
    loader state object free            912    0.0             0            0
    longop free list parent             479    0.0             0           60    1.7
    message pool operations             100    1.0    0.0      0            0
    messages                      8,036,523    0.3    0.0      0            0
    mostly latch-free SCN           878,016    1.0    0.0      0            0
    multiblock read objects         922,048    0.1    0.0      0            0
    ncodef allocation latch             230    0.0             0            0
    object stats modificatio          2,100    0.0             0            0
    post/wait queue                 709,603    0.0    0.0      0      334,338    0.0
    process allocation                  576    0.0             0          288    0.0
    process group creation              576    0.2    0.0      0            0
    redo allocation              18,881,467    0.7    0.0      0            0
    redo copy                             0                    0   17,155,579    0.1
    redo writing                  4,513,716    0.2    0.0      0            0
    resumable state object               48    0.0             0            0
    row cache enqueue latch       3,556,148    0.1    0.0      0            0
    row cache objects             6,671,783    0.1    0.0      0            0
                                               Pct    Avg   Wait                 Pct
                                  Get          Get   Slps   Time       NoWait NoWait
    Latch                       Requests      Miss  /Miss    (s)     Requests   Miss
    sequence cache                1,533,482    0.0    0.0      0            0
    session allocation           15,194,281    0.1    0.0      0            0
    session idle bit              3,005,477    0.0    0.0      0            0
    session switching                   230    0.0             0            0
    session timer                     4,825    0.0             0            0
    shared pool                   2,114,153    0.0    0.0      0            0
    sim partition latch                   0                    0       10,243    0.6
    simulator hash latch          8,460,492    0.0    0.0      0            0
    simulator lru latch             223,868    0.0    0.3      0      470,589    0.1
    sort extent pool                  1,823    0.5    0.0      0            0
    temporary table state ob             16    0.0             0            0
    transaction allocation          533,964    0.0             0            0
    transaction branch alloc      1,259,723    0.1    0.0      0            0
    undo global data             17,460,173    0.0    0.0      0       14,976    0.0
    user lock                           906    0.1    0.0      0            0
                                          Get                            Spin &
    Latch Name                       Requests      Misses      Sleeps Sleeps 1->4
    cache buffers chains          387,508,854   2,447,832      12,701 2435135/1269
                                                                      3/4/0/0
    redo allocation                18,881,467     131,460         343 131118/341/1
                                                                      /0/0
    enqueues                        9,380,299      62,436         122 62314/122/0/
                                                                      0/0
    library cache                  29,540,874      38,344          80 38264/80/0/0
                                                                      /0
    messages                        8,036,523      25,266          28 25238/28/0/0
                                                                      /0
    dml lock allocation             5,900,238      19,220          25 19195/25/0/0
                                                                      /0
    enqueue hash chains            19,444,854      10,510          27 10483/27/0/0
                                                                      /0
    active checkpoint queue la      1,905,423       9,896          36 9860/36/0/0/
                                                                      0
    library cache pin              24,145,848       8,451          17 8434/17/0/0/
                                                                      0
    mostly latch-free SCN             878,016       8,423          17 8406/17/0/0/
                                                                      0
    session allocation             15,194,281       8,290          33 8257/33/0/0/
                                                                      0
    redo writing                    4,513,716       7,235          13 7222/13/0/0/
                                                                      0
    undo global data               17,460,173       4,113           8 4105/8/0/0/0
    row cache objects               6,671,783       3,680           4 3676/4/0/0/0
    row cache enqueue latch         3,556,148       2,015           1 2014/1/0/0/0
    checkpoint queue latch        209,244,852       1,756           6 1750/6/0/0/0
    transaction branch allocat      1,259,723       1,236           7 1229/7/0/0/0
    shared pool                     2,114,153         808           2 806/2/0/0/0
    library cache pin allocati      3,988,997         649           1 648/1/0/0/0
    cache buffers lru chain         1,946,036         588          13 575/13/0/0/0
    multiblock read objects           922,048         469           8 461/8/0/0/0
    sequence cache                  1,533,482         333           1 332/1/0/0/0
    session idle bit                3,005,477          97           1 96/1/0/0/0
    Consistent RBA                    866,126          55           1 54/1/0/0/0
    simulator lru latch               223,868          33           9 24/9/0/0/0
    post/wait queue                   709,603          27           1 26/1/0/0/0
                                                         NoWait              Waiter
    Latch Name               Where                       Misses     Sleeps   Sleeps
    active checkpoint queue  kcbbacq: scan active check       0         36       36
    cache buffers chains     kcbgtcr: kslbegin excl           0     10,706   10,084
    cache buffers chains     kcbrls: kslbegin                 0        937    1,577
    cache buffers chains     kcbzwb                           0        371      388
    cache buffers chains     kcbgtcr: fast path               0        223      174
    cache buffers chains     kcbgcur: kslbegin                0         86      114
    cache buffers chains     kcbget: pin buffer               0         57       47
    cache buffers chains     kcbzib: finish free bufs         0         54       27
    cache buffers chains     kcbchg: kslbegin: bufs not       0         52       89
    cache buffers chains     kcbnlc                           0         46       37
    cache buffers chains     kcbzgb: scan from tail. no       0         42        0
    cache buffers chains     kcbzib: multi-block read:        0         25        0
    cache buffers chains     kcbget: exchange rls             0         12        3
    cache buffers chains     kcbchg: kslbegin: call CR        0         12       80
    cache buffers chains     kcbget: exchange                 0         10        8
    cache buffers chains     kcbnew                           0          9        0
    cache buffers chains     kcbcge                           0          1        0
    cache buffers chains     kcbgtcr                          0          1        0
    cache buffers chains     kcbbxsv                          0          1       16
    cache buffers chains     kcbkzs                           0          1        3
    cache buffers chains     kcbbic2                          0          1        2
    cache buffers chains     kcbbic1                          0          1        5
    cache buffers lru chain  kcbzgb: multiple sets nowa  10,344         10        0
    cache buffers lru chain  kcbbiop: lru scan              112          3        0
    checkpoint queue latch   kcbklbc: Link buffer into        0          6        0
    dml lock allocation      ktaiam                           0         15       16
    dml lock allocation      ktaidm                           0         10        9
    enqueue hash chains      ksqgtl3                          0         17       11
    enqueue hash chains      ksqrcl                           0          9       15
    enqueue hash chains      ksqcnl                           0          1        1
    enqueues                 ksqdel                           0         60       50
    enqueues                 ksqgel: create enqueue           0         60       56
    enqueues                 ksqies                           0          2       16
    lgwr LWN SCN             kcs023                           0          9        0
    library cache            kglpnc: child                    0         29       29
    library cache            kglupc: child                    0         25       22
    library cache            kgllkdl: child: cleanup          0         11        2
    library cache            kglpndl: child: before pro       0          5        4
    library cache            kglhdgn: child:                  0          3       13
    library cache            kglhdgc: child:                  0          2        1
    library cache            kglpndl: child: after proc       0          2        0
    library cache            kgldte: child 0                  0          1        2
    library cache            kglpin: child: heap proces       0          1        0
    library cache            kglobpn: child:                  0          1        5
    library cache pin        kglpndl                          0          6        1
    library cache pin        kglpnc: child                    0          6        8
    library cache pin        kglupc                           0          5        4
    library cache pin alloca kglpnal                          0          1        1
    messages                 ksarcv                           0         12        5
    messages                 ksarcv: after wait               0          8       19
    messages                 ksaamb: after wakeup             0          8        4
    mostly latch-free SCN    kcslcu3                          0          8       17
    mostly latch-free SCN    kcsnew_scn_rba                   0          1        0
    Latch Name               Where                       Misses     Sleeps   Sleeps
    multiblock read objects  kcbzib: mbr get                  0          4        4
    multiblock read objects  kcbzib: normal mbr free          0          4        4
    post/wait queue          ksliwat:add:nowait               0          1        0
    redo allocation          kcrfwr                           0        322      248
    redo allocation          kcrfwi: more space               0         14       88
    redo allocation          kcrfwi: before write             0          7        7
    redo writing             kcrfwcr                          0          9       12
    redo writing             kcrfwint: rba scn pair           0          2        0
    redo writing             kcrfwint: after write            0          2        6
    row cache enqueue latch  kqreqa                           0          1        1
    row cache objects        kqrpre: find obj                 0          3        1
    row cache objects        kqrpfl: not dirty                0          1        1
    sequence cache           kdnss                            0          1        1
    session allocation       ksuprc                           0         14        5
    session allocation       ksudlc                           0         10        7
    session allocation       ksucri                           0          6       14
    session allocation       ksuxds: not user session         0          3        7
    session idle bit         ksupuc: clear busy               0          1        0
    shared pool              kghalo                           0          1        0
    shared pool              kghupr1                          0          1        2
    simulator lru latch      kcbs_simulate: simulate se       0          8        9
    simulator lru latch      kcbs_lookup_setid                0          1        0
    transaction branch alloc ktcbba                           0          4        2
    transaction branch alloc ktcbod                           0          2        3
    transaction branch alloc ksupuc                           0          1        2
    undo global data         ktudba: KSLBEGIN                 0          8        7
    Top 5 Logical Reads per Segment for DB: XXXXX  Instance: XXXXX  Snaps: 35980
    -> End Segment Logical Reads Threshold:     10000
                                               Subobject  Obj.       Logical
    Owner      Tablespace Object Name          Name       Type         Reads  %Total
    xxxxxxxx    XXXXXDATA      TAB1TABLE                      TABLE   39,838,592   37.65
    Top 5 Physical Reads per Segment for DB: XXXXX  Instance: XXXXX  Snaps: 3598
    -> End Segment Physical Reads Threshold:      1000
                                               Subobject  Obj.      Physical
    Owner      Tablespace Object Name          Name       Type         Reads  %Total
    xxxxxxxx    XXXXXDATA          TAB1TABLE                        TABLE    3,568,038   58.64
    Top 5 Buf. Busy Waits per Segment for DB: XXXXX  Instance: XXXXX  Snaps: 359
    -> End Segment Buffer Busy Waits Threshold:       100
                                                                      Buffer
                                               Subobject  Obj.          Busy
    Owner      Tablespace Object Name          Name       Type         Waits  %Total
    xxxxxxxx    XXXXXDATA                                           TAB1TABLE    1,421,043   91.65
    xxxxxxxx    XXXXXDATA IDX_SOMEIDX               INDEX       62,638    4.04
    xxxxxxxx    XXXXXTABLE                                TABLE       26,914    1.74
    ----------So, for me It looks like :
    TAB1TABLE is buffer busy deliver but there are no inserts reported for that table,
    I've checked that TAB1TABLE tablespace is NO ASSM and extent management is local with uniform size 1M .
    So this is not obvious free list problem . Kind of strange for me .
    Any ideas greatly appreciated :).
    Regards.
    Greg

    user10388717 wrote:
    Hi,
    I've got statspack report from 9.2.0.8 DB, cpu_count = 12 , there is 'buffer busy waits' in top 5 .
    Is there a problem ?
    DB Name         DB Id    Instance     Inst Num Release     Cluster Host
    XXXX         138180125   XXXX            1 9.2.0.8.0   NO      X1
    Snap Id     Snap Time      Sessions Curs/Sess Comment
    Begin Snap:     35980 14-Jul-10 01:00:02       17       8.8
    End Snap:     35984 14-Jul-10 05:00:01       17       8.8
    Elapsed:              239.98 (mins)
    ----------So, for me It looks like :
    TAB1TABLE is buffer busy deliver but there are no inserts reported for that table,
    I've checked that TAB1TABLE tablespace is NO ASSM and extent management is local with uniform size 1M .
    So this is not obvious free list problem . Kind of strange for me .
    Any ideas greatly appreciated :).We can't tell if you have a problem - only you (or your users) can know.
    But you've shown us a statspack snapshot covering 4 hours and in that time you've reported about 30 hours of database time (sum foreground "in-database" waits and CPU), with a fairly small number of sessions which (allowing for background sessions) means most of your foreground sessions seem to be working pretty much non-stop for the entire period. I could take a guess and say that you would like some of the work that they're doing completed sooner.
    The largest fraction of your time goes into waiting for messages from db link, with an average time of 55ms - maybe you have a network problem, maybe you have a query that has a bad choice of execution path that is doing lots of unnecessary trips to the remote db, maybe the queries that get to the remote db could be made much more efficient. (Look for 'sql ordered by executions' in a statspack from the remote db for clues).
    pl/sql lock timer is the next big chunk of time - but this is deliberately coded waits in pl/sql (dbms_lock.sleep) maybe that's supposed to be happening, but you could check the logic to see if some "slow" processes are actually coded to sleep much longer than necessary.
    Your db file sequential reads (single block reads) are, on average taking 5.5 ms - which is reasonable, so you have to ask if the number (and we know which table a lot of them are hitting) is reasonable. This brings us to your buffer busy waits: these can be caused by updates and deletes as well as inserts, but in 9i they are also caused by "read by other session" - so the buffer busy wait may simple be one session waiting for another session to complete a db file sequential read.
    I'd look at your "SQL ordered by Reads" to see if you have some inefficient execution plans (or poorly defined indexes) that result in large amounts of the critical table being constantly re-read. It's possible that you can eliminate redundant visits to this table and reduce your I/O, BBW, and CPU in one shot.
    Regards
    Jonathan Lewis

Maybe you are looking for

  • How can i convert an AAC file to an mp3 file on my new 2015 IMAC? The option to convert to mp3 isn't available on iTunes controls anymore.

    How do i convert an AAC file into an Mp3 file on my New 2015 iMac? The option to "convert to mp3" isn't available in controls. Thank you.

  • K7t Turbo2..Do I have a bad one?

    Am new to the forums and MSI, but have been building and repairing 'puters for a few. Have a new K7T Turbo2 MS 6330 bios v 3.4 My question is: Is this a bad MBO, or is it in need of a bios update or something? THE PROBLEM: Incomplete POST, stalling w

  • Can I download Numbers 9 anywhere?

    This sounds crazy, even to me, but I recently had to reinstall Mavericks.  Afterwards although the iWork 09 folder is still in my Application it is empty.  I lost my previous versions of Numbers, Pages and Keynote.  Is there any place I can download

  • JSP Build Problem

    I am trying to build the following jsp page: <jsp:include page="/reports/RWServlet" flush="true" > <jsp:param name="desformat" value="<%= session.getId() %>" /> </jsp:include> It will not build with the following error: Error: variable reports$_jsp_S

  • Global exception handler not being called

    Hi, I have a wli process that has a global excepton handler. One of the controls throws a NullPointerException but it is not being handled by the group exception handler nor the global exception handler. Does anyone have any idea what may be causing