Redo Space wait

oracle : 9i
os : linux
log : archive
dg : primary
Production : Yes
The instance is up. the performace is poor due to redospace wait.
I checked the following.
1. select (sw.value)*100/lw.valuefrom v$sysstat sw, v$sysstat lwwhere sw.name='redo log space requests' and lw.name='redo writes';
24.9544131
2. Parameters
log_checkpoint_interval integer 0
log_checkpoint_timeout integer 1800
log_checkpoints_to_alert boolean TRUE
log_buffer integer 524288
3. SELECT name, value
FROM SYS.v_$sysstat
WHERE NAME in ('redo buffer allocation retries',
'redo log space wait time');
redo buffer allocation retries 5216940
redo log space wait time 40519121
4. Select name, value from v$sysstat
Where name in ('redo log space requests', 'redo entries'); 2
redo entries 785620472
redo log space requests 1295431
Other than restarting the server, Any action can be taken ?

1. Make sure your archiving destination isn't getting
full. If ARCH can't archive, LGWR can't switch, the
log buffer fills up and users then have to wait until
it unclogs.
space is enough . File is getting generated and able to transport also.
2. Make sure you have sufficient ARCH processes. As a
rough rule of thumb, LGWR can fill a log about three
times faster than ARCH can copy it. Therefore, there
is always a risk of ARCH getting bogged down with a
backlog of unarchived logs. If that happens, LGWR
will eventually get to a point where it can't
switch... and the log buffer fills up etc etc. ARCH
is supposed to be self-tuning (that is, extra ARCH
processes are spawned automatically). But you may
want to set log_archive_max_processes to provide a
minimum number of processes to start with (yeah, the
name of the parameter and its job are very confusing!
MAX_PROCESSES actually specifies a minimum (and
initial) number of processes!)
log_archive_max_processes integer 2
need to increase further?
3. Make sure you have sufficient log groups.If you've
only the minimum two, for example, LGWR will likely
be unable to switch back to log 1 when it's finished
in log 2, because log 1 is still being archived and
checkpointed. If LGWR waits, the log buffer fills up,
users have to wait until space becomes free once
more... With lots of log groups, however, LGWR can
switch into group 3, group 4, group 5 and so on
before having to switch back to attempt to re-use
group 1. I'd generally recommend a minimum of 4
groups.
SQL> select group#, (bytes)/(1024* 1024) from v$log;
GROUP# (BYTES)/(1024*1024)
1 5
2 5
3 5
I wll add 1 more group as per your suggestion
4. Make sure your logs are sufficiently large. Small
logs switch quicker, and hence LGWR can catch up with
itself more readily. If the logs are big, the rate of
switching slows down, but incremental checkpointing
means you don't build up a huge backlog of checkpoint
work to perform when the switch finally happens. Few
switches that don't have to do much will help keep
LGWR able to work, and if LGWR can keep clearing the
log buffer, users won't experience redo space waits.
5. Put your redo logs on your best hardware. If
you've got a mix of very fast disks and very ordinary
disks, put your online logs on the good stuff. If
you've got RAID 5 for the data files, don't put your
redo logs on it. Use RAID1+0 ideally, or RAID0 with
multiplexing. Keep the redo subsystem fast, in other
words.
running on RAID0 and archive logs are on san device
6. Related: keep the IO done to redo logs away from
the IO done to data files and anything else. Anything
which disrupts LGWR's ability to clear the log buffer
efficiently will increase your risk of redo space
waits.

Similar Messages

  • How to reduce redo space wait

    os:x86_64 x86_64 x86_64 GNU/Linux
    oracle:9.2.0.6
    running : Data guard
    Problem : Redo space wait is very high
    Init.ora paramaeters
    *.background_dump_dest='/u01/app/oracle/admin/PBPR01/bdump'
    *.compatible='9.2.0'
    *.control_files='/s410/oradata/PBPR01/control01.ctl','/s420/oradata/PBPR01/control02.ctl','/s430/oradata/PBPR01/control03.ctl'
    *.core_dump_dest='/u01/app/oracle/admin/PBPR01/cdump'
    *.cursor_space_for_time=true
    *.db_block_size=8192
    *.db_cache_size=576000000
    *.db_domain='cc.com'
    *.db_file_multiblock_read_count=16
    *.db_files=150
    *.db_name='PBPR01'
    *.db_writer_processes=1
    *.dbwr_io_slaves=2
    *.disk_asynch_io=false
    *.fast_start_mttr_target=1800
    *.java_pool_size=10485760
    *.job_queue_processes=5
    *.log_archive_dest_1='LOCATION=/s470/oraarch/PBPR01'
    *.log_archive_dest_3='service=DR_PBPR01 LGWR ASYNC=20480'
    *.log_archive_format='PBPR01_%t_%s.arc'
    *.log_archive_start=true
    *.log_buffer=524288
    *.log_checkpoints_to_alert=true
    *.max_dump_file_size='500000'
    *.object_cache_max_size_percent=20
    *.object_cache_optimal_size=512000
    *.open_cursors=500
    *.optimizer_mode='CHOOSE'
    *.processes=500
    *.pga_aggregate_target=414187520
    *.replication_dependency_tracking=false
    *.undo_management=AUTO
    *.undo_retention=10800
    *.undo_tablespace=UNDOTBS1
    *.undo_suppress_errors=TRUE
    *.session_cached_cursors=20
    *.shared_pool_size=450000000
    *.user_dump_dest='/u01/app/oracle/admin/PBPR01/udump'
    SGA :
    SQL> show sga
    Total System Global Area 1108839248 bytes
    Fixed Size 744272 bytes
    Variable Size 520093696 bytes
    Database Buffers 587202560 bytes
    Redo Buffers 798720 bytes
    SQL>
    I created log groups with 2 memebers each and with size 25 mb.
    Redo space waits shows as
    SQL> SELECT name, value
    FROM v$sysstat
    WHERE name = 'redo log space requests';
    NAME VALUE
    redo log space requests 152797
    this is running between 140000 and 160000
    some of the trace file error
    [oracle@hipclora6b bdump]$ cat PBPR01_lns0_23689.trc
    Dump file /u01/app/oracle/admin/PBPR01/bdump/PBPR01_lns0_23689.trc
    Oracle9i Enterprise Edition Release 9.2.0.6.0 - 64bit Production
    With the Partitioning, OLAP and Oracle Data Mining options
    JServer Release 9.2.0.6.0 - Production
    ORACLE_HOME = /u01/app/oracle/product/9.2.0.6
    System name: Linux
    Node name: hipclora6b.clickipc.hipc.clickcommerce.com
    Release: 2.4.21-37.EL
    Version: #1 SMP Wed Sep 7 13:32:18 EDT 2005
    Machine: x86_64
    Instance name: PBPR01
    Redo thread mounted by this instance: 1
    Oracle process number: 34
    Unix process pid: 23689, image: [email protected]
    *** SESSION ID:(82.51071) 2008-04-14 23:40:04.122
    *** 2008-04-14 23:40:04.122 46512 kcrr.c
    NetServer 0: initializing for LGWR communication
    NetServer 0: connecting to KSR channel
    : success
    NetServer 0: subscribing to KSR channel
    : success
    *** 2008-04-14 23:40:04.162 46559 kcrr.c
    NetServer 0: initialized successfully
    *** 2008-04-14 23:40:04.172 46819 kcrr.c
    NetServer 0: Request to Perform KCRRNSUPIAHM
    NetServer 0: connecting to remote destination DR_PBPR01
    *** 2008-04-14 23:40:04.412 46866 kcrr.c
    NetServer 0: connect status = 0
    A Sample alert Log
    Thread 1 advanced to log sequence 275496
    Current log# 1 seq# 275496 mem# 0: /s420/oradata/PBPR01/redo01a.log
    Current log# 1 seq# 275496 mem# 1: /s420/oradata/PBPR01/redo01b.log
    Tue Apr 15 09:10:03 2008
    ARC0: Evaluating archive log 4 thread 1 sequence 275495
    ARC0: Archive destination LOG_ARCHIVE_DEST_3: Previously completed
    ARC0: Beginning to archive log 4 thread 1 sequence 275495
    Creating archive destination LOG_ARCHIVE_DEST_1: '/s470/oraarch/PBPR01/PBPR01_1_275495.arc'
    Tue Apr 15 09:10:03 2008
    Beginning global checkpoint up to RBA [0x43428.3.10], SCN: 0x0000.3c1594fd
    Completed checkpoint up to RBA [0x43428.2.10], SCN: 0x0000.3c1594fa
    Completed checkpoint up to RBA [0x43428.3.10], SCN: 0x0000.3c1594fd
    Tue Apr 15 09:10:03 2008
    ARC0: Completed archiving log 4 thread 1 sequence 275495
    Tue Apr 15 09:29:15 2008
    LGWR: Completed archiving log 1 thread 1 sequence 275496
    Creating archive destination LOG_ARCHIVE_DEST_3: 'DR_PBPR01'
    LGWR: Beginning to archive log 5 thread 1 sequence 275497
    Beginning log switch checkpoint up to RBA [0x43429.2.10], SCN: 0x0000.3c15bc33
    Tue Apr 15 09:29:16 2008
    ARC1: Evaluating archive log 1 thread 1 sequence 275496
    ARC1: Archive destination LOG_ARCHIVE_DEST_3: Previously completed
    ARC1: Beginning to archive log 1 thread 1 sequence 275496
    Creating archive destination LOG_ARCHIVE_DEST_1: '/s470/oraarch/PBPR01/PBPR01_1_275496.arc'
    Tue Apr 15 09:29:16 2008
    Thread 1 advanced to log sequence 275497
    Current log# 5 seq# 275497 mem# 0: /s420/oradata/PBPR01/redo05a.log
    Current log# 5 seq# 275497 mem# 1: /s420/oradata/PBPR01/redo05b.log
    Tue Apr 15 09:29:16 2008
    ARC1: Completed archiving log 1 thread 1 sequence 275496
    Log file size
    SQL> select GROUP#,MEMBERS ,sum(bytes)/(1024*1024) from v$log group by
    2 GROUP#,MEMBERS;
    GROUP# MEMBERS SUM(BYTES)/(1024*1024)
    1 2 25
    2 2 25
    3 2 25
    4 2 25
    5 2 25
    Pl. give your view what can be thought of to reduce redospace wait

    Below are my suggestion:
    Increase log buffer between [ 5Mb and 15Mb]
    differ the the commit: COMMIT_WRITE=NOWAIT,BATCH
    You can also increase your redo log fil, but read the following
    Sizing Redo Logs with Oracle 10g
    Oracle has introduced a Redo Logfile Sizing Advisor that will recommend a size for our redo logs that limit excessive log switches, incomplete and excessive checkpoints, log archiving issues, DBWR performance and excessive disk I/O. All these issues result in transactions bottlenecking within redo and performance degradation. While many DBAs' first thought is throughput of the transaction base, not very many give thought to the recovery time required in relation to the size of redo generated or the actual size of the redo log groups. With the introduction of Oracle's Mean Time to Recovery features, DBAs can now specify through the FAST_START_MTTR_TARGET initialization variable just how long a crash recovery should take. Oracle will then try its best to issue the proper checkpoints during normal system operation to help meet this target. Since the size of redo logs and the checkpointing of data have a key role in Oracle's ability to recover within a desired time frame, Oracle will now use the value of FAST_START_MTTR_TARGET to suggest an optimal redo log size. In actuality, the setting of FAST_START_MTTR_TARGET is what triggers the new redo logfile sizing advisor, and if you do not set it, Oracle will not provide a suggestion for your redo logs. If you do not have any real time requirement for recovery you should at least set this to its maximum value of 3600 seconds, or one hour and you will then be able to take advantage of the advisory. After setting the FAST_START_MTTR_TARGET initialization parameter a DBA need only query the V$INSTANCE_RECOVERY view for the column OPTIMAL_LOGFILE_SIZE value, in MEG, and then rebuild the redo log groups with this recommendation.
    Simple query to show the optimal size for redo logs
    SQL> SELECT OPTIMAL_LOGFILE_SIZE
    FROM V$INSTANCE_RECOVERY
    OPTIMAL_LOGFILE_SIZE
    64
    A few notes about setting FAST_START_MTTR_TARGET
    •     Specify a value in seconds (0-3600) that you wish Oracle to perform recovery within.
    •     Is overridden by LOG_CHECKPOINT_INTERVAL:
    Since LOG_CHECKPOINT_INTERVAL requests Oracle to checkpoint after a specified amount of redo blocks have been written, and FAST_START_MTTR_TARGET basically attempts to size the redo logs in such a way as to perform a checkpoint when they switch, you can easily see that these two parameters are of conflicting interest. You will need to unset LOG_CHECKPOINT_INTERVAL if you wish to use the redo log sizing advisor and have checkpoints occur with log switches. This is how it was recommended to be done in the v7 days and really I can't quite see any reason for anything else.
    •     Is overridden by LOG_CHECKPOINT_TIMEOUT:
    LOG_CHECKPOINT_TIMEOUT controls the amount of time in between checkpoints if a log switch or the amount of redo generated has not yet triggered a checkpoint. Since our focus is now on Mean Time to Recovery (MTTR) this parameter is no longer of concern because we are asking Oracle to determine when to checkpoint based on our crash recovery requirements.
    •     Is overridden by FAST_START_IO_TARGET:
    Actually, the FAST_START_IO_TARGET parameter is deprecated and you should switch over to the FAST_START_MTTR_TARGET parameter
    Thanks

  • High redo log space wait time

    Hello,
    Our DB is having very high redo log space wait time :
    redo log space requests 867527
    redo log space wait time 67752674
    LOG_BUFFER is 14 MB and having 6 redo logs groups and the size of redo log file is 500MB for each log file.
    Also, the amount of redo generated per hour :
    START_DATE START NUM_LOGS MBYTES DBNAME
    2008-07-03 10:00 2 1000 TKL
    2008-07-03 11:00 4 2000 TKL
    2008-07-03 12:00 3 1500 TKL
    Does increasing the size of LOG_BUFFER will help to reduce the redo log space wait ?
    Thanks in advance ,
    Regards,
    Aman

    Looking quickly over the AWR report provided the following information could be helpful:
    1. You are currently targeting approx. 6GB of memory with this single instance and the report tells that physical memory is 8GB. According to the advisories it looks like you could decrease your memory allocation without tampering your performance.
    In particular the large_pool_size setting seems to be quite high although you're using shared servers.
    Since you're using 10.2.0.4 it might be worth to think about using the single SGA_TARGET parameter instead of the specifying all the single parameters. This allows Oracle to size the shared pool components within the given target dynamically.
    2. You are currently using a couple of underscore parameters. In particular the "_optimizer_max_permutations" parameter is set to 200 which might reduce significantly the number of execution plans permutations Oracle is investigating while optimizing the statement and could lead to suboptimal plans. It could be worth to check why this has been set.
    In addition you are using a non-default setting of "_shared_pool_reserved_pct" which might no longer be necessary if you are using the SGA_TARGET parameter as mentioned above.
    3. You are using non-default settings for the "optimizer_index_caching" and "optimizer_index_cost_adj" parameters which favor index-access paths / nested loops. Since the "db file sequntial read" is the top wait event it might be worth to check if the database is doing too excessive index access. Also most of the rows have been fetched by rowid (table fetch by rowid) which could also be an indicator for excessive index access/nested loop usage.
    4. You database has been working quite a lot during the 30min. snapshot interval: It processed 123.000.000 logical blocks, which means almost 0.5GB per second. Check the top SQLs, there are a few that are responsible for most of the blocks processed. E.g. there is a anonymous PL/SQL block that has been executed almost 17.000 times during the interval representing 75% of the blocks processed. The statements executed as part of these procedures might be worth to check if they could be tuned to require less logical I/Os. This could be related to the non-default optimizer parameters mentioned above.
    5. You are still using the compatible = 9.2.0 setting which means this database could still be opened by a 9i instance. If this is no longer required, you might lift this to the default value of 10g. This will also convert the REDO format to 10g I think which could lead to less amount of redo generated. But be aware of the fact that this is a one-way operation, you can only go back to 9i then via a restore once the compatible has been set to 10.x.
    6. Your undo retention is set quite high (> 6000 secs), although your longest query in the AWR period was 151 seconds. It might be worth to check if this setting is reasonable, as you might have quite a large undo tablespace at present. Oracle 10g ignores the setting if it isn't able to honor the setting given the current Undo tablespace size.
    7. "parallel_max_servers" has been set to 0, so no parallel operations can take place. This might be intentional but it's something to keep in mind.
    Regards,
    Randolf
    Oracle related stuff:
    http://oracle-randolf.blogspot.com/
    SQLTools++ for Oracle:
    http://www.sqltools-plusplus.org:7676/
    http://sourceforge.net/projects/sqlt-pp/

  • Lot of Redo log wait

    Dear all,
    In st04 I see Redo log wait is this a problem. Please suggest how to solve it
    Please find the details.
    Size (kB)                            14,352
    Entries                          42,123,046
    Allocation retries                    9,103
    Alloc fault rate(%)                     0.0
    Redo log wait (s)                       486
    Log files (in use)                8 (   8 )
    DB_INST_ID     Instance ID     1
    DB_INSTANCE     DB instance name     prd
    DB_NODE     Database node     A
    DB_RELEASE     Database release     10.2.0.4.0
    DB_SYS_TIMESTAMP     Day, Time     06.04.2010 13:07:10
    DB_SYSDATE     DB System date     20100406
    DB_SYSTIME     DB System time     130710
    DB_STARTUP_TIMESTAMP     Start up at     22.03.2010 03:51:02
    DB_STARTDATE     DB Startup date     20100322
    DB_STARTTIME     DB Startup time     35102
    DB_ELAPSED     Seconds since start     1329368
    DB_SNAPDIFF     Sec. btw. snapshots     1329368
    DATABUFFERSIZE     Size (kB)     3784704
    DBUFF_QUALITY     Quality (%)     96.3
    DBUFF_LOGREADS     Logical reads     5615573538
    DBUFF_PHYSREADS     Physical reads     207302988
    DBUFF_PHYSWRITES     Physical writes     7613263
    DBUFF_BUSYWAITS     Buffer busy waits     878188
    DBUFF_WAITTIME     Buffer wait time (s)     3583
    SHPL_SIZE     Size (kB)     1261568
    SHPL_CAQUAL     DD-cache Quality (%)     95.1
    SHPL_GETRATIO     SQL area getratio(%)     98.4
    SHPL_PINRATIO     SQL area pinratio(%)     99.9
    SHPL_RELOADSPINS     SQLA.Reloads/pins(%)     0.0042
    LGBF_SIZE     Size (kB)     14352
    LGBF_ENTRIES     Entries     42123046
    LGBF_ALLORETR     Allocation retries     9103
    LGBF_ALLOFRAT     Alloc fault rate(%)     0
    LGBF_REDLGWT     Redo log wait (s)     486
    LGBF_LOGFILES     Log files     8
    LGBF_LOGFUSE     Log files (in use)     8
    CLL_USERCALLS     User calls     171977181
    CLL_USERCOMM     User commits     1113161
    CLL_USERROLLB     User rollbacks     34886
    CLL_RECURSIVE     Recursive calls     36654755
    CLL_PARSECNT     Parse count     10131732
    CLL_USR_PER_RCCLL     User/recursive calls     4.7
    CLL_RDS_PER_UCLL     Log.Reads/User Calls     32.7
    TIMS_BUSYWT     Busy wait time (s)     389991
    TIMS_CPUTIME     CPU time session (s)     134540
    TIMS_TIM_PER_UCLL     Time/User call (ms)     3
    TIMS_SESS_BUSY     Sessions busy (%)     0.94
    TIMS_CPUUSAGE     CPU usage (%)     2.53
    TIMS_CPUCOUNT     Number of CPUs     4
    RDLG_WRITES     Redo writes     1472363
    RDLG_OSBLCKWRT     OS blocks written     54971892
    RDLG_LTCHTIM     Latching time (s)     19
    RDLG_WRTTIM     Redo write time (s)     2376
    RDLG_MBWRITTEN     MB written     25627
    TABSF_SHTABSCAN     Short table scans     12046230
    TABSF_LGTABSCAN     Long table scans     6059
    TABSF_FBYROWID     Table fetch by rowid     1479714431
    TABSF_FBYCONTROW     Fetch by contin. row     2266031
    SORT_MEMORY     Sorts (memory)     3236898
    SORT_DISK     Sorts (disk)     89
    SORT_ROWS     Sorts (rows)     5772889843
    SORT_WAEXOPT     WA exec. optim. mode     1791746
    SORT_WAEXONEP     WA exec. one pass m.     93
    SORT_WAEXMULTP     WA exec. multipass m     0
    IEFF_SOFTPARSE     Soft parse ratio     0.9921
    IEFF_INMEM_SORT     In-memory sort ratio     1
    IEFF_PARSTOEXEC     Parse to exec. ratio     0.9385
    IEFF_PARSCPUTOTOT     Parse CPU to Total     0.9948
    IEFF_PTCPU_PTELPS     PTime CPU / PT elps.     0.1175
    Regards,
    Kumar

    Hi,
    If the redo buffers are not large enough, the Oracle log-writer process waits for space to become available. This wait time becomes wait time for the end user. Hence this may cause perfromance problem at database end and hence need to be tuned. 
    The size of the redo log buffer is defined in the init.ora file using the 'LOG_BUFFER' parameter. The statistic 'redo log space requests' reflects the number of times a user process waits for space in the redo log buffer.
    If the size of redo log buffer is not big enough causing this wait, recommendation is to increase the size of redo log buffer in such a way that the value of "redo log space requests" should be near to zero.
    regards,
    rakesh

  • Redo log wait

    Dear All,
    We are usinf ecc5 ans the databse oacle 9i on wondows 2003I have notice that the
    Redo log wait S has been suddenly increase in number   690
    Please suggest what si the problem and to solve it.
    Data buffer
    Size              kb      1,261,568
    Quality            %           96.2
    Reads                 4,234,462,711
    Physical reads          160,350,516
              writes           3,160,751
    Buffer busy waits         1,117,697
    Buffer wait time   s          3,507
    Shared Pool
    Size              kb        507,904
    DD-Cache quality   %           84.3
    SQL Area getratio  %           95.6
             pinratio  %           98.8
          reloads/pins %         0.0297
    Log buffer
    Size              kb          1,176
    Entries                  11,757,027
    Allocation retries              722
    Alloc fault rate   %            0.0
    *Redo log wait      s            690*
    Log files (in use)            8( 8)
    Calls
    User calls               41,615,763
         commits                367,243
         rollbacks                7,890
    Recursive calls         100,067,593
    Parses                    7,822,590
    User/Recursive calls            0.4
    Reads / User calls            101.8
    Time statistics
    Busy wait time     s        697,392
    CPU time           s         42,505
    Time/User call    ms             18
      Sessions busy      %           9.26
      CPU usage          %           4.51
      CPU count                         2
    Redo logging
    Writes                    1,035,582
    OS-Blocks written        14,276,056
    Latching time      s              1
    Sessions busy      %           9.26
    CPU usage          %           4.51
    CPU count                         2
    Redo logging
    Writes                    1,035,582
    OS-Blocks written        14,276,056
    Latching time      s              1
      Write time         s            806
      Mb written                    6,574
    Table scans & fetches
    Short table scans           607,891
    Long table scans             32,468
    Fetch by rowid        1,620,054,083
       by continued row         761,131
    Sorts
    Memory                    3,046,669
    Disk                             32
    Rows sorted             446,593,854
    Regards,
    Shiva

    Hi Stefan,
    As per the doc you have suggest. The details are as following.
    In the day there is only 24 log switch , but in hour there is no more than 10 to 15 as per doc ,so ti is very less.
    The DD-Cache quality   %           84.1 is less
    The elapsed time since start
    Elapsed since start (s)       540,731
      Log buffer
      Size              kb          1,176
      Entries                  13,449,901
      Allocation retries              767
      Alloc fault rate   %            0.0
    *Redo log wait      s            696*
       Log files (in use)            8( 8)
    Check DB Wait times
    TCode ST04->Detail Analysis Menu->Wait Events
    Statistics on total waits for an event
    Elapsed time:             985  s
    since reset at 09:34:06
    Type   Client   Sessions      Busy wait            Total wait           Busy wait
                                time (ms)    time (ms)            time (%)
    USER   User          40            1,028,710           17,594,230        5.85
    BACK   ARC0           1                2,640            1,264,410        0.21
    BACK   ARC1           1                  540            1,020,400        0.05
    BACK   CKPT           1                  950              987,490        0.10
    BACK   DBW0           1                  130              983,920        0.01
    BACK   LGWR           1                  160              986,430        0.02
    BACK   PMON           1                    0              987,000        0.00
    BACK   RECO           1                   10            1,800,010        0.00
    BACK   SMON           1                3,820            1,179,410        0.32
    Disk based sorts
    Sorts
    Memory                    3,443,693
    Disk                             41
    Rows sorted             921,591,847
    Check DB Shared Pool Quality
    Shared Pool
    Size              kb        507,904
    DD-Cache quality   %           84.1
    SQL Area getratio  %           95.6
      pinratio  %                           98.8
          reloads/pins %         0.0278
      V$LOGHIST
    THREAD#   SEQUENCE#   FIRST_CHANGE#   FIRST_TIME            SWITCH_CHANGE#
    1         31612       381284375       2008/11/13 00:01:29   381293843
    1         31613       381293843       2008/11/13 00:12:12   381305142
    1         31614       381305142       2008/11/13 03:32:39   381338724
    1         31615       381338724       2008/11/13 06:29:21   381362057
    1         31616       381362057       2008/11/13 07:00:39   381371178
    1         31617       381371178       2008/11/13 07:13:01   381457916
    1         31618       381457916       2008/11/13 09:26:17   381469012
    1         31619       381469012       2008/11/13 10:27:19   381478636
    1         31620       381478636       2008/11/13 10:59:54   381488508
    1         31621       381488508       2008/11/13 11:38:33   381498759
    1         31622       381498759       2008/11/13 12:05:14   381506545
    1         31623       381506545       2008/11/13 12:33:48   381513732
    1         31624       381513732       2008/11/13 13:08:10   381521338
    1         31625       381521338       2008/11/13 13:50:15   381531371
    1         31626       381531371       2008/11/13 14:38:36   381540689
    1         31627       381540689       2008/11/13 15:02:19   381549493
    1         31628       381549493       2008/11/13 15:43:39   381556307
    1         31629       381556307       2008/11/13 16:07:47   381564737
    1         31630       381564737       2008/11/13 16:39:45   381571786
    1         31631       381571786       2008/11/13 17:07:07   381579026
    1         31632       381579026       2008/11/13 17:37:26   381588121
    1         31633       381588121       2008/11/13 18:28:58   381595963
    1         31634       381595963       2008/11/13 20:00:41   381602469
    1         31635       381602469       2008/11/13 22:23:20   381612866
    1         31636       381612866       2008/11/14 00:01:28   381622652
    1         31637       381622652       2008/11/14 00:09:52   381634720
    1         31638       381634720       2008/11/14 03:32:00   381688156
    1         31639       381688156       2008/11/14 07:00:30   381703441
    14.11.2008         Log File information from control file                                10:01:32
      Group     Thread    Sequence   Size         Nr of     Archive          First           Time 1st SCN
      Nr        Nr        Nr         (bytes)      Members        Status      Change Nr       in log
      1         1         31638      52428800     2         YES  INACTIVE    381634720       2008/11/14 03:32:00
      2         1         31639      52428800     2         YES  INACTIVE    381688156       2008/11/14 07:00:30
      3         1         31641      52428800     2         NO   CURRENT     381783353       2008/11/14 09:50:09
      4         1         31640      52428800     2         YES  ACTIVE      381703441       2008/11/14 07:15:07
    Regards,

  • Running out of REDO space locks the database

    Hi All,
    My 11gR2 RAC database running on two RH5.7 nodes got locked. I relate the cause to the REDO disk space being full as running the ASMCA shows that all the REDO space is used (40GB!!!)
    Furthermore as I try to access the database via sqlplus i get the ORA-00257 (http://ora-00257.ora-code.com/) that mentions
    "The most likely cause of this message is the destination device is out of space to store the redo log file"
    Now is it possible that an Oracle RAC is unrecoverable just because it can't write logs? And that as we can't access it we can't delete these logs?
    Please help

    920550 wrote:
    Now is it possible that an Oracle RAC is unrecoverable just because it can't write logs?No it's the contrary: Oracle instance is blocking any new database write in order to be recoverable.
    And that as we can't access it we can't delete these logs?
    Even if archive destination is stored in ASM you can access it (for example with asmcmd utility with command line interface).
    What you must do is:
    1. backup archived redo logs
    2. remove archived redo logs that have been backed up.
    For example if you are using RMAN you can try
    backup archivelog all delete input;

  • Redo log wait event

    Hi,
    in my top evens i've:
    Top 5 Timed Events Avg %Total
    ~~~~~~~~~~~~~~~~~~ wait Call
    Event Waits Time (s) (ms) Time Wait Class
    CPU time 1,894 36.1
    log file sync 36,862 1,008 27 19.2 Commit
    db file scattered read 165,508 970 6 18.5 User I/O
    db file sequential read 196,596 857 4 16.3 User I/O
    log file parallel write 35,847 565 16 10.8 System I/O
    Log file are on a separate disks, with no activity, only 1 redo per group, and 4 groups.
    I think that 27ms for log file synch is high.
    I raised commits in sqlloader putting rows=100000 instead 30000 but it's always high.
    Which check i can perform?
    I'm on AIX 5.3 and database in 10.2.0.4.4

    Log File Sync
    The “log file sync” wait event is triggered when a user session issues a commit (or a rollback). The user session will signal or post the LGWR to write the log buffer to the redo log file. When the LGWR has finished writing, it will post the user session. The wait is entirely dependent on LGWR to write out the necessary redo blocks and send confirmation of its completion back to the user session. The wait time includes the writing of the log buffer and the post, and is sometimes called “commit latency”.
    The P1 parameter in <View:V$SESSION_WAIT> is defined as follows for this wait event:
    P1 = buffer#
    All changes up to this buffer number (in the log buffer) must be flushed to disk and the writes confirmed to ensure that the transaction is committed and will be kept on an instance crash. The wait is for LGWR to flush up to this buffer#.
    Reducing Waits / Wait times:
    If a SQL statement is encountering a significant amount of total time for this event, the average wait time should be examined. If the average wait time is low, but the number of waits is high, then the application might be committing after every row, rather than batching COMMITs. Applications can reduce this wait by committing after “n” rows so there are fewer distinct COMMIT operations. Each commit has to be confirmed to make sure the relevant REDO is on disk. Although commits can be "piggybacked" by Oracle, reducing the overall number of commits by batching transactions can be very beneficial.
    If the SQL statement is a SELECT statement, review the Oracle Auditing settings. If Auditing is enabled for SELECT statements, Oracle could be spending time writing and commit data to the AUDIT$ table.
    If the average time waited is high, then examine the other log related waits for the session, to see where the session is spending most of its time. If a session continues to wait on the same
    If the average time waited is high, then examine the other log related waits for the session, to see where the session is spending most of its time. If a session continues to wait on the same buffer# then the SEQ# column of V$SESSION_WAIT should increment every second. If not then the local session has a problem with wait event timeouts. If the SEQ# column is incrementing then the blocking process is the LGWR process. Check to see what LGWR is waiting on as it may be stuck. If the waits are because of slow I/O, then try the following:
    Reduce other I/O activity on the disks containing the redo logs, or use dedicated disks.
    Try to reduce resource contention. Check the number of transactions (commits + rollbacks) each second, from V$SYSSTAT.
    Alternate redo logs on different disks to minimize the effect of the archiver on the log writer.
    Move the redo logs to faster disks or a faster I/O subsystem (for example, switch from RAID 5 to RAID 1).
    Consider using raw devices (or simulated raw devices provided by disk vendors) to speed up the writes.
    See if any activity can safely be done with NOLOGGING / UNRECOVERABLE options in order to reduce the amount of redo being written.
    See if any of the processing can use the COMMIT NOWAIT option (be sure to understand the semantics of this before using it).
    Check the size of the log buffer as it may be so large that LGWR is writing too many blocks at one time. 

  • Impdp showing log buffer space wait

    I am doing an import using datapump.
    I can see a lot of waits related to "log buffer space" while running the impdp.
    I have *.log_buffer paramter set to 10485760 (10MB)
    Does it mean that I should increase this parameter and how to know what value will be good.
    Thanks!

    user608897 wrote:
    I am doing an import using datapump.
    I can see a lot of waits related to "log buffer space" while running the impdp.
    I have *.log_buffer paramter set to 10485760 (10MB)
    Does it mean that I should increase this parameter and how to know what value will be good.
    Thanks!http://asktom.oracle.com/pls/asktom/f?p=100:11:3952768239352725::::P11_QUESTION_ID:1724586308922
    Your log_buffer is generally 5meg or less.
    Since the log buffer is flushed:
    o every 3 seconds
    o every commit
    o when 1/3 full
    o when 1meg full

  • Redo log space requests VALUE high

    SELECT name||' = '||value
    FROM v$sysstat
    WHERE name = 'redo log space requests';
    I am noticing 40+ space requests for some of my Oracle 9.2.0.5 databases.
    On another 7.3.4 DB I see this over 140 but this DB shutdown only on weekends so this cumulative value increases I presume.
    I have 20MB of 5 groups already. Do I still add another 2 more groups or increase their sizes ?
    I did read somewhere that I'd have to increase the log_buffer parameter. So how do we deal with this issue ? Any repercussions if I let this as it is for now ?
    The cause of this would be redo logs are not big enough or otherwise ?
    Thanks.

    user4874781 wrote:
    Thanks for your response Charles.
    So if I understand this correctly ... Redolog Space requests corresponds to a either an incorrectly sized redo log file / DBWR / CKPT needs to be tuned.
    Maybe I was interpreting this the wrong way. (Possibly)
    " The statistic 'redo log space requests' reflects the number of times a user process waits for space in the redo log buffer. " If that is the case, if there was longer waits for this event, I was under the assumption that log_buffer needs to be increased.
    http://www.idevelopment.info/data/Oracle/DBA_tips/Tuning/TUNING_6.shtml
    * Yes, the waits have increased to 70 as of now (since 40 yesterday .. DB was started Saturday night and will run till weekend) Less activity as of now, since the day has just started ; so it would definitely rise by end of the day. I took a look at the above article, and I think I understand why the article is slightly confusing. With due respect to the author, the article was last modified 16-Apr-2001, which I believe is before the Oracle documentation was better clarified regarding these statistics. From:
    http://download.oracle.com/docs/cd/B14117_01/server.101/b10755/stats002.htm
    "redo log space requests: Number of times the active log file is full and Oracle must wait for disk space to be allocated for the redo log entries. Such space is created by performing a log switch. Log files that are small in relation to the size of the SGA or the commit rate of the work load can cause problems. When the log switch occurs, Oracle must ensure that all committed dirty buffers are written to disk before switching to a new log file. If you have a large SGA full of dirty buffers and small redo log files, a log switch must wait for DBWR to write dirty buffers to disk before continuing."
    "redo log space wait time: Total elapsed waiting time for "redo log space requests" in 10s of milliseconds"
    It is quite possible that the "redo log space requests" will increase with every redo log file switch, which should not be too much of a concern. You may want to focus a little more on the "redo log space wait time" statistic, which indicates how much wait time was involved waiting. You might also want to focus on the system-wide wait event interface, examining how the accumulated wait time increases from one sampling of each of the statistics to the next.
    * I have 1 log switch every 11 minutes ; BTW ; I have 5 log groups of 20 MB each as of now. So I am assuming 40 MB of 4 or 5 log groups should be fine as per your suggestion ?If you have the disk space, considering that it is an ancient AIX box, you may want to set the redo log files to an even larger size, possibly 100MB (or larger). You may then want to force periodic switches of the redo log, for instance once an hour, or once every 30 minutes.
    * This is an ancient AIX box with 512 MB Ram. Is the redo log located on a fast device ? I'd have to find that out ( any hints on that ? )
    * Decreasing the log_buffer is possible on weekends since I'd have to bounce it for it to take effect.
    I will increase the log files accordingly and hopefully the space waits will reduce. Thanks again.Someone else, possibly Sybrand, on the forums might be familiar with AIX and be able to provide you with an answer. If you are seeing system-wide increasing wait time for redo related waits, then that might be an indication that the redo logs are not located on a fast device.
    Charles Hooper
    IT Manager/Oracle DBA
    K&M Machine-Fabricating, Inc.

  • Buffer busy waits

    Hi,
    Version : Oracle 9i
    I am getting buffer busy waits on some tables. Will increase in inittrans & pctfree of those tables reduce buffer busy waits?
    Tablespace is having segment space mgmt auto & extent management local.
    cursor_sharing is similar.
    Users are not experiencing any problem.Is there any problem other than this in statspack report?
    STATSPACK report for
    DB Name DB Id Instance Inst Num Release Cluster Host
    AHD 3712247982 ahd 1 9.2.0.1.0 NO SBGSDPRI
    Snap Id Snap Time Sessions Curs/Sess Comment
    Begin Snap: 20 13-Feb-07 14:48:35 33 9.9
    End Snap: 21 13-Feb-07 15:12:19 34 10.4
    Elapsed: 23.73 (mins)
    Cache Sizes (end)
    ~~~~~~~~~~~~~~~~~
    Buffer Cache: 656M Std Block Size: 8K
    Shared Pool Size: 152M Log Buffer: 768K
    Load Profile
    ~~~~~~~~~~~~ Per Second Per Transaction
    Redo size: 5,960.83 2,761.29
    Logical reads: 2,376.85 1,101.05
    Block changes: 35.48 16.44
    Physical reads: 97.56 45.20
    Physical writes: 1.15 0.53
    User calls: 92.63 42.91
    Parses: 20.00 9.27
    Hard parses: 0.29 0.13
    Sorts: 4.80 2.22
    Logons: 0.01 0.00
    Executes: 23.14 10.72
    Transactions: 2.16
    % Blocks changed per Read: 1.49 Recursive Call %: 14.69
    Rollback per transaction %: 0.00 Rows per Sort: 472.64
    Instance Efficiency Percentages (Target 100%)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Buffer Nowait %: 99.65 Redo NoWait %: 100.00
    Buffer Hit %: 95.90 In-memory Sort %: 100.00
    Library Hit %: 99.15 Soft Parse %: 98.55
    Execute to Parse %: 13.57 Latch Hit %: 99.70
    Parse CPU to Parse Elapsd %: 90.83 % Non-Parse CPU: 96.58
    Shared Pool Statistics Begin End
    Memory Usage %: 84.68 84.76
    % SQL with executions>1: 77.32 79.22
    % Memory for SQL w/exec>1: 90.74 92.81
    Top 5 Timed Events
    ~~~~~~~~~~~~~~~~~~ % Total
    Event Waits Time (s) Ela Time
    CPU time 125 54.23
    db file sequential read 83,110 69 30.14
    db file scattered read 23,196 27 11.75
    buffer busy waits 11,760 6 2.42
    log file sync 3,078 1 .45
    Wait Events for DB: AHD Instance: ahd Snaps: 20 -21
    -> s - second
    -> cs - centisecond - 100th of a second
    -> ms - millisecond - 1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc (idle events last)
    Avg
    Total Wait wait Waits
    Event Waits Timeouts Time (s) (ms) /txn
    db file sequential read 83,110 0 69 1 27.0
    db file scattered read 23,196 0 27 1 7.5
    buffer busy waits 11,760 0 6 0 3.8
    log file sync 3,078 0 1 0 1.0
    log file parallel write 5,216 4,841 1 0 1.7
    control file sequential read 1,390 0 1 1 0.5
    control file parallel write 462 0 0 1 0.2
    db file parallel write 672 336 0 0 0.2
    latch free 54 24 0 2 0.0
    SQL*Net more data to client 1,026 0 0 0 0.3
    LGWR wait for redo copy 12 0 0 0 0.0
    SQL*Net message from client 131,863 0 22,857 173 42.9
    virtual circuit status 48 48 1,497 31188 0.0
    wakeup time manager 45 45 1,446 32123 0.0
    SQL*Net message to client 131,864 0 0 0 42.9
    SQL*Net more data from clien 27 0 0 0 0.0
    Background Wait Events for DB: AHD Instance: ahd Snaps: 20 -21
    -> ordered by wait time desc, waits desc (idle events last)
    Avg
    Total Wait wait Waits
    Event Waits Timeouts Time (s) (ms) /txn
    log file parallel write 5,216 4,841 1 0 1.7
    control file parallel write 462 0 0 1 0.2
    control file sequential read 184 0 0 2 0.1
    db file parallel write 672 336 0 0 0.2
    log file sync 24 0 0 0 0.0
    db file sequential read 1 0 0 8 0.0
    LGWR wait for redo copy 12 0 0 0 0.0
    rdbms ipc message 12,386 7,345 10,752 868 4.0
    SQL*Net message from client 384 0 1,498 3901 0.1
    smon timer 4 4 1,229 ###### 0.0
    SQL*Net message to client 384 0 0 0 0.1
    SQL ordered by Gets for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Buffer Gets Threshold: 10000
    -> Note that resources reported for PL/SQL includes the resources used by
    all SQL statements called within the PL/SQL code. As individual SQL
    statements are also reported, it is possible and valid for the summed
    total % to exceed 100
    CPU Elapsd
    Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value
    1,269,773 52 24,418.7 37.5 27.03 76.26 3370382957
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1" ) AND ( (
    381,394 44 8,668.0 11.3 21.30 22.94 3653016280
    SELECT count(*) FROM call_req, ctct, loc, site, z_zo, z_lho WHER
    E ( call_req.customer = ctct.id AND ctct.c_l_id = loc.id AND l
    oc.l_si_id = site.id AND site.z_si_zo_id = z_zo.id AND z_zo.zo
    lhoid = z_lho.id AND z_lho.lho_name LIKE :"SYS_B_0" AND cal
    l_req.active_flag = :"SYS_B_1" ) AND ( ( call_req.group_id != :"
    239,582 10 23,958.2 7.1 5.95 17.44 1650906216
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_0" ) AND ( ( call_req.group_id != :"SYS_B_1" ) and
    146,016 9 16,224.0 4.3 2.58 12.01 977739309
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.group_id IN ( SELECT id FROM ctct WHERE id = :"SYS_B_00" O
    R id = :"SYS_B_01" OR id = :"SYS_B_02" OR id = :"SYS_B_03" OR id
    = :"SYS_B_04" OR id = :"SYS_B_05" OR id = :"SYS_B_06" OR id = :
    "SYS_B_07" OR id = :"SYS_B_08" OR id = :"SYS_B_09" OR id = :"SYS
    117,569 7 16,795.6 3.5 0.52 0.52 1972089848
    SELECT call_req.open_date, call_req.id FROM call_req, ctct WHERE
    ( call_req.status = :"SYS_B_00" AND call_req.group_id = ctct.id
    AND ctct.c_last_name LIKE :"SYS_B_01" AND ( call_req.assigne
    e IS NULL ) ) AND ( call_req.group_id IN ( SELECT id FROM ctct W
    HERE id = :"SYS_B_02" OR id = :"SYS_B_03" OR id = :"SYS_B_04" OR
    100,276 4 25,069.0 3.0 2.77 6.95 771782876
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho, ctct cn01 WHERE ( call_req.customer = ctct.i
    d AND ctct.c_l_id = loc.id AND loc.l_si_id = site.id AND site
    .z_si_zo_id = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.
    lho_name LIKE :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1"
    95,832 4 23,958.0 2.8 2.13 3.80 1755292198
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_00" AND call_req.active_flag = :"SYS_B_01" AND ( ca
    86,680 10 8,668.0 2.6 7.69 8.21 3407388950
    SQL ordered by Gets for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Buffer Gets Threshold: 10000
    -> Note that resources reported for PL/SQL includes the resources used by
    all SQL statements called within the PL/SQL code. As individual SQL
    statements are also reported, it is possible and valid for the summed
    total % to exceed 100
    CPU Elapsd
    Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value
    SELECT count(*) FROM call_req, ctct, loc, site, z_zo, z_lho WHER
    E ( call_req.customer = ctct.id AND ctct.c_l_id = loc.id AND l
    oc.l_si_id = site.id AND site.z_si_zo_id = z_zo.id AND z_zo.zo
    lhoid = z_lho.id AND z_lho.lho_name LIKE :"SYS_B_0" ) AND (
    ( call_req.group_id != :"SYS_B_1" ) and ( call_req.group_id !=
    71,839 3 23,946.3 2.1 2.73 6.07 1599404397
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo WHERE ( call_req.active_flag = :"SYS_B_0" AND call_r
    eq.customer = ctct.id AND ctct.c_l_id = loc.id AND loc.l_si_id
    = site.id AND site.z_si_zo_id = z_zo.id AND z_zo.zo_name LIK
    E :"SYS_B_1" ) AND ( ( call_req.group_id != :"SYS_B_2" ) and (
    60,507 9 6,723.0 1.8 1.47 1.49 632450130
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho, ctct cn01 WHERE ( call_req.customer = ctct.i
    d AND ctct.c_l_id = loc.id AND loc.l_si_id = site.id AND site
    .z_si_zo_id = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.
    lho_name LIKE :"SYS_B_0" AND call_req.status = :"SYS_B_1" AND
    57,682 191 302.0 1.7 3.48 3.52 484128938
    SELECT cnote.posted_date, cnote.text FROM cnote WHERE ( ( cnote.
    loc_id = :"SYS_B_0" ) OR cnote.loc_id IS NULL ) AND ( cnote.inte
    rnal IS NULL OR cnote.internal != :"SYS_B_1" ) ORDER BY cnote.p
    osted_date DESC
    52,146 3 17,382.0 1.5 1.22 3.60 930247717
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.group_id IN ( SELECT id FROM ctct WHERE id = :"SYS_B_00" O
    R id = :"SYS_B_01" OR id = :"SYS_B_02" OR id = :"SYS_B_03" OR id
    = :"SYS_B_04" OR id = :"SYS_B_05" OR id = :"SYS_B_06" OR id = :
    "SYS_B_07" OR id = :"SYS_B_08" OR id = :"SYS_B_09" OR id = :"SYS
    43,534 4 10,883.5 1.3 2.05 2.10 2363733805
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho, prob_ctg, ctct cn01 WHERE ( call_req.custome
    r = ctct.id AND ctct.c_l_id = loc.id AND loc.l_si_id = site.id
    AND site.z_si_zo_id = z_zo.id AND z_zo.zo_lho_id = z_lho.id A
    ND z_lho.lho_name LIKE :"SYS_B_00" AND call_req.active_flag =
    SQL ordered by Reads for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Disk Reads Threshold: 1000
    CPU Elapsd
    Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
    81,653 52 1,570.3 58.8 27.03 76.26 3370382957
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1" ) AND ( (
    15,402 10 1,540.2 11.1 5.95 17.44 1650906216
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_0" ) AND ( ( call_req.group_id != :"SYS_B_1" ) and
    13,371 9 1,485.7 9.6 2.58 12.01 977739309
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.group_id IN ( SELECT id FROM ctct WHERE id = :"SYS_B_00" O
    R id = :"SYS_B_01" OR id = :"SYS_B_02" OR id = :"SYS_B_03" OR id
    = :"SYS_B_04" OR id = :"SYS_B_05" OR id = :"SYS_B_06" OR id = :
    "SYS_B_07" OR id = :"SYS_B_08" OR id = :"SYS_B_09" OR id = :"SYS
    6,157 4 1,539.3 4.4 2.77 6.95 771782876
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho, ctct cn01 WHERE ( call_req.customer = ctct.i
    d AND ctct.c_l_id = loc.id AND loc.l_si_id = site.id AND site
    .z_si_zo_id = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.
    lho_name LIKE :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1"
    6,152 4 1,538.0 4.4 2.13 3.80 1755292198
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_00" AND call_req.active_flag = :"SYS_B_01" AND ( ca
    4,622 3 1,540.7 3.3 2.73 6.07 1599404397
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo WHERE ( call_req.active_flag = :"SYS_B_0" AND call_r
    eq.customer = ctct.id AND ctct.c_l_id = loc.id AND loc.l_si_id
    = site.id AND site.z_si_zo_id = z_zo.id AND z_zo.zo_name LIK
    E :"SYS_B_1" ) AND ( ( call_req.group_id != :"SYS_B_2" ) and (
    2,982 3 994.0 2.1 1.22 3.60 930247717
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.group_id IN ( SELECT id FROM ctct WHERE id = :"SYS_B_00" O
    R id = :"SYS_B_01" OR id = :"SYS_B_02" OR id = :"SYS_B_03" OR id
    = :"SYS_B_04" OR id = :"SYS_B_05" OR id = :"SYS_B_06" OR id = :
    "SYS_B_07" OR id = :"SYS_B_08" OR id = :"SYS_B_09" OR id = :"SYS
    1,566 44 35.6 1.1 21.30 22.94 3653016280
    SELECT count(*) FROM call_req, ctct, loc, site, z_zo, z_lho WHER
    E ( call_req.customer = ctct.id AND ctct.c_l_id = loc.id AND l
    oc.l_si_id = site.id AND site.z_si_zo_id = z_zo.id AND z_zo.zo
    _lho_id = z_lho.id AND z_lho.lho_name LIKE :"SYS_B_0" AND cal
    SQL ordered by Reads for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Disk Reads Threshold: 1000
    CPU Elapsd
    Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
    l_req.active_flag = :"SYS_B_1" ) AND ( ( call_req.group_id != :"
    1,540 1 1,540.0 1.1 0.56 1.64 2582352638
    SELECT call_req.open_date, call_req.id FROM call_req, ctct, loc,
    site, z_zo, z_lho WHERE ( call_req.customer = ctct.id AND ctct
    .c_l_id = loc.id AND loc.l_si_id = site.id AND site.z_si_zo_id
    = z_zo.id AND z_zo.zo_lho_id = z_lho.id AND z_lho.lho_name L
    IKE :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1" AND ( call
    1,106 2 553.0 0.8 1.25 3.01 548248759
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( ( c
    all_req.assignee IS NOT NULL OR call_req.group_id IS NOT NULL )
    AND ( call_req.type = :"SYS_B_00" OR call_req.type = :"SYS_B_01"
    OR call_req.type IS NULL ) AND call_req.active_flag = :"SYS_B_0
    2" ) AND ( call_req.group_id IN ( SELECT id FROM ctct WHERE id =
    875 2 437.5 0.6 0.94 2.95 1195215130
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( ( c
    all_req.assignee IS NULL AND call_req.group_id IS NULL ) AND ( c
    all_req.type = :"SYS_B_00" OR call_req.type = :"SYS_B_01" OR cal
    l_req.type IS NULL ) AND call_req.active_flag = :"SYS_B_02" ) AN
    D ( call_req.group_id IN ( SELECT id FROM ctct WHERE id = :"SYS_
    473 1 473.0 0.3 1.80 5.57 3376831664
    BEGIN statspack.snap; END;
    357 10 35.7 0.3 7.69 8.21 3407388950
    SELECT count(*) FROM call_req, ctct, loc, site, z_zo, z_lho WHER
    E ( call_req.customer = ctct.id AND ctct.c_l_id = loc.id AND l
    oc.l_si_id = site.id AND site.z_si_zo_id = z_zo.id AND z_zo.zo
    _lho_id = z_lho.id AND z_lho.lho_name LIKE :"SYS_B_0" ) AND (
    ( call_req.group_id != :"SYS_B_1" ) and ( call_req.group_id !=
    177 5 35.4 0.1 1.81 2.08 920690862
    SELECT ctct.c_last_name, ctct.c_first_name, ctct.c_middle_name,
    ctct.c_public_phone, ctct.c_contact_num, ctct.c_org_id, ctct.c_l
    _id, ctct.id FROM ctct, ct_ty WHERE ( ctct.c_ctp_id = ct_ty.id A
    SQL ordered by Executions for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Executions Threshold: 100
    CPU per Elap per
    Executions Rows Processed Rows per Exec Exec (s) Exec (s) Hash Value
    7,741 7,738 1.0 0.00 0.00 1060224445
    SELECT grpmem.group_id , grpmem.manager_flag , grpmem.member , g
    rpmem.notify_flag FROM grpmem WHERE grpmem.id = :"SYS_B_0"
    2,459 2,459 1.0 0.00 0.00 3026674282
    SELECT act_log.action_desc , act_log.analyst , act_log.call_req_
    id , act_log.description , act_log.internal , act_log.knowledge_
    session , act_log.knowledge_tool , act_log.last_mod_dt , act_log
    .persid , act_log.system_time , act_log.time_spent , act_log.tim
    e_stamp , act_log.type FROM act_log WHERE act_log.id = :"SYS_B_0
    1,449 1,449 1.0 0.00 0.00 3299996875
    SELECT att_evt.cancel_time , att_evt.event_tmpl , att_evt.fire_t
    ime , att_evt.first_fire_time , att_evt.group_name , att_evt.las
    t_mod_dt , att_evt.num_fire , att_evt.obj_id , att_evt.persid ,
    att_evt.start_time , att_evt.status_flag , att_evt.user_smag , a
    tt_evt.wait_time FROM att_evt WHERE att_evt.id = :"SYS_B_0"
    1,336 1,336 1.0 0.00 0.00 3034229510
    SELECT cr_prp.description , cr_prp.label , cr_prp.last_mod_by ,
    cr_prp.last_mod_dt , cr_prp.owning_cr , cr_prp.persid , cr_prp.r
    equired , cr_prp.sample , cr_prp.sequence , cr_prp.value FROM cr
    prp WHERE crprp.id = :"SYS_B_0"
    968 968 1.0 0.00 0.00 3460529092
    select t.name, (select owner_instance from sys.aq$_queue_table_
    affinities where table_objno = t.objno) from system.aq$_queue
    _tables t where t.name = :1 and t.schema = :2 for update skip lo
    cked
    808 808 1.0 0.00 0.00 3346182257
    SELECT call_req.active_flag , call_req.affected_rc , call_req.as
    signee , call_req.call_back_date , call_req.call_back_flag , cal
    l_req.category , call_req.change , call_req.charge_back_id , cal
    l_req.close_date , call_req.created_via , call_req.customer , ca
    ll_req.description , call_req.event_token , call_req.extern_ref
    720 720 1.0 0.00 0.00 140137628
    Module: Spotlight On Oracle, classic
    SELECT DECODE(:b1,'BL','Buffer hash table instance lock','CF','C
    ontrol file schema global enqueue lock','CI','Cross-instance fun
    ction invocation instance lock','CS','Control file schema global
    enqueue lock','CU','Cursor bind lock','DF','Data file instance
    lock','DL','Direct loader parallel index create','DM','Mount/sta
    718 718 1.0 0.00 0.00 4078915446
    SELECT options.app_name, options.sym, options.id FROM options WH
    ERE ( options.sym = :"SYS_B_0" ) AND ( options.del = :"SYS_B_1"
    ) ORDER BY options.app_name
    634 634 1.0 0.00 0.00 1199698393
    SELECT loc.alias , loc.del , loc.l_addr1 , loc.l_addr2 , loc.l_a
    ddr3 , loc.l_addr4 , loc.l_addr5 , loc.l_addr6 , loc.l_details ,
    loc.l_name , loc.l_si_id , loc.last_mod , loc.persid , loc.z_cb
    SQL ordered by Executions for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Executions Threshold: 100
    CPU per Elap per
    Executions Rows Processed Rows per Exec Exec (s) Exec (s) Hash Value
    l1 , loc.zcb_l2 , loc.z_cb_l3 , loc.z_l_code , loc.z_ro_code ,
    loc.z_zo_code FROM loc WHERE loc.id = :"SYS_B_0"
    531 208 0.4 0.00 0.00 800192270
    SELECT lrel.l_persid, lrel.l_attr, lrel.l_sql, lrel.r_persid, lr
    el.r_attr, lrel.r_sql, lrel.id FROM lrel WHERE lrel.l_persid = :
    "SYS_B_0" and lrel.l_attr = :"SYS_B_1" ORDER BY lrel.l_persid ,
    lrel.l_attr , lrel.l_sql
    438 438 1.0 0.00 0.00 1317334374
    Select PROPERTY_NAME,PROPERTY_VALUE,PROPERTY_TYPE from CI_PROPER
    TIES where PROPERTY_NAME=:"SYS_B_0"
    429 8,151 19.0 0.00 0.00 1976028604
    SELECT cr_stat.sym, cr_stat.code FROM cr_stat WHERE cr_stat.del
    = :"SYS_B_0" ORDER BY cr_stat.sym
    383 383 1.0 0.00 0.00 2599265718
    DELETE FROM anima WHERE id = :"SYS_B_0"
    359 359 1.0 0.00 0.00 1719939797
    DELETE FROM att_evt WHERE id = :"SYS_B_0"
    337 337 1.0 0.00 0.00 3069423312
    SELECT anima.a_act , anima.a_delta , anima.a_lock , anima.a_name
    , anima.a_org , anima.a_string , anima.a_time , anima.t_method
    , anima.t_persid , anima.t_type FROM anima WHERE anima.id = :"SY
    S_B_0"
    332 331 1.0 0.00 0.00 1549656119
    SELECT crsq.id FROM crsq WHERE crsq.code = :"SYS_B_0"
    315 315 1.0 0.00 0.00 1734736338
    UPDATE cr_prp SET last_mod_by = :"SYS_B_0" , last_mod_dt = :"SYS
    _B_1" WHERE id = :"SYS_B_2"
    308 1,580 5.1 0.00 0.00 618252548
    SELECT cr_prp.sequence, cr_prp.id FROM cr_prp WHERE cr_prp.ownin
    g_cr = :"SYS_B_0" ORDER BY cr_prp.sequence
    279 1,716 6.2 0.00 0.00 749386807
    SELECT call_req.open_date, call_req.id FROM call_req WHERE call_
    req.customer = :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1"
    ORDER BY call_req.open_date DESC
    277 277 1.0 0.00 0.00 321149819
    INSERT INTO anima ( a_act, a_delta, a_lock, a_name, a_org, a_str
    ing, a_time, t_method, t_persid, t_type, id ) VALUES ( :"SYS_B_
    0" , :"SYS_B_1" , :"SYS_B_2" , :"SYS_B_3" , :"SYS_B_4" , nu
    ll , :"SYS_B_5" , :"SYS_B_6" , :"SYS_B_7" , :"SYS_B_8" , :"
    SQL ordered by Parse Calls for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Parse Calls Threshold: 1000
    % Total
    Parse Calls Executions Parses Hash Value
    7,733 7,741 27.15 1060224445
    SELECT grpmem.group_id , grpmem.manager_flag , grpmem.member , g
    rpmem.notify_flag FROM grpmem WHERE grpmem.id = :"SYS_B_0"
    2,459 2,459 8.63 3026674282
    SELECT act_log.action_desc , act_log.analyst , act_log.call_req_
    id , act_log.description , act_log.internal , act_log.knowledge_
    session , act_log.knowledge_tool , act_log.last_mod_dt , act_log
    .persid , act_log.system_time , act_log.time_spent , act_log.tim
    e_stamp , act_log.type FROM act_log WHERE act_log.id = :"SYS_B_0
    1,449 1,449 5.09 3299996875
    SELECT att_evt.cancel_time , att_evt.event_tmpl , att_evt.fire_t
    ime , att_evt.first_fire_time , att_evt.group_name , att_evt.las
    t_mod_dt , att_evt.num_fire , att_evt.obj_id , att_evt.persid ,
    att_evt.start_time , att_evt.status_flag , att_evt.user_smag , a
    tt_evt.wait_time FROM att_evt WHERE att_evt.id = :"SYS_B_0"
    1,336 1,336 4.69 3034229510
    SELECT cr_prp.description , cr_prp.label , cr_prp.last_mod_by ,
    cr_prp.last_mod_dt , cr_prp.owning_cr , cr_prp.persid , cr_prp.r
    equired , cr_prp.sample , cr_prp.sequence , cr_prp.value FROM cr
    prp WHERE crprp.id = :"SYS_B_0"
    808 808 2.84 3346182257
    SELECT call_req.active_flag , call_req.affected_rc , call_req.as
    signee , call_req.call_back_date , call_req.call_back_flag , cal
    l_req.category , call_req.change , call_req.charge_back_id , cal
    l_req.close_date , call_req.created_via , call_req.customer , ca
    ll_req.description , call_req.event_token , call_req.extern_ref
    718 718 2.52 4078915446
    SELECT options.app_name, options.sym, options.id FROM options WH
    ERE ( options.sym = :"SYS_B_0" ) AND ( options.del = :"SYS_B_1"
    ) ORDER BY options.app_name
    634 634 2.23 1199698393
    SELECT loc.alias , loc.del , loc.l_addr1 , loc.l_addr2 , loc.l_a
    ddr3 , loc.l_addr4 , loc.l_addr5 , loc.l_addr6 , loc.l_details ,
    loc.l_name , loc.l_si_id , loc.last_mod , loc.persid , loc.z_cb
    l1 , loc.zcb_l2 , loc.z_cb_l3 , loc.z_l_code , loc.z_ro_code ,
    loc.z_zo_code FROM loc WHERE loc.id = :"SYS_B_0"
    531 531 1.86 800192270
    SELECT lrel.l_persid, lrel.l_attr, lrel.l_sql, lrel.r_persid, lr
    el.r_attr, lrel.r_sql, lrel.id FROM lrel WHERE lrel.l_persid = :
    "SYS_B_0" and lrel.l_attr = :"SYS_B_1" ORDER BY lrel.l_persid ,
    lrel.l_attr , lrel.l_sql
    438 438 1.54 1317334374
    Select PROPERTY_NAME,PROPERTY_VALUE,PROPERTY_TYPE from CI_PROPER
    TIES where PROPERTY_NAME=:"SYS_B_0"
    429 429 1.51 1976028604
    SQL ordered by Parse Calls for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Parse Calls Threshold: 1000
    % Total
    Parse Calls Executions Parses Hash Value
    SELECT cr_stat.sym, cr_stat.code FROM cr_stat WHERE cr_stat.del
    = :"SYS_B_0" ORDER BY cr_stat.sym
    383 383 1.34 2599265718
    DELETE FROM anima WHERE id = :"SYS_B_0"
    359 359 1.26 1719939797
    DELETE FROM att_evt WHERE id = :"SYS_B_0"
    337 337 1.18 3069423312
    SELECT anima.a_act , anima.a_delta , anima.a_lock , anima.a_name
    , anima.a_org , anima.a_string , anima.a_time , anima.t_method
    , anima.t_persid , anima.t_type FROM anima WHERE anima.id = :"SY
    S_B_0"
    330 332 1.16 1549656119
    SELECT crsq.id FROM crsq WHERE crsq.code = :"SYS_B_0"
    315 315 1.11 1734736338
    UPDATE cr_prp SET last_mod_by = :"SYS_B_0" , last_mod_dt = :"SYS
    _B_1" WHERE id = :"SYS_B_2"
    308 308 1.08 618252548
    SELECT cr_prp.sequence, cr_prp.id FROM cr_prp WHERE cr_prp.ownin
    g_cr = :"SYS_B_0" ORDER BY cr_prp.sequence
    277 277 0.97 321149819
    INSERT INTO anima ( a_act, a_delta, a_lock, a_name, a_org, a_str
    ing, a_time, t_method, t_persid, t_type, id ) VALUES ( :"SYS_B_
    0" , :"SYS_B_1" , :"SYS_B_2" , :"SYS_B_3" , :"SYS_B_4" , nu
    ll , :"SYS_B_5" , :"SYS_B_6" , :"SYS_B_7" , :"SYS_B_8" , :"
    SYS_B_9" )
    277 279 0.97 749386807
    SELECT call_req.open_date, call_req.id FROM call_req WHERE call_
    req.customer = :"SYS_B_0" AND call_req.active_flag = :"SYS_B_1"
    ORDER BY call_req.open_date DESC
    275 275 0.97 2816620377
    INSERT INTO att_evt ( cancel_time, event_tmpl, fire_time, first_
    fire_time, group_name, last_mod_dt, num_fire, obj_id, persid, st
    art_time, status_flag, user_smag, wait_time, id ) VALUES ( null
    , :"SYS_B_00" , :"SYS_B_01" , :"SYS_B_02" , :"SYS_B_03" ,
    :"SYS_B_04" , :"SYS_B_05" , :"SYS_B_06" , :"SYS_B_07" , :"SY
    269 269 0.94 3605948696
    SELECT slatpl.del , slatpl.elapsed , slatpl.event , slatpl.last_
    mod_by , slatpl.last_mod_dt , slatpl.object_type , slatpl.persid
    , slatpl.service_type , slatpl.sym FROM slatpl WHERE slatpl.id
    SQL ordered by Sharable Memory for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Sharable Memory Threshold: 1048576
    Sharable Mem (b) Executions % Total Hash Value
    23,912,520 231 13.6 139964375
    SELECT anima.a_name, anima.t_persid, anima.t_method, anima.id FR
    OM anima WHERE anima.t_persid LIKE :"SYS_B_0" ORDER BY anima.
    a_name
    18,314,292 26 10.4 380755726
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.ref_num LIKE :"SYS_B_00" ) AND ( call_req.group_id IN (
    SELECT id FROM ctct WHERE id = :"SYS_B_01" OR id = :"SYS_B_02" O
    R id = :"SYS_B_03" OR id = :"SYS_B_04" OR id = :"SYS_B_05" OR id
    = :"SYS_B_06" OR id = :"SYS_B_07" OR id = :"SYS_B_08" OR id = :
    12,365,844 107 7.0 1877135209
    SELECT chg.open_date, chg.chg_ref_num, chg.id FROM chg WHERE ( c
    hg.affected_contact = :"SYS_B_0" and chg.active_flag = :"SYS_B_1
    " ) AND ( chg.affected_contact = :"SYS_B_2" ) ORDER BY chg.open
    _date DESC
    2,692,852 17 1.5 4181730075
    SELECT ctct.c_last_name, ctct.c_first_name, ctct.c_middle_name,
    ctct.c_public_phone, ctct.c_contact_num, ctct.c_org_id, ctct.c_l
    _id, ctct.id FROM ctct, ct_ty WHERE ( ctct.c_last_name LIKE :"
    SYS_B_0" AND ctct.c_ctp_id = ct_ty.id AND ct_ty.id = :"SYS_B_1"
    AND ctct.del = :"SYS_B_2" AND ctct.id IN ( SELECT member FROM g
    2,048,083 10 1.2 153455816
    SELECT ctct.c_last_name, ctct.c_first_name, ctct.c_middle_name,
    ctct.c_public_phone, ctct.c_contact_num, ctct.c_org_id, ctct.c_l
    _id, ctct.id FROM ctct WHERE ( ctct.c_last_name LIKE :"SYS_B_0
    " ) AND ( ( ctct.del = :"SYS_B_1" ) AND ( ctct.c_ctp_id = :"SYS_
    B_2" AND ctct.alias = -:"SYS_B_3" ) ) ORDER BY ctct.c_last_name
    1,653,628 3 0.9 1096419296
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.ref_num LIKE :"SYS_B_0" ) AND ( ( call_req.group_id IN (
    SELECT group_id FROM grpmem WHERE member = :"SYS_B_1" ) ) or ca
    ll_req.assignee = :"SYS_B_2" or call_req.customer = :"SYS_B_3" )
    ORDER BY call_req.open_date DESC
    SQL ordered by Version Count for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Version Count Threshold: 20
    Version
    Count Executions Hash Value
    349 231 139964375
    SELECT anima.a_name, anima.t_persid, anima.t_method, anima.id FR
    OM anima WHERE anima.t_persid LIKE :"SYS_B_0" ORDER BY anima.
    a_name
    196 107 1877135209
    SELECT chg.open_date, chg.chg_ref_num, chg.id FROM chg WHERE ( c
    hg.affected_contact = :"SYS_B_0" and chg.active_flag = :"SYS_B_1
    " ) AND ( chg.affected_contact = :"SYS_B_2" ) ORDER BY chg.open
    _date DESC
    127 26 380755726
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.ref_num LIKE :"SYS_B_00" ) AND ( call_req.group_id IN (
    SELECT id FROM ctct WHERE id = :"SYS_B_01" OR id = :"SYS_B_02" O
    R id = :"SYS_B_03" OR id = :"SYS_B_04" OR id = :"SYS_B_05" OR id
    = :"SYS_B_06" OR id = :"SYS_B_07" OR id = :"SYS_B_08" OR id = :
    36 17 4181730075
    SELECT ctct.c_last_name, ctct.c_first_name, ctct.c_middle_name,
    ctct.c_public_phone, ctct.c_contact_num, ctct.c_org_id, ctct.c_l
    _id, ctct.id FROM ctct, ct_ty WHERE ( ctct.c_last_name LIKE :"
    SYS_B_0" AND ctct.c_ctp_id = ct_ty.id AND ct_ty.id = :"SYS_B_1"
    AND ctct.del = :"SYS_B_2" AND ctct.id IN ( SELECT member FROM g
    33 10 153455816
    SELECT ctct.c_last_name, ctct.c_first_name, ctct.c_middle_name,
    ctct.c_public_phone, ctct.c_contact_num, ctct.c_org_id, ctct.c_l
    _id, ctct.id FROM ctct WHERE ( ctct.c_last_name LIKE :"SYS_B_0
    " ) AND ( ( ctct.del = :"SYS_B_1" ) AND ( ctct.c_ctp_id = :"SYS_
    B_2" AND ctct.alias = -:"SYS_B_3" ) ) ORDER BY ctct.c_last_name
    26 3 1096419296
    SELECT call_req.open_date, call_req.id FROM call_req WHERE ( cal
    l_req.ref_num LIKE :"SYS_B_0" ) AND ( ( call_req.group_id IN (
    SELECT group_id FROM grpmem WHERE member = :"SYS_B_1" ) ) or ca
    ll_req.assignee = :"SYS_B_2" or call_req.customer = :"SYS_B_3" )
    ORDER BY call_req.open_date DESC
    Instance Activity Stats for DB: AHD Instance: ahd Snaps: 20 -21
    Statistic Total per Second per Trans
    CPU used by this session 12,450 8.7 4.1
    CPU used when call started 12,515 8.8 4.1
    CR blocks created 53 0.0 0.0
    DBWR buffers scanned 0 0.0 0.0
    DBWR checkpoint buffers written 1,644 1.2 0.5
    DBWR checkpoints 0 0.0 0.0
    DBWR free buffers found 0 0.0 0.0
    DBWR lru scans 0 0.0 0.0
    DBWR make free requests 0 0.0 0.0
    DBWR summed scan depth 0 0.0 0.0
    DBWR transaction table writes 10 0.0 0.0
    DBWR undo block writes 238 0.2 0.1
    SQL*Net roundtrips to/from client 131,833 92.6 42.9
    active txn count during cleanout 130 0.1 0.0
    background checkpoints completed 0 0.0 0.0
    background checkpoints started 0 0.0 0.0
    background timeouts 2,161 1.5 0.7
    branch node splits 0 0.0 0.0
    buffer is not pinned count 3,147,925 2,210.6 1,024.1
    buffer is pinned count 638,155 448.1 207.6
    bytes received via SQL*Net from c 20,116,711 14,126.9 6,544.2
    bytes sent via SQL*Net to client 33,961,169 23,849.1 11,047.9
    calls to get snapshot scn: kcmgss 76,324 53.6 24.8
    calls to kcmgas 6,266 4.4 2.0
    calls to kcmgcs 110 0.1 0.0
    change write time 25 0.0 0.0
    cleanout - number of ktugct calls 145 0.1 0.1
    cleanouts and rollbacks - consist 0 0.0 0.0
    cleanouts only - consistent read 0 0.0 0.0
    cluster key scan block gets 1,361 1.0 0.4
    cluster key scans 1,146 0.8 0.4
    commit cleanout failures: buffer 0 0.0 0.0
    commit cleanout failures: callbac 3 0.0 0.0
    commit cleanout failures: cannot 0 0.0 0.0
    commit cleanouts 14,837 10.4 4.8
    commit cleanouts successfully com 14,834 10.4 4.8
    commit txn count during cleanout 106 0.1 0.0
    consistent changes 2,123 1.5 0.7
    consistent gets 3,336,864 2,343.3 1,085.5
    consistent gets - examination 197,061 138.4 64.1
    cursor authentications 71 0.1 0.0
    data blocks consistent reads - un 2,123 1.5 0.7
    db block changes 50,525 35.5 16.4
    db block gets 47,774 33.6 15.5
    deferred (CURRENT) block cleanout 7,940 5.6 2.6
    dirty buffers inspected 0 0.0 0.0
    enqueue conversions 29 0.0 0.0
    enqueue releases 14,210 10.0 4.6
    enqueue requests 14,210 10.0 4.6
    enqueue waits 0 0.0 0.0
    execute count 32,955 23.1 10.7
    free buffer inspected 16 0.0 0.0
    free buffer requested 140,283 98.5 45.6
    hot buffers moved to head of LRU 950 0.7 0.3
    immediate (CR) block cleanout app 0 0.0 0.0
    immediate (CURRENT) block cleanou 2,804 2.0 0.9
    Instance Activity Stats for DB: AHD Instance: ahd Snaps: 20 -21
    Statistic Total per Second per Trans
    index fast full scans (full) 157 0.1 0.1
    index fetch by key 70,378 49.4 22.9
    index scans kdiixs1 28,181 19.8 9.2
    leaf node 90-10 splits 10 0.0 0.0
    leaf node splits 76 0.1 0.0
    logons cumulative 11 0.0 0.0
    messages received 5,452 3.8 1.8
    messages sent 5,452 3.8 1.8
    no buffer to keep pinned count 0 0.0 0.0
    no work - consistent read gets 3,085,481 2,166.8 1,003.7
    opened cursors cumulative 4,561 3.2 1.5
    parse count (failures) 0 0.0 0.0
    parse count (hard) 412 0.3 0.1
    parse count (total) 28,484 20.0 9.3
    parse time cpu 426 0.3 0.1
    parse time elapsed 469 0.3 0.2
    physical reads 138,930 97.6 45.2
    physical reads direct 0 0.0 0.0
    physical writes 1,644 1.2 0.5
    physical writes direct 0 0.0 0.0
    physical writes non checkpoint 232 0.2 0.1
    pinned buffers inspected 7 0.0 0.0
    prefetched blocks 32,732 23.0 10.7
    process last non-idle time 12,884,949,552 9,048,419.6 4,191,590.6
    recursive calls 22,718 16.0 7.4
    recursive cpu usage 226 0.2 0.1
    redo blocks written 19,178 13.5 6.2
    redo buffer allocation retries 0 0.0 0.0
    redo entries 27,265 19.2 8.9
    redo log space requests 0 0.0 0.0
    redo log space wait time 0 0.0 0.0
    redo size 8,488,216 5,960.8 2,761.3
    redo synch time 74 0.1 0.0
    redo synch writes 3,078 2.2 1.0
    redo wastage 1,040,788 730.9 338.6
    redo write time 75 0.1 0.0
    redo writer latching time 0 0.0 0.0
    redo writes 5,216 3.7 1.7
    rollback changes - undo records a 6 0.0 0.0
    rollbacks only - consistent read 233 0.2 0.1
    rows fetched via callback 54,581 38.3 17.8
    session connect time 12,884,949,552 9,048,419.6 4,191,590.6
    session logical reads 3,384,638 2,376.9 1,101.1
    session pga memory max 6,168,536 4,331.8 2,006.7
    session uga memory 599,984 421.3 195.2
    session uga memory max 9,592,864 6,736.6 3,120.7
    shared hash latch upgrades - no w 27,737 19.5 9.0
    shared hash latch upgrades - wait 84 0.1 0.0
    sorts (disk) 0 0.0 0.0
    sorts (memory) 6,834 4.8 2.2
    sorts (rows) 3,229,994 2,268.3 1,050.8
    summed dirty queue length 0 0.0 0.0
    switch current to new buffer 990 0.7 0.3
    table fetch by rowid 474,673 333.3 154.4
    table fetch continued row 8 0.0 0.0
    table scan blocks gotten 2,751,375 1,932.2 895.1
    Instance Activity Stats for DB: AHD Instance: ahd Snaps: 20 -21
    Statistic Total per Second per Trans
    table scan rows gotten 55,928,200 39,275.4 18,194.0
    table scans (long tables) 245 0.2 0.1
    table scans (short tables) 3,383 2.4 1.1
    transaction rollbacks 3 0.0 0.0
    transaction tables consistent rea 0 0.0 0.0
    transaction tables consistent rea 0 0.0 0.0
    user calls 131,904 92.6 42.9
    user commits 3,074 2.2 1.0
    user rollbacks 0 0.0 0.0
    workarea executions - onepass 0 0.0 0.0
    workarea executions - optimal 8,438 5.9 2.7
    write clones created in backgroun 0 0.0 0.0
    write clones created in foregroun 0 0.0 0.0
    Tablespace IO Stats for DB: AHD Instance: ahd Snaps: 20 -21
    ->ordered by IOs (Reads + Writes) desc
    Tablespace
    Av Av Av Av Buffer Av Buf
    Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)
    AHD1_DATA
    105,869 74 0.9 1.3 828 1 11,740 0.5
    AHD1_IDX
    38 0 7.4 1.0 563 0 0 0.0
    PERFSTAT
    372 0 3.6 1.0 0 0 0 0.0
    UNDOTBS1
    0 0 0.0 248 0 0 0.0
    SYSTEM
    6 0 6.7 1.0 5 0 0 0.0
    File IO Stats for DB: AHD Instance: ahd Snaps: 20 -21
    ->ordered by Tablespace, File
    Tablespace Filename
    Av Av Av Av Buffer Av Buf
    Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)
    AHD1_DATA E:\ORACLE\ORADATA\AHD\AHD1_DATA.ORA
    53,454 38 0.9 1.3 432 0 5,949 0.5
    E:\ORACLE\ORADATA\AHD\AHD2_DATA.ORA
    52,415 37 0.9 1.3 396 0 5,791 0.5
    AHD1_IDX E:\ORACLE\ORADATA\AHD\AHD1_IDX.ORA
    38 0 7.4 1.0 563 0 0
    PERFSTAT E:\ORACLE\ORADATA\AHD\PERFSTAT.ORA
    372 0 3.6 1.0 0 0 0
    SYSTEM E:\ORACLE\ORADATA\AHD\SYSTEM01.DBF
    6 0 6.7 1.0 5 0 0
    UNDOTBS1 E:\ORACLE\ORADATA\AHD\UNDOTBS01.DBF
    0 0 248 0 0
    Buffer Pool Statistics for DB: AHD Instance: ahd Snaps: 20 -21
    -> Standard block size Pools D: default, K: keep, R: recycle
    -> Default Pools for other block sizes: 2k, 4k, 8k, 16k, 32k
    Free Write Buffer
    Number of Cache Buffer Physical Physical Buffer Complete Busy
    P Buffers Hit % Gets Reads Writes Waits Waits Waits
    D 82,082 97.8 6,327,007 138,971 1,644 0 0 11,760
    Instance Recovery Stats for DB: AHD Instance: ahd Snaps: 20 -21
    -> B: Begin snapshot, E: End snapshot
    Targt Estd Log File Log Ckpt Log Ckpt
    MTTR MTTR Recovery Actual Target Size Timeout Interval
    (s) (s) Estd IOs Redo Blks Redo Blks Redo Blks Redo Blks Redo Blks
    B 75 26 2354 18057 17632 184320 17632
    E 75 27 2967 23569 22952 184320 22952
    Buffer Pool Advisory for DB: AHD Instance: ahd End Snap: 21
    -> Only rows with estimated physical reads >0 are displayed
    -> ordered by Block Size, Buffers For Estimate
    Size for Size Buffers for Est Physical Estimated
    P Estimate (M) Factr Estimate Read Factor Physical Reads
    D 64 .1 8,008 261.38 4,357,231,706
    D 128 .2 16,016 207.44 3,458,029,385
    D 192 .3 24,024 143.22 2,387,570,894
    D 256 .4 32,032 2.29 38,243,018
    D 320 .5 40,040 1.89 31,541,321
    D 384 .6 48,048 1.74 29,023,767
    D 448 .7 56,056 1.69 28,232,064
    D 512 .8 64,064 1.20 19,951,481
    D 576 .9 72,072 1.11 18,529,925
    D 640 1.0 80,080 1.04 17,367,752
    D 656 1.0 82,082 1.00 16,670,129
    D 704 1.1 88,088 0.97 16,124,256
    D 768 1.2 96,096 0.91 15,155,822
    D 832 1.3 104,104 0.90 15,055,099
    D 896 1.4 112,112 0.89 14,839,567
    D 960 1.5 120,120 0.88 14,668,682
    D 1,024 1.6 128,128 0.87 14,479,726
    D 1,088 1.7 136,136 0.84 13,988,866
    D 1,152 1.8 144,144 0.70 11,723,518
    D 1,216 1.9 152,152 0.61 10,156,857
    D 1,280 2.0 160,160 0.20 3,281,883
    Buffer wait Statistics for DB: AHD Instance: ahd Snaps: 20 -21
    -> ordered by wait time desc, waits desc
    Tot Wait Avg
    Class Waits Time (s) Time (ms)
    data block 11,754 6 0
    PGA Aggr Target Stats for DB: AHD Instance: ahd Snaps: 20 -21
    -> B: Begin snap E: End snap (rows dentified with B or E contain data
    which is absolute i.e. not diffed over the interval)
    -> PGA cache hit % - percentage of W/A (WorkArea) data processed only in-memory
    -> Auto PGA Target - actual workarea memory target
    -> W/A PGA Used - amount of memory used for all Workareas (manual + auto)
    -> %PGA W/A Mem - percentage of PGA memory allocated to workareas
    -> %Auto W/A Mem - percentage of workarea memory controlled by Auto Mem Mgmt
    -> %Man W/A Mem - percentage of workarea memory under manual control
    PGA Cache Hit % W/A MB Processed Extra W/A MB Read/Written
    100.0 1,169 0
    %PGA %Auto %Man
    PGA Aggr Auto PGA PGA Mem W/A PGA W/A W/A W/A Global Mem
    Target(M) Target(M) Alloc(M) Used(M) Mem Mem Mem Bound(K)
    B 350 293 37.6 0.0 .0 .0 .0 17,920
    E 350 293 37.5 0.2 .6 100.0 .0 17,920
    PGA Aggr Target Histogram for DB: AHD Instance: ahd Snaps: 20 -21
    -> Optimal Executions are purely in-memory operations
    Low High
    Optimal Optimal Total Execs Optimal Execs 1-Pass Execs M-Pass Execs
    8K 16K 6,809 6,809 0 0
    16K 32K 148 148 0 0
    32K 64K 90 90 0 0
    64K 128K 154 154 0 0
    128K 256K 73 73 0 0
    256K 512K 308 308 0 0
    512K 1024K 374 374 0 0
    1M 2M 171 171 0 0
    2M 4M 217 217 0 0
    4M 8M 10 10 0 0
    PGA Memory Advisory for DB: AHD Instance: ahd End Snap: 21
    -> When using Auto Memory Mgmt, minimally choose a pga_aggregate_target value
    where Estd PGA Overalloc Count is 0
    Estd Extra Estd PGA Estd PGA
    PGA Target Size W/A MB W/A MB Read/ Cache Overalloc
    Est (MB) Factr Processed Written to Disk Hit % Count
    44 0.1 180,060.5 42,218.7 81.0 4
    88 0.3 180,060.5 23,194.7 89.0 0
    175 0.5 180,060.5 9,436.8 95.0 0
    263 0.8 180,060.5 9,356.7 95.0 0
    350 1.0 180,060.5 9,274.8 95.0 0
    420 1.2 180,060.5 9,169.9 95.0 0
    490 1.4 180,060.5 9,148.0 95.0 0
    560 1.6 180,060.5 9,148.0 95.0 0
    630 1.8 180,060.5 9,148.0 95.0 0
    700 2.0 180,060.5 9,148.0 95.0 0
    1,050 3.0 180,060.5 9,148.0 95.0 0
    1,400 4.0 180,060.5 9,148.0 95.0 0
    2,100 6.0 180,060.5 3,983.3 98.0 0
    2,800 8.0 180,060.5 3,983.3 98.0 0
    Rollback Segment Stats for DB: AHD Instance: ahd Snaps: 20 -21
    ->A high value for "Pct Waits" suggests more rollback segments may be required
    ->RBS stats may not be accurate between begin and end snaps when using Auto Undo
    managment, as RBS may be dynamically created and dropped as needed
    Trans Table Pct Undo Bytes
    RBS No Gets Waits Written Wraps Shrinks Extends
    0 29.0 0.00 0 0 0 0
    1 975.0 0.00 122,796 0 0 0
    2 1,244.0 0.00 1,094,706 10 0 5
    3 816.0 0.00 118,596 0 0 0
    4 1,430.0 0.00 212,754 2 0 0
    5 1,716.0 0.00 291,940 2 0 0
    6 1,287.0 0.00 197,900 0 0 0
    7 1,674.0 0.00 279,160 0 0 0
    8 1,031.0 0.00 148,216 0 0 0
    9 947.0 0.00 141,870 0 0 0
    10 834.0 0.00 117,422 0 0 0
    Rollback Segment Storage for DB: AHD Instance: ahd Snaps: 20 -21
    ->Optimal Size should be larger than Avg Active
    RBS No Segment Size Avg Active Optimal Size Maximum Size
    0 385,024 0 385,024
    1 2,220,032 455,412 2,220,032
    2 2,088,960 333,026 2,220,032
    3 2,220,032 456,101 2,220,032
    4 2,220,032 474,584 3,268,608
    5 2,220,032 480,865 3,268,608
    6 2,220,032 513,967 3,268,608
    7 2,220,032 480,785 2,220,032
    8 2,220,032 496,182 2,220,032
    9 2,220,032 486,763 2,220,032
    10 2,220,032 430,016 6,414,336
    Undo Segment Summary for DB: AHD Instance: ahd Snaps: 20 -21
    -> Undo segment block stats:
    -> uS - unexpired Stolen, uR - unexpired Released, uU - unexpired reUsed
    -> eS - expired Stolen, eR - expired Released, eU - expired reUsed
    Undo Undo Num Max Qry Max Tx Snapshot Out of uS/uR/uU/
    TS# Blocks Trans Len (s) Concurcy Too Old Space eS/eR/eU
    1 395 2,900,725 5 1 0 0 0/0/0/0/0/0
    Undo Segment Stats for DB: AHD Instance: ahd Snaps: 20 -21
    -> ordered by Time desc
    Undo Num Max Qry Max Tx Snap Out of uS/uR/uU/
    End Time Blocks Trans Len (s) Concy Too Old Space eS/eR/eU
    13-Feb 15:04 96 ######## 4 1 0 0 0/0/0/0/0/0
    13-Feb 14:54 299 ######## 5 1 0 0 0/0/0/0/0/0
    Latch Activity for DB: AHD Instance: ahd Snaps: 20 -21
    ->"Get Requests", "Pct Get Miss" and "Avg Slps/Miss" are statistics for
    willing-to-wait latch get requests
    ->"NoWait Requests", "Pct NoWait Miss" are for no-wait latch get requests
    ->"Pct Misses" for both should be very close to 0.0
    Pct Avg Wait Pct
    Get Get Slps Time NoWait NoWait
    Latch Requests Miss /Miss (s) Requests Miss
    Consistent RBA 5,216 0.0 0 0
    FOB s.o list latch 34 0.0 0 0
    SQL memory manager latch 1 0.0 0 462 0.0
    SQL memory manager worka 40,347 0.0 0 0
    active checkpoint queue 1,261 0.0 0 0
    archive control 163 0.0 0 0
    archive process latch 29 0.0 0 0
    cache buffer handles 378 0.0 0 0
    cache buffers chains 6,836,244 0.4 0.0 0 266,617 0.0
    cache buffers lru chain 244,157 0.0 0.0 0 140,432 0.0
    channel handle pool latc 21 0.0 0 0
    channel operations paren 960 0.0 0 0
    checkpoint queue latch 86,982 0.0 0 2,337 0.0
    child cursor hash table 6,464 0.0 0.0 0 0
    dml lock allocation 15,005 0.0 0 0
    dummy allocation 21 0.0 0 0
    enqueue hash chains 28,447 0.0 0 0
    enqueues 8,689 0.0 0 0
    event group latch 11 0.0 0 0
    file number translation 4,079 0.0 0 0
    hash table column usage 38 0.0 0 187,596 0.0
    hash table modification 1 0.0 0 0
    job_queue_processes para 23 0.0 0 0
    ktm global data 4 0.0 0 0
    kwqit: protect wakeup ti 45 0.0 0 0
    lgwr LWN SCN 5,328 0.4 0.0 0 0
    library cache 342,865 0.2 0.0 0 342 0.6
    library cache load lock 452 0.0 0 0
    library cache pin 197,662 0.0 0.0 0 0
    library cache pin alloca 124,035 0.0 0.0 0 0
    list of block allocation 55 0.0 0 0
    messages 30,779 0.0 0.0 0 0
    mostly latch-free SCN 5,459 1.8 0.0 0 0
    multiblock read objects 194,822 0.0 0.0 0 0
    ncodef allocation latch 23 0.0 0 0
    object stats modificatio 618 0.0 0 0
    post/wait queue 10,441 0.0 0 3,078 0.0
    process allocation 11 0.0 0 11 0.0
    process group creation 21 0.0 0 0
    redo allocation 37,773 0.0 0.0 0 0
    redo copy 0 0 27,274 0.0
    redo writing 17,880 0.0 0 0
    row cache enqueue latch 169,423 0.0 0.0 0 0
    row cache objects 169,795 0.0 0 3 0.0
    sequence cache 38 0.0 0 0
    session allocation 15,580 0.0 0 0
    session idle bit 269,419 0.0 0.0 0 0
    session switching 23 0.0 0 0
    session timer 478 0.0 0 0
    shared pool 104,427 0.1 0.0 0 0
    Latch Activity for DB: AHD Instance: ahd Snaps: 20 -21
    ->"Get Requests", "Pct Get Miss" and "Avg Slps/Miss" are statistics for
    willing-to-wait latch get requests
    ->"NoWait Requests", "Pct NoWait Miss" are for no-wait latch get requests
    ->"Pct Misses" for both should be very close to 0.0
    Pct Avg Wait Pct
    Get Get Slps Time NoWait NoWait
    Latch Requests Miss /Miss (s) Requests Miss
    sim partition latch 0 0 32 0.0
    simulator hash latch 217,119 0.0 0.0 0 0
    simulator lru latch 16,247 0.0 0 902 0.4
    sort extent pool 29 0.0 0 0
    transaction allocation 36 0.0 0 0
    transaction branch alloc 23 0.0 0 0
    undo global data 19,973 0.0 0 0
    user lock 42 0.0 0 0
    Latch Sleep breakdown for DB: AHD Instance: ahd Snaps: 20 -21
    -> ordered by misses desc
    Get Spin &
    Latch Name Requests Misses Sleeps Sleeps 1->4
    cache buffers chains 6,836,244 26,201 46 0/0/0/0/0
    library cache 342,865 778 5 773/5/0/0/0
    shared pool 104,427 125 3 122/3/0/0/0
    Latch Miss Sources for DB: AHD Instance: ahd Snaps: 20 -21
    -> only latches with sleeps are shown
    -> ordered by name, sleeps desc
    NoWait Waiter
    Latch Name Where Misses Sleeps Sleeps
    cache buffers chains kcbgtcr: kslbegin excl 0 32 30
    cache buffers chains kcbrls: kslbegin 0 7 13
    cache buffers chains kcbzwb 0 4 3
    cache buffers chains kcbgtcr: fast path 0 3 0
    library cache kglic 0 2 0
    library cache kglobpn: child: 0 2 0
    library cache kgllkdl: child: cleanup 0 1 0
    shared pool kghalo 0 2 0
    shared pool kghalp 0 1 0
    Child Latch Statistics DB: AHD Instance: ahd Snaps: 20 -21
    -> only latches with sleeps/gets > 1/100000 are shown
    -> ordered by name, gets desc
    Child Get Spin &
    Latch Name Num Requests Misses Sleeps Sleeps 1->4
    cache buffers chains 439 28,269 1,276 1 0/0/0/0/0
    cache buffers chains 269 26,297 842 1 0/0/0/0/0
    cache buffers chains 1010 17,482 49 2 0/0/0/0/0
    cache buffers chains 260 11,141 20 1 0/0/0/0/0
    cache buffers chains 324 9,454 29 1 0/0/0/0/0
    cache buffers chains 840 7,235 20 1 0/0/0/0/0
    cache buffers chains 46 6,868 25 1 0/0/0/0/0
    cache buffers chains 835 6,799 26 2 0/0/0/0/0
    cache buffers chains 202 6,768 17 1 0/0/0/0/0
    cache buffers chains 740 6,573 38 2 0/0/0/0/0
    cache buffers chains 592 6,508 30 1 0/0/0/0/0
    cache buffers chains 436 6,485 25 2 0/0/0/0/0
    cache buffers chains 513 6,443 16 1 0/0/0/0/0
    cache buffers chains 844 6,436 28 1 0/0/0/0/0
    cache buffers chains 117 6,423 25 1 0/0/0/0/0
    cache buffers chains 389 6,381 25 1 0/0/0/0/0
    cache buffers chains 116 6,349 29 1 0/0/0/0/0
    cache buffers chains 51 6,340 34 1 0/0/0/0/0
    cache buffers chains 914 6,259 31 1 0/0/0/0/0
    cache buffers chains 713 6,249 24 1 0/0/0/0/0
    cache buffers chains 465 6,198 27 2 0/0/0/0/0
    cache buffers chains 416 6,193 27 1 0/0/0/0/0
    cache buffers chains 432 6,155 34 1 0/0/0/0/0
    cache buffers chains 583 6,152 23 2 0/0/0/0/0
    cache buffers chains 126 6,147 35 1 0/0/0/0/0
    cache buffers chains 879 6,043 21 1 0/0/0/0/0
    cache buffers chains 110 6,010 25 1 0/0/0/0/0
    cache buffers chains 138 6,010 25 1 0/0/0/0/0
    cache buffers chains 472 6,002 31 1 0/0/0/0/0
    cache buffers chains 908 5,964 20 1 0/0/0/0/0
    cache buffers chains 860 5,950 23 1 0/0/0/0/0
    cache buffers chains 71 5,945 29 3 0/0/0/0/0
    cache buffers chains 20 5,780 28 1 0/0/0/0/0
    cache buffers chains 932 5,759 25 1 0/0/0/0/0
    cache buffers chains 866 5,610 22 1 0/0/0/0/0
    cache buffers chains 989 5,454 34 2 0/0/0/0/0
    cache buffers chains 1005 5,434 40 1 0/0/0/0/0
    library cache 6 47,067 52 3 49/3/0/0/0
    shared pool 1 99,771 124 3 121/3/0/0/0
    Top 5 Logical Reads per Segment for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Segment Logical Reads Threshold: 10000
    Subobject Obj. Logical
    Owner Tablespace Object Name Name Type Reads %Total
    AHD AHD1_DATA CALL_REQ TABLE 1,714,928 51.19
    AHD AHD1_DATA CTCT TABLE 1,169,360 34.90
    AHD AHD1_IDX SYS_C003707 INDEX 89,152 2.66
    AHD AHD1_DATA CNOTE TABLE 66,272 1.98
    AHD AHD1_IDX CALL_REQ_X5 INDEX 61,360 1.83
    Top 5 Physical Reads per Segment for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Segment Physical Reads Threshold: 1000
    Subobject Obj. Physical
    Owner Tablespace Object Name Name Type Reads %Total
    AHD AHD1_DATA CALL_REQ TABLE 132,989 95.95
    AHD AHD1_DATA CTCT TABLE 5,325 3.84
    AHD AHD1_DATA CI_AUDIT_TRAILS_GU_I INDEX 43 .03
    AHD AHD1_DATA ACT_LOG TABLE 38 .03
    AHD AHD1_DATA CI_EXT_CALLS_GUID INDEX 36 .03
    Top 5 Buf. Busy Waits per Segment for DB: AHD Instance: ahd Snaps: 20 -21
    -> End Segment Buffer Busy Waits Threshold: 100
    Buffer
    Subobject Obj. Busy
    Owner Tablespace Object Name Name Type Waits %Total
    AHD AHD1_DATA CALL_REQ TABLE 11,751 99.95
    AHD AHD1_DATA CTCT TABLE 6 .05
    Dictionary Cache Stats for DB: AHD Instance: ahd Snaps: 20 -21
    ->"Pct Misses" should be very low (< 2% in most cases)
    ->"Cache Usage" is the number of cache entries being used
    ->"Pct SGA" is the ratio of usage to allocated size for that cache
    Get Pct Scan Pct Mod Final
    Cache Requests Miss Reqs Miss Reqs Usage
    dc_files 30 0.0 0 0 15
    dc_histogram_defs 3,022 3.9 0 0 1,919
    dc_object_ids 22,961 0.1 0 0 1,181
    dc_objects 1,092 9.2 0 0 1,026
    dc_profiles 11 0.0 0 0 1
    dc_rollback_segments 168 0.0 0 0 22
    dc_segments 5,519 0.1 0 0 1,334
    dc_sequences 1 0.0 0 1 2
    dc_tablespace_quotas 3 0.0 0 3 2
    dc_tablespaces 25,902 0.0 0 0 16
    dc_user_grants 127 0.0 0 0 22
    dc_usernames 110 0.0 0 0 18
    dc_users 26,077 0.0 0 0 30
    Library Cache Activity for DB: AHD Instance: ahd Snaps: 20 -21
    ->"Pct Misses" should be very low
    Get Pct Pin Pct Invali-
    Namespace Requests Miss Requests Miss Reloads dations
    CLUSTER 19 0.0 16 0.0 0 0
    INDEX 315 0.0 315 0.0 0 0
    SQL AREA 27,908 0.0 94,300 0.5 38 0
    TABLE/PROCEDURE 3,793 2.6 6,017 6.5 55 0
    TRIGGER 20 0.0 20 0.0 0 0
    Shared Pool Advisory for DB: AHD Instance: ahd End Snap: 21
    -> Note there is often a 1:Many correlation between a single logical object
    in the Library Cache, and the physical number of memory objects associated
    with it. Therefore comparing the number of Lib Cache objects (e.g. in
    v$librarycache), with the number of Lib Cache Memory Objects is invalid
    Estd
    Shared Pool SP Estd Estd Estd Lib LC Time
    Size for Size Lib Cache Lib Cache Cache Time Saved Estd Lib Cache
    Estim (M) Factr Size (M) Mem Obj Saved (s) Factr Mem Obj Hits
    88 .6 81 11,169 59,229 1.0 6,202,663
    104 .7 96 13,308 59,237 1.0 6,207,373
    120 .8 112 15,603 59,277 1.0 6,228,405
    136 .9 127 18,086 59,348 1.0 6,265,370
    152 1.0 142 19,501 59,379 1.0 6,295,279
    168 1.1 157 21,035 59,426 1.0 6,314,861
    184 1.2 172 22,038 59,455 1.0 6,325,903
    200 1.3 187 23,807 59,459 1.0 6,328,446
    216 1.4 202 25,911 59,460 1.0 6,329,386
    232 1.5 217 28,194 59,461 1.0 6,330,245
    248 1.6 232 29,884 59,462 1.0 6,330,914
    264 1.7 248 31,127 59,462 1.0 6,331,222
    280 1.8 263 32,878 59,463 1.0 6,331,563
    296 1.9 278 34,121 59,463 1.0 6,331,898
    312 2.1 295 36,139 59,463 1.0 6,332,102
    SGA Memory Summary for DB: AHD Instance: ahd Snaps: 20 -21
    SGA regions Size in Bytes
    Database Buffers 687,865,856
    Fixed Size 455,196
    Redo Buffers 929,792
    Variable Size 293,601,280
    sum 982,852,124
    SGA breakdown difference for DB: AHD Instance: ahd Snaps: 20 -21
    Pool Name Begin value End value % Diff
    java free memory 75,497,472 75,497,472 0.00
    large free memory 41,943,040 41,943,040 0.00
    shared 1M buffer 2,098,176 2,098,176 0.00
    shared Checkpoint queue 846,912 846,912 0.00
    shared FileOpenBlock 695,504 695,504 0.00
    shared KGK heap 3,756 3,756 0.00
    shared KGLS heap 1,230,944 1,438,740 16.88
    shared KQR L PO 2,064 2,064 0.00
    shared KQR M PO 2,480,924 2,514,220 1.34
    shared KQR S PO 383,036 383,036 0.00
    shared KQR S SO 5,636 5,636 0.00
    shared KSXR pending messages que 841,036 841,036 0.00
    shared KSXR receive buffers 1,033,000 1,033,000 0.00
    shared MTTR advisory 97,412 97,412 0.00
    shared PL/SQL DIANA 624,112 624,112 0.00
    shared PL/SQL MPCODE 422,640 422,640 0.00
    shared PLS non-lib hp 2,068 2,068 0.00
    shared character set object 323,724 323,724 0.00
    shared dictionary cache 1,610,880 1,610,880 0.00
    shared errors 35,964 35,964 0.00
    shared event statistics per sess 1,718,360 1,718,360 0.00
    shared fixed allocation callback 300 300 0.00
    shared free memory 26,982,004 26,841,956 -0.52
    shared joxs heap init 4,220 4,220 0.00
    shared kgl simulator 3,980,240 3,996,976 0.42
    shared library cache 54,425,164 53,999,624 -0.78
    shared message pool freequeue 834,752 834,752 0.00
    shared miscellaneous 8,126,704 8,177,516 0.63
    shared parameters 1,632 1,632 0.00
    shared sessions 410,720 410,720 0.00
    shared sim memory hea 377,656 377,656 0.00
    shared sql area 66,513,080 66,768,476 0.38
    shared subheap 45,216 45,216 0.00
    shared table definiti 1,200 2,752 129.33
    shared trigger defini 340 340 0.00
    shared trigger inform 1,292 1,292 0.00
    shared trigger source 100 100 0.00
    buffer_cache 687,865,856 687,865,856 0.00
    fixed_sga 455,196 455,196 0.00
    log_buffer 918,528 918,528 0.00
    init.ora Parameters for DB: AHD Instance: ahd Snaps: 20 -21
    End value
    Parameter Name Begin value (if different)
    aq_tm_processes 1
    background_dump_dest E:\oracle\admin\ahd\bdump
    compatible 9.2.0.0.0
    control_files E:\oracle\oradata\ahd\CONTROL01.C
    core_dump_dest E:\oracle\admin\ahd\cdump
    cursor_sharing SIMILAR
    db_block_size 8192
    db_cache_size 687865856
    db_domain
    db_file_multiblock_read_count 8
    db_name ahd
    db_writer_processes 2
    dispatchers (PROTOCOL=TCP) (SERVICE=ahdXDB)
    fast_start_mttr_target 300
    hash_join_enabled TRUE
    instance_name ahd
    java_pool_size 75497472
    job_queue_processes 10
    large_pool_size 41943040
    log_archive_dest_1 location=c:\archive
    log_archive_format arc%d_%t_%s.arc
    log_archive_start TRUE
    open_cursors 300
    pga_aggregate_target 367001600
    processes 150
    query_rewrite_enabled FALSE
    remote_login_passwordfile EXCLUSIVE
    shared_pool_size 159383552
    sort_area_size 10485760
    star_transformation_enabled FALSE
    timed_statistics TRUE
    undo_management AUTO
    undo_retention 3600
    undo_tablespace UNDOTBS1
    user_dump_dest E:\oracle\admin\ahd\udump
    End of Report

    I am getting buffer busy waits on some tables.
    Users are not experiencing any problem.Looks like you got bit by the CTD troll while sleeping.
    Note also that (if I'm reading the report alright) out of 23 mins you have 6 seconds accounted to buffer busy waits.
    Read the sample chapter here.

  • Statspack interpreting help - buffer busy waits

    Hi,
    I've got statspack report from 9.2.0.8 DB, cpu_count = 12 , there is 'buffer busy waits' in top 5 .
    Is there a problem ?
    DB Name         DB Id    Instance     Inst Num Release     Cluster Host
    XXXX         138180125   XXXX            1 9.2.0.8.0   NO      X1
                  Snap Id     Snap Time      Sessions Curs/Sess Comment
    Begin Snap:     35980 14-Jul-10 01:00:02       17       8.8
      End Snap:     35984 14-Jul-10 05:00:01       17       8.8
       Elapsed:              239.98 (mins)
    Cache Sizes (end)
    ~~~~~~~~~~~~~~~~~
                   Buffer Cache:     3,072M      Std Block Size:          8K
               Shared Pool Size:       512M          Log Buffer:      4,096K
    Load Profile
    ~~~~~~~~~~~~                            Per Second       Per Transaction
                      Redo size:            317,746.23              4,498.59
                  Logical reads:             11,150.77                157.87
                  Block changes:              2,134.89                 30.23
                 Physical reads:                466.05                  6.60
                Physical writes:                133.62                  1.89
                     User calls:                 82.42                  1.17
                         Parses:                 67.92                  0.96
                    Hard parses:                  0.02                  0.00
                          Sorts:                106.77                  1.51
                         Logons:                  0.03                  0.00
                       Executes:                516.58                  7.31
                   Transactions:                 70.63
      % Blocks changed per Read:   19.15    Recursive Call %:     95.00
    Rollback per transaction %:    0.00       Rows per Sort:      4.34
    Instance Efficiency Percentages (Target 100%)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Buffer Nowait %:   99.03       Redo NoWait %:    100.00
                Buffer  Hit   %:   95.91    In-memory Sort %:    100.00
                Library Hit   %:  100.00        Soft Parse %:     99.98
             Execute to Parse %:   86.85         Latch Hit %:     99.65
    Parse CPU to Parse Elapsd %:    7.82     % Non-Parse CPU:     99.91
    Shared Pool Statistics        Begin   End
                 Memory Usage %:   43.53   43.92
        % SQL with executions>1:   64.89   70.00
      % Memory for SQL w/exec>1:   55.95   61.64
    Top 5 Timed Events
    ~~~~~~~~~~~~~~~~~~                                                     % Total
    Event                                               Waits    Time (s) Ela Time
    SQL*Net message from dblink                       797,760      44,575    41.69
    PL/SQL lock timer                                   1,207      34,992    32.73
    db file sequential read                         3,297,249      17,047    15.94
    buffer busy waits                               1,558,995       3,987     3.73
    CPU time                                                        3,204     3.00
    Statistic                                      Total     per Second    per Trans
    enqueue timeouts                                 299            0.0          0.0
    enqueue waits                                    425            0.0          0.0
    exchange deadlocks                                41            0.0          0.0
    execute count                              7,438,297          516.6          7.3
    failed probes on index block recl                 13            0.0          0.0
    free buffer inspected                        107,385            7.5          0.1
    free buffer requested                      7,344,870          510.1          7.2
    hot buffers moved to head of LRU           2,332,802          162.0          2.3
    immediate (CR) block cleanout app            356,492           24.8          0.4
    immediate (CURRENT) block cleanou          1,751,731          121.7          1.7
    index crx upgrade (found)                          7            0.0          0.0
    index crx upgrade (positioned)                25,604            1.8          0.0
    index fast full scans (full)                     888            0.1          0.0
    index fetch by key                         6,008,269          417.3          5.9
    index scans kdiixs1                        2,343,163          162.7          2.3
    leaf node 90-10 splits                           330            0.0          0.0
    leaf node splits                              13,681            1.0          0.0
    logons cumulative                                447            0.0          0.0
    messages received                          2,760,503          191.7          2.7
    messages sent                              2,760,503          191.7          2.7
    no buffer to keep pinned count                     0            0.0          0.0
    no work - consistent read gets            89,143,249        6,190.9         87.7
    opened cursors cumulative                    978,462           68.0          1.0
    parse count (failures)                             0            0.0          0.0
    parse count (hard)                               243            0.0          0.0
    parse count (total)                          977,939           67.9          1.0
    parse time cpu                                   276            0.0          0.0
    parse time elapsed                             3,531            0.3          0.0
    physical reads                             6,710,684          466.1          6.6
    physical reads direct                        140,520            9.8          0.1
    physical writes                            1,924,011          133.6          1.9
    physical writes direct                       149,434           10.4          0.2
    physical writes non checkpoint             1,160,293           80.6          1.1
    pinned buffers inspected                      88,165            6.1          0.1
    prefetched blocks                          2,965,135          205.9          2.9
    prefetched blocks aged out before              1,485            0.1          0.0
    process last non-idle time                    14,401            1.0          0.0
    recovery blocks read                               0            0.0          0.0
    recursive calls                           22,566,381        1,567.2         22.2
    recursive cpu usage                          314,662           21.9          0.3
    redo blocks written                        9,712,190          674.5          9.6
    redo buffer allocation retries                   483            0.0          0.0
    redo entries                              17,147,344        1,190.9         16.9
    redo log space requests                          760            0.1          0.0
    redo log space wait time                       1,255            0.1          0.0
    redo ordering marks                               21            0.0          0.0
    redo size                              4,575,228,028      317,746.2      4,498.6
    redo synch time                               73,190            5.1          0.1
    redo synch writes                            333,440           23.2          0.3
    redo wastage                             240,517,096       16,703.7        236.5
    redo write time                              136,628            9.5          0.1
    redo writer latching time                         56            0.0          0.0
    redo writes                                  865,653           60.1          0.9
    rollback changes - undo records a             60,510            4.2          0.1
    rows fetched via callback                  3,948,006          274.2          3.9
    session connect time                               0            0.0          0.0
    Statistic                                      Total     per Second    per Trans
    session logical reads                    160,559,938       11,150.8        157.9
    session pga memory                       223,020,424       15,488.6        219.3
    session pga memory max                   841,058,240       58,410.9        827.0
    session uga memory                   682,912,005,944   47,427,738.5    671,472.1
    session uga memory max                   505,627,192       35,115.4        497.2
    shared hash latch upgrades - no w          1,661,152          115.4          1.6
    shared hash latch upgrades - wait                101            0.0          0.0
    sorts (disk)                                       2            0.0          0.0
    sorts (memory)                             1,537,403          106.8          1.5
    sorts (rows)                               6,669,072          463.2          6.6
    summed dirty queue length                     71,613            5.0          0.1
    switch current to new buffer                  80,971            5.6          0.1
    table fetch by rowid                      79,047,167        5,489.8         77.7
    table fetch continued row                  5,013,545          348.2          4.9
    table scan blocks gotten                  10,328,271          717.3         10.2
    table scan rows gotten                   381,848,913       26,519.1        375.5
    table scans (long tables)                         82            0.0          0.0
    table scans (short tables)                 1,117,114           77.6          1.1
    transaction rollbacks                         32,437            2.3          0.0
    transaction tables consistent rea                 39            0.0          0.0
    transaction tables consistent rea             82,904            5.8          0.1
    user calls                                 1,186,828           82.4          1.2
    user commits                               1,017,037           70.6          1.0
    user rollbacks                                     0            0.0          0.0
    workarea executions - onepass                      7            0.0          0.0
    workarea executions - optimal              2,291,005          159.1          2.3
    write clones created in backgroun                  3            0.0          0.0
    write clones created in foregroun                711            0.1          0.0
    Class                    Waits   Time (s) Time (ms)
    data block           1,549,301      4,015         3
    segment header             253          1         2
    undo block               2,574          0         0
    undo header              2,209          0         0
    extent map                   2          0         5
    Eq     Requests    Succ Gets Failed Gets       Waits   Time (ms)     Time (s)
    TX    1,749,961    1,749,961           0         202          6.47            1
    HW       20,789       20,789           0         223           .40            0
                                               Pct    Avg   Wait                 Pct
                                  Get          Get   Slps   Time       NoWait NoWait
    Latch                       Requests      Miss  /Miss    (s)     Requests   Miss
    Consistent RBA                  866,126    0.0    0.0      0            0
    FIB s.o chain latch                 594    0.0             0            0
    FOB s.o list latch                2,891    0.0    0.0      0            0
    SQL memory manager latch              4    0.0             0        4,793    0.0
    SQL memory manager worka      3,266,221    0.0    0.0      0            0
    active checkpoint queue       1,905,423    0.5    0.0      0            0
    archive control                      28    0.0             0            0
    cache buffer handles             12,070    0.0             0            0
    cache buffers chains        387,508,854    0.6    0.0    134   11,212,815    0.1
    cache buffers lru chain       1,946,036    0.0    0.0      0   14,681,284    0.2
    channel handle pool latc            668    0.0             0            0
    channel operations paren         12,460    0.0    0.0      0            0
    checkpoint queue latch      209,244,852    0.0    0.0      0    1,775,314    0.0
    child cursor hash table           3,240    0.7    0.0      0            0
    commit callback allocati             16    0.0             0            0
    dictionary lookup                    13    0.0             0            0
    dml lock allocation           5,900,238    0.3    0.0      0            0
    dummy allocation                    894    0.2    0.0      0            0
    enqueue hash chains          19,444,854    0.1    0.0      0            0
    enqueues                      9,380,299    0.7    0.0      0            0
    event group latch                   288    0.0             0            0
    event range base latch                3    0.0             0            0
    global tx hash mapping        4,988,645    0.0    0.0      0            0
    hash table column usage              82    0.0             0        1,783    0.0
    job workq parent latch                1  100.0    0.0      0          242   15.3
    job_queue_processes para            287    0.0             0            0
    ktm global data                     265    0.0             0            0
    lgwr LWN SCN                    868,053    0.1    0.0      0            0
    library cache                29,540,874    0.1    0.0      0           37    0.0
    library cache load lock             102    0.0             0            0
    library cache pin            24,145,848    0.0    0.0      0            0
    library cache pin alloca      3,988,997    0.0    0.0      0            0
    list of block allocation        363,684    0.0    0.0      0            0
    loader state object free            912    0.0             0            0
    longop free list parent             479    0.0             0           60    1.7
    message pool operations             100    1.0    0.0      0            0
    messages                      8,036,523    0.3    0.0      0            0
    mostly latch-free SCN           878,016    1.0    0.0      0            0
    multiblock read objects         922,048    0.1    0.0      0            0
    ncodef allocation latch             230    0.0             0            0
    object stats modificatio          2,100    0.0             0            0
    post/wait queue                 709,603    0.0    0.0      0      334,338    0.0
    process allocation                  576    0.0             0          288    0.0
    process group creation              576    0.2    0.0      0            0
    redo allocation              18,881,467    0.7    0.0      0            0
    redo copy                             0                    0   17,155,579    0.1
    redo writing                  4,513,716    0.2    0.0      0            0
    resumable state object               48    0.0             0            0
    row cache enqueue latch       3,556,148    0.1    0.0      0            0
    row cache objects             6,671,783    0.1    0.0      0            0
                                               Pct    Avg   Wait                 Pct
                                  Get          Get   Slps   Time       NoWait NoWait
    Latch                       Requests      Miss  /Miss    (s)     Requests   Miss
    sequence cache                1,533,482    0.0    0.0      0            0
    session allocation           15,194,281    0.1    0.0      0            0
    session idle bit              3,005,477    0.0    0.0      0            0
    session switching                   230    0.0             0            0
    session timer                     4,825    0.0             0            0
    shared pool                   2,114,153    0.0    0.0      0            0
    sim partition latch                   0                    0       10,243    0.6
    simulator hash latch          8,460,492    0.0    0.0      0            0
    simulator lru latch             223,868    0.0    0.3      0      470,589    0.1
    sort extent pool                  1,823    0.5    0.0      0            0
    temporary table state ob             16    0.0             0            0
    transaction allocation          533,964    0.0             0            0
    transaction branch alloc      1,259,723    0.1    0.0      0            0
    undo global data             17,460,173    0.0    0.0      0       14,976    0.0
    user lock                           906    0.1    0.0      0            0
                                          Get                            Spin &
    Latch Name                       Requests      Misses      Sleeps Sleeps 1->4
    cache buffers chains          387,508,854   2,447,832      12,701 2435135/1269
                                                                      3/4/0/0
    redo allocation                18,881,467     131,460         343 131118/341/1
                                                                      /0/0
    enqueues                        9,380,299      62,436         122 62314/122/0/
                                                                      0/0
    library cache                  29,540,874      38,344          80 38264/80/0/0
                                                                      /0
    messages                        8,036,523      25,266          28 25238/28/0/0
                                                                      /0
    dml lock allocation             5,900,238      19,220          25 19195/25/0/0
                                                                      /0
    enqueue hash chains            19,444,854      10,510          27 10483/27/0/0
                                                                      /0
    active checkpoint queue la      1,905,423       9,896          36 9860/36/0/0/
                                                                      0
    library cache pin              24,145,848       8,451          17 8434/17/0/0/
                                                                      0
    mostly latch-free SCN             878,016       8,423          17 8406/17/0/0/
                                                                      0
    session allocation             15,194,281       8,290          33 8257/33/0/0/
                                                                      0
    redo writing                    4,513,716       7,235          13 7222/13/0/0/
                                                                      0
    undo global data               17,460,173       4,113           8 4105/8/0/0/0
    row cache objects               6,671,783       3,680           4 3676/4/0/0/0
    row cache enqueue latch         3,556,148       2,015           1 2014/1/0/0/0
    checkpoint queue latch        209,244,852       1,756           6 1750/6/0/0/0
    transaction branch allocat      1,259,723       1,236           7 1229/7/0/0/0
    shared pool                     2,114,153         808           2 806/2/0/0/0
    library cache pin allocati      3,988,997         649           1 648/1/0/0/0
    cache buffers lru chain         1,946,036         588          13 575/13/0/0/0
    multiblock read objects           922,048         469           8 461/8/0/0/0
    sequence cache                  1,533,482         333           1 332/1/0/0/0
    session idle bit                3,005,477          97           1 96/1/0/0/0
    Consistent RBA                    866,126          55           1 54/1/0/0/0
    simulator lru latch               223,868          33           9 24/9/0/0/0
    post/wait queue                   709,603          27           1 26/1/0/0/0
                                                         NoWait              Waiter
    Latch Name               Where                       Misses     Sleeps   Sleeps
    active checkpoint queue  kcbbacq: scan active check       0         36       36
    cache buffers chains     kcbgtcr: kslbegin excl           0     10,706   10,084
    cache buffers chains     kcbrls: kslbegin                 0        937    1,577
    cache buffers chains     kcbzwb                           0        371      388
    cache buffers chains     kcbgtcr: fast path               0        223      174
    cache buffers chains     kcbgcur: kslbegin                0         86      114
    cache buffers chains     kcbget: pin buffer               0         57       47
    cache buffers chains     kcbzib: finish free bufs         0         54       27
    cache buffers chains     kcbchg: kslbegin: bufs not       0         52       89
    cache buffers chains     kcbnlc                           0         46       37
    cache buffers chains     kcbzgb: scan from tail. no       0         42        0
    cache buffers chains     kcbzib: multi-block read:        0         25        0
    cache buffers chains     kcbget: exchange rls             0         12        3
    cache buffers chains     kcbchg: kslbegin: call CR        0         12       80
    cache buffers chains     kcbget: exchange                 0         10        8
    cache buffers chains     kcbnew                           0          9        0
    cache buffers chains     kcbcge                           0          1        0
    cache buffers chains     kcbgtcr                          0          1        0
    cache buffers chains     kcbbxsv                          0          1       16
    cache buffers chains     kcbkzs                           0          1        3
    cache buffers chains     kcbbic2                          0          1        2
    cache buffers chains     kcbbic1                          0          1        5
    cache buffers lru chain  kcbzgb: multiple sets nowa  10,344         10        0
    cache buffers lru chain  kcbbiop: lru scan              112          3        0
    checkpoint queue latch   kcbklbc: Link buffer into        0          6        0
    dml lock allocation      ktaiam                           0         15       16
    dml lock allocation      ktaidm                           0         10        9
    enqueue hash chains      ksqgtl3                          0         17       11
    enqueue hash chains      ksqrcl                           0          9       15
    enqueue hash chains      ksqcnl                           0          1        1
    enqueues                 ksqdel                           0         60       50
    enqueues                 ksqgel: create enqueue           0         60       56
    enqueues                 ksqies                           0          2       16
    lgwr LWN SCN             kcs023                           0          9        0
    library cache            kglpnc: child                    0         29       29
    library cache            kglupc: child                    0         25       22
    library cache            kgllkdl: child: cleanup          0         11        2
    library cache            kglpndl: child: before pro       0          5        4
    library cache            kglhdgn: child:                  0          3       13
    library cache            kglhdgc: child:                  0          2        1
    library cache            kglpndl: child: after proc       0          2        0
    library cache            kgldte: child 0                  0          1        2
    library cache            kglpin: child: heap proces       0          1        0
    library cache            kglobpn: child:                  0          1        5
    library cache pin        kglpndl                          0          6        1
    library cache pin        kglpnc: child                    0          6        8
    library cache pin        kglupc                           0          5        4
    library cache pin alloca kglpnal                          0          1        1
    messages                 ksarcv                           0         12        5
    messages                 ksarcv: after wait               0          8       19
    messages                 ksaamb: after wakeup             0          8        4
    mostly latch-free SCN    kcslcu3                          0          8       17
    mostly latch-free SCN    kcsnew_scn_rba                   0          1        0
    Latch Name               Where                       Misses     Sleeps   Sleeps
    multiblock read objects  kcbzib: mbr get                  0          4        4
    multiblock read objects  kcbzib: normal mbr free          0          4        4
    post/wait queue          ksliwat:add:nowait               0          1        0
    redo allocation          kcrfwr                           0        322      248
    redo allocation          kcrfwi: more space               0         14       88
    redo allocation          kcrfwi: before write             0          7        7
    redo writing             kcrfwcr                          0          9       12
    redo writing             kcrfwint: rba scn pair           0          2        0
    redo writing             kcrfwint: after write            0          2        6
    row cache enqueue latch  kqreqa                           0          1        1
    row cache objects        kqrpre: find obj                 0          3        1
    row cache objects        kqrpfl: not dirty                0          1        1
    sequence cache           kdnss                            0          1        1
    session allocation       ksuprc                           0         14        5
    session allocation       ksudlc                           0         10        7
    session allocation       ksucri                           0          6       14
    session allocation       ksuxds: not user session         0          3        7
    session idle bit         ksupuc: clear busy               0          1        0
    shared pool              kghalo                           0          1        0
    shared pool              kghupr1                          0          1        2
    simulator lru latch      kcbs_simulate: simulate se       0          8        9
    simulator lru latch      kcbs_lookup_setid                0          1        0
    transaction branch alloc ktcbba                           0          4        2
    transaction branch alloc ktcbod                           0          2        3
    transaction branch alloc ksupuc                           0          1        2
    undo global data         ktudba: KSLBEGIN                 0          8        7
    Top 5 Logical Reads per Segment for DB: XXXXX  Instance: XXXXX  Snaps: 35980
    -> End Segment Logical Reads Threshold:     10000
                                               Subobject  Obj.       Logical
    Owner      Tablespace Object Name          Name       Type         Reads  %Total
    xxxxxxxx    XXXXXDATA      TAB1TABLE                      TABLE   39,838,592   37.65
    Top 5 Physical Reads per Segment for DB: XXXXX  Instance: XXXXX  Snaps: 3598
    -> End Segment Physical Reads Threshold:      1000
                                               Subobject  Obj.      Physical
    Owner      Tablespace Object Name          Name       Type         Reads  %Total
    xxxxxxxx    XXXXXDATA          TAB1TABLE                        TABLE    3,568,038   58.64
    Top 5 Buf. Busy Waits per Segment for DB: XXXXX  Instance: XXXXX  Snaps: 359
    -> End Segment Buffer Busy Waits Threshold:       100
                                                                      Buffer
                                               Subobject  Obj.          Busy
    Owner      Tablespace Object Name          Name       Type         Waits  %Total
    xxxxxxxx    XXXXXDATA                                           TAB1TABLE    1,421,043   91.65
    xxxxxxxx    XXXXXDATA IDX_SOMEIDX               INDEX       62,638    4.04
    xxxxxxxx    XXXXXTABLE                                TABLE       26,914    1.74
    ----------So, for me It looks like :
    TAB1TABLE is buffer busy deliver but there are no inserts reported for that table,
    I've checked that TAB1TABLE tablespace is NO ASSM and extent management is local with uniform size 1M .
    So this is not obvious free list problem . Kind of strange for me .
    Any ideas greatly appreciated :).
    Regards.
    Greg

    user10388717 wrote:
    Hi,
    I've got statspack report from 9.2.0.8 DB, cpu_count = 12 , there is 'buffer busy waits' in top 5 .
    Is there a problem ?
    DB Name         DB Id    Instance     Inst Num Release     Cluster Host
    XXXX         138180125   XXXX            1 9.2.0.8.0   NO      X1
    Snap Id     Snap Time      Sessions Curs/Sess Comment
    Begin Snap:     35980 14-Jul-10 01:00:02       17       8.8
    End Snap:     35984 14-Jul-10 05:00:01       17       8.8
    Elapsed:              239.98 (mins)
    ----------So, for me It looks like :
    TAB1TABLE is buffer busy deliver but there are no inserts reported for that table,
    I've checked that TAB1TABLE tablespace is NO ASSM and extent management is local with uniform size 1M .
    So this is not obvious free list problem . Kind of strange for me .
    Any ideas greatly appreciated :).We can't tell if you have a problem - only you (or your users) can know.
    But you've shown us a statspack snapshot covering 4 hours and in that time you've reported about 30 hours of database time (sum foreground "in-database" waits and CPU), with a fairly small number of sessions which (allowing for background sessions) means most of your foreground sessions seem to be working pretty much non-stop for the entire period. I could take a guess and say that you would like some of the work that they're doing completed sooner.
    The largest fraction of your time goes into waiting for messages from db link, with an average time of 55ms - maybe you have a network problem, maybe you have a query that has a bad choice of execution path that is doing lots of unnecessary trips to the remote db, maybe the queries that get to the remote db could be made much more efficient. (Look for 'sql ordered by executions' in a statspack from the remote db for clues).
    pl/sql lock timer is the next big chunk of time - but this is deliberately coded waits in pl/sql (dbms_lock.sleep) maybe that's supposed to be happening, but you could check the logic to see if some "slow" processes are actually coded to sleep much longer than necessary.
    Your db file sequential reads (single block reads) are, on average taking 5.5 ms - which is reasonable, so you have to ask if the number (and we know which table a lot of them are hitting) is reasonable. This brings us to your buffer busy waits: these can be caused by updates and deletes as well as inserts, but in 9i they are also caused by "read by other session" - so the buffer busy wait may simple be one session waiting for another session to complete a db file sequential read.
    I'd look at your "SQL ordered by Reads" to see if you have some inefficient execution plans (or poorly defined indexes) that result in large amounts of the critical table being constantly re-read. It's possible that you can eliminate redundant visits to this table and reduce your I/O, BBW, and CPU in one shot.
    Regards
    Jonathan Lewis

  • USER I/O Wait (Please help kind of stuck here from long time)

    I have a delete statement running from more than 24 hrs now and the session info says its waiting on user I/O. There are no blocking sessions and its doing a full table scan of a table having around 500000 records. I dont understand what exactly its waiting on and how to check that and why it taking more than 24 hrs to FTS of 1 table? Here are some of the statistics:
    SQL> select blocking_session, event, wait_class, wait_time, seconds_in_wait, state from v$session where sid=1026;
    BLOCKING_SESSION EVENT WAIT_CLASS WAIT_TIME SECONDS_IN_WAIT
    STATE
    db file scattered read User I/O 0 0
    WAITING
    SQL> select * from table(dbms_xplan.display_cursor('1g5k0k3qpy8j2'));
    PLAN_TABLE_OUTPUT
    SQL_ID 1g5k0k3qpy8j2, child number 0
    DELETE FROM RX_TX WHERE ID IN (SELECT ID FROM TEMP_PURGE WHERE TABLE_NAME = 'rx_
    tx')
    Plan hash value: 3126475949
    PLAN_TABLE_OUTPUT
    | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)
    | Time |
    | 0 | DELETE STATEMENT | | | | | 17239 (100)
    | |
    | 1 | DELETE | RX_TX | | | |
    | |
    PLAN_TABLE_OUTPUT
    |* 2 | HASH JOIN RIGHT SEMI| | 513K| 123M| 14M| 17239 (2)
    | 00:03:27 |
    |* 3 | TABLE ACCESS FULL | TEMP_PURGE | 557K| 8717K| | 2789 (2)
    | 00:00:34 |
    | 4 | TABLE ACCESS FULL | RX_TX | 578K| 130M| | 6918 (2)
    | 00:01:24 |
    PLAN_TABLE_OUTPUT
    Predicate Information (identified by operation id):
    2 - access("ID"="ID")
    3 - filter("TABLE_NAME"='rx_tx')
    22 rows selected.
    SQL> select b.name, a.value from v$sesstat a, v$statname b
    where a.statistic# = b.statistic#
    and a.value > 0 2 3
    4 and b.name like '%wait%'
    5 and a.sid=1026;
    NAME VALUE
    concurrency wait time 1615
    application wait time 388
    user I/O wait time 13403000
    enqueue waits 1
    shared hash latch upgrades - no wait 7924935
    redo log space wait time 2852
    6 rows selected.
    Any help would be appreciable.
    This deletes more that 60% of the records from this table so indexed should be out of question here, i think.
    Daljit Singh
    Message was edited by:
    Daljit

    Thanks for replying Reega, here is the required
    info:
    SQL> select p1text, p1, p2text, p2, p3text, p3 from
    v$session where sid=1026;
    P1TEXT
    1
    P2TEXT
    2
    P3TEXT
    3
    file#
    block#
    16937
    blocks
    6
    Actually Reega had a good point, not sure why he didn't go down the route.
    You may want to find out what's that table/index your session is waiting from the value, something like
    select   owner||'.'||segment_name,  segment_type
    from dba_extents
    where file_id=4 and (block_id between 116937 and 116937+66)This might be a long run query if you have many objects.
    Actually the better view to query is v$session_wait instead of v$session.
    Check article you might find useful,
    Oracle wait tuning with v$session_wait

  • SMON generating lot of redo

    DB Version 10.2.0.4
    OS HP-UX
    Hello we are facing problem with wait event Redo Log space Wait
    when i checked for the session which are generating lot of Redo i found SMON at the top..
    Just couldn't understand what SMON would be doing so it is generating highest amount of Redo
    Please Advise ..
    Regards
    Vinayak

    I would use DBMS_LOGMNR package to see what kind of activity is performed. For details about this package see Oracle® Database PL/SQL Packages and Types Reference 10g Release 2 (10.2).
    Kind regards,
    Joze

  • Log file sync waits

    10.2.0.2 aix 5.3 64bit archivelog mode.
    I'm going to attempt to describe the system first and then outline the issue: The database is about 1Gb in size of which only about 400Mb is application data. There is only one table in the schema that is very active with all transactions inserting and or updating a row to log the user activity. The rest of the tables are used primarily for reads by the users and periodically updated by the application administrator with application code. There's about 1.2G of archive logs generated per day, from 3 50Mb redo logs all on the same filesystem.
    The problem: We randomly have issues with users being kicked out of the application or hung up for a period of time. This application is used at a remote site and many times we can attribute the users issues to network delays or problems with a terminal server they are logging into. Today however they called and I noticed an abnormally high amount of 'log file sync' waits.
    I asked the application admin if there could have been more activity during that time frame and more frequent commits than normal, but he says there was not. My next thought was that there might be an issue with the IO sub-system that the logs are on. So I went to our aix admin to find out the activity of that file system during that time frame. She had an nmon report generated that shows the RAID-1 disk group peak activity during that time was only 10%.
    Now I took two awr reports and compared some of the metrics to see if indeed there was the same amount of activity, and it does look like the load was the same. With the same amount of activity & commits during both time periods wouldn't that lead to it being time spent waiting on writes to the disk that the redo logs are on? If so, why wouldn't the nmon report show a higher percentage of disk activity?
    I can provide more values from the awr reports if needed.
              per sec          per trx
    Redo size:     31,226.81     2,334.25
    Logical reads:     646.11          48.30
    Block changes:     190.80          14.26
    Physical reads:     0.65          0.05
    Physical writes:     3.19          0.24
    User calls:     69.61          5.20
    Parses:          34.34          2.57
    Hard parses:     19.45          1.45
    Sorts:          14.36          1.07
    Logons:          0.01          0.00
    Executes:     36.49          2.73
    Transactions:     13.38
    Redo size:     33,639.71      2,347.93
    Logical reads:     697.58          48.69
    Block changes:     215.83          15.06
    Physical reads:     0.86          0.06
    Physical writes:     3.26          0.23
    User calls:     71.06          4.96
    Parses:          36.78          2.57
    Hard parses:     21.03          1.47
    Sorts:          15.85          1.11
    Logons:          0.01          0.00
    Executes:     39.53          2.76
    Transactions:     14.33
                        Total          Per sec          Per Trx
    redo blocks written           252,046      70.52           5.27
    redo buffer allocation retries      7           0.00           0.00
    redo entries                167,349      46.82           3.50
    redo log space requests      7           0.00           0.00
    redo log space wait time      49           0.01           0.00
    redo ordering marks           2,765           0.77           0.06
    redo size                111,612,156      31,226.81      2,334.25
    redo subscn max counts      5,443           1.52           0.11
    redo synch time           47,910           13.40           1.00
    redo synch writes           64,433           18.03           1.35
    redo wastage                13,535,756      3,787.03      283.09
    redo write time                27,642           7.73           0.58
    redo writer latching time      2           0.00           0.00
    redo writes                48,507           13.57           1.01
    user commits                47,815           13.38           1.00
    user rollbacks                0           0.00           0.00
    redo blocks written           273,363      76.17           5.32
    redo buffer allocation retries      6           0.00           0.00
    redo entries                179,992      50.15           3.50
    redo log space requests      6           0.00           0.00
    redo log space wait time      18           0.01           0.00
    redo ordering marks           2,997           0.84           0.06
    redo size                120,725,932      33,639.71      2,347.93
    redo subscn max counts      5,816           1.62           0.11
    redo synch time           12,977           3.62           0.25
    redo synch writes           66,985           18.67           1.30
    redo wastage                14,665,132      4,086.37      285.21
    redo write time                11,358           3.16           0.22
    redo writer latching time      6           0.00           0.00
    redo writes                52,521           14.63           1.02
    user commits                51,418           14.33           1.00
    user rollbacks                0           0.00           0.00Edited by: PktAces on Oct 1, 2008 1:45 PM

    Mr Lewis,
    Here's the results from the histogram query, the two sets of values were gathered about 15 minutes apart, during a slower than normal activity time.
    105     log file parallel write     1     714394
    105     log file parallel write     2     289538
    105     log file parallel write     4     279550
    105     log file parallel write     8     58805
    105     log file parallel write     16     28132
    105     log file parallel write     32     10851
    105     log file parallel write     64     3833
    105     log file parallel write     128     1126
    105     log file parallel write     256     316
    105     log file parallel write     512     192
    105     log file parallel write     1024     78
    105     log file parallel write     2048     49
    105     log file parallel write     4096     31
    105     log file parallel write     8192     35
    105     log file parallel write     16384     41
    105     log file parallel write     32768     9
    105     log file parallel write     65536     1
    105     log file parallel write     1     722787
    105     log file parallel write     2     295607
    105     log file parallel write     4     284524
    105     log file parallel write     8     59671
    105     log file parallel write     16     28412
    105     log file parallel write     32     10976
    105     log file parallel write     64     3850
    105     log file parallel write     128     1131
    105     log file parallel write     256     316
    105     log file parallel write     512     192
    105     log file parallel write     1024     78
    105     log file parallel write     2048     49
    105     log file parallel write     4096     31
    105     log file parallel write     8192     35
    105     log file parallel write     16384     41
    105     log file parallel write     32768     9
    105     log file parallel write     65536     1

  • Tuning of Redo logs in data warehouses (dwh)

    Hi everybody,
    I'm looking for some guidance to configure redo logs in data warehouse environments.
    Of course we are running in noarchive log mode and use direct path inserts (nologging) whereever possible.
    Nevertheless every etl process (one process per day) produces 150 GB of redo logs. That seems quite a lot compared to the overall data volume (1 TB tables + indexes).
    Actually im not sure if there is a tuning problem, but because of the large amount of redo I'm interested in examining it.
    Here are the facts:
    - Oracle 10g, 32 GB RAM
    - 6 GB SGA, 20 GB PGA
    - 5 log groups each with 1 Gb log file
    - 4 MB Log buffer
    - every day ca 150 logswitches (with peaks: some logswitches after 10 seconds)
    - some sysstat metrics after one etl load:
    Select name, to_char(value, '9G999G999G999G999G999G999') from v$sysstat Where name like 'redo %';
    "NAME" "TO_CHAR(VALUE,'9G999G999G999G999G999G999')"
    "redo synch writes" " 300.636"
    "redo synch time" " 61.421"
    "redo blocks read for recovery"" 0"
    "redo entries" " 327.090.445"
    "redo size" " 159.588.263.420"
    "redo buffer allocation retries"" 95.901"
    "redo wastage" " 212.996.316"
    "redo writer latching time" " 1.101"
    "redo writes" " 807.594"
    "redo blocks written" " 321.102.116"
    "redo write time" " 183.010"
    "redo log space requests" " 10.903"
    "redo log space wait time" " 28.501"
    "redo log switch interrupts" " 0"
    "redo ordering marks" " 2.253.328"
    "redo subscn max counts" " 4.685.754"
    So the questions:
    Does anybody can see tuning needs? Should the Redo logs be increased or incremented? What about placing redo logs on Solid state disks?
    kind regards,
    Mirko

    user5341252 wrote:
    I'm looking for some guidance to configure redo logs in data warehouse environments.
    Of course we are running in noarchive log mode and use direct path inserts (nologging) whereever possible.Why "of course" ? What's your recovery strategy if you wreck the database ?
    Nevertheless every etl process (one process per day) produces 150 GB of redo logs. That seems quite a lot compared to the overall data volume (1 TB tables + indexes).This may be an indication that you need to do something to reduce index maintenance during data loading
    >
    Actually im not sure if there is a tuning problem, but because of the large amount of redo I'm interested in examining it.
    For a quick check you might be better off running statspack (or AWR) snapshots across the start and end of batch to get an idea of what work goes on and where the most time goes. A better strategy would be to examine specific jobs in detail, though).
    "redo synch time" " 61.421"
    "redo log space wait time" " 28.501" Rough guideline - if the redo is slowing you down, then you've lost less than 15 minutes across the board to the log writer. Given the number of processes loading and the elapsed time to load, is this significant ?
    "redo buffer allocation retries"" 95.901" This figure tells us how OFTEN we couldn't get space in the log buffer - but not how much time we lost as a result. We also need to see your 'log buffer space' wait time.
    Does anybody can see tuning needs? Should the Redo logs be increased or incremented? What about placing redo logs on Solid state disks?Based on the information you've given so far, I don't think anyone should be giving you concrete recommendations on what to do; only suggestions on where to look or what to tell us.
    Regards
    Jonathan Lewis

Maybe you are looking for