In Memory Undo

Hi All.
Does IMU generate redo for ALL data changed?
I guess that in case of the IMU transaction undo segment header is modified at the begining ( for ITL entries) and so that would need redo.
But since the actual IMU is in the shared pool, no redo is generated , and when an IMU flush happens the proper undo block is updated with the "collapsed" data.
I went through two quality materials on this subject
http://www.oracle.com/education/america_eblasts/fy09/q1/craig_shallahamer_all_about_oracle_s_in_memory_undo.pdf
and
http://www.teamycc.com/RMOUG_2008_Conference/PDF/Haisley.pdf
Both seem to suggest that redo is generated, but the wordings in the second paper suggests that ALL redo for the undo "blocks" are recorded in the redo stream, whereas the first paper suggests that redo is generated only for the "collapsed" values.
I am doubting whether my confusion arises out of the "throwaway" undo and "normal" IMU operations.
Any assistance would be welcome and thanks in advance!!
Edited by: indhar on Dec 13, 2009 6:39 PM
Edited by: indhar on Dec 13, 2009 6:40 PM

Hi,
Both seem to suggest that redo is generated, but the wordings in the second paper suggests that ALL redo for the undo "blocks" are recorded in the redo stream, whereas the first paper suggests that redo is generated only for the "collapsed" values.
I am doubting whether my confusion arises out of the "throwaway" undo and "normal" IMU operations.
Any assistance would be welcome and thanks in advance!!AFAIK, Undo will not be recorded in Redo until the transactions are committed. See Advantage of IMU, its removing of burden of Maintaining the Chain of Undo buffer Management Separately in memory, instead of that existing IMU is maintaing with help of pointer and updating the ITL with the memory. So it will be advantage when compared to regular flow
- Pavan Kumar N
Oracle 9i/10g - OCP
http://oracleinternals.blogspot.com/

Similar Messages

  • Update Statement Simply hanged but doing db file sequential read

    Hi,
    Last night we had issue with one of the prod server where we updating one of table which contains large number records in millions.Same identical machine completed in1 hour and other box never completed but doing db file sequential read but in the long ops the last statement it was done 20:16 after that nothing is happening but i ran few trace on that user.
    /u01/app/oracle/admin/SURV2/udump/surv2_ora_10048.trc
    Oracle Database 10g Release 10.2.0.4.0 - Production
    ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
    System name:     SunOS
    Node name:     prdfa001
    Release:     5.10
    Version:     Generic_139556-08
    Machine:     i86pc
    Instance name: SURV2
    Redo thread mounted by this instance: 1
    Oracle process number: 18
    Unix process pid: 10048, image: oracle@prdfa001
    *** 2010-09-09 23:37:07.484
    *** ACTION NAME:() 2010-09-09 23:37:07.473
    *** MODULE NAME:(SQL*Plus) 2010-09-09 23:37:07.473
    *** SERVICE NAME:(SURV2) 2010-09-09 23:37:07.473
    *** SESSION ID:(289.54) 2010-09-09 23:37:07.473
    Received ORADEBUG command 'unlimit' from process Unix process pid: 3983, image:
    *** 2010-09-09 23:37:20.315
    Received ORADEBUG command 'event 10046 trace name context forever, level 12' from process Unix process pid: 3983, image:
    WAIT #7: nam='db file sequential read' ela= 11160 file#=13 block#=2252349 blocks=1 obj#=166421 tim=12499462835161
    WAIT #7: nam='db file sequential read' ela= 2857 file#=13 block#=2249751 blocks=1 obj#=166421 tim=12499462838137
    WAIT #7: nam='db file sequential read' ela= 3810 file#=13 block#=2251361 blocks=1 obj#=166421 tim=12499462842048
    WAIT #7: nam='db file sequential read' ela= 4459 file#=13 block#=2247059 blocks=1 obj#=166421 tim=12499462846564
    WAIT #7: nam='db file sequential read' ela= 2841 file#=13 block#=2247507 blocks=1 obj#=166421 tim=12499462849468
    WAIT #7: nam='db file sequential read' ela= 427 file#=13 block#=2247568 blocks=1 obj#=166421 tim=12499462850032
    WAIT #7: nam='db file sequential read' ela= 1187 file#=13 block#=2248264 blocks=1 obj#=166421 tim=12499462851327
    WAIT #7: nam='db file sequential read' ela= 2687 file#=13 block#=2250707 blocks=1 obj#=166421 tim=12499462854178
    WAIT #7: nam='db file sequential read' ela= 3657 file#=13 block#=2249697 blocks=1 obj#=166421 tim=12499462857896
    WAIT #7: nam='db file sequential read' ela= 4139 file#=13 block#=2247074 blocks=1 obj#=166421 tim=12499462862093
    WAIT #7: nam='db file sequential read' ela= 4180 file#=47 block#=3649690 blocks=1 obj#=166421 tim=12499509270445
    WAIT #7: nam='db file sequential read' ela= 4802 file#=47 block#=3649309 blocks=1 obj#=166421 tim=12499509275327
    WAIT #7: nam='db file sequential read' ela= 2459 file#=47 block#=3652697 blocks=1 obj#=166421 tim=12499509277859
    WAIT #7: nam='db file sequential read' ela= 4015 file#=47 block#=3652826 blocks=1 obj#=166421 tim=12499509281948
    WAIT #7: nam='db file sequential read' ela= 2248 file#=47 block#=3651610 blocks=1 obj#=166421 tim=12499509284269
    WAIT #7: nam='db file sequential read' ela= 4824 file#=47 block#=3654297 blocks=1 obj#=166421 tim=12499509289166
    WAIT #7: nam='db file sequential read' ela= 2008 file#=47 block#=3652312 blocks=1 obj#=166421 tim=12499509291248
    WAIT #7: nam='db file sequential read' ela= 1925 file#=47 block#=3654490 blocks=1 obj#=166421 tim=12499509293246
    WAIT #7: nam='db file sequential read' ela= 2859 file#=47 block#=3648458 blocks=1 obj#=166421 tim=12499509296178
    WAIT #7: nam='db file sequential read' ela= 1740 file#=47 block#=3648212 blocks=1 obj#=166421 tim=12499509297991
    WAIT #7: nam='db file sequential read' ela= 2566 file#=47 block#=3648411 blocks=1 obj#=166421 tim=12499509300631
    WAIT #7: nam='db file sequential read' ela= 50772 file#=5 block#=480749 blocks=1 obj#=166421 tim=12499509351477
    WAIT #7: nam='db file sequential read' ela= 12928 file#=5 block#=477177 blocks=1 obj#=166421 tim=12499509364482
    WAIT #7: nam='db file sequential read' ela= 11116 file#=5 block#=479412 blocks=1 obj#=166421 tim=12499509375672
    WAIT #7: nam='db file sequential read' ela= 4803 file#=5 block#=483440 blocks=1 obj#=166421 tim=12499509380549
    WAIT #7: nam='db file sequential read' ela= 6900 file#=5 block#=481454 blocks=1 obj#=166421 tim=12499509387522
    Received ORADEBUG command 'event 10046 trace name context off' from process Unix process pid: 3983, image:
    /u01/app/oracle/admin/SURV2/udump/surv2_ora_1545.trc
    Oracle Database 10g Release 10.2.0.4.0 - Production
    ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
    System name:     SunOS
    Node name:     prdfa001
    Release:     5.10
    Version:     Generic_139556-08
    Machine:     i86pc
    Instance name: SURV2
    Redo thread mounted by this instance: 1
    Oracle process number: 22
    Unix process pid: 1545, image: oracle@prdfa001 (TNS V1-V3)
    *** ACTION NAME:() 2010-09-09 23:20:13.485
    *** MODULE NAME:(sqlplus@prdfa001 (TNS V1-V3)) 2010-09-09 23:20:13.485
    *** SERVICE NAME:(SYS$USERS) 2010-09-09 23:20:13.485
    *** SESSION ID:(290.697) 2010-09-09 23:20:13.485
    ===================================================
    SYSTEM STATE
    System global information:
         processes: base 47819b480, size 300, cleanup 4781a5638
         allocation: free sessions 47f1d6148, free calls 0
         control alloc errors: 0 (process), 0 (session), 0 (call)
         PMON latch cleanup depth: 0
         seconds since PMON's last scan for dead processes: 20
         system statistics:
    1171 logons cumulative
    19 logons current
    89219 opened cursors cumulative
    86 opened cursors current
    15095069 user commits
    5 user rollbacks
    58632904 user calls
    44023255 recursive calls
    224311 recursive cpu usage
    201424173 session logical reads
    0 session stored procedure space
    901812 CPU used when call started
    995437 CPU used by this session
    6814196 DB time
    0 cluster wait time
    22542300822 concurrency wait time
    3095 application wait time
    16479074661 user I/O wait time
    1284052668 session connect time
    1284067190 process last non-idle time
    189018343568 session uga memory
    1249667216 session uga memory max
    26059216 messages sent
    26059220 messages received
    239739 background timeouts
    162399896 session pga memory
    189662872 session pga memory max
    4 enqueue timeouts
    901146 enqueue waits
    0 enqueue deadlocks
    32122711 enqueue requests
    17819 enqueue conversions
    32122676 enqueue releases
    0 global enqueue gets sync
    0 global enqueue gets async
    0 global enqueue get time
    0 global enqueue releases
    2865667 physical read total IO requests
    262620 physical read total multi block requests
    270093476864 physical read total bytes
    select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'SERVICE_NAME'), INSTANCE_NUMBER, STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN') from v$instance where INSTANCE_NAME=SYS_CONTEXT('USERENV', 'INSTANCE_NAME')
          hash=550c95f3d0cfa8290e60ea8382d3a2ca timestamp=09-09-2010 04:24:19
          namespace=CRSR flags=RON/KGHP/TIM/PN0/LRG/KST/DBN/MTX/[100100d1]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=9 hpc=0582 hlc=0582
          lwt=47df576e8[47df576e8,47df576e8] ltm=47df576f8[47df576f8,47df576f8]
          pwt=47df576b0[47df576b0,47df576b0] ptm=47df576c0[47df576c0,47df576c0]
          ref=47df57718[47df57718,47df57718] lnd=47df57730[47df57730,47df57730]
            LIBRARY OBJECT: object=471ee1d38
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            CHILDREN: size=16
            child#    table reference   handle
                 0 471ee1800 471ee1470 47df7dce0
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47df7de48 471ee1e50 I/P/A/-/-    0 NONE   00
          SO: 473691d60, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=473691d60 handle=47bb22fa0 mode=N
          call pin=0 session pin=0 hpc=0000 hlc=0000
          htl=473691de0[4735dbcb8,476cfbf58] htb=476cfbf58 ssga=476cfb6a0
          user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x0
          LIBRARY OBJECT HANDLE: handle=47bb22fa0 mtx=47bb230d0(0) cdp=0
          namespace=CRSR flags=RON/KGHP/PN0/EXP/[10010100]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=fd84 hlc=fd84
          lwt=47bb23048[47bb23048,47bb23048] ltm=47bb23058[47bb23058,47bb23058]
          pwt=47bb23010[47bb23010,47bb23010] ptm=47bb23020[47bb23020,47bb23020]
          ref=47bb23078[472f8de18,472f8de18] lnd=47bb23090[47bb23090,47bb23090]
            LIBRARY OBJECT: object=472f8d9d8
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            DEPENDENCIES: count=1 size=16
            AUTHORIZATIONS: count=1 size=16 minimum entrysize=16
            ACCESSES: count=1 size=16
            TRANSLATIONS: count=1 size=16
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47bb22ee0 472f8daf0 I/P/A/-/-    0 NONE   00
                6 472f8e508 46be86250 I/-/A/-/E    0 NONE   00
          SO: 4735dbc38, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=4735dbc38 handle=47bb231c8 mode=N
          call pin=0 session pin=0 hpc=0000 hlc=0000
          htl=4735dbcb8[476cfbf58,473691de0] htb=476cfbf58 ssga=476cfb6a0
          user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x4c894f8b
          LIBRARY OBJECT HANDLE: handle=47bb231c8 mtx=47bb232f8(1) cdp=1
          name=select value$ from props$ where name = 'GLOBAL_DB_NAME'
          hash=4bb432d65c5a391a42a5c3fa74472c7a timestamp=09-09-2010 04:24:12
          namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/KST/DBN/MTX/[120100d0]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=0584 hlc=0584
          lwt=47bb23270[47bb23270,47bb23270] ltm=47bb23280[47bb23280,47bb23280]
          pwt=47bb23238[47bb23238,47bb23238] ptm=47bb23248[47bb23248,47bb23248]
          ref=47bb232a0[47bb232a0,47bb232a0] lnd=47bb232b8[47bb232b8,47bb232b8]
            LIBRARY OBJECT: object=472f8e6e0
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            CHILDREN: size=16
            child#    table reference   handle
                 0 472f8e1a8 472f8de18 47bb22fa0
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47bb23108 472f8e7f8 I/P/A/-/-    0 NONE   00
          SO: 473644348, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=473644348 handle=47bbde418 mode=N
          call pin=0 session pin=0 hpc=0000 hlc=0000
          htl=4736443c8[476cfc0b8,476cfc0b8] htb=476cfc0b8 ssga=476cfb6a0
          user=47924e810 session=47924e810 count=1 flags=[0000] savepoint=0x4c894f8b
          LIBRARY OBJECT HANDLE: handle=47bbde418 mtx=47bbde548(0) cdp=0
          name=ALTER SESSION SET TIME_ZONE='+02:00'
          hash=3878dff8839e71e3dd05a2e75fbd6390 timestamp=09-09-2010 04:24:04
          namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/DBN/[12010040]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=11 hpc=04e8 hlc=04e8
          lwt=47bbde4c0[47bbde4c0,47bbde4c0] ltm=47bbde4d0[47bbde4d0,47bbde4d0]
          pwt=47bbde488[47bbde488,47bbde488] ptm=47bbde498[47bbde498,47bbde498]
          ref=47bbde4f0[47bbde4f0,47bbde4f0] lnd=47bbde508[47bbde508,47bbde508]
            LIBRARY OBJECT: object=472fffc08
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47bbde320 472fffd20 I/P/A/-/-    0 NONE   00
          SO: 47aecf9e8, type: 41, owner: 47924e810, flag: INIT/-/-/0x00
          (dummy) nxc=0, nlb=0  
        SO: 47f290540, type: 11, owner: 4781a7dc0, flag: INIT/-/-/0x00
        (broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: 4781a7dc0,
                           event: 1132, last message event: 1132,
                           last message waited event: 1132,                        next message: 0(0), messages read: 0
                           channel: (47a2df4f8) system events broadcast channel
                                    scope: 2, event: 1132, last mesage event: 18,
                                    publishers/subscribers: 0/17,
                                    messages published: 1
        SO: 47826b228, type: 3, owner: 4781a7dc0, flag: INIT/-/-/0x00
        (call) sess: cur 47924e810, rec 0, usr 47924e810; depth: 0
        SO: 476c52968, type: 16, owner: 4781a7dc0, flag: INIT/-/-/0x00
        (osp req holder)
    PSEUDO PROCESS for group DEFAULT:
      SO: 47a1eb7d0, type: 2, owner: 0, flag: INIT/-/-/0x00
      (process) Oracle pid=0, calls cur/top: 0/0, flag: (20) PSEUDO
                int error: 0, call error: 0, sess error: 0, txn error 0
      (post info) last post received: 0 0 0
                  last post received-location: No post
                  last process to post me: none
                  last post sent: 0 0 0
                  last post sent-location: No post
                  last process posted by me: none
        (latch info) wait_event=0 bits=0
        Process Group: DEFAULT, pseudo proc: 47a1eb7d0
        O/S info: user: , term: , ospid:  (DEAD)
        OSD pid info: Unix process pid: 0, image: PSEUDO
    Dump of memory from 0x00000004791BF538 to 0x00000004791BF740
    4791BF530                   00000000 00000000          [........]
    4791BF540 00000000 00000000 00000000 00000000  [................]
      Repeat 31 times
    NO DETACHED BRANCHES.
    NO DETACHED NETWORK CONNECTIONS.
    CLEANUP STATE OBJECTS:
    SO: 47f0cd038, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: instance enqueue anchor state
    latch: 0x380009890
      SO: 4782cf080, type: 5, owner: 47f0cd038, flag: INIT/-/-/0x00
      (enqueue) TA-00000006-00000001     DID: 0001-000F-0000000B
      lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x2
      res: 0x47a28d020, mode: X, lock_flag: 0x0
      own: 0x0, sess: 0x0, prv: 0x47a28d030
    SO: 47f0cd098, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: switchable channel handle anch
    latch: 0x38000ac98
      SO: 47f28f868, type: 11, owner: 47f0cd098, flag: INIT/-/-/0x00
      (broadcast handle) flag: (c2) ACTIVE SUBSCRIBER, owner: 0,
                         event: 1, last message event: 1,
                         last message waited event: 1,                      next message: 0(0), messages read: 0
                         channel: (47a2e4190) KPON channel
                                  scope: 2, event: 1, last mesage event: 0,
                                  publishers/subscribers: 0/1,
                                  messages published: 0
    SO: 47f0cd0f8, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: TT shared object cleanup SO
    latch: 0x38001c6b8
    SO: 47f0cd158, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: SS shared object cleanup SO
    latch: 0x38001cd48
    END OF SYSTEM STATE
    Top 5 Timed Events                                         Avg %Total
    ~~~~~~~~~~~~~~~~~~                                        wait   Call
    Event                                 Waits    Time (s)   (ms)   Time Wait Class
    db file sequential read           2,347,652       9,215      4   64.5   User I/O
    db file scattered read              245,687       4,199     17   29.4   User I/O
    CPU time                                            974           6.8
    db file parallel write               50,082         408      8    2.9 System I/O
    log file parallel write               6,963          52      7    0.4 System I/O
    Time Model Statistics                DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> Total time in database user-calls (DB Time): 14286.4s
    -> Statistics including the word "background" measure background process
       time, and so do not contribute to the DB time statistic
    -> Ordered by % or DB time desc, Statistic name
    Statistic Name                                       Time (s) % of DB Time
    sql execute elapsed time                             14,280.3        100.0
    DB CPU                                                  974.5          6.8
    PL/SQL execution elapsed time                           531.8          3.7
    parse time elapsed                                       30.5           .2
    hard parse elapsed time                                  27.1           .2
    connection management call elapsed time                  14.9           .1
    hard parse (sharing criteria) elapsed time                3.4           .0
    hard parse (bind mismatch) elapsed time                   3.1           .0
    PL/SQL compilation elapsed time                           2.4           .0
    failed parse elapsed time                                 0.0           .0
    repeated bind elapsed time                                0.0           .0
    sequence load elapsed time                                0.0           .0
    DB time                                              14,286.4          N/A
    background elapsed time                                 670.2          N/A
    background cpu time                                     186.1          N/A
    Wait Class                            DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> s  - second
    -> cs - centisecond -     100th of a second
    -> ms - millisecond -    1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc
                                                                      Avg
                                           %Time       Total Wait    wait     Waits
    Wait Class                      Waits  -outs         Time (s)    (ms)      /txn
    User I/O                    2,593,484     .0           13,415       5     150.0
    System I/O                     87,506     .0              515       6       5.1
    Other                             839   11.4                6       7       0.0
    Commit                          3,225     .1                6       2       0.2
    Concurrency                     1,033     .0                5       5       0.1
    Configuration                   2,514   99.4                0       0       0.1
    Network                        47,559     .0                0       0       2.8
    Application                         7     .0                0       0       0.0
    Wait Events                          DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> s  - second
    -> cs - centisecond -     100th of a second
    -> ms - millisecond -    1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc (idle events last)
                                                                       Avg
                                                 %Time  Total Wait    wait     Waits
    Event                                 Waits  -outs    Time (s)    (ms)      /txn
    db file sequential read           2,347,652     .0       9,215       4     135.8
    db file scattered read              245,687     .0       4,199      17      14.2
    db file parallel write               50,082     .0         408       8       2.9
    log file parallel write               6,963     .0          52       7       0.4
    control file parallel write           6,203     .0          44       7       0.4
    control file sequential read         24,242     .0          11       0       1.4
    log file sync                         3,225     .1           6       2       0.2
    latch free                               84     .0           4      47       0.0
    os thread startup                        25     .0           3     120       0.0
    latch: session allocation                39     .0           1      33       0.0
    db file parallel read                    12     .0           1      92       0.0
    enq: TX - index contention              186     .0           1       3       0.0
    latch: shared pool                       47     .0           1      11       0.0
    LGWR wait for redo copy                 319    3.1           0       1       0.0
    library cache load lock                   2     .0           0     172       0.0
    buffer busy waits                       590     .0           0       0       0.0
    log file switch completion                6     .0           0      29       0.0
    SGA: allocation forcing comp             11   54.5           0      14       0.0
    latch: library cache lock                50     .0           0       3       0.0
    read by other session                    38     .0           0       4       0.0
    direct path read                         42     .0           0       3       0.0
    SQL*Net message to client            44,807     .0           0       0       2.6
    rdbms ipc reply                         207     .0           0       0       0.0
    SQL*Net more data from clien          1,014     .0           0       0       0.1
    latch: cache buffers chains              24     .0           0       1       0.0
    latch: library cache                     29     .0           0       1       0.0
    log file sequential read                  8     .0           0       3       0.0
    direct path write                        50     .0           0       0       0.0
    SQL*Net more data to client             398     .0           0       0       0.0
    latch: object queue header o             12     .0           0       1       0.0
    latch: In memory undo latch              78     .0           0       0       0.0
    undo segment extension                2,507   99.7           0       0       0.1
    latch: cache buffers lru cha              4     .0           0       1       0.0
    log file single write                     8     .0           0       0       0.0
    local write wait                          3     .0           0       1       0.0
    enq: RO - fast object reuse               3     .0           0       1       0.0
    buffer deadlock                          87   92.0           0       0       0.0
    enq: JS - queue lock                      1     .0           0       1       0.0
    cursor: pin S                            70     .0           0       0       0.0
    latch: row cache objects                  2     .0           0       1       0.0
    SQL*Net message to dblink             1,338     .0           0       0       0.1
    latch: checkpoint queue latc              2     .0           0       0       0.0
    reliable message                          3     .0           0       0       0.0
    log buffer space                          1     .0           0       1       0.0
    SQL*Net break/reset to clien              4     .0           0       0       0.0
    SQL*Net more data from dblin              2     .0           0       0       0.0
    SQL*Net message from client          44,949     .0     155,701    3464       2.6
    virtual circuit status                  621  100.0      18,156   29237       0.0
    Streams AQ: qmn slave idle w            664     .0      18,127   27299       0.0
    Streams AQ: qmn coordinator           1,339   50.4      18,099   13517       0.1
    Streams AQ: waiting for time             12  100.0       8,741  728394       0.0
    jobq slave wait                         130  100.0         380    2927       0.0
    PL/SQL lock timer                         1  100.0           1     978       0.0
    SQL*Net message from dblink           1,338     .0           0       0       0.1
    single-task message                       1     .0           0      38       0.0
    class slave wait                         11     .0           0       1       0.0
    SQL ordered by Elapsed Time          DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> Resources reported for PL/SQL code includes the resources used by all SQL
       statements called by the code.
    -> % Total DB Time is the Elapsed Time of the SQL statement divided
       into the Total Database Time multiplied by 100
      Elapsed      CPU                  Elap per  % Total
      Time (s)   Time (s)  Executions   Exec (s)  DB Time    SQL Id
        13,664        906            0        N/A    95.6 gr2cx6athc5j5
    Module: SQL*Plus
    BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
         8,792        195            0        N/A    61.5 986fzxtzr52u5
    Module: SQL*Plus
    UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
         2,524        368            1     2524.1    17.7 c4uf0x6hdgnwq
    Module: SQL*Plus
    UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
    SYS_B_1"
         1,414        177            1     1414.4     9.9 cbg09ma34kq8w
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
           742        137            1      742.2     5.2 g0sg6v994wssq
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
           274         11            1      274.2     1.9 6mcpb06rctk0x
    Module: DBMS_SCHEDULER
    call dbms_space.auto_space_advisor_job_proc ( )
           264          8           27        9.8     1.8 8szmwam7fysa3
    Module: DBMS_SCHEDULER
    insert into wri$_adv_objspace_trend_data select timepoint, space_usage, space_a
    lloc, quality from table(dbms_space.object_growth_trend(:1, :2, :3, :4, NULL, N
    ULL, NULL, 'FALSE', :5, 'FALSE'))
            99          1            1       99.4     0.7 1z0x41f66nvjr
    Module: SQL*Plus
    UPDATE TIBEX_INSTRUMENTADMIN SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
    SYS_B_1"
            21         10            1       21.5     0.2 bbc1ck8594kvj
    Module: SQL*Plus
    UPDATE TIBEX_INSTRUMENTDAILYHIST SET ADJOPEN=NVL(ADJOPEN,OPEN), ADJHIGH=NVL(ADJH
    IGH,HIGH), ADJLOW=NVL(ADJLOW,LOW), ADJMID=NVL(ADJMID,MID), ADJCLOSE=NVL(ADJCLOSE
    ,CLOSE), ADJVOLUME=NVL(ADJVOLUME,VOLUME), ADJCLOSINGBID=NVL(ADJCLOSINGBID,CLOSIN
    GBID), ADJCLOSINGOFFER=NVL(ADJCLOSINGOFFER,CLOSINGOFFER)
            12          0            1       12.5     0.1 6xm9p9uy5kaap
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_INSTRUMENTSTATE WHERE INSTRUMENTID=:"SYS_B_0"
    SQL ordered by CPU Time              DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> Resources reported for PL/SQL code includes the resources used by all SQL
       statements called by the code.
    -> % Total DB Time is the Elapsed Time of the SQL statement divided
       into the Total Database Time multiplied by 100
        CPU      Elapsed                  CPU per  % Total
      Time (s)   Time (s)  Executions     Exec (s) DB Time    SQL Id
           906     13,664            0         N/A    95.6 gr2cx6athc5j5
    Module: SQL*Plus
    BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
           368      2,524            1      367.51    17.7 c4uf0x6hdgnwq
    Module: SQL*Plus
    UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
    SYS_B_1"
           195      8,792            0         N/A    61.5 986fzxtzr52u5
    Module: SQL*Plus
    UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
           177      1,414            1      176.93     9.9 cbg09ma34kq8w
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
           137        742            1      137.38     5.2 g0sg6v994wssq
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
            11        274            1       10.82     1.9 6mcpb06rctk0x
    Module: DBMS_SCHEDULER
    call dbms_space.auto_space_advisor_job_proc ( )
            10         21            1        9.65     0.2 bbc1ck8594kvjEdited by: NM on 10-Sep-2010 07:39

    Hi,
    Last night we had issue with one of the prod server where we updating one of table which contains large number records in millions.Same identical machine completed in1 hour and other box never completed but doing db file sequential read but in the long ops the last statement it was done 20:16 after that nothing is happening but i ran few trace on that user.
    /u01/app/oracle/admin/SURV2/udump/surv2_ora_10048.trc
    Oracle Database 10g Release 10.2.0.4.0 - Production
    ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
    System name:     SunOS
    Node name:     prdfa001
    Release:     5.10
    Version:     Generic_139556-08
    Machine:     i86pc
    Instance name: SURV2
    Redo thread mounted by this instance: 1
    Oracle process number: 18
    Unix process pid: 10048, image: oracle@prdfa001
    *** 2010-09-09 23:37:07.484
    *** ACTION NAME:() 2010-09-09 23:37:07.473
    *** MODULE NAME:(SQL*Plus) 2010-09-09 23:37:07.473
    *** SERVICE NAME:(SURV2) 2010-09-09 23:37:07.473
    *** SESSION ID:(289.54) 2010-09-09 23:37:07.473
    Received ORADEBUG command 'unlimit' from process Unix process pid: 3983, image:
    *** 2010-09-09 23:37:20.315
    Received ORADEBUG command 'event 10046 trace name context forever, level 12' from process Unix process pid: 3983, image:
    WAIT #7: nam='db file sequential read' ela= 11160 file#=13 block#=2252349 blocks=1 obj#=166421 tim=12499462835161
    WAIT #7: nam='db file sequential read' ela= 2857 file#=13 block#=2249751 blocks=1 obj#=166421 tim=12499462838137
    WAIT #7: nam='db file sequential read' ela= 3810 file#=13 block#=2251361 blocks=1 obj#=166421 tim=12499462842048
    WAIT #7: nam='db file sequential read' ela= 4459 file#=13 block#=2247059 blocks=1 obj#=166421 tim=12499462846564
    WAIT #7: nam='db file sequential read' ela= 2841 file#=13 block#=2247507 blocks=1 obj#=166421 tim=12499462849468
    WAIT #7: nam='db file sequential read' ela= 427 file#=13 block#=2247568 blocks=1 obj#=166421 tim=12499462850032
    WAIT #7: nam='db file sequential read' ela= 1187 file#=13 block#=2248264 blocks=1 obj#=166421 tim=12499462851327
    WAIT #7: nam='db file sequential read' ela= 2687 file#=13 block#=2250707 blocks=1 obj#=166421 tim=12499462854178
    WAIT #7: nam='db file sequential read' ela= 3657 file#=13 block#=2249697 blocks=1 obj#=166421 tim=12499462857896
    WAIT #7: nam='db file sequential read' ela= 4139 file#=13 block#=2247074 blocks=1 obj#=166421 tim=12499462862093
    WAIT #7: nam='db file sequential read' ela= 4180 file#=47 block#=3649690 blocks=1 obj#=166421 tim=12499509270445
    WAIT #7: nam='db file sequential read' ela= 4802 file#=47 block#=3649309 blocks=1 obj#=166421 tim=12499509275327
    WAIT #7: nam='db file sequential read' ela= 2459 file#=47 block#=3652697 blocks=1 obj#=166421 tim=12499509277859
    WAIT #7: nam='db file sequential read' ela= 4015 file#=47 block#=3652826 blocks=1 obj#=166421 tim=12499509281948
    WAIT #7: nam='db file sequential read' ela= 2248 file#=47 block#=3651610 blocks=1 obj#=166421 tim=12499509284269
    WAIT #7: nam='db file sequential read' ela= 4824 file#=47 block#=3654297 blocks=1 obj#=166421 tim=12499509289166
    WAIT #7: nam='db file sequential read' ela= 2008 file#=47 block#=3652312 blocks=1 obj#=166421 tim=12499509291248
    WAIT #7: nam='db file sequential read' ela= 1925 file#=47 block#=3654490 blocks=1 obj#=166421 tim=12499509293246
    WAIT #7: nam='db file sequential read' ela= 2859 file#=47 block#=3648458 blocks=1 obj#=166421 tim=12499509296178
    WAIT #7: nam='db file sequential read' ela= 1740 file#=47 block#=3648212 blocks=1 obj#=166421 tim=12499509297991
    WAIT #7: nam='db file sequential read' ela= 2566 file#=47 block#=3648411 blocks=1 obj#=166421 tim=12499509300631
    WAIT #7: nam='db file sequential read' ela= 50772 file#=5 block#=480749 blocks=1 obj#=166421 tim=12499509351477
    WAIT #7: nam='db file sequential read' ela= 12928 file#=5 block#=477177 blocks=1 obj#=166421 tim=12499509364482
    WAIT #7: nam='db file sequential read' ela= 11116 file#=5 block#=479412 blocks=1 obj#=166421 tim=12499509375672
    WAIT #7: nam='db file sequential read' ela= 4803 file#=5 block#=483440 blocks=1 obj#=166421 tim=12499509380549
    WAIT #7: nam='db file sequential read' ela= 6900 file#=5 block#=481454 blocks=1 obj#=166421 tim=12499509387522
    Received ORADEBUG command 'event 10046 trace name context off' from process Unix process pid: 3983, image:
    /u01/app/oracle/admin/SURV2/udump/surv2_ora_1545.trc
    Oracle Database 10g Release 10.2.0.4.0 - Production
    ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
    System name:     SunOS
    Node name:     prdfa001
    Release:     5.10
    Version:     Generic_139556-08
    Machine:     i86pc
    Instance name: SURV2
    Redo thread mounted by this instance: 1
    Oracle process number: 22
    Unix process pid: 1545, image: oracle@prdfa001 (TNS V1-V3)
    *** ACTION NAME:() 2010-09-09 23:20:13.485
    *** MODULE NAME:(sqlplus@prdfa001 (TNS V1-V3)) 2010-09-09 23:20:13.485
    *** SERVICE NAME:(SYS$USERS) 2010-09-09 23:20:13.485
    *** SESSION ID:(290.697) 2010-09-09 23:20:13.485
    ===================================================
    SYSTEM STATE
    System global information:
         processes: base 47819b480, size 300, cleanup 4781a5638
         allocation: free sessions 47f1d6148, free calls 0
         control alloc errors: 0 (process), 0 (session), 0 (call)
         PMON latch cleanup depth: 0
         seconds since PMON's last scan for dead processes: 20
         system statistics:
    1171 logons cumulative
    19 logons current
    89219 opened cursors cumulative
    86 opened cursors current
    15095069 user commits
    5 user rollbacks
    58632904 user calls
    44023255 recursive calls
    224311 recursive cpu usage
    201424173 session logical reads
    0 session stored procedure space
    901812 CPU used when call started
    995437 CPU used by this session
    6814196 DB time
    0 cluster wait time
    22542300822 concurrency wait time
    3095 application wait time
    16479074661 user I/O wait time
    1284052668 session connect time
    1284067190 process last non-idle time
    189018343568 session uga memory
    1249667216 session uga memory max
    26059216 messages sent
    26059220 messages received
    239739 background timeouts
    162399896 session pga memory
    189662872 session pga memory max
    4 enqueue timeouts
    901146 enqueue waits
    0 enqueue deadlocks
    32122711 enqueue requests
    17819 enqueue conversions
    32122676 enqueue releases
    0 global enqueue gets sync
    0 global enqueue gets async
    0 global enqueue get time
    0 global enqueue releases
    2865667 physical read total IO requests
    262620 physical read total multi block requests
    270093476864 physical read total bytes
    select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'SERVICE_NAME'), INSTANCE_NUMBER, STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN') from v$instance where INSTANCE_NAME=SYS_CONTEXT('USERENV', 'INSTANCE_NAME')
          hash=550c95f3d0cfa8290e60ea8382d3a2ca timestamp=09-09-2010 04:24:19
          namespace=CRSR flags=RON/KGHP/TIM/PN0/LRG/KST/DBN/MTX/[100100d1]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=9 hpc=0582 hlc=0582
          lwt=47df576e8[47df576e8,47df576e8] ltm=47df576f8[47df576f8,47df576f8]
          pwt=47df576b0[47df576b0,47df576b0] ptm=47df576c0[47df576c0,47df576c0]
          ref=47df57718[47df57718,47df57718] lnd=47df57730[47df57730,47df57730]
            LIBRARY OBJECT: object=471ee1d38
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            CHILDREN: size=16
            child#    table reference   handle
                 0 471ee1800 471ee1470 47df7dce0
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47df7de48 471ee1e50 I/P/A/-/-    0 NONE   00
          SO: 473691d60, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=473691d60 handle=47bb22fa0 mode=N
          call pin=0 session pin=0 hpc=0000 hlc=0000
          htl=473691de0[4735dbcb8,476cfbf58] htb=476cfbf58 ssga=476cfb6a0
          user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x0
          LIBRARY OBJECT HANDLE: handle=47bb22fa0 mtx=47bb230d0(0) cdp=0
          namespace=CRSR flags=RON/KGHP/PN0/EXP/[10010100]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=fd84 hlc=fd84
          lwt=47bb23048[47bb23048,47bb23048] ltm=47bb23058[47bb23058,47bb23058]
          pwt=47bb23010[47bb23010,47bb23010] ptm=47bb23020[47bb23020,47bb23020]
          ref=47bb23078[472f8de18,472f8de18] lnd=47bb23090[47bb23090,47bb23090]
            LIBRARY OBJECT: object=472f8d9d8
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            DEPENDENCIES: count=1 size=16
            AUTHORIZATIONS: count=1 size=16 minimum entrysize=16
            ACCESSES: count=1 size=16
            TRANSLATIONS: count=1 size=16
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47bb22ee0 472f8daf0 I/P/A/-/-    0 NONE   00
                6 472f8e508 46be86250 I/-/A/-/E    0 NONE   00
          SO: 4735dbc38, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=4735dbc38 handle=47bb231c8 mode=N
          call pin=0 session pin=0 hpc=0000 hlc=0000
          htl=4735dbcb8[476cfbf58,473691de0] htb=476cfbf58 ssga=476cfb6a0
          user=47924e810 session=47f2310f0 count=1 flags=[0000] savepoint=0x4c894f8b
          LIBRARY OBJECT HANDLE: handle=47bb231c8 mtx=47bb232f8(1) cdp=1
          name=select value$ from props$ where name = 'GLOBAL_DB_NAME'
          hash=4bb432d65c5a391a42a5c3fa74472c7a timestamp=09-09-2010 04:24:12
          namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/KST/DBN/MTX/[120100d0]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=0584 hlc=0584
          lwt=47bb23270[47bb23270,47bb23270] ltm=47bb23280[47bb23280,47bb23280]
          pwt=47bb23238[47bb23238,47bb23238] ptm=47bb23248[47bb23248,47bb23248]
          ref=47bb232a0[47bb232a0,47bb232a0] lnd=47bb232b8[47bb232b8,47bb232b8]
            LIBRARY OBJECT: object=472f8e6e0
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            CHILDREN: size=16
            child#    table reference   handle
                 0 472f8e1a8 472f8de18 47bb22fa0
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47bb23108 472f8e7f8 I/P/A/-/-    0 NONE   00
          SO: 473644348, type: 53, owner: 47924e810, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=473644348 handle=47bbde418 mode=N
          call pin=0 session pin=0 hpc=0000 hlc=0000
          htl=4736443c8[476cfc0b8,476cfc0b8] htb=476cfc0b8 ssga=476cfb6a0
          user=47924e810 session=47924e810 count=1 flags=[0000] savepoint=0x4c894f8b
          LIBRARY OBJECT HANDLE: handle=47bbde418 mtx=47bbde548(0) cdp=0
          name=ALTER SESSION SET TIME_ZONE='+02:00'
          hash=3878dff8839e71e3dd05a2e75fbd6390 timestamp=09-09-2010 04:24:04
          namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/DBN/[12010040]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=11 hpc=04e8 hlc=04e8
          lwt=47bbde4c0[47bbde4c0,47bbde4c0] ltm=47bbde4d0[47bbde4d0,47bbde4d0]
          pwt=47bbde488[47bbde488,47bbde488] ptm=47bbde498[47bbde498,47bbde498]
          ref=47bbde4f0[47bbde4f0,47bbde4f0] lnd=47bbde508[47bbde508,47bbde508]
            LIBRARY OBJECT: object=472fffc08
            type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
            DATA BLOCKS:
            data#     heap  pointer    status pins change whr
                0 47bbde320 472fffd20 I/P/A/-/-    0 NONE   00
          SO: 47aecf9e8, type: 41, owner: 47924e810, flag: INIT/-/-/0x00
          (dummy) nxc=0, nlb=0  
        SO: 47f290540, type: 11, owner: 4781a7dc0, flag: INIT/-/-/0x00
        (broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: 4781a7dc0,
                           event: 1132, last message event: 1132,
                           last message waited event: 1132,                        next message: 0(0), messages read: 0
                           channel: (47a2df4f8) system events broadcast channel
                                    scope: 2, event: 1132, last mesage event: 18,
                                    publishers/subscribers: 0/17,
                                    messages published: 1
        SO: 47826b228, type: 3, owner: 4781a7dc0, flag: INIT/-/-/0x00
        (call) sess: cur 47924e810, rec 0, usr 47924e810; depth: 0
        SO: 476c52968, type: 16, owner: 4781a7dc0, flag: INIT/-/-/0x00
        (osp req holder)
    PSEUDO PROCESS for group DEFAULT:
      SO: 47a1eb7d0, type: 2, owner: 0, flag: INIT/-/-/0x00
      (process) Oracle pid=0, calls cur/top: 0/0, flag: (20) PSEUDO
                int error: 0, call error: 0, sess error: 0, txn error 0
      (post info) last post received: 0 0 0
                  last post received-location: No post
                  last process to post me: none
                  last post sent: 0 0 0
                  last post sent-location: No post
                  last process posted by me: none
        (latch info) wait_event=0 bits=0
        Process Group: DEFAULT, pseudo proc: 47a1eb7d0
        O/S info: user: , term: , ospid:  (DEAD)
        OSD pid info: Unix process pid: 0, image: PSEUDO
    Dump of memory from 0x00000004791BF538 to 0x00000004791BF740
    4791BF530                   00000000 00000000          [........]
    4791BF540 00000000 00000000 00000000 00000000  [................]
      Repeat 31 times
    NO DETACHED BRANCHES.
    NO DETACHED NETWORK CONNECTIONS.
    CLEANUP STATE OBJECTS:
    SO: 47f0cd038, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: instance enqueue anchor state
    latch: 0x380009890
      SO: 4782cf080, type: 5, owner: 47f0cd038, flag: INIT/-/-/0x00
      (enqueue) TA-00000006-00000001     DID: 0001-000F-0000000B
      lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x2
      res: 0x47a28d020, mode: X, lock_flag: 0x0
      own: 0x0, sess: 0x0, prv: 0x47a28d030
    SO: 47f0cd098, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: switchable channel handle anch
    latch: 0x38000ac98
      SO: 47f28f868, type: 11, owner: 47f0cd098, flag: INIT/-/-/0x00
      (broadcast handle) flag: (c2) ACTIVE SUBSCRIBER, owner: 0,
                         event: 1, last message event: 1,
                         last message waited event: 1,                      next message: 0(0), messages read: 0
                         channel: (47a2e4190) KPON channel
                                  scope: 2, event: 1, last mesage event: 0,
                                  publishers/subscribers: 0/1,
                                  messages published: 0
    SO: 47f0cd0f8, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: TT shared object cleanup SO
    latch: 0x38001c6b8
    SO: 47f0cd158, type: 1, owner: 0, flag: INIT/-/-/0x00
    (cleanup state object) description: SS shared object cleanup SO
    latch: 0x38001cd48
    END OF SYSTEM STATE
    Top 5 Timed Events                                         Avg %Total
    ~~~~~~~~~~~~~~~~~~                                        wait   Call
    Event                                 Waits    Time (s)   (ms)   Time Wait Class
    db file sequential read           2,347,652       9,215      4   64.5   User I/O
    db file scattered read              245,687       4,199     17   29.4   User I/O
    CPU time                                            974           6.8
    db file parallel write               50,082         408      8    2.9 System I/O
    log file parallel write               6,963          52      7    0.4 System I/O
    Time Model Statistics                DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> Total time in database user-calls (DB Time): 14286.4s
    -> Statistics including the word "background" measure background process
       time, and so do not contribute to the DB time statistic
    -> Ordered by % or DB time desc, Statistic name
    Statistic Name                                       Time (s) % of DB Time
    sql execute elapsed time                             14,280.3        100.0
    DB CPU                                                  974.5          6.8
    PL/SQL execution elapsed time                           531.8          3.7
    parse time elapsed                                       30.5           .2
    hard parse elapsed time                                  27.1           .2
    connection management call elapsed time                  14.9           .1
    hard parse (sharing criteria) elapsed time                3.4           .0
    hard parse (bind mismatch) elapsed time                   3.1           .0
    PL/SQL compilation elapsed time                           2.4           .0
    failed parse elapsed time                                 0.0           .0
    repeated bind elapsed time                                0.0           .0
    sequence load elapsed time                                0.0           .0
    DB time                                              14,286.4          N/A
    background elapsed time                                 670.2          N/A
    background cpu time                                     186.1          N/A
    Wait Class                            DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> s  - second
    -> cs - centisecond -     100th of a second
    -> ms - millisecond -    1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc
                                                                      Avg
                                           %Time       Total Wait    wait     Waits
    Wait Class                      Waits  -outs         Time (s)    (ms)      /txn
    User I/O                    2,593,484     .0           13,415       5     150.0
    System I/O                     87,506     .0              515       6       5.1
    Other                             839   11.4                6       7       0.0
    Commit                          3,225     .1                6       2       0.2
    Concurrency                     1,033     .0                5       5       0.1
    Configuration                   2,514   99.4                0       0       0.1
    Network                        47,559     .0                0       0       2.8
    Application                         7     .0                0       0       0.0
    Wait Events                          DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> s  - second
    -> cs - centisecond -     100th of a second
    -> ms - millisecond -    1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc (idle events last)
                                                                       Avg
                                                 %Time  Total Wait    wait     Waits
    Event                                 Waits  -outs    Time (s)    (ms)      /txn
    db file sequential read           2,347,652     .0       9,215       4     135.8
    db file scattered read              245,687     .0       4,199      17      14.2
    db file parallel write               50,082     .0         408       8       2.9
    log file parallel write               6,963     .0          52       7       0.4
    control file parallel write           6,203     .0          44       7       0.4
    control file sequential read         24,242     .0          11       0       1.4
    log file sync                         3,225     .1           6       2       0.2
    latch free                               84     .0           4      47       0.0
    os thread startup                        25     .0           3     120       0.0
    latch: session allocation                39     .0           1      33       0.0
    db file parallel read                    12     .0           1      92       0.0
    enq: TX - index contention              186     .0           1       3       0.0
    latch: shared pool                       47     .0           1      11       0.0
    LGWR wait for redo copy                 319    3.1           0       1       0.0
    library cache load lock                   2     .0           0     172       0.0
    buffer busy waits                       590     .0           0       0       0.0
    log file switch completion                6     .0           0      29       0.0
    SGA: allocation forcing comp             11   54.5           0      14       0.0
    latch: library cache lock                50     .0           0       3       0.0
    read by other session                    38     .0           0       4       0.0
    direct path read                         42     .0           0       3       0.0
    SQL*Net message to client            44,807     .0           0       0       2.6
    rdbms ipc reply                         207     .0           0       0       0.0
    SQL*Net more data from clien          1,014     .0           0       0       0.1
    latch: cache buffers chains              24     .0           0       1       0.0
    latch: library cache                     29     .0           0       1       0.0
    log file sequential read                  8     .0           0       3       0.0
    direct path write                        50     .0           0       0       0.0
    SQL*Net more data to client             398     .0           0       0       0.0
    latch: object queue header o             12     .0           0       1       0.0
    latch: In memory undo latch              78     .0           0       0       0.0
    undo segment extension                2,507   99.7           0       0       0.1
    latch: cache buffers lru cha              4     .0           0       1       0.0
    log file single write                     8     .0           0       0       0.0
    local write wait                          3     .0           0       1       0.0
    enq: RO - fast object reuse               3     .0           0       1       0.0
    buffer deadlock                          87   92.0           0       0       0.0
    enq: JS - queue lock                      1     .0           0       1       0.0
    cursor: pin S                            70     .0           0       0       0.0
    latch: row cache objects                  2     .0           0       1       0.0
    SQL*Net message to dblink             1,338     .0           0       0       0.1
    latch: checkpoint queue latc              2     .0           0       0       0.0
    reliable message                          3     .0           0       0       0.0
    log buffer space                          1     .0           0       1       0.0
    SQL*Net break/reset to clien              4     .0           0       0       0.0
    SQL*Net more data from dblin              2     .0           0       0       0.0
    SQL*Net message from client          44,949     .0     155,701    3464       2.6
    virtual circuit status                  621  100.0      18,156   29237       0.0
    Streams AQ: qmn slave idle w            664     .0      18,127   27299       0.0
    Streams AQ: qmn coordinator           1,339   50.4      18,099   13517       0.1
    Streams AQ: waiting for time             12  100.0       8,741  728394       0.0
    jobq slave wait                         130  100.0         380    2927       0.0
    PL/SQL lock timer                         1  100.0           1     978       0.0
    SQL*Net message from dblink           1,338     .0           0       0       0.1
    single-task message                       1     .0           0      38       0.0
    class slave wait                         11     .0           0       1       0.0
    SQL ordered by Elapsed Time          DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> Resources reported for PL/SQL code includes the resources used by all SQL
       statements called by the code.
    -> % Total DB Time is the Elapsed Time of the SQL statement divided
       into the Total Database Time multiplied by 100
      Elapsed      CPU                  Elap per  % Total
      Time (s)   Time (s)  Executions   Exec (s)  DB Time    SQL Id
        13,664        906            0        N/A    95.6 gr2cx6athc5j5
    Module: SQL*Plus
    BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
         8,792        195            0        N/A    61.5 986fzxtzr52u5
    Module: SQL*Plus
    UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
         2,524        368            1     2524.1    17.7 c4uf0x6hdgnwq
    Module: SQL*Plus
    UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
    SYS_B_1"
         1,414        177            1     1414.4     9.9 cbg09ma34kq8w
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
           742        137            1      742.2     5.2 g0sg6v994wssq
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
           274         11            1      274.2     1.9 6mcpb06rctk0x
    Module: DBMS_SCHEDULER
    call dbms_space.auto_space_advisor_job_proc ( )
           264          8           27        9.8     1.8 8szmwam7fysa3
    Module: DBMS_SCHEDULER
    insert into wri$_adv_objspace_trend_data select timepoint, space_usage, space_a
    lloc, quality from table(dbms_space.object_growth_trend(:1, :2, :3, :4, NULL, N
    ULL, NULL, 'FALSE', :5, 'FALSE'))
            99          1            1       99.4     0.7 1z0x41f66nvjr
    Module: SQL*Plus
    UPDATE TIBEX_INSTRUMENTADMIN SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
    SYS_B_1"
            21         10            1       21.5     0.2 bbc1ck8594kvj
    Module: SQL*Plus
    UPDATE TIBEX_INSTRUMENTDAILYHIST SET ADJOPEN=NVL(ADJOPEN,OPEN), ADJHIGH=NVL(ADJH
    IGH,HIGH), ADJLOW=NVL(ADJLOW,LOW), ADJMID=NVL(ADJMID,MID), ADJCLOSE=NVL(ADJCLOSE
    ,CLOSE), ADJVOLUME=NVL(ADJVOLUME,VOLUME), ADJCLOSINGBID=NVL(ADJCLOSINGBID,CLOSIN
    GBID), ADJCLOSINGOFFER=NVL(ADJCLOSINGOFFER,CLOSINGOFFER)
            12          0            1       12.5     0.1 6xm9p9uy5kaap
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_INSTRUMENTSTATE WHERE INSTRUMENTID=:"SYS_B_0"
    SQL ordered by CPU Time              DB/Inst: SURV2/SURV2  Snaps: 19172-19178
    -> Resources reported for PL/SQL code includes the resources used by all SQL
       statements called by the code.
    -> % Total DB Time is the Elapsed Time of the SQL statement divided
       into the Total Database Time multiplied by 100
        CPU      Elapsed                  CPU per  % Total
      Time (s)   Time (s)  Executions     Exec (s) DB Time    SQL Id
           906     13,664            0         N/A    95.6 gr2cx6athc5j5
    Module: SQL*Plus
    BEGIN DBMS_OUTPUT.PUT_LINE(equiduct.eod(NULL,NULL)); END;
           368      2,524            1      367.51    17.7 c4uf0x6hdgnwq
    Module: SQL*Plus
    UPDATE TIBEX_FIXSESSIONSTATE SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"
    SYS_B_1"
           195      8,792            0         N/A    61.5 986fzxtzr52u5
    Module: SQL*Plus
    UPDATE TIBEX_ORDER SET INSTRUMENTID=:"SYS_B_0" WHERE INSTRUMENTID=:"SYS_B_1"
           177      1,414            1      176.93     9.9 cbg09ma34kq8w
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_ORDER WHERE INSTRUMENTID=:"SYS_B_0"
           137        742            1      137.38     5.2 g0sg6v994wssq
    Module: SQL*Plus
    SELECT count(*) FROM TIBEX_FIXSESSIONSTATE WHERE INSTRUMENTID=:"SYS_B_0"
            11        274            1       10.82     1.9 6mcpb06rctk0x
    Module: DBMS_SCHEDULER
    call dbms_space.auto_space_advisor_job_proc ( )
            10         21            1        9.65     0.2 bbc1ck8594kvjEdited by: NM on 10-Sep-2010 07:39

  • Performance issue showing read by other session Event

    Hi All,
    we are having a severe performance issue in my database when we are running batch jobs.
    This was a new database(11.2.0.2) and we are testing the performance by running some batch jobs. These batch jobs included some inserts and updates.
    I am seeing read by other session in top 5 timed events and cache buffers chains in Latch Miss Sources section.
    Please help me to solve this out.
    Inst Num Startup Time    Release     RAC
    1 27-Feb-12 09:03 11.2.0.2.0  NO
    Platform                         CPUs Cores Sockets Memory(GB)
    Linux x86 64-bit                    8     8       8      48.00           
    Snap Id      Snap Time      Sessions Curs/Sess
    Begin Snap:      5605 29-Feb-12 03:00:27        63       4.5
      End Snap:      5614 29-Feb-12 12:00:47        63       4.3
       Elapsed:              540.32 (mins)
       DB Time:            1,774.23 (mins)
    Cache Sizes                       Begin        End
    ~~~~~~~~~~~                  ---------- ----------
                   Buffer Cache:     1,952M     1,952M  Std Block Size:        16K
               Shared Pool Size:     1,024M     1,024M      Log Buffer:    18,868K
    Load Profile              Per Second    Per Transaction   Per Exec   Per Call
    ~~~~~~~~~~~~         ---------------    --------------- ---------- ----------
          DB Time(s):                3.3                0.8       0.02       0.05
           DB CPU(s):                1.1                0.3       0.01       0.02
           Redo size:           55,763.8           13,849.3
       Logical reads:           23,906.6            5,937.4
       Block changes:              325.7               80.9
      Physical reads:              665.6              165.3
    Physical writes:               40.4               10.0
          User calls:               60.7               15.1
              Parses:               10.6                2.6
         Hard parses:                1.1                0.3
    W/A MB processed:                0.6                0.2
              Logons:                0.1                0.0
            Executes:              151.2               37.6
           Rollbacks:                0.0                0.0
        Transactions:                4.0
    Instance Efficiency Percentages (Target 100%)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Buffer Nowait %:   99.94       Redo NoWait %:  100.00
                Buffer  Hit   %:   97.90    In-memory Sort %:  100.00
                Library Hit   %:   98.06        Soft Parse %:   90.16
             Execute to Parse %:   92.96         Latch Hit %:  100.00
    Parse CPU to Parse Elapsd %:   76.71     % Non-Parse CPU:   98.57
    Shared Pool Statistics        Begin    End
                 Memory Usage %:   89.38   87.96
        % SQL with executions>1:   97.14   95.15
      % Memory for SQL w/exec>1:   96.05   92.46
    Top 5 Timed Foreground Events
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                                               Avg
                                                              wait   % DB
    Event                                 Waits     Time(s)   (ms)   time Wait Class
    db file sequential read          14,092,706      65,613      5   61.6 User I/O
    DB CPU                                           34,819          32.7
    read by other session               308,534       1,260      4    1.2 User I/O
    direct path read                     97,454         987     10     .9 User I/O
    db file scattered read               71,870         910     13     .9 User I/O
    Host CPU (CPUs:    8 Cores:    8 Sockets:    8)
    ~~~~~~~~         Load Average
                   Begin       End     %User   %System      %WIO     %Idle
                    0.43      0.36      13.7       0.6       9.7      85.7
    Instance CPU
    ~~~~~~~~~~~~
                  % of total CPU for Instance:      13.5
                  % of busy  CPU for Instance:      94.2
      %DB time waiting for CPU - Resource Mgr:       0.0
    Memory Statistics
    ~~~~~~~~~~~~~~~~~                       Begin          End
                      Host Mem (MB):     49,152.0     49,152.0
                       SGA use (MB):      3,072.0      3,072.0
                       PGA use (MB):        506.5        629.1
        % Host Mem used for SGA+PGA:         7.28         7.53
    Time Model Statistics             
    -> Total time in database user-calls (DB Time): 106453.8s
    -> Statistics including the word "background" measure background process
       time, and so do not contribute to the DB time statistic
    -> Ordered by % or DB time desc, Statistic name
    Statistic Name                                       Time (s) % of DB Time
    sql execute elapsed time                            105,531.1         99.1
    DB CPU                                               34,818.8         32.7
    parse time elapsed                                      714.7           .7
    hard parse elapsed time                                 684.8           .6
    PL/SQL execution elapsed time                           161.9           .2
    PL/SQL compilation elapsed time                          44.2           .0
    connection management call elapsed time                  16.9           .0
    hard parse (sharing criteria) elapsed time               10.2           .0
    hard parse (bind mismatch) elapsed time                   9.4           .0
    sequence load elapsed time                                2.9           .0
    repeated bind elapsed time                                0.5           .0
    failed parse elapsed time                                 0.0           .0
    DB time                                             106,453.8
    background elapsed time                               1,753.9
    background cpu time                                      61.7
    Operating System Statistics        
    -> *TIME statistic values are diffed.
       All others display actual values.  End Value is displayed if different
    -> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name
    Statistic                                  Value        End Value
    BUSY_TIME                              3,704,415
    IDLE_TIME                             22,203,740
    IOWAIT_TIME                            2,517,864
    NICE_TIME                                      3
    SYS_TIME                                 145,696
    USER_TIME                              3,557,758
    LOAD                                           0                0
    RSRC_MGR_CPU_WAIT_TIME                         0
    VM_IN_BYTES                      358,813,045,760
    VM_OUT_BYTES                      29,514,830,848
    PHYSICAL_MEMORY_BYTES             51,539,607,552
    NUM_CPUS                                       8
    NUM_CPU_CORES                                  8
    NUM_CPU_SOCKETS                                8
    GLOBAL_RECEIVE_SIZE_MAX                4,194,304
    GLOBAL_SEND_SIZE_MAX                   1,048,586
    TCP_RECEIVE_SIZE_DEFAULT                  87,380
    TCP_RECEIVE_SIZE_MAX                   4,194,304
    TCP_RECEIVE_SIZE_MIN                       4,096
    TCP_SEND_SIZE_DEFAULT                     16,384
    TCP_SEND_SIZE_MAX                      4,194,304
    TCP_SEND_SIZE_MIN                          4,096
    Operating System Statistics -
    Snap Time           Load    %busy    %user     %sys    %idle  %iowait
    29-Feb 03:00:27      0.4      N/A      N/A      N/A      N/A      N/A
    29-Feb 04:00:35      1.4     11.9     11.2      0.6     88.1     14.3
    29-Feb 05:00:41      1.7     13.8     13.2      0.6     86.2     15.8
    29-Feb 06:00:48      1.5     14.0     13.5      0.6     86.0     12.3
    29-Feb 07:01:00      1.8     16.3     15.8      0.5     83.7     10.4
    29-Feb 08:00:12      2.6     23.2     22.5      0.6     76.8     12.6
    29-Feb 09:00:26      1.3     16.6     16.0      0.5     83.4      5.7
    29-Feb 10:00:33      1.2     13.8     13.3      0.5     86.2      2.0
    29-Feb 11:00:43      1.3     14.5     14.0      0.5     85.5      3.8
    29-Feb 12:00:47      0.4      4.9      4.2      0.7     95.1     10.6
    Foreground Wait Class              
    -> s  - second, ms - millisecond -    1000th of a second
    -> ordered by wait time desc, waits desc
    -> %Timeouts: value of 0 indicates value was < .5%.  Value of null is truly 0
    -> Captured Time accounts for         97.9%  of Total DB time     106,453.79 (s)
    -> Total FG Wait Time:            69,415.64 (s)  DB CPU time:      34,818.79 (s)
                                                                      Avg
                                          %Time       Total Wait     wait
    Wait Class                      Waits -outs         Time (s)     (ms)  %DB time
    User I/O                   14,693,843     0           69,222        5      65.0
    DB CPU                                                34,819               32.7
    Commit                         40,629     0              119        3       0.1
    System I/O                     26,504     0               57        2       0.1
    Network                     1,945,010     0               11        0       0.0
    Other                         125,200    99                4        0       0.0
    Application                     2,673     0                2        1       0.0
    Concurrency                     3,059     0                1        0       0.0
    Configuration                      31    19                0       15       0.0
    Foreground Wait Events            
    -> s  - second, ms - millisecond -    1000th of a second
    -> Only events with Total Wait Time (s) >= .001 are shown
    -> ordered by wait time desc, waits desc (idle events last)
    -> %Timeouts: value of 0 indicates value was < .5%.  Value of null is truly 0
                                                                 Avg
                                            %Time Total Wait    wait    Waits   % DB
    Event                             Waits -outs   Time (s)    (ms)     /txn   time
    db file sequential read      14,092,706     0     65,613       5    108.0   61.6
    read by other session           308,534     0      1,260       4      2.4    1.2
    direct path read                 97,454     0        987      10      0.7     .9
    db file scattered read           71,870     0        910      13      0.6     .9
    db file parallel read            35,001     0        372      11      0.3     .3
    log file sync                    40,629     0        119       3      0.3     .1
    control file sequential re       26,504     0         57       2      0.2     .1
    direct path read temp            14,499     0         49       3      0.1     .0
    direct path write temp            9,186     0         28       3      0.1     .0
    SQL*Net message to client     1,923,973     0          5       0     14.7     .0
    SQL*Net message from dblin        1,056     0          5       5      0.0     .0
    Disk file operations I/O          8,848     0          2       0      0.1     .0
    ASM file metadata operatio           36     0          2      54      0.0     .0
    SQL*Net break/reset to cli        2,636     0          1       1      0.0     .0
    ADR block file read                 472     0          1       1      0.0     .0
    os thread startup                     8     0          1      74      0.0     .0
    SQL*Net more data to clien       17,656     0          1       0      0.1     .0
    asynch descriptor resize        123,852   100          0       0      0.9     .0
    local write wait                    110     0          0       4      0.0     .0
    utl_file I/O                     55,635     0          0       0      0.4     .0
    log file switch (private s            8     0          0      52      0.0     .0
    cursor: pin S wait on X               2     0          0     142      0.0     .0
    enq: KO - fast object chec           13     0          0      20      0.0     .0
    PX Deq: Slave Session Stat          248     0          0       1      0.0     .0
    enq: RO - fast object reus           18     0          0      11      0.0     .0
    latch: cache buffers chain        2,511     0          0       0      0.0     .0
    latch: shared pool                  195     0          0       1      0.0     .0
    CSS initialization                   12     0          0       8      0.0     .0
    PX qref latch                        54   100          0       2      0.0     .0
    SQL*Net more data from cli          995     0          0       0      0.0     .0
    SQL*Net more data from dbl          300     0          0       0      0.0     .0
    kksfbc child completion               1   100          0      56      0.0     .0
    library cache: mutex X              244     0          0       0      0.0     .0
    PX Deq: Signal ACK RSG              124     0          0       0      0.0     .0
    undo segment extension                6   100          0       7      0.0     .0
    PX Deq: Signal ACK EXT              124     0          0       0      0.0     .0
    library cache load lock               3     0          0       9      0.0     .0
    ADR block file write                 45     0          0       1      0.0     .0
    CSS operation: action                12     0          0       2      0.0     .0
    reliable message                     28     0          0       1      0.0     .0
    CSS operation: query                 72     0          0       0      0.0     .0
    latch: row cache objects             14     0          0       1      0.0     .0
    enq: SQ - contention                 17     0          0       0      0.0     .0
    latch free                           32     0          0       0      0.0     .0
    buffer busy waits                    52     0          0       0      0.0     .0
    enq: PS - contention                 16     0          0       0      0.0     .0
    enq: TX - row lock content            6     0          0       1      0.0     .0
    SQL*Net message to dblink         1,018     0          0       0      0.0     .0
    cursor: pin S                        23     0          0       0      0.0     .0
    latch: cache buffers lru c            8     0          0       0      0.0     .0
    SQL*Net message from clien    1,923,970     0    944,508     491     14.7
    jobq slave wait                  66,732   100     33,334     500      0.5
    Streams AQ: waiting for me        6,481   100     32,412    5001      0.0
    wait for unread message on       32,858    98     32,411     986      0.3
    PX Deq: Execution Msg             1,448     0        190     131      0.0
    PX Deq: Execute Reply             1,196     0         74      62      0.0
    HS message to agent                 228     0          4      19      0.0
    single-task message                  42     0          4      97      0.0
    PX Deq Credit: send blkd            904     0          2       3      0.0
    PX Deq Credit: need buffer          205     0          1       3      0.0
    Foreground Wait Events            
    -> s  - second, ms - millisecond -    1000th of a second
    -> Only events with Total Wait Time (s) >= .001 are shown
    -> ordered by wait time desc, waits desc (idle events last)
    -> %Timeouts: value of 0 indicates value was < .5%.  Value of null is truly 0
                                                                 Avg
                                            %Time Total Wait    wait    Waits   % DB
    Event                             Waits -outs   Time (s)    (ms)     /txn   time
    PX Deq: Table Q Normal            4,291     0          1       0      0.0
    PX Deq: Join ACK                    124     0          0       1      0.0
    PX Deq: Parse Reply                 124     0          0       0      0.0
    KSV master wait                     256     0          0       0      0.0
    Latch Miss Sources                
    -> only latches with sleeps are shown
    -> ordered by name, sleeps desc
                                                         NoWait              Waiter
    Latch Name               Where                       Misses     Sleeps   Sleeps
    ASM map operation freeli kffmTranslate2                   0          2        0
    DML lock allocation      ktadmc                           0          2        0
    FOB s.o list latch       ksfd_allfob                      0          2        2
    In memory undo latch     ktiFlushMe                       0          5        0
    In memory undo latch     ktichg: child                    0          3        0
    PC and Classifier lists  No latch                         0          6        0
    Real-time plan statistic keswxAddNewPlanEntry             0         20       20
    SQL memory manager worka qesmmIRegisterWorkArea:1         0          1        1
    active service list      kswslogon: session logout        0         23       12
    active service list      kswssetsvc: PX session swi       0          6        1
    active service list      kswsite: service iterator        0          1        0
    archive process latch    kcrrgpll                         0          3        3
    cache buffers chains     kcbgtcr_2                        0      1,746      573
    cache buffers chains     kcbgtcr: fast path (cr pin       0      1,024    2,126
    cache buffers chains     kcbgcur_2                        0         60        8
    cache buffers chains     kcbchg1: kslbegin: bufs no       0         16        3
    cache buffers chains     kcbgtcr: fast path               0         14       20
    cache buffers chains     kcbzibmlt: multi-block rea       0         10        0
    cache buffers chains     kcbrls_2                         0          9       53
    cache buffers chains     kcbgtcr: kslbegin shared         0          8        1
    cache buffers chains     kcbrls_1                         0          7       84
    cache buffers chains     kcbgtcr: kslbegin excl           0          6       14
    cache buffers chains     kcbnew: new latch again          0          6        0
    cache buffers chains     kcbzgb: scan from tail. no       0          6        0
    cache buffers chains     kcbzwb                           0          5        8
    cache buffers chains     kcbgcur: fast path (shr)         0          3        0
    cache buffers chains     kcbget: pin buffer               0          3        0
    cache buffers chains     kcbzhngcbk2_1                    0          1        0
    cache buffers lru chain  kcbzgws                          0         19        0
    cache buffers lru chain  kcbo_link_q                      0          3        0
    call allocation          ksuxds                           0         14       10
    call allocation          ksudlp: top call                 0          2        3
    enqueue hash chains      ksqgtl3                          0          2        1
    enqueue hash chains      ksqrcl                           0          1        2
    enqueues                 ksqgel: create enqueue           0          1        0
    object queue header oper kcbo_unlink_q                    0          5        2
    object queue header oper kcbo_sw_buf                      0          2        0
    object queue header oper kcbo_link_q                      0          1        2
    object queue header oper kcbo_switch_cq                   0          1        2
    object queue header oper kcbo_switch_mq_bg                0          1        4
    parallel query alloc buf kxfpbalo                         0          1        1
    process allocation       ksucrp:1                         0          2        0
    process queue reference  kxfpqrsnd                        0          1        0
    qmn task queue latch     kwqmnmvtsks: delay to read       0          1        0
    redo allocation          kcrfw_redo_gen: redo alloc       0         17        0
    row cache objects        kqreqd: reget                    0          6        0
    row cache objects        kqrpre: find obj                 0          6       13
    row cache objects        kqrso                            0          2        0
    row cache objects        kqreqd                           0          1        2
    row cache objects        kqrpre: init complete            0          1        1
    shared pool              kghalo                           0        199      106
    shared pool              kghupr1                          0         39      109
    shared pool              kghfre                           0         18       19
    shared pool              kghalp                           0          7       29
    space background task la ktsj_grab_task                   0         21       27
    Mutex Sleep Summary                
    -> ordered by number of sleeps desc
                                                                             Wait
    Mutex Type            Location                               Sleeps    Time (ms)
    Library Cache         kglhdgn2 106                              338           12
    Library Cache         kgllkc1   57                              259           10
    Library Cache         kgllkdl1  85                              123           21
    Cursor Pin            kkslce [KKSCHLPIN2]                        70          286
    Library Cache         kglget2   2                                31            1
    Library Cache         kglhdgn1  62                               31            2
    Library Cache         kglpin1   4                                26            1
    Library Cache         kglpnal1  90                               18            0
    Library Cache         kglpndl1  95                               15            2
    Library Cache         kgllldl2 112                                6            0
    Library Cache         kglini1   32                                1            0
              -------------------------------------------------------------Thanks in advance.

    Hi,
    Thanks for reply.
    I provided one hour report.
    Inst Num Startup Time    Release     RAC
    1 27-Feb-12 09:03 11.2.0.2.0  NO
      Platform                         CPUs Cores Sockets Memory(GB)
    Linux x86 64-bit                    8     8       8      48.00
                  Snap Id      Snap Time      Sessions Curs/Sess
    Begin Snap:      5606 29-Feb-12 04:00:35        63       3.7
      End Snap:      5607 29-Feb-12 05:00:41        63       3.6
       Elapsed:               60.11 (mins)
       DB Time:              382.67 (mins)
    Cache Sizes                       Begin        End
    ~~~~~~~~~~~                  ---------- ----------
                   Buffer Cache:     1,952M     1,952M  Std Block Size:        16K
               Shared Pool Size:     1,024M     1,024M      Log Buffer:    18,868K
    Load Profile              Per Second    Per Transaction   Per Exec   Per Call
    ~~~~~~~~~~~~         ---------------    --------------- ---------- ----------
          DB Time(s):                6.4                0.8       0.03       0.03
           DB CPU(s):                1.0                0.1       0.00       0.00
           Redo size:           84,539.3           10,425.6
       Logical reads:           23,345.6            2,879.1
       Block changes:              386.5               47.7
      Physical reads:            1,605.0              197.9
    Physical writes:                7.1                0.9
          User calls:              233.9               28.9
              Parses:                4.0                0.5
         Hard parses:                0.1                0.0
    W/A MB processed:                0.1                0.0
              Logons:                0.1                0.0
            Executes:              210.9               26.0
           Rollbacks:                0.0                0.0
        Transactions:                8.1
    Instance Efficiency Percentages (Target 100%)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Buffer Nowait %:   99.62       Redo NoWait %:  100.00
                Buffer  Hit   %:   95.57    In-memory Sort %:  100.00
                Library Hit   %:   99.90        Soft Parse %:   98.68
             Execute to Parse %:   98.10         Latch Hit %:   99.99
    Parse CPU to Parse Elapsd %:   32.08     % Non-Parse CPU:   99.90
    Shared Pool Statistics        Begin    End
                 Memory Usage %:   89.25   89.45
        % SQL with executions>1:   96.79   97.52
      % Memory for SQL w/exec>1:   95.67   96.56
    Top 5 Timed Foreground Events
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                                               Avg
                                                              wait   % DB
    Event                                 Waits     Time(s)   (ms)   time Wait Class
    db file sequential read           3,054,464      17,002      6   74.0 User I/O
    DB CPU                                            3,748          16.3
    read by other session               199,603         796      4    3.5 User I/O
    direct path read                     46,301         439      9    1.9 User I/O
    db file scattered read               21,113         269     13    1.2 User I/O
    Host CPU (CPUs:    8 Cores:    8 Sockets:    8)
    ~~~~~~~~         Load Average
                   Begin       End     %User   %System      %WIO     %Idle
                    1.45      1.67      13.2       0.6      15.8      86.2
    Instance CPU
    ~~~~~~~~~~~~
                  % of total CPU for Instance:      13.0
                  % of busy  CPU for Instance:      94.7
      %DB time waiting for CPU - Resource Mgr:       0.0
    Memory Statistics
    ~~~~~~~~~~~~~~~~~                       Begin          End
                      Host Mem (MB):     49,152.0     49,152.0
                       SGA use (MB):      3,072.0      3,072.0
                       PGA use (MB):        513.5        467.7
        % Host Mem used for SGA+PGA:         7.29         7.20
    Time Model Statistics            
    -> Total time in database user-calls (DB Time): 22960.5s
    -> Statistics including the word "background" measure background process
       time, and so do not contribute to the DB time statistic
    -> Ordered by % or DB time desc, Statistic name
    Statistic Name                                       Time (s) % of DB Time
    sql execute elapsed time                             22,835.9         99.5
    DB CPU                                                3,748.4         16.3
    parse time elapsed                                       15.4           .1
    hard parse elapsed time                                  14.3           .1
    PL/SQL execution elapsed time                             7.5           .0
    PL/SQL compilation elapsed time                           6.0           .0
    connection management call elapsed time                   1.6           .0
    sequence load elapsed time                                0.4           .0
    hard parse (sharing criteria) elapsed time                0.0           .0
    repeated bind elapsed time                                0.0           .0
    failed parse elapsed time                                 0.0           .0
    DB time                                              22,960.5
    background elapsed time                                 238.1
    background cpu time                                       4.9
    Operating System Statistics        
    -> *TIME statistic values are diffed.
       All others display actual values.  End Value is displayed if different
    -> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), Name
    Statistic                                  Value        End Value
    BUSY_TIME                                396,506
    IDLE_TIME                              2,483,725
    IOWAIT_TIME                              455,495
    NICE_TIME                                      0
    SYS_TIME                                  16,163
    USER_TIME                                380,052
    LOAD                                           1                2
    RSRC_MGR_CPU_WAIT_TIME                         0
    VM_IN_BYTES                       95,646,943,232
    VM_OUT_BYTES                       1,686,059,008
    PHYSICAL_MEMORY_BYTES             51,539,607,552
    NUM_CPUS                                       8
    NUM_CPU_CORES                                  8
    NUM_CPU_SOCKETS                                8
    GLOBAL_RECEIVE_SIZE_MAX                4,194,304
    GLOBAL_SEND_SIZE_MAX                   1,048,586
    TCP_RECEIVE_SIZE_DEFAULT                  87,380
    TCP_RECEIVE_SIZE_MAX                   4,194,304
    TCP_RECEIVE_SIZE_MIN                       4,096
    TCP_SEND_SIZE_DEFAULT                     16,384
    TCP_SEND_SIZE_MAX                      4,194,304
    TCP_SEND_SIZE_MIN                          4,096
    Operating System Statistics -
    Snap Time           Load    %busy    %user     %sys    %idle  %iowait
    29-Feb 04:00:35      1.4      N/A      N/A      N/A      N/A      N/A
    29-Feb 05:00:41      1.7     13.8     13.2      0.6     86.2     15.8
    Foreground Wait Class              
    -> s  - second, ms - millisecond -    1000th of a second
    -> ordered by wait time desc, waits desc
    -> %Timeouts: value of 0 indicates value was < .5%.  Value of null is truly 0
    -> Captured Time accounts for         97.6%  of Total DB time      22,960.46 (s)
    -> Total FG Wait Time:            18,651.75 (s)  DB CPU time:       3,748.35 (s)
                                                                      Avg
                                          %Time       Total Wait     wait
    Wait Class                      Waits -outs         Time (s)     (ms)  %DB time
    User I/O                    3,327,253     0           18,576        6      80.9
    DB CPU                                                 3,748               16.3
    Commit                         23,882     0               69        3       0.3
    System I/O                      1,035     0                3        3       0.0
    Network                       842,393     0                2        0       0.0
    Other                          10,120    99                0        0       0.0
    Configuration                       3     0                0       58       0.0
    Application                       264     0                0        1       0.0
    Concurrency                     1,482     0                0        0       0.0
    Foreground Wait Events            
    -> s  - second, ms - millisecond -    1000th of a second
    -> Only events with Total Wait Time (s) >= .001 are shown
    -> ordered by wait time desc, waits desc (idle events last)
    -> %Timeouts: value of 0 indicates value was < .5%.  Value of null is truly 0
                                                                 Avg
                                            %Time Total Wait    wait    Waits   % DB
    Event                             Waits -outs   Time (s)    (ms)     /txn   time
    db file sequential read       3,054,464     0     17,002       6    104.5   74.0
    read by other session           199,603     0        796       4      6.8    3.5
    direct path read                 46,301     0        439       9      1.6    1.9
    db file scattered read           21,113     0        269      13      0.7    1.2
    log file sync                    23,882     0         69       3      0.8     .3
    db file parallel read             4,727     0         68      14      0.2     .3
    control file sequential re        1,035     0          3       3      0.0     .0
    SQL*Net message to client       840,792     0          2       0     28.8     .0
    direct path read temp                95     0          2      18      0.0     .0
    local write wait                     79     0          0       4      0.0     .0
    Disk file operations I/O            870     0          0       0      0.0     .0
    ASM file metadata operatio            4     0          0      50      0.0     .0
    log file switch (private s            3     0          0      58      0.0     .0
    ADR block file read                  36     0          0       3      0.0     .0
    enq: RO - fast object reus            5     0          0      16      0.0     .0
    latch: cache buffers chain        1,465     0          0       0      0.1     .0
    SQL*Net break/reset to cli          256     0          0       0      0.0     .0
    asynch descriptor resize         10,059   100          0       0      0.3     .0
    SQL*Net more data to clien        1,510     0          0       0      0.1     .0
    enq: KO - fast object chec            3     0          0       8      0.0     .0
    SQL*Net more data from cli           91     0          0       0      0.0     .0
    latch: shared pool                   14     0          0       0      0.0     .0
    ADR block file write                  5     0          0       1      0.0     .0
    reliable message                      8     0          0       0      0.0     .0
    direct path write temp                1     0          0       2      0.0     .0
    SQL*Net message from clien      840,794     0     68,885      82     28.8
    jobq slave wait                   7,365   100      3,679     499      0.3
    Streams AQ: waiting for me          721   100      3,605    5000      0.0
    wait for unread message on        3,648    98      3,603     988      0.1
    KSV master wait                      20     0          0       0      0.0
    Background Wait Events            
    -> ordered by wait time desc, waits desc (idle events last)
    -> Only events with Total Wait Time (s) >= .001 are shown
    -> %Timeouts: value of 0 indicates value was < .5%.  Value of null is truly 0
                                                                 Avg
                                            %Time Total Wait    wait    Waits   % bg
    Event                             Waits -outs   Time (s)    (ms)     /txn   time
    log file parallel write          29,353     0         83       3      1.0   34.8
    db file parallel write            5,753     0         17       3      0.2    6.9
    db file sequential read           1,638     0         15       9      0.1    6.1
    control file sequential re        5,142     0         13       2      0.2    5.4
    os thread startup                   140     0          8      58      0.0    3.4
    control file parallel writ        1,440     0          8       6      0.0    3.4
    log file sequential read            304     0          8      26      0.0    3.3
    db file scattered read              214     0          2       9      0.0     .8
    ASM file metadata operatio        1,199     0          1       1      0.0     .3
    direct path write                    35     0          0       6      0.0     .1
    direct path read                     41     0          0       5      0.0     .1
    kfk: async disk IO                    6     0          0       9      0.0     .0
    Disk file operations I/O          1,266     0          0       0      0.0     .0
    ADR block file read                  16     0          0       2      0.0     .0
    read by other session                 3     0          0       8      0.0     .0
    Log archive I/O                       2     0          0      10      0.0     .0
    log file sync                         3     0          0       5      0.0     .0
    asynch descriptor resize            341   100          0       0      0.0     .0
    CSS initialization                    1     0          0       6      0.0     .0
    log file single write                 4     0          0       1      0.0     .0
    latch: redo allocation                3     0          0       1      0.0     .0
    ADR block file write                  5     0          0       1      0.0     .0
    LGWR wait for redo copy              45     0          0       0      0.0     .0
    CSS operation: query                  6     0          0       0      0.0     .0
    CSS operation: action                 1     0          0       1      0.0     .0
    SQL*Net message to client           420     0          0       0      0.0     .0
    rdbms ipc message                47,816    39     61,046    1277      1.6
    DIAG idle wait                    7,200   100      7,200    1000      0.2
    Space Manager: slave idle         1,146    98      5,674    4951      0.0
    class slave wait                    284     0      3,983   14026      0.0
    dispatcher timer                     61   100      3,660   60006      0.0
    Streams AQ: qmn coordinato          258    50      3,613   14003      0.0
    Streams AQ: qmn slave idle          130     0      3,613   27789      0.0
    Streams AQ: waiting for ti            7    71      3,608  515430      0.0
    wait for unread message on        3,605   100      3,606    1000      0.1
    pmon timer                        1,201   100      3,604    3001      0.0
    smon timer                           15    73      3,603  240207      0.0
    ASM background timer                754     0      3,602    4777      0.0
    shared server idle wait             120   100      3,601   30006      0.0
    SQL*Net message from clien          554     0          4       7      0.0
    KSV master wait                     101     0          0       2      0.0
    Wait Event Histogram              
    -> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
    -> % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
    -> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
    -> Ordered by Event (idle events last)
                                                        % of Waits
                               Total
    Event                      Waits  <1ms  <2ms  <4ms  <8ms <16ms <32ms  <=1s   >1s
    ADR block file read           52  73.1   1.9   9.6  13.5               1.9
    ADR block file write          10 100.0
    ADR file lock                 12 100.0
    ARCH wait for archivelog l     3 100.0
    ASM file metadata operatio  1203  97.3    .5    .7    .3    .2          .9
    CSS initialization             1                   100.0
    CSS operation: action          1       100.0
    CSS operation: query           6  83.3  16.7
    Disk file operations I/O    2118  95.4   4.5    .1
    LGWR wait for redo copy       45 100.0
    Log archive I/O                2                         100.0
    SQL*Net break/reset to cli   256  99.6    .4
    SQL*Net message to client  839.9 100.0    .0
    SQL*Net more data from cli    91 100.0
    SQL*Net more data to clien  1503 100.0
    asynch descriptor resize   10.4K 100.0
    buffer busy waits              2 100.0
    control file parallel writ  1440   5.7  35.1  24.0  16.3  12.0   5.5   1.5
    control file sequential re  6177  69.4   7.5   5.9   8.1   7.1   1.7    .3
    db file parallel read       4727   1.7   3.2   3.2  10.1  46.6  33.3   1.8
    db file parallel write      5755  42.3  21.3  18.6  11.2   4.6   1.4    .5
    db file scattered read     21.5K   8.4   4.3  11.9  18.9  26.3  25.3   4.9
    db file sequential read    3053.  28.7  15.1  11.1  17.9  21.5   5.4    .3    .0
    direct path read           46.3K   9.9   8.8  18.5  21.7  22.8  15.7   2.7
    direct path read temp         95               9.5   9.5  23.2  49.5   8.4
    direct path write             35  11.4  31.4  17.1  22.9  11.4   2.9   2.9
    direct path write temp         1       100.0
    enq: KO - fast object chec     3                    66.7  33.3
    enq: RO - fast object reus     5  20.0              20.0  20.0  20.0  20.0
    kfk: async disk IO             6  50.0  16.7              16.7        16.7
    latch free                     3 100.0
    latch: cache buffers chain  1465 100.0
    latch: cache buffers lru c     1 100.0
    latch: object queue header     2 100.0
    latch: redo allocation         3  33.3  33.3  33.3
    latch: row cache objects       2 100.0
    latch: shared pool            15  93.3   6.7
    local write wait              79        35.4  34.2  21.5   8.9
    log file parallel write    29.4K  47.8  21.7  11.9   9.9   6.8   1.6    .3
    log file sequential read     304   6.3   3.0   3.6  10.2  23.4  24.3  29.3
    log file single write          4  25.0  75.0
    log file switch (private s     3                                     100.0
    log file sync              23.9K  40.9  28.0  12.9   9.7   6.7   1.5    .3
    os thread startup            140                                     100.0
    read by other session      199.6  37.1  19.9  12.9  13.1  13.8   3.1    .2
    reliable message               8 100.0
    ASM background timer         755   2.9    .4    .1    .1    .3    .1    .3  95.8
    DIAG idle wait              7196                                     100.0
    KSV master wait              121  88.4   2.5   3.3   2.5    .8    .8   1.7
    SQL*Net message from clien 840.1  97.1   1.8    .5    .2    .2    .1    .0    .1
    Space Manager: slave idle   1147    .1                                  .5  99.4
    Streams AQ: qmn coordinato   258  49.6                .4                    50.0
    Streams AQ: qmn slave idle   130    .8                                      99.2
    Streams AQ: waiting for me   721                                           100.0
    Streams AQ: waiting for ti     7  28.6                                42.9  28.6
    class slave wait             283  39.9   2.5   2.5   3.5   4.9   9.2  15.2  22.3
    dispatcher timer              60                                           100.0
    jobq slave wait             7360    .0    .0    .0                    99.9
    pmon timer                  1201                                           100.0
    rdbms ipc message          47.8K   2.7  31.6  17.4   1.1   1.1    .9  20.9  24.3
    Wait Event Histogram               DB/Inst: I2KPROD/I2KPROD  Snaps: 5606-5607
    -> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000
    -> % of Waits: value of .0 indicates value was <.05%; value of null is truly 0
    -> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms
    -> Ordered by Event (idle events last)
                                                        % of Waits
                               Total
    Event                      Waits  <1ms  <2ms  <4ms  <8ms <16ms <32ms  <=1s   >1s
    shared server idle wait      120                                           100.0
    smon timer                    16                                       6.3  93.8
    wait for unread message on  7250                                  .1  99.9
    Latch Miss Sources                
    -> only latches with sleeps are shown
    -> ordered by name, sleeps desc
                                                         NoWait              Waiter
    Latch Name               Where                       Misses     Sleeps   Sleeps
    In memory undo latch     ktichg: child                    0          1        0
    active service list      kswslogon: session logout        0          2        0
    cache buffers chains     kcbgtcr_2                        0      1,123      483
    cache buffers chains     kcbgtcr: fast path (cr pin       0        496    1,131
    cache buffers chains     kcbrls_2                         0          5        6
    cache buffers chains     kcbgcur_2                        0          4        0
    cache buffers chains     kcbgtcr: fast path               0          3        1
    cache buffers chains     kcbzwb                           0          2        4
    cache buffers chains     kcbchg1: kslbegin: bufs no       0          1        0
    cache buffers chains     kcbnew: new latch again          0          1        0
    cache buffers chains     kcbrls_1                         0          1        6
    cache buffers chains     kcbzgb: scan from tail. no       0          1        0
    cache buffers lru chain  kcbzgws                          0          1        0
    object queue header oper kcbo_switch_cq                   0          1        0
    object queue header oper kcbo_switch_mq_bg                0          1        2
    redo allocation          kcrfw_redo_gen: redo alloc       0          3        0
    row cache objects        kqrpre: find obj                 0          1        1
    row cache objects        kqrso                            0          1        0
    shared pool              kghalo                           0         13        3
    shared pool              kghupr1                          0          4       15
    shared pool              kghalp                           0          1        0
    space background task la ktsj_grab_task                   0          2        2
              -------------------------------------------------------------

  • Performance issue with high CPU and IO

    Hi guys,
    I am encountering huge user response time on a production system and I don’t know how to solve it.
    Doing some extra tests and using the instrumentation that we have in the code we concluded that the DB is the bottleneck.
    We generated some AWR reports and noticed the CPU was in top wait events. Also noticed that in a random manner some simple sql take a long time to execute. We activated the sql trace on the system and noticed that for very simple SQLs (unique index access on one table) we have huge exec times. 9s
    In the trace file the huge time we had it in fetch area: 9.1s cpu and elapsed 9.2.
    And no or very small waits for this specific SQL.
    it seems like the bottle neck is on the CPU but at that point there were very few processes running on the DB. Why can we have such a big cpu wait on a simple select? This is a machine with 128 cores. We have quicker responses on machines smaller/busier than this.
    We noticed that we had a huge db_cache_size (12G) and after we scale it down we noticed some improvements but not enough. How can I prove that there is a link between high CPU and big cache_size? (there was not wait involved in SQL execution). what can we do in the case we need big DB cache size?
    The second issue is that I tried to execute an sql on a big table (FTS on a big table. no join). Again on that smaller machine it runs in 30 seconds and on this machine it runs in 1038 seconds.
    Also generated a trace for this SQL on the problematic machine:
    call     count       cpu    elapsed       disk      query    current        rows
    Parse        1      0.00       0.00          0          0          0           0
    Execute      1      0.00       0.00          0          0          0           0
    Fetch        1    402.08    1038.31    1842916    6174343          0           1
    total        3    402.08    1038.32    1842916    6174343          0           1
      db file sequential read                     12419        0.21         40.02
      i/o slave wait                             135475        0.51        613.03
      db file scattered read                     135475        0.52        675.15
      log file switch completion                      5        0.06          0.18
      latch: In memory undo latch                     6        0.00          0.00
      latch: object queue header operation            1        0.00          0.00
    ********************************************************************************The high CPU is present here also but here I have huge wait on db file scattered read.
    Looking at the session with the select the AWG_wait for db scattered read was 0.5. on the other machine it is like 0.07.
    I though this is an IO issue. I did some IO tests at SO level and it seems like the read and writes operation are very fast…much faster than the machine that has the awg_wait smaller. Why the difference in waits?
    One difference between these two DBs is that the problem one has the db block size = 16k and the other one has 8k.
    I received some reports done at OS level on CPU and IO usage on the problematic machine (in normal operations). It seems like the CPU is very used and the IO stays very low.
    On the other machine, the smaller and the faster one, it is other way around.
    What is the problem here? How can I test further? Can I link the high CPU to low/slow IO?
    we have 10G on sun os with ASM.
    Thanks in advance.

    Yes, there are many things you can and should do to isolate this. But first check MOS Poor Performance With Oracle9i and 10g Releases When Using Dynamic Intimate Shared Memory (DISM) [ID 1018855.1] isn't messing you up to start.
    Also, be sure and post exact patch levels for both Oracle and OS.
    Be sure and check all your I/O settings and see what MOS has to say about those.
    Are you using ASSM? See Long running update
    Since it got a little better with shrinking the SGA size, that might indicate (wild speculation here, something like) one of the problems is simply too much thrashing within the SGA, as oracle decides "small" objects being full scanned in memory is faster than range scans (or whatever) from disk, overloading the cpu, not allowing the cpu to ask for other full scans from I/O. Possibly made worse by row level locking, or some other app issue that just does too much cpu.
    You probably have more than one thing wrong. High fetch count might mean you need to adjust the array size on the clients.
    Now that that is all out of the way, if you still haven't found the problem, go through http://oracle-randolf.blogspot.com/2009/02/basic-sql-statement-performance.html
    Edit: Oh, see Solaris 10 memory management conflicts with Automatic PGA Memory Management [ID 460424.1] too.
    Edited by: jgarry on Nov 15, 2011 1:45 PM

  • AWR Report - suggestions to improve performance.

    Hi Folks,
    Web application server version - 10.1.2.0.2
    Database version - 10.1.0.4.0
    OS version - AIX 5.3
    WORKLOAD REPOSITORY report for
    DB Name         DB Id    Instance     Inst Num Release     Cluster Host
    xxxx      3419432639 xxxx            1 10.1.0.4.0  NO      hostname
                  Snap Id      Snap Time      Sessions Curs/Sess
    Begin Snap:     16598 07-Jul-09 16:30:54        23      60.7
      End Snap:     16599 07-Jul-09 17:30:46        36      94.1
       Elapsed:               59.88 (mins)
       DB Time:               44.99 (mins)
    Cache Sizes (end)
    ~~~~~~~~~~~~~~~~~
                   Buffer Cache:     1,536M      Std Block Size:         8K
               Shared Pool Size:     1,248M          Log Buffer:     1,024K
    Load Profile
    ~~~~~~~~~~~~                            Per Second       Per Transaction
                      Redo size:             30,495.86            138,156.74
                  Logical reads:              1,735.49              7,862.39
                  Block changes:                197.66                895.45
                 Physical reads:                564.70              2,558.28
                Physical writes:                 54.64                247.52
                     User calls:                  2.00                  9.04
                         Parses:                 14.32                 64.87
                    Hard parses:                  1.37                  6.22
                          Sorts:                  5.83                 26.42
                         Logons:                  0.12                  0.54
                       Executes:                 72.71                329.38
                   Transactions:                  0.22
      % Blocks changed per Read:   11.39    Recursive Call %:    99.64
    Rollback per transaction %:   17.53       Rows per Sort:  5933.99
    Instance Efficiency Percentages (Target 100%)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Buffer Nowait %:  100.00       Redo NoWait %:  100.00
                Buffer  Hit   %:   71.24    In-memory Sort %:   99.79
                Library Hit   %:   96.22        Soft Parse %:   90.41
             Execute to Parse %:   80.31         Latch Hit %:   99.99
    Parse CPU to Parse Elapsd %:   21.82     % Non-Parse CPU:   96.80
    Shared Pool Statistics        Begin    End
                 Memory Usage %:   13.13   27.55
        % SQL with executions>1:   61.04   15.52
      % Memory for SQL w/exec>1:   60.96   12.31
    Top 5 Timed Events
    ~~~~~~~~~~~~~~~~~~                                        % Total
    Event                                 Waits    Time (s)   DB Time     Wait Class
    db file scattered read              135,977       1,204     44.62       User I/O
    CPU time                                            547     20.26
    db file sequential read              99,521         187      6.92       User I/O
    log file parallel write              10,995          16       .59     System I/O
    direct path write temp                7,225          13       .47       User I/O
    Wait Events  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> s  - second
    -> cs - centisecond -     100th of a second
    -> ms - millisecond -    1000th of a second
    -> us - microsecond - 1000000th of a second
    -> ordered by wait time desc, waits desc (idle events last)
                                                                       Avg
                                                        Total Wait    wait     Waits
    Event                              Waits   Timeouts   Time (s)    (ms)      /txn
    db file scattered read           135,977          0      1,204       9     171.5
    db file sequential read           99,521          0        187       2     125.5
    log file parallel write           10,995          0         16       1      13.9
    direct path write temp             7,225          0         13       2       9.1
    process startup                      117          1          7      58       0.1
    latch: library cache                 306          0          5      15       0.4
    direct path read temp            152,490          0          4       0     192.3
    db file parallel write               342          0          4      11       0.4
    log file sync                        309          3          4      13       0.4
    db file parallel read                359          0          3       8       0.5
    latch: cache buffers lru ch           80          0          2      19       0.1
    control file parallel write        1,325          0          1       1       1.7
    latch: row cache objects              26          0          1      51       0.0
    latch: cache buffers chains          138        138          1       9       0.2
    log file switch completion             3          0          1     399       0.0
    buffer busy waits                     94          0          1      10       0.1
    latch: shared pool                    44          0          0       6       0.1
    control file sequential rea        1,258          0          0       0       1.6
    latch: object queue header             3          0          0      41       0.0
    SQL*Net message to client          5,866          0          0       0       7.4
    LGWR wait for redo copy              495          1          0       0       0.6
    direct path read                      58          0          0       1       0.1
    latch: session allocation              6          0          0       6       0.0
    latch: In memory undo latch            3          0          0      10       0.0
    SQL*Net break/reset to clie           62          0          0       0       0.1
    control file single write             40          0          0       1       0.1
    latch: redo allocation                12          0          0       2       0.0
    latch free                             2          0          0      11       0.0
    enq: TX - index contention             1          0          0      21       0.0
    latch: messages                        8          0          0       2       0.0
    log file single write                  2          0          0       4       0.0
    direct path write                     99          0          0       0       0.1
    SQL*Net more data to client          237          0          0       0       0.3
    log file sequential read               2          0          0       2       0.0
    latch: library cache lock              1          0          0       3       0.0
    latch: enqueue hash chains             3          0          0       0       0.0
    SQL*Net more data from clie            2          0          0       0       0.0
    SQL*Net message from client        5,855          0     23,607    4032       7.4
    jobq slave wait                    3,441      3,334      9,925    2884       4.3
    Queue Monitor Wait                 1,537        623      3,532    2298       1.9
    Queue Monitor Slave Wait             503          0      3,532    7022       0.6
    wakeup time manager                   12          0      3,527  293898       0.0
    virtual circuit status               120        120      3,512   29266       0.2
    class slave wait                       2          2         10    4883       0.0
    Background Wait Events  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> ordered by wait time desc, waits desc (idle events last)
                                                                       Avg
                                                        Total Wait    wait     Waits
    Event                              Waits   Timeouts   Time (s)    (ms)      /txn
    log file parallel write           10,995          0         16       1      13.9
    process startup                      117          1          7      58       0.1
    db file sequential read            1,255          0          4       3       1.6
    db file parallel write               342          0          4      11       0.4
    control file parallel write        1,329          0          1       1       1.7
    db file scattered read               151          0          1       7       0.2
    latch: cache buffers lru ch           32          0          0      15       0.0
    latch: library cache                   5          0          0      69       0.0
    control file sequential rea          213          0          0       1       0.3
    latch: cache buffers chains            2          2          0      34       0.0
    LGWR wait for redo copy              495          1          0       0       0.6
    control file single write             40          0          0       1       0.1
    latch: redo allocation                12          0          0       2       0.0
    latch: messages                        8          0          0       2       0.0
    latch: object queue header             2          0          0       8       0.0
    log file single write                  2          0          0       4       0.0
    direct path write                     25          0          0       0       0.0
    buffer busy waits                      1          0          0       7       0.0
    latch: shared pool                     2          0          0       2       0.0
    log file sequential read               2          0          0       2       0.0
    direct path read                      22          0          0       0       0.0
    rdbms ipc message                 15,461     10,346     26,675    1725      19.5
    Queue Monitor Wait                 1,537        623      3,532    2298       1.9
    Queue Monitor Slave Wait             503          0      3,532    7022       0.6
    wakeup time manager                   12          0      3,527  293898       0.0
    smon timer                           175          8      3,418   19530       0.2
    Time Model Statistics  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> ordered by Time (seconds) desc
                                                           Time      % Total
    Statistic Name                                     (seconds)     DB Time
    sql execute elapsed time                            2,708.75      100.35
    DB time                                             2,699.22      100.00
    DB CPU                                                546.94       20.26
    background elapsed time                               113.64        4.21
    parse time elapsed                                     90.88        3.37
    hard parse elapsed time                                59.03        2.19
    PL/SQL execution elapsed time                          56.46        2.09
    Java execution elapsed time                             8.99         .33
    background cpu time                                     8.18         .30
    PL/SQL compilation elapsed time                         4.55         .17
    connection management call elapsed time                 1.08         .04
    failed parse elapsed time                                .73         .03
    hard parse (sharing criteria) elapsed time               .70         .03
    hard parse (bind mismatch) elapsed time                  .61         .02
    sequence load elapsed time                               .41         .02
    inbound PL/SQL rpc elapsed time                          .00         .00
    failed parse (out of shared memory) elapsed t            .00         .00
    Operating System Statistics  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    Statistic Name                                   Value
    AVG_BUSY_TICKS                                 771,364
    AVG_IDLE_TICKS                                 346,024
    AVG_IN_BYTES                             3,812,632,576
    AVG_IOWAIT_TICKS                                38,470
    AVG_OUT_BYTES                            1,141,137,408
    AVG_SYS_TICKS                                    2,786
    AVG_USER_TICKS                                  13,588
    BUSY_TICKS                                   3,085,837
    IDLE_TICKS                                   1,384,395
    IN_BYTES                                15,250,530,304
    IOWAIT_TICKS                                   154,103
    OS_CPU_WAIT_TIME                                70,200
    OUT_BYTES                                4,564,549,632
    RSRC_MGR_CPU_WAIT_TIME                               0
    SYS_TICKS                                       11,439
    USER_TICKS                                      54,613
    Service Statistics  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> ordered by DB Time
                                                                 Physical    Logical
    Service Name                      DB Time (s)   DB CPU (s)      Reads      Reads
    xxxx                              2,032.9        519.7  1,804,948  5,709,611
    SYS$USERS                               666.3         27.2    219,320    501,732
    SYS$BACKGROUND                            0.0          0.0      4,565     28,708
    xxxxXDB                               0.0          0.0          0          0
    Service Wait Class Stats  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> Wait Class info for services in the Service Statistics section.
    -> Total Waits and Time Waited displayed for the following wait
       classes:  User I/O, Concurrency, Administrative, Network
    -> Time Waited (Wt Time) in centisecond (100th of a second)
    Service Name
    User I/O  User I/O  Concurcy  Concurcy     Admin     Admin   Network   Network
    Total Wts   Wt Time Total Wts   Wt Time Total Wts   Wt Time Total Wts   Wt Time
    xxxx
       374696    127922         0         0         0         0      3701         7
    SYS$USERS
        18820     12406       172       222         0         0      2129         0
    SYS$BACKGROUND
         2218       783         0         0         0         0         0         0
    SQL ordered by Elapsed Time  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> Resources reported for PL/SQL code includes the resources used by all SQL
       statements called by the code.
    -> % Total DB Time is the Elapsed Time of the SQL statement divided
       into the Total Database Time multiplied by 100
    <<<<<<<<<<<<<<<<<<<<<<<<<<<SQLs Deleted >>>>>>>>>>>>      
    SQL ordered by Sharable Memory  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
                      No data exists for this section of the report.
    SQL ordered by Version Count  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
                      No data exists for this section of the report.
    Instance Activity Stats  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    Statistic                                     Total     per Second     per Trans
    CPU used by this session                     53,470           14.9          67.4
    CPU used when call started                   51,928           14.5          65.5
    CR blocks created                             5,298            1.5           6.7
    Cached Commit SCN referenced                192,226           53.5         242.4
    Commit SCN cached                             3,613            1.0           4.6
    DB time                                   1,253,838          349.0       1,581.1
    DBWR checkpoint buffers written               4,304            1.2           5.4
    DBWR checkpoints                                  1            0.0           0.0
    DBWR transaction table writes                    59            0.0           0.1
    DBWR undo block writes                        5,557            1.6           7.0
    IMU CR rollbacks                                  1            0.0           0.0
    IMU Flushes                                     191            0.1           0.2
    IMU Redo allocation size                    860,820          239.6       1,085.5
    IMU commits                                     381            0.1           0.5
    IMU contention                                    5            0.0           0.0
    IMU ktichg flush                                  1            0.0           0.0
    IMU undo allocation size                  1,839,872          512.1       2,320.1
    SQL*Net roundtrips to/from clien              5,591            1.6           7.1
    active txn count during cleanout                948            0.3           1.2
    application wait time                             4            0.0           0.0
    background checkpoints completed                  1            0.0           0.0
    background checkpoints started                    1            0.0           0.0
    background timeouts                          10,366            2.9          13.1
    buffer is not pinned count                2,387,595          664.6       3,010.8
    buffer is pinned count                      792,051          220.5         998.8
    bytes received via SQL*Net from             333,467           92.8         420.5
    bytes sent via SQL*Net to client          2,449,972          682.0       3,089.5
    calls to get snapshot scn: kcmgs            319,869           89.0         403.4
    calls to kcmgas                              17,576            4.9          22.2
    calls to kcmgcs                               1,912            0.5           2.4
    change write time                             2,175            0.6           2.7
    cleanout - number of ktugct call              4,589            1.3           5.8
    cluster key scan block gets                 579,340          161.3         730.6
    cluster key scans                           283,013           78.8         356.9
    commit cleanout failures: block                   1            0.0           0.0
    commit cleanout failures: buffer                  1            0.0           0.0
    commit cleanout failures: callba                  1            0.0           0.0
    commit cleanout failures: cannot                  5            0.0           0.0
    commit cleanouts                             25,713            7.2          32.4
    commit cleanouts successfully co             25,705            7.2          32.4
    commit txn count during cleanout              4,110            1.1           5.2
    concurrency wait time                           650            0.2           0.8
    consistent changes                            7,352            2.1           9.3
    consistent gets                           5,509,829        1,533.7       6,948.1
    consistent gets - examination             1,061,141          295.4       1,338.1
    consistent gets direct                           39            0.0           0.1
    consistent gets from cache                5,509,790        1,533.7       6,948.0
    cursor authentications                          310            0.1           0.4
    data blocks consistent reads - u              6,747            1.9           8.5
    db block changes                            710,092          197.7         895.5
    db block gets                               725,044          201.8         914.3
    db block gets direct                         14,789            4.1          18.7
    db block gets from cache                    710,255          197.7         895.7
    deferred (CURRENT) block cleanou             14,036            3.9          17.7
    dirty buffers inspected                       9,309            2.6          11.7
    enqueue conversions                             926            0.3           1.2
    enqueue releases                             92,643           25.8         116.8
    enqueue requests                             92,643           25.8         116.8
    enqueue waits                                     1            0.0           0.0
    execute count                               261,199           72.7         329.4
    Instance Activity Stats  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    Statistic                                     Total     per Second     per Trans
    free buffer inspected                     1,719,681          478.7       2,168.6
    free buffer requested                     1,804,893          502.4       2,276.0
    heap block compress                           1,617            0.5           2.0
    hot buffers moved to head of LRU            295,841           82.4         373.1
    immediate (CR) block cleanout ap              3,664            1.0           4.6
    immediate (CURRENT) block cleano              6,108            1.7           7.7
    index fast full scans (full)                    533            0.2           0.7
    index fetch by key                          582,804          162.2         734.9
    index scans kdiixs1                         328,534           91.5         414.3
    leaf node 90-10 splits                          190            0.1           0.2
    leaf node splits                                353            0.1           0.5
    logons cumulative                               426            0.1           0.5
    messages received                             5,699            1.6           7.2
    messages sent                                 5,699            1.6           7.2
    no buffer to keep pinned count                    0            0.0           0.0
    no work - consistent read gets            4,088,134        1,137.9       5,155.3
    opened cursors cumulative                    46,867           13.1          59.1
    parse count (failures)                           27            0.0           0.0
    parse count (hard)                            4,933            1.4           6.2
    parse count (total)                          51,443           14.3          64.9
    parse time cpu                                1,748            0.5           2.2
    parse time elapsed                            8,012            2.2          10.1
    physical read IO requests                   390,490          108.7         492.4
    physical reads                            2,028,713          564.7       2,558.3
    physical reads cache                      1,793,229          499.2       2,261.3
    physical reads cache prefetch             1,557,452          433.5       1,964.0
    physical reads direct                       235,484           65.6         297.0
    physical reads direct (lob)                      36            0.0           0.1
    physical reads direct temporary             235,426           65.5         296.9
    physical reads prefetch warmup                6,031            1.7           7.6
    physical write IO requests                   15,092            4.2          19.0
    physical writes                             196,287           54.6         247.5
    physical writes direct                      179,731           50.0         226.7
    physical writes direct (lob)                     39            0.0           0.1
    physical writes direct temporary            179,664           50.0         226.6
    physical writes from cache                   16,556            4.6          20.9
    physical writes non checkpoint              194,791           54.2         245.6
    pinned buffers inspected                          2            0.0           0.0
    prefetch warmup blocks aged out               1,292            0.4           1.6
    prefetched blocks aged out befor            293,795           81.8         370.5
    process last non-idle time                    1,954            0.5           2.5
    recursive calls                           1,997,982          556.1       2,519.5
    recursive cpu usage                          50,834           14.2          64.1
    redo blocks written                         224,755           62.6         283.4
    redo buffer allocation retries                    3            0.0           0.0
    redo entries                                357,063           99.4         450.3
    redo log space requests                           3            0.0           0.0
    redo log space wait time                        123            0.0           0.2
    redo ordering marks                           5,391            1.5           6.8
    redo size                               109,558,292       30,495.9     138,156.7
    redo synch time                                 398            0.1           0.5
    redo synch writes                               300            0.1           0.4
    redo wastage                              1,486,252          413.7       1,874.2
    redo write time                               1,657            0.5           2.1
    redo writer latching time                         7            0.0           0.0
    redo writes                                   5,499            1.5           6.9
    rollback changes - undo records               5,688            1.6           7.2
    rows fetched via callback                    96,659           26.9         121.9
    session connect time                              0            0.0           0.0
    session logical reads                     6,234,873        1,735.5       7,862.4
    Instance Activity Stats  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    Statistic                                     Total     per Second     per Trans
    session pga memory                      162,220,520       45,154.5     204,565.6
    session pga memory max                  341,641,032       95,096.7     430,821.0
    session uga memory                  790,286,168,264  219,978,374.3 #############
    session uga memory max                  267,584,080       74,482.8     337,432.6
    shared hash latch upgrades - no             327,865           91.3         413.5
    shared hash latch upgrades - wai                 47            0.0           0.1
    sorts (disk)                                     44            0.0           0.1
    sorts (memory)                               20,909            5.8          26.4
    sorts (rows)                            124,334,820       34,609.0     156,790.4
    summed dirty queue length                    10,849            3.0          13.7
    switch current to new buffer                    125            0.0           0.2
    table fetch by rowid                        362,785          101.0         457.5
    table fetch continued row                     1,811            0.5           2.3
    table scan blocks gotten                  2,989,662          832.2       3,770.1
    table scan rows gotten                  102,945,486       28,655.2     129,817.8
    table scans (long tables)                        65            0.0           0.1
    table scans (short tables)                    8,082            2.3          10.2
    transaction rollbacks                            72            0.0           0.1
    undo change vector size                  40,524,316       11,280.1      51,102.5
    user I/O wait time                          140,959           39.2         177.8
    user calls                                    7,168            2.0           9.0
    user commits                                    654            0.2           0.8
    user rollbacks                                  139            0.0           0.2
    workarea executions - onepass                    30            0.0           0.0
    workarea executions - optimal                13,486            3.8          17.0
    write clones created in foregrou                  6            0.0           0.0
              -------------------------------------------------------------contd on next ...
    Edited by: LazyDBA10g on Jul 8, 2009 4:50 AM

    Instance Activity Stats - Absolute Values  DB/Inst: xxxx/xxxx  Snaps: 1
    -> Statistics with absolute values (should not be diffed)
    -> Statistics identified by '(derived)' come from sources other than SYSSTAT
    Statistic                            Begin Value       End Value
    opened cursors current                     1,397           3,387
    logons current                                23              36
    Instance Activity Stats - Thread Activity  DB/Inst: xxxx/xxxx  Snaps: 16
    Statistic                                     Total  per Hour
    log switches (derived)                            1      1.00
    Tablespace IO Stats  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> ordered by IOs (Reads + Writes) desc
    Tablespace
                     Av      Av     Av                       Av     Buffer Av Buf
             Reads Reads/s Rd(ms) Blks/Rd       Writes Writes/s      Waits Wt(ms)
    TEMP2
           153,326      43    0.2     1.6        6,040        2          0    0.0
    AA_DATA
           115,909      32    5.8     7.6          217        0          0    0.0
    AA_WEB
            57,358      16    9.1    12.8        2,200        1          0    0.0
    AA_INDEX
            43,576      12    3.3     3.3          738        0          0    0.0
    AA_WEB_IDX
            11,475       3    2.9     1.1            6        0          0    0.0
    SYSTEM
             5,575       2    3.8     1.4        2,528        1         61   16.1
    SYSAUX
             3,017       1    5.3     1.4        2,401        1          5    0.0
    UNDOTBS1
               262       0    5.0     1.0          955        0         28    0.0
    AA_LOB
                36       0   13.6     5.5            3        0          0    0.0
    USERS
                 3       0   33.3     1.0            3        0          0    0.0
    EXAMPLE
                 1       0   40.0     1.0            1        0          0    0.0
    File IO Stats  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> ordered by Tablespace, File
    Tablespace               Filename
                     Av      Av     Av                       Av     Buffer Av Buf
             Reads Reads/s Rd(ms) Blks/Rd       Writes Writes/s      Waits Wt(ms)
    EXAMPLE                  /AAA/oradata/xxxx/example01.dbf
                 1       0   40.0     1.0            1        0          0    0.0
    SYSAUX                   /AAA/oradata/xxxx/sysaux01.dbf
             3,017       1    5.3     1.4        2,401        1          5    0.0
    SYSTEM                   /AAA/oradata/xxxx/system01.dbf
             5,575       2    3.8     1.4        2,528        1         61   16.1
    TEMP2                    /AAA/oradata/xxxx/TEMP02.DBFf
           153,324      43    0.2     1.6        6,040        2          0
    TEMP2                    /AAA/oradata/xxxx/TEMP03.DBF
                 2       0    5.0     1.0            0        0          0
    UNDOTBS1                 /AAA/oradata/xxxx/undotbs01.dbf
               181       0    4.9     1.0          307        0          3    0.0
    UNDOTBS1                 /AAA/oradata/xxxx/undotbs02.dbf
                70       0    5.6     1.0          157        0         20    0.0
    UNDOTBS1                 /AAA/oradata/xxxx/undotbs021.dbf
                11       0    2.7     1.0          491        0          5    0.0
    USERS                    /AAA/oradata/xxxx/users01.dbf
                 1       0   50.0     1.0            1        0          0    0.0
    USERS                    /AAA/oradata/xxxx/users02.dbf
                 1       0   30.0     1.0            1        0          0    0.0
    USERS                    /AAA/oradata/xxxx/users03.dbf
                 1       0   20.0     1.0            1        0          0    0.0
    AA_DATA              /AAA/oradata/xxxx/AA_DATA01.DBF
            28,830       8    5.8     7.6           80        0          0    0.0
    AA_DATA              /AAA/oradata/xxxx/AA_DATA02.DBF
            30,068       8    5.8     7.5           34        0          0    0.0
    AA_DATA              /AAA/oradata/xxxx/AA_DATA03.DBF
            28,669       8    5.8     7.5           56        0          0    0.0
    AA_DATA              /AAA/oradata/xxxx/AA_DATA04.DBF
            28,342       8    5.9     7.7           47        0          0    0.0
    AA_INDEX             /AAA/oradata/xxxx/AA_INDEX01.DBF
            22,286       6    3.1     3.2          439        0          0    0.0
    AA_INDEX             /AAA/oradata/xxxx/AA_INDEX02.DBF
            21,290       6    3.4     3.3          299        0          0    0.0
    AA_LOB               /AAA/oradata/xxxx/AA_LOB01.DBF
                20       0   11.0     5.4            1        0          0    0.0
    AA_LOB               /AAA/oradata/xxxx/AA_LOB02.DBF
                11       0   17.3     5.5            1        0          0    0.0
    AA_LOB               /AAA/oradata/xxxx/AA_LOB03.DBF
                 5       0   16.0     6.2            1        0          0    0.0
    AA_WEB               /AAA/oradata/xxxx/AA_WEB01.DBF
            28,640       8    9.2    12.8        1,079        0          0    0.0
    AA_WEB               /AAA/oradata/xxxx/AA_WEB02.DBF
            28,718       8    9.0    12.8        1,121        0          0    0.0
    AA_WEB_IDX           /AAA/oradata/xxxx/AA_WEB_IDX01.DBF
             5,068       1    2.8     1.2            2        0          0    0.0
    AA_WEB_IDX           /AAA/oradata/xxxx/AA_WEB_IDX02.DBF
             6,407       2    2.9     1.1            4        0          0    0.0
    Buffer Pool Statistics  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> Standard block size Pools  D: default,  K: keep,  R: recycle
    -> Default Pools for other block sizes: 2k, 4k, 8k, 16k, 32k
                                                                Free Writ     Buffer
         Number of Pool         Buffer     Physical    Physical Buff Comp       Busy
    P      Buffers Hit%           Gets        Reads      Writes Wait Wait      Waits
    D      190,848   71      6,220,953    1,793,255      16,556    0    0         94
    Instance Recovery Stats  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> B: Begin snapshot,  E: End snapshot
      Targt  Estd                                  Log File Log Ckpt     Log Ckpt
      MTTR   MTTR   Recovery  Actual    Target       Size    Timeout     Interval
       (s)    (s)   Estd IOs Redo Blks Redo Blks  Redo Blks Redo Blks   Redo Blks
    B     0     8        155       955      5359     360000      5359
    E     0     9        311      1177      4572     360000      4572
    Buffer Pool Advisory  DB/Inst: xxxx/xxxx  Snap: 16599
    -> Only rows with estimated physical reads >0 are displayed
    -> ordered by Block Size, Buffers For Estimate
            Size for  Size      Buffers for  Est Physical          Estimated
    P   Estimate (M) Factr         Estimate   Read Factor     Physical Reads
    D            144    .1           17,892          1.97          3,571,363
    D            288    .2           35,784          1.62          2,936,281
    D            432    .3           53,676          1.57          2,850,391
    D            576    .4           71,568          1.44          2,616,052
    D            720    .5           89,460          1.33          2,414,217
    D            864    .6          107,352          1.15          2,091,986
    D          1,008    .7          125,244          1.04          1,882,876
    D          1,152    .8          143,136          1.02          1,857,287
    D          1,296    .8          161,028          1.02          1,844,981
    D          1,440    .9          178,920          1.01          1,830,107
    D          1,536   1.0          190,848          1.00          1,816,003
    D          1,584   1.0          196,812          0.98          1,785,023
    D          1,728   1.1          214,704          0.93          1,688,213
    D          1,872   1.2          232,596          0.89          1,613,261
    D          2,016   1.3          250,488          0.88          1,594,176
    D          2,160   1.4          268,380          0.87          1,584,933
    D          2,304   1.5          286,272          0.87          1,582,965
    D          2,448   1.6          304,164          0.86          1,569,802
    D          2,592   1.7          322,056          0.86          1,569,169
    D          2,736   1.8          339,948          0.86          1,568,758
    D          2,880   1.9          357,840          0.86          1,568,194
    PGA Aggr Summary  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> PGA cache hit % - percentage of W/A (WorkArea) data processed only in-memory
    PGA Cache Hit %   W/A MB Processed  Extra W/A MB Read/Written
               88.0              4,058                        552
    PGA Aggr Target Stats  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> B: Begin snap   E: End snap (rows dentified with B or E contain data
       which is absolute i.e. not diffed over the interval)
    -> Auto PGA Target - actual workarea memory target
    -> W/A PGA Used    - amount of memory used for all Workareas (manual + auto)
    -> %PGA W/A Mem    - percentage of PGA memory allocated to workareas
    -> %Auto W/A Mem   - percentage of workarea memory controlled by Auto Mem Mgmt
    -> %Man W/A Mem    - percentage of workarea memory under manual control
                                                    %PGA  %Auto   %Man
        PGA Aggr   Auto PGA   PGA Mem    W/A PGA     W/A    W/A    W/A Global Mem
       Target(M)  Target(M)  Alloc(M)    Used(M)     Mem    Mem    Mem   Bound(K)
    B        500        432       57.5        0.0     .0     .0     .0     25,600
    E        500        428       70.2        0.0     .0     .0     .0     25,600
    PGA Aggr Target Histogram  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> Optimal Executions are purely in-memory operations
      Low     High
    Optimal Optimal    Total Execs  Optimal Execs 1-Pass Execs M-Pass Execs
         2K      4K         11,726         11,726            0            0
        64K    128K            256            256            0            0
       128K    256K            206            206            0            0
       256K    512K            583            583            0            0
       512K   1024K            363            363            0            0
         1M      2M            124            124            0            0
         2M      4M            107            107            0            0
         4M      8M             66             64            2            0
         8M     16M             49             35           14            0
        16M     32M             52             39           13            0
        32M     64M              3              2            1            0
        64M    128M              4              4            0            0
       128M    256M              2              2            0            0
    PGA Memory Advisory  DB/Inst: xxxx/xxxx  Snap: 16599
    -> When using Auto Memory Mgmt, minimally choose a pga_aggregate_target value
       where Estd PGA Overalloc Count is 0
                                           Estd Extra    Estd PGA   Estd PGA
    PGA Target    Size           W/A MB   W/A MB Read/      Cache  Overalloc
      Est (MB)   Factr        Processed Written to Disk     Hit %      Count
            63     0.1          4,098.7          2,609.3     61.0          0
           125     0.3          4,098.7          1,521.6     73.0          0
           250     0.5          4,098.7          1,354.7     75.0          0
           375     0.8          4,098.7          1,109.3     79.0          0
           500     1.0          4,098.7            373.4     92.0          0
           600     1.2          4,098.7             71.6     98.0          0
           700     1.4          4,098.7             41.6     99.0          0
           800     1.6          4,098.7             41.6     99.0          0
           900     1.8          4,098.7              0.0    100.0          0
         1,000     2.0          4,098.7              0.0    100.0          0
         1,500     3.0          4,098.7              0.0    100.0          0
         2,000     4.0          4,098.7              0.0    100.0          0
         3,000     6.0          4,098.7              0.0    100.0          0
         4,000     8.0          4,098.7              0.0    100.0          0
    Shared Pool Advisory  DB/Inst: xxxx/xxxx  Snap: 16599
    -> SP: Shared Pool     Est LC: Estimated Library Cache   Factr: Factor
    -> Note there is often a 1:Many correlation between a single logical object
       in the Library Cache, and the physical number of memory objects associated
       with it.  Therefore comparing the number of Lib Cache objects (e.g. in
       v$librarycache), with the number of Lib Cache Memory Objects is invalid.
                                            Est LC Est LC  Est LC Est LC
        Shared    SP   Est LC                 Time   Time    Load   Load      Est LC
          Pool  Size     Size       Est LC   Saved  Saved    Time   Time         Mem
       Size(M) Factr      (M)      Mem Obj     (s)  Factr     (s)  Factr    Obj Hits
           224    .2      130        9,174     380    1.0      91    1.0     408,867
           352    .3      171       12,647     381    1.0      90    1.0     409,226
           480    .4      171       12,647     381    1.0      90    1.0     409,226
           608    .5      171       12,647     381    1.0      90    1.0     409,226
           736    .6      171       12,647     381    1.0      90    1.0     409,226
           864    .7      171       12,647     381    1.0      90    1.0     409,226
           992    .8      171       12,647     381    1.0      90    1.0     409,226
         1,120    .9      171       12,647     381    1.0      90    1.0     409,226
         1,248   1.0      171       12,647     381    1.0      90    1.0     409,226
         1,376   1.1      171       12,647     381    1.0      90    1.0     409,226
         1,504   1.2      171       12,647     381    1.0      90    1.0     409,226
         1,632   1.3      171       12,647     381    1.0      90    1.0     409,226
         1,760   1.4      171       12,647     381    1.0      90    1.0     409,226
         1,888   1.5      171       12,647     381    1.0      90    1.0     409,226
         2,016   1.6      171       12,647     381    1.0      90    1.0     409,226
         2,144   1.7      171       12,647     381    1.0      90    1.0     409,226
         2,272   1.8      171       12,647     381    1.0      90    1.0     409,226
         2,400   1.9      171       12,647     381    1.0      90    1.0     409,226
         2,528   2.0      171       12,647     381    1.0      90    1.0     409,226
    Java Pool Advisory  DB/Inst: xxxx/xxxx  Snap: 16599
                                            Est LC Est LC  Est LC Est LC
          Java    JP   Est LC                 Time   Time    Load   Load      Est LC
          Pool  Size     Size       Est LC   Saved  Saved    Time   Time         Mem
       Size(M) Factr      (M)      Mem Obj     (s)  Factr     (s)  Factr    Obj Hits
            32    .1        6          142       2    1.0      23    1.0         142
            64    .3        6          142       2    1.0      23    1.0         142
            96    .4        6          142       2    1.0      23    1.0         142
           128    .5        6          142       2    1.0      23    1.0         142
           160    .6        6          142       2    1.0      23    1.0         142
           192    .8        6          142       2    1.0      23    1.0         142
           224    .9        6          142       2    1.0      23    1.0         142
           256   1.0        6          142       2    1.0      23    1.0         142
           288   1.1        6          142       2    1.0      23    1.0         142
           320   1.3        6          142       2    1.0      23    1.0         142
           352   1.4        6          142       2    1.0      23    1.0         142
           384   1.5        6          142       2    1.0      23    1.0         142
           416   1.6        6          142       2    1.0      23    1.0         142
           448   1.8        6          142       2    1.0      23    1.0         142
           480   1.9        6          142       2    1.0      23    1.0         142
           512   2.0        6          142       2    1.0      23    1.0         142
    Buffer Wait Statistics  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> ordered by wait time desc, waits desc
    Class                    Waits Total Wait Time (s)  Avg Time (ms)
    data block                  66                   1             15
    undo block                  25                   0              0
    undo header                  3                   0              0
    Enqueue Activity  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> Enqueue stats gathered prior to 10i should not be compared with 10i data
    -> ordered by Wait Time desc, Waits desc
    Enqueue Type (Request Reason)
        Requests    Succ Gets Failed Gets       Waits  Wt Time (s) Av Wt Time(ms)
    TX-Transaction (index contention)
               1            1           0           1            0          22.00
    Undo Segment Summary  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> Undo segment block stats:
    -> uS - unexpired Stolen,   uR - unexpired Released,   uU - unexpired reUsed
    -> eS - expired   Stolen,   eR - expired   Released,   eU - expired   reUsed
    Undo         Undo             Num  Max Qry   Max Tx   Snap OutOf    uS/uR/uU/
    TS#       Blocks           Trans  Len (s) Concurcy TooOld Space    eS/eR/eU
       1        5,497           6,716    2,708        4      0     0 0/0/0/0/0/0
    Undo Segment Stats  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> ordered by Time desc
                        Undo          Num Max Qry  Max Tx   Snap OutOf   uS/uR/uU/
    End Time          Blocks        Trans Len (s)   Concy TooOld Space   eS/eR/eU
    07-Jul 17:24          26          161   2,708       1      0     0 0/0/0/0/0/0
    07-Jul 17:14          27          143   2,108       2      0     0 0/0/0/0/0/0
    07-Jul 17:04          27          226   1,508       1      0     0 0/0/0/0/0/0
    07-Jul 16:54       1,565        1,734     204       4      0     0 0/0/0/0/0/0
    07-Jul 16:44       2,985        3,078     145       4      0     0 0/0/0/0/0/0
    07-Jul 16:34         867        1,374       0       4      0     0 0/0/0/0/0/0
    Latch Activity  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> "Get Requests", "Pct Get Miss" and "Avg Slps/Miss" are statistics for
       willing-to-wait latch get requests
    -> "NoWait Requests", "Pct NoWait Miss" are for no-wait latch get requests
    -> "Pct Misses" for both should be very close to 0.0
                                               Pct    Avg   Wait                 Pct
                                        Get    Get   Slps   Time       NoWait NoWait
    Latch Name                     Requests   Miss  /Miss    (s)     Requests   Miss
    Consistent RBA                    5,501    0.0             0            0
    FOB s.o list latch                  523    0.0             0            0
    In memory undo latch              4,267    0.1    1.0      0          728    1.2
    JOX SGA heap latch                   22    0.0             0            0
    JS queue state obj latch         25,992    0.0             0            0
    JS slv state obj latch              338    0.0             0            0
    KTF sga enqueue                       8    0.0             0        1,092    0.0
    KWQMN job cache list lat             12    0.0             0            0
    KWQP Prop Status                    415    0.0             0            0
    MQL Tracking Latch                    0                    0           72    0.0
    Memory Management Latch          18,075    0.0             0        1,207    0.0
    PL/SQL warning settings           1,607    0.0             0            0
    SQL memory manager latch              1    0.0             0        1,206    0.0
    SQL memory manager worka        127,457    0.0    0.0      0            0
    SWRF Alerted Metric Elem         13,029    0.0             0            0
    Shared B-Tree                     2,523    0.0             0            0
    active checkpoint queue           1,560    0.0             0            0
    active service list               6,791    0.0             0            0
    archive control                       8    0.0             0            0
    begin backup scn array               40    0.0             0            0
    cache buffer handles              4,560    0.0             0            0
    cache buffers chains         14,704,024    0.0    0.2      1    3,447,993    0.0
    cache buffers lru chain          62,381    0.3    0.4      2    3,121,799    0.4
    channel handle pool latc            759    0.0             0            0
    channel operations paren         14,505    0.0             0            0
    checkpoint queue latch           67,923    0.0             0       16,481    0.0
    child cursor hash table          36,508    0.0             0            0
    client/application info              16    0.0             0            0
    compile environment latc          1,063    0.0             0            0
    cursor bind value captur            125    0.0             0          384    1.0
    dictionary lookup                   107    0.0             0            0
    dml lock allocation              52,017    0.1    0.0      0            0
    dummy allocation                    839    0.4    0.0      0            0
    enqueue hash chains             186,347    0.0    0.1      0            0
    enqueues                        122,225    0.0    0.0      0            0
    event group latch                   230    0.0             0            0
    file cache latch                    153    0.0             0            0
    global KZLD latch for me            110    0.0             0            0
    hash table column usage             512    0.0             0      138,833    0.0
    hash table modification             125    0.0             0            0
    internal temp table obje              3    0.0             0            0
    job workq parent latch                0                    0          388    0.0
    job_queue_processes para            172    0.0             0            0
    ksuosstats global area              243    0.0             0            0
    ktm global data                     175    0.0             0            0
    kwqbsn:qsga                       2,523    0.0             0            0
    lgwr LWN SCN                      6,225    0.0             0            0
    library cache                 1,069,757    0.1    0.3      5        2,857  227.5
    library cache load lock          10,002    0.0             0            0
    library cache lock              277,733    0.0    0.0      0            0
    library cache lock alloc          4,092    0.0             0            0
    library cache pin               776,264    0.0    0.0      0            0
    library cache pin alloca          4,018    0.0             0            0
    list of block allocation          1,044    0.0             0            0
    loader state object free            460    0.0             0            0
    longop free list parent           1,357    0.0             0       19,547    0.0
    message pool operations             334    0.0             0            0
    messages                         37,979    0.0    0.9      0            0
    mostly latch-free SCN             6,237    0.0    0.0      0            0
    multiblock read objects         351,202    0.0    0.0      0            0
    Latch Activity  DB/Inst: xxxx/xxxx  Snaps: 16598-16599
    -> "Get Requests", "Pct Get Miss" and "Avg Slps/Miss" are statistics for
       willing-to-wait latch get requests
    -> "NoWait Requests", "Pct NoWait Miss" are for no-wait latch get requestscontd..............
    Edited by: LazyDBA10g on Jul 8, 2009 4:51 AM

  • Need help to analysis "foreground and background wait events" on statspack report for oracle database 11.2.0.4 on AIX

    Hi: I'm analyzing this STATSPACK report: it is "volume test" on our UAT server, so most input is from 'bind variables'.  Our shared pool is well utilized in oracle.  Oracle redo logs is not appropriately configured on this server, as in 'Top 5 wait events' there are 2 for redos.
    I need to know what else information can be dig-out from 'foreground wait events' & 'background wait events', and what can assist us to better understanding, in combination of 'Top 5 wait event's, that how the server/test went?  it could be overwelming No. of wait events, so appreciate any helpful diagnostic or analysis.  Database is oracle 11.2.0.4 upgraded from 11.2.0.3, on IBM AIX power system 64bit, level 6.x
    STATSPACK report for
    Database    DB Id    Instance     Inst Num  Startup Time   Release     RAC
    ~~~~~~~~ ----------- ------------ -------- --------------- ----------- ---
    700000XXX   XXX              1 22-Apr-15 12:12 11.2.0.4.0  NO
    Host Name             Platform                CPUs Cores Sockets   Memory (G)
    ~~~~ ---------------- ---------------------- ----- ----- ------- ------------
         dXXXX_XXX    AIX-Based Systems (64-     2     1       0         16.0
    Snapshot       Snap Id     Snap Time      Sessions Curs/Sess Comment
    ~~~~~~~~    ---------- ------------------ -------- --------- ------------------
    Begin Snap:       5635 22-Apr-15 13:00:02      114       4.6
      End Snap:       5636 22-Apr-15 14:00:01      128       8.8
       Elapsed:      59.98 (mins) Av Act Sess:       0.6
       DB time:      35.98 (mins)      DB CPU:      19.43 (mins)
    Cache Sizes            Begin        End
    ~~~~~~~~~~~       ---------- ----------
        Buffer Cache:     2,064M              Std Block Size:         8K
         Shared Pool:     3,072M                  Log Buffer:    13,632K
    Load Profile              Per Second    Per Transaction    Per Exec    Per Call
    ~~~~~~~~~~~~      ------------------  ----------------- ----------- -----------
          DB time(s):                0.6                0.0        0.00        0.00
           DB CPU(s):                0.3                0.0        0.00        0.00
           Redo size:          458,720.6            8,755.7
       Logical reads:           12,874.2              245.7
       Block changes:            1,356.4               25.9
      Physical reads:                6.6                0.1
    Physical writes:               61.8                1.2
          User calls:            2,033.7               38.8
              Parses:              286.5                5.5
         Hard parses:                0.5                0.0
    W/A MB processed:                1.7                0.0
              Logons:                1.2                0.0
            Executes:              801.1               15.3
           Rollbacks:                6.1                0.1
        Transactions:               52.4
    Instance Efficiency Indicators
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Buffer Nowait %:  100.00       Redo NoWait %:  100.00
                Buffer  Hit   %:   99.98  Optimal W/A Exec %:  100.00
                Library Hit   %:   99.77        Soft Parse %:   99.82
             Execute to Parse %:   64.24         Latch Hit %:   99.98
    Parse CPU to Parse Elapsd %:   53.15     % Non-Parse CPU:   98.03
    Shared Pool Statistics        Begin   End
                 Memory Usage %:   10.50   12.79
        % SQL with executions>1:   69.98   78.37
      % Memory for SQL w/exec>1:   70.22   81.96
    Top 5 Timed Events                                                    Avg %Total
    ~~~~~~~~~~~~~~~~~~                                                   wait   Call
    Event                                            Waits    Time (s)   (ms)   Time
    CPU time                                                       847          50.2
    enq: TX - row lock contention                    4,480         434     97   25.8
    log file sync                                  284,169         185      1   11.0
    log file parallel write                        299,537         164      1    9.7
    log file sequential read                           698          16     24    1.0
    Host CPU  (CPUs: 2  Cores: 1  Sockets: 0)
    ~~~~~~~~              Load Average
                          Begin     End      User  System    Idle     WIO     WCPU
                           1.16    1.84     19.28   14.51   66.21    1.20   82.01
    Instance CPU
    ~~~~~~~~~~~~                                       % Time (seconds)
                         Host: Total time (s):                  7,193.8
                      Host: Busy CPU time (s):                  2,430.7
                       % of time Host is Busy:      33.8
                 Instance: Total CPU time (s):                  1,203.1
              % of Busy CPU used for Instance:      49.5
            Instance: Total Database time (s):                  2,426.4
      %DB time waiting for CPU (Resource Mgr):       0.0
    Memory Statistics                       Begin          End
    ~~~~~~~~~~~~~~~~~                ------------ ------------
                      Host Mem (MB):     16,384.0     16,384.0
                       SGA use (MB):      7,136.0      7,136.0
                       PGA use (MB):        282.5        361.4
        % Host Mem used for SGA+PGA:         45.3         45.8
    Foreground Wait Events  DB/Inst: XXXXXs  Snaps: 5635-5636
    -> Only events with Total Wait Time (s) >= .001 are shown
    -> ordered by Total Wait Time desc, Waits desc (idle events last)
                                                                 Avg          %Total
                                              %Tim Total Wait   wait    Waits   Call
    Event                               Waits  out   Time (s)   (ms)     /txn   Time
    enq: TX - row lock contentio        4,480    0        434     97      0.0   25.8
    log file sync                     284,167    0        185      1      1.5   11.0
    Disk file operations I/O            8,741    0          4      0      0.0     .2
    direct path write                  13,247    0          3      0      0.1     .2
    db file sequential read             6,058    0          1      0      0.0     .1
    buffer busy waits                   1,800    0          1      1      0.0     .1
    SQL*Net more data to client        29,161    0          1      0      0.2     .1
    direct path read                    7,696    0          1      0      0.0     .0
    db file scattered read                316    0          1      2      0.0     .0
    latch: shared pool                    144    0          0      2      0.0     .0
    CSS initialization                     30    0          0      3      0.0     .0
    cursor: pin S                          10    0          0      9      0.0     .0
    row cache lock                         41    0          0      2      0.0     .0
    latch: row cache objects               19    0          0      3      0.0     .0
    log file switch (private str            8    0          0      7      0.0     .0
    library cache: mutex X                 28    0          0      2      0.0     .0
    latch: cache buffers chains            54    0          0      1      0.0     .0
    latch free                            290    0          0      0      0.0     .0
    control file sequential read        1,568    0          0      0      0.0     .0
    log file switch (checkpoint             4    0          0      6      0.0     .0
    direct path sync                        8    0          0      3      0.0     .0
    latch: redo allocation                 60    0          0      0      0.0     .0
    SQL*Net break/reset to clien           34    0          0      1      0.0     .0
    latch: enqueue hash chains             45    0          0      0      0.0     .0
    latch: cache buffers lru cha            7    0          0      2      0.0     .0
    latch: session allocation               5    0          0      1      0.0     .0
    latch: object queue header o            6    0          0      1      0.0     .0
    ASM file metadata operation            30    0          0      0      0.0     .0
    latch: In memory undo latch            15    0          0      0      0.0     .0
    latch: undo global data                 8    0          0      0      0.0     .0
    SQL*Net message from client     6,362,536    0    278,225     44     33.7
    jobq slave wait                     7,270  100      3,635    500      0.0
    SQL*Net more data from clien        7,976    0         15      2      0.0
    SQL*Net message to client       6,362,544    0          8      0     33.7
    Background Wait Events  DB/Inst: XXXXXs  Snaps: 5635-5636
    -> Only events with Total Wait Time (s) >= .001 are shown
    -> ordered by Total Wait Time desc, Waits desc (idle events last)
                                                                 Avg          %Total
                                              %Tim Total Wait   wait    Waits   Call
    Event                               Waits  out   Time (s)   (ms)     /txn   Time
    log file parallel write           299,537    0        164      1      1.6    9.7
    log file sequential read              698    0         16     24      0.0    1.0
    db file parallel write              9,556    0         13      1      0.1     .8
    os thread startup                     146    0         10     70      0.0     .6
    control file parallel write         2,037    0          2      1      0.0     .1
    Log archive I/O                        35    0          1     30      0.0     .1
    LGWR wait for redo copy             2,447    0          0      0      0.0     .0
    db file async I/O submit            9,556    0          0      0      0.1     .0
    db file sequential read               145    0          0      2      0.0     .0
    Disk file operations I/O              349    0          0      0      0.0     .0
    db file scattered read                 30    0          0      4      0.0     .0
    control file sequential read        5,837    0          0      0      0.0     .0
    ADR block file read                    19    0          0      4      0.0     .0
    ADR block file write                    5    0          0     15      0.0     .0
    direct path write                      14    0          0      2      0.0     .0
    direct path read                        3    0          0      7      0.0     .0
    latch: shared pool                      3    0          0      6      0.0     .0
    log file single write                  56    0          0      0      0.0     .0
    latch: redo allocation                 53    0          0      0      0.0     .0
    latch: active service list              1    0          0      3      0.0     .0
    latch free                             11    0          0      0      0.0     .0
    rdbms ipc message                 314,523    5     57,189    182      1.7
    Space Manager: slave idle wa        4,086   88     18,996   4649      0.0
    DIAG idle wait                      7,185  100      7,186   1000      0.0
    Streams AQ: waiting for time            2   50      4,909 ######      0.0
    Streams AQ: qmn slave idle w          129    0      3,612  28002      0.0
    Streams AQ: qmn coordinator           258   50      3,612  14001      0.0
    smon timer                             43    2      3,605  83839      0.0
    pmon timer                          1,199   99      3,596   2999      0.0
    SQL*Net message from client        17,019    0         31      2      0.1
    SQL*Net message to client          12,762    0          0      0      0.1
    class slave wait                       28    0          0      0      0.0
    thank you very much!

    Hi: just know it now: it is a large amount of 'concurrent transaction' designed in this "Volume Test" - to simulate large incoming transaction volme, so I guess wait in eq:TX - row is expected.
    The fact: (1) redo logs at uat server is known to not well-tune for configurations (2) volume test slow 5%, however data amount in its test is kept the same by each time import  production data, by the team. So why it slowed 5% this year?
    The wait histogram is pasted below, any one interest to take a look?  any ideas?
    Wait Event Histogram  DB/Inst: XXXX/XXXX  Snaps: 5635-5636
    -> Total Waits - units: K is 1000, M is 1000000, G is 1000000000
    -> % of Waits - column heading: <=1s is truly <1024ms, >1s is truly >=1024ms
    -> % of Waits - value: .0 indicates value was <.05%, null is truly 0
    -> Ordered by Event (idle events last)
                               Total ----------------- % of Waits ------------------
    Event                      Waits  <1ms  <2ms  <4ms  <8ms <16ms <32ms  <=1s   >1s
    ADR block file read          19   26.3   5.3  10.5  57.9
    ADR block file write          5                     40.0        60.0
    ADR file lock                 6  100.0
    ARCH wait for archivelog l   14  100.0
    ASM file metadata operatio   30  100.0
    CSS initialization           30              100.0
    Disk file operations I/O   9090   97.2   1.4    .6    .4    .2    .1    .1
    LGWR wait for redo copy    2447   98.5    .5    .4    .2    .2    .2    .1
    Log archive I/O              35   40.0         8.6  25.7   2.9        22.9
    SQL*Net break/reset to cli   34   85.3   8.8         5.9
    SQL*Net more data to clien   29K  99.9    .0    .0    .0          .0    .0
    buffer busy waits          1800   96.8    .7    .7    .6    .3    .4    .5
    control file parallel writ 2037   90.7   5.0   2.1    .8   1.0    .3    .1
    control file sequential re 7405  100.0                      .0
    cursor: pin S                10   10.0                    90.0
    db file async I/O submit   9556   99.9    .0                .0          .0
    db file parallel read         1  100.0
    db file parallel write     9556   62.0  32.4   1.7    .8   1.5   1.3    .1
    db file scattered read      345   72.8   3.8   2.3  11.6   9.0    .6
    db file sequential read    6199   97.2    .2    .3   1.6    .7    .0    .0
    direct path read           7699   99.1    .4    .2    .1    .1    .0
    direct path sync              8   25.0  37.5  12.5  25.0
    direct path write            13K  97.8    .9    .5    .4    .3    .1    .0
    enq: TX - row lock content 4480     .4    .7   1.3   3.0   6.8  12.3  75.4    .1
    latch free                  301   98.3    .3    .7    .7
    latch: In memory undo latc   15   93.3   6.7
    latch: active service list    1              100.0
    latch: cache buffers chain   55   94.5                     3.6   1.8
    latch: cache buffers lru c    9   88.9                    11.1
    latch: call allocation        6  100.0
    latch: checkpoint queue la    3  100.0
    latch: enqueue hash chains   45   97.8                     2.2
    latch: messages               4  100.0
    latch: object queue header    7   85.7        14.3
    latch: redo allocation      113   97.3               1.8    .9
    latch: row cache objects     19   89.5                           5.3   5.3
    latch: session allocation     5   80.0              20.0
    latch: shared pool          147   90.5   1.4   2.7   1.4    .7   1.4   2.0
    latch: undo global data       8  100.0
    library cache: mutex X       28   89.3         3.6         3.6         3.6
    log file parallel write     299K  95.6   2.6   1.0    .4    .3    .2    .0
    log file sequential read    698   29.5    .1               4.6  46.8  18.9
    log file single write        56  100.0
    log file switch (checkpoin    4               25.0  50.0  25.0
    log file switch (private s    8         12.5        37.5  50.0
    log file sync               284K  93.3   3.7   1.4    .7    .5    .3    .1
    os thread startup           146                                      100.0
    row cache lock               41   85.4   9.8               2.4         2.4
    DIAG idle wait             7184                                      100.0
    SQL*Net message from clien 6379K  86.6   5.1   2.9   1.3    .7    .3   2.8    .3
    SQL*Net message to client  6375K 100.0    .0    .0    .0    .0    .0    .0
    Wait Event Histogram  DB/Inst: XXXX/xxxx  Snaps: 5635-5636
    -> Total Waits - units: K is 1000, M is 1000000, G is 1000000000
    -> % of Waits - column heading: <=1s is truly <1024ms, >1s is truly >=1024ms
    -> % of Waits - value: .0 indicates value was <.05%, null is truly 0
    -> Ordered by Event (idle events last)
                               Total ----------------- % of Waits ------------------
    Event                      Waits  <1ms  <2ms  <4ms  <8ms <16ms <32ms  <=1s   >1s
    SQL*Net more data from cli 7976   99.7    .1    .1    .0                      .1
    Space Manager: slave idle  4086     .1    .2    .0    .0    .3         3.2  96.1
    Streams AQ: qmn coordinato  258   49.2                .8                    50.0
    Streams AQ: qmn slave idle  129                                            100.0
    Streams AQ: waiting for ti    2   50.0                                      50.0
    class slave wait             28   92.9   3.6   3.6
    jobq slave wait            7270     .0                               100.0
    pmon timer                 1199                                            100.0
    rdbms ipc message           314K  10.3   7.3  39.7  15.4  10.6   5.3   8.2   3.3
    smon timer                   43                                            100.0

  • Performance Problems

    Hi,
    I´m new with tuning and i have a little problem, i read a lot of blogs and articles about awr and stackspack. Here is part of my awr report.
    Server – DELL 2850 – Windows 2003 SP2 x86 – 4GB RAM (/3GB) 2CPU.
    Snap Id Snap Time Sessions Curs/Sess
    Begin Snap: 11592 20-May-09 09:00:49 110 50.3
    End Snap: 11761 27-May-09 10:01:00 115 45.3
    Elapsed: 10,140.18 (mins)
    DB Time: 19,151.07 (mins)
    Cache Sizes
    ~~~~~~~~~~~ Begin End
    Buffer Cache: 316M 316M Std Block Size: 8K
    Shared Pool Size: 576M 576M Log Buffer: 6,924K
    Load Profile
    ~~~~~~~~~~~~ Per Second Per Transaction
    Redo size: 196,591.16 6,808.93
    Logical reads: 68,060.52 2,357.27
    Block changes: 1,664.88 57.66
    Physical reads: 157.76 5.46
    Physical writes: 34.21 1.18
    User calls: 1,689.77 58.53
    Parses: 675.63 23.40
    Hard parses: 2.86 0.10
    Sorts: 7,059.70 244.51
    Logons: 2.41 0.08
    Executes: 2,377.07 82.33
    Transactions: 28.87
    % Blocks changed per Read: 2.45 Recursive Call %: 74.45
    Rollback per transaction %: 76.03 Rows per Sort: 0.73
    Instance Efficiency Percentages (Target 100%)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Buffer Nowait %: 99.99 Redo NoWait %: 100.00
    Buffer Hit %: 99.77 In-memory Sort %: 100.00
    Library Hit %: 102.97 Soft Parse %: 99.58
    Execute to Parse %: 71.58 Latch Hit %: 99.93
    Parse CPU to Parse Elapsd %: 78.73 % Non-Parse CPU: 98.55
    Shared Pool Statistics Begin End
    Memory Usage %: 47.32 47.64
    % SQL with executions>1: 58.04 69.45
    % Memory for SQL w/exec>1: 68.71 76.32
    Top 5 Timed Events Avg %Total
    ~~~~~~~~~~~~~~~~~~ wait Call
    Event Waits Time (s) (ms) Time Wait Class
    CPU time 873,110 76.0
    log file sync 5,161,812 76,211 15 6.6 Commit
    enq: UL – contention 243,039 54,470 224 4.7 Applicatio
    db file sequential read 12,534,382 41,279 3 3.6 User I/O
    log file parallel write 8,215,709 25,760 3 2.2 System I/O
    I see that CPU time is very high and the other wait events not are very significant.
    My Database reports Rollback per transaction %age to 76%.
    Statistic Total per Second per Trans
    user commits 4,210,155 6.9 0.2
    user rollbacks 13,356,226 22.0 0.8
    Transacciones = user commits + user rollbacks
    Transacciones = 4.210.155 + 13.356.226 = 17.566.381
    Rollbacks por transacción = (100*13.356.226/17.566.381)= 76,03%
    I review this other statistics.
    Statistic Total per Second per Trans
    db block changes 1,012,928,251 1,664.9 57.7
    rollback changes – undo records 177,118,776 291.1 10.1
    In the section on sql statistics i don´t see any query with delete statement or something that is producing the rollback.
    For the other hand, i saw high values for latches:
    Latch Activity
    Pct Avg Wait Pct
    Get Get Slps Time NoWait NoWait
    Latch Name Requests Miss /Miss (s) Requests Miss
    cache buffer handles 594,141,662 0.0 0.0 0 0 N/A
    cache buffers chains ############## 0.0 0.0 24 256,952,500 0.0
    cache buffers lru chain 57,432,611 0.1 0.0 0 407,926,828 0.0
    cache table scan latch 0 N/A N/A 0 7,211,848 0.0
    channel handle pool latc 248,878 0.0 0.0 0 0 N/A
    channel operations paren 4,191,138 0.0 0.0 0 0 N/A
    checkpoint queue latch 93,701,454 0.0 0.0 0 18,705,338 0.0
    client/application info 74,841,233 0.0 0.0 0 0 N/A
    In memory undo latch 319,086,961 0.2 0.0 3 48,465,421 0.1
    row cache objects 2,204,749,411 0.1 0.0 0 630,441 0.0
    lgwr LWN SCN 8,230,336 0.0 0.0 0 0 N/A
    library cache 2,606,820,844 0.2 0.0 1106 8,043,804 0.4
    library cache load lock 841,281 0.0 0.0 0 74 0.0
    library cache lock 611,525,102 0.1 0.0 0 205 0.0
    library cache lock alloc 32,549,163 0.0 0.0 0 0 N/A
    library cache pin 1,349,787,907 0.1 0.0 1 4,386 0.0
    library cache pin alloca 2,617,831 0.0 0.0 0 0 N/A
    Latch Name
    Get Requests Misses Sleeps Spin Gets Sleep1 Sleep2 Sleep3
    cache buffers chains
    ############## 22,397,249 127,771 ########## 0 0 0
    session allocation
    -953,498,830 5,328,052 341 5,327,714 0 0 0
    library cache
    2,606,820,844 5,076,626 82,635 4,995,862 0 0 0
    simulator lru latch
    -3,426,351,079 4,816,176 244 4,815,941 0 0 0
    row cache objects
    2,204,749,411 2,659,953 25 2,659,928 0 0 0
    library cache pin
    1,349,787,907 1,566,144 2,726 1,563,447 0 0 0
    library cache lock
    611,525,102 801,491 75 801,416 0 0 0
    In memory undo latch
    319,086,961 724,884 5,850 719,534 0 0 0
    enqueues
    256,944,623 546,555 1,748 544,980 0 0 0
    shared pool
    190,887,687 515,397 78,393 439,711 0 0 0
    session idle bit
    2,097,791,961 512,800 2,111 510,722 0 0 0
    enqueue hash chains
    325,688,224 316,885 2,067 314,956 0 0 0
    undo global data
    395,144,808 249,509 395 249,136 0 0 0
    dml lock allocation
    82,685,629 38,906 213 38,738 0 0 0
    Latch Miss Sources
    NoWait Waiter
    Latch Name Where Misses Sleeps Sleeps
    cache buffers chains kcbchg: kslbegin: bufs not 0 124,497 72,893
    cache buffers chains kcbgtcr: fast path 0 84,073 73,823
    cache buffers chains kcbgtcr: kslbegin excl 0 69,785 47,848
    cache buffers chains kcbzwb 0 47,844 28,553
    cache buffers chains kcbgcur: kslbegin 0 46,897 10,928
    cache buffers chains kcbrls: kslbegin 0 11,584 58,227
    cache buffers chains kcbchg: kslbegin: call CR 0 5,786 50,971
    cache buffers chains kcbget: pin buffer 0 3,802 691
    cache buffers chains kcb_is_private 0 2,362 42,467
    cache buffers chains kcbbxsv 0 1,421 79
    cache buffers chains kcbnlc 0 1,197 5,431
    library cache kglpnp: child 0 25,131 56,619
    library cache kglpndl: child: after proc 0 8,800 216
    library cache kglhdgn: child: 0 7,515 16,401
    library cache kglLockCursor 0 3,962 1,328
    library cache kglpndl: child: before pro 0 990 5,918
    library cache kglic 0 750 28
    shared pool kghasp 0 54,200 40
    shared pool kghupr1 0 13,535 56,783
    shared pool kghalo 0 6,161 12,243
    shared pool kghfre 0 3,081 8,630
    shared pool kgh_next_free 0 887 9
    I´m a bit lost… i need to look for hot blocks, sql statments that are consuming a lot of cpu, set pct_free of hot_blocks to a high value¿?
    Edited by: Roberto Marotta on Jun 22, 2009 3:45 AM

    Well,
    DELL -2850 4GB RAM 2CPU
    I´ll try to explain better. Sorry for my explains and my english because my english is not very well.
    This environment had serious memory problems some time ago. Sometimes database hangs because memory problems. We resolve the problem with /3GB setting in the Windows OS.
    Now we are doing a study "tuning" of the database server looking for gain performance.
    I find that the redo size is not the optimal size. Actual = 50M Recommended= 6GB.
    Statistic l
    log switches (derived) Tota 2,682 per Hour 15.87
    OPTIMAL_LOGFILE_SIZE
    6008
    In one week the server had processed 17M of transactions.
    Statistic Total per Second per Trans
    user commits 4,210,155 6.9 0.2
    user rollbacks 13,356,226 22.0 0.8
    Transacciones = user commits + user rollbacks
    Transacciones = 4.210.155 + 13.356.226 = 17.566.381
    Rollbacks por transacción = (100*13.356.226/17.566.381)= 76,03%
    Statistic Total per Second per Trans
    db block changes 1,012,928,251 1,664.9 57.7
    rollback changes - undo records 177,118,776 291.1 10.1
    I think that is a problem with the logic of application but i don´t find anything like delete statements or something that alerts me doing much rollback.
    For the other hand i detect high values for some latches "cache buffer chains", "library cache", "row objects cache".
    More things:
    METRIC_NAME
    Database Wait Time Ratio value 24.3862681
    Database CPU Time Ratio value 75.6137319
    METRIC_NAME MINIMUN MAXIMUN AVERAGE
    CPU Usage Per Sec 0 158.290545 109.032138
    CPU Usage Per Txn 0 6.15517349 3.72228913
    Database CPU Time Ratio 0 88.7685616 77.766078
    Database Wait Time Ratio 0 88.8858063 22.233922
    Executions Per Sec 0 5624.66667 2633.58251
    Executions Per Txn 0 218.716785 90.4001077
    Response Time Per Txn (secs) 0 .31 .05
    SQL Service Response Time (secs) 0 0 0
    User Transaction Per Sec 0 53.1166667 29.6726776
    STAT_NAME PCT_TIME
    sql execute elapsed time 33.84
    DB time 32
    DB CPU 7670898.16 26.44
    PL/SQL execution elapsed time 1202699.05 4.15
    background elapsed time 560082.574 1.93
    parse time elapsed 227501.434 .78
    hard parse elapsed time 87775.528 .3
    PL/SQL compilation elapsed time 53801.08 .19
    background cpu time 47749.065 .16
    failed parse elapsed time 37235.995 .13
    hard parse (sharing criteria) elapsed time 5800.741 .02
    inbound PL/SQL rpc elapsed time 3930.999 .01
    RMAN cpu time (backup/restore) 3828.488 .01
    connection management call elapsed time 3332.64 .01
    sequence load elapsed time 2438.323 .01
    hard parse (bind mismatch) elapsed time 1337.938 0
    repeated bind elapsed time 510.387 0
    Java execution elapsed time 18.669 0
    failed parse (out of shared memory) elapsed time 0 0
    I think that the server expend much time executing sql statements...
    Best Regards,
    Edited by: Roberto Marotta on Jun 22, 2009 5:20 AM
    Edited by: Roberto Marotta on Jun 22, 2009 5:20 AM
    Edited by: Roberto Marotta on Jun 22, 2009 5:22 AM
    Edited by: Roberto Marotta on Jun 22, 2009 5:23 AM
    Edited by: Roberto Marotta on Jun 22, 2009 5:25 AM
    Edited by: Roberto Marotta on Jun 22, 2009 5:25 AM

  • Performance crippled with audit_trail=db in 12.1.0.2 CDB/PDB

    Performance tanked as soon as I set audit_trail=db, and all I see in EM Express is buffer busy waits.  Tracked it to object sys.aud$ with 100% of buffer busy waits.  I have been using Oracle auditing for years and never see this type of impact.  Anyone else see this?
    Oracle 12.1.0.2 on 64-bit Oracle Linux 6.5 with kernel 3.8.13-44.el6uek.x86_64.
    Created container database using DBCA with one pluggable database. 
    Ran benchmarks to baseline performance.  Very good.
    set audit_trail=db, bounced the database, logged into the PDB and enabled audit on select, insert, update, delete on all my application tables.
    Transactions per minute dropped 95% and write I/O throughput dropped 75%.
    I collected AWR reports with and without audit_trail set.  These are shown below.
    Each benchmark is 5 minutes with 2 minute ramp-up time, and 40 users no key/think time.
    BEFORE SETTING AUDIT_TRAIL (IT IS UNSET).  Buffer Busy Waits is only 1.2 seconds total.
    Top 10 Foreground Events by Total Wait Time
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                               Total Wait       Wait   % DB Wait
    Event                                Waits Time (sec)    Avg(ms)   time Class
    log file sync                    2,500,409     6224.2       2.49   54.9 Commit
    DB CPU                                         5166.3              45.6
    enq: TX - row lock contention       16,155       14.3       0.88     .1 Applicat
    library cache: mutex X              83,598       13.6       0.16     .1 Concurre
    db file sequential read             87,145       10.4       0.12     .1 User I/O
    SQL*Net message to client        2,739,811        8.3       0.00     .1 Network
    cursor: pin S                        3,631        3.9       1.07     .0 Concurre
    latch: In memory undo latch         86,392        2.9       0.03     .0 Concurre
    buffer busy waits                   45,740        1.2       0.03     .0 Concurre
    cursor: mutex X                      1,004         .9       0.94     .0 Concurre
    NOW RE-RUN SAME BENCHMARK WITH AUDIT_TRAIL = DB.  Buffer Busy Waits over 8000 seconds!
    Top 10 Foreground Events by Total Wait Time
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                               Total Wait       Wait   % DB Wait
    Event                                Waits Time (sec)    Avg(ms)   time Class
    buffer busy waits               12,024,693     8002.1       0.67   68.1 Concurre
    DB CPU                                         3253.5              27.7
    log file sync                      151,536      255.6       1.69    2.2 Commit
    enq: TX - row lock contention        5,549      199.6      35.98    1.7 Applicat
    enq: TX - contention                   959      110.3     114.98     .9 Other
    enq: HW - contention               177,020       95.9       0.54     .8 Configur
    Data file init write                 2,232        1.9       0.84     .0 User I/O
    db file scattered read              19,037        1.7       0.09     .0 User I/O
    db file sequential read             19,363        1.4       0.07     .0 User I/O
    library cache: mutex X              36,060        1.2       0.03     .0 Concurre
    Buffer Wait Statistics                        DB/Inst: NORM/NORM  Snaps: 60-61
    -> ordered by wait time desc, waits desc
    Class                    Waits Total Wait Time (s)  Avg Time (ms)
    data block          14,204,039               7,922              1
    segment header       1,288,059                  22              0
    undo header            286,303                   2              0
    1st level bmb                1                   0              0
    Segments by Buffer Busy Waits                 DB/Inst: NORM/NORM  Snaps: 60-61
    -> % of Capture shows % of Buffer Busy Waits for each top segment compared
    -> with total Buffer Busy Waits for all segments captured by the Snapshot
    -> When ** MISSING ** occurs, some of the object attributes may not be available
                         Tablespace
    Owner                   Name
                                                                      Buffer
                         Subobject  Obj.                                Busy    % of
    Object Name            Name     Type        Obj#   Dataobj#        Waits Capture
    SYS                  SYSTEM
    AUD$                            TABLE        499        499   15,496,081  100.00

    Hi,
    Happened to me that parameter values of a workspace were defaulting to the values in <configdir>/defaults.xml instead of the ones in the <workspace>,xml.
    When you setup a database (e.g..java -jar ords.war setup --database mydb), you will have mydb.xml, mydb_al.xml and mydb_rt.xml under the <configdir>/ords created for APEX_PUBLIC_USER, APEX_LISTENER and APEX_REST_PUBLIC_USER, respectively. When a config entry is not present in mydb_al.xml or mydb_rt.xml,  one would expect that it would be read from mydb.xml. However, that is not what happens. It reads from defaults.xml under the configdir.
    That is the case for db connection parameters as well.
    So, inspect all three xmls and make sure entries with
    key="db.hostname"
    key="db.password"
    key="db.port"
    key="db.servicename"
    exist in all three xmls.
    P.S. That should be the case for other parameters as well, but haven't verified.
    Ege

  • Reduce buffer busy waits

    Can you please provide me the suggestion on how to reduce the number of buffer busy waits for the below query?
    Please find the query where the buffer busy waits is taking 11091 and 13160 seconds.
    INSERT INTO RPM_CLEARANCE (CLEARANCE_ID, CLEARANCE_DISPLAY_ID, STATE,
    REASON_CODE, CLEARANCE_RESET_ID, RESET_IND, ITEM, ZONE_ID, LOCATION,
    ZONE_NODE_TYPE, EFFECTIVE_DATE, OUT_OF_STOCK_DATE, RESET_DATE, CHANGE_TYPE,
    CHANGE_AMOUNT, CHANGE_PERCENT, CHANGE_CURRENCY, VENDOR_FUNDED_IND,
    CREATE_DATE, CREATE_ID, APPROVAL_DATE, APPROVAL_ID, TSL_EVENT_REF,
    TSL_MARKDOWN_REF, TSL_EVENT_PHASE, TSL_COVER_GROUP, TSL_END_DATE,
    TSL_EVENT_POS_IND, TSL_EVENT_SEL_IND, TSL_HOPOS_TEMPLATE_ID )
    VALUES
    (:B23 , 'reset:'||:B22 , :B21 , :B20 , NULL, '1', :B19 , :B18 , :B17 , :B16 ,
    :B15 , :B15 - 1, NULL, :B14 , :B13 , :B12 , :B11 , '0', :B10 , :B9 , :B10 ,
    :B9 , :B8 , :B7 , :B6 , :B5 , :B4 , :B3 , :B2 , :B1 )
    call count cpu elapsed disk query current rows
    Parse 1849 0.06 0.05 0 0 0 0
    Execute 1895091 2539.06 2606.23 32 119694 31084693 1895091
    Fetch 0 0.00 0.00 0 0 0 0
    total 1896940 2539.12 2606.28 32 119694 31084693 1895091
    Misses in library cache during parse: 0
    Optimizer mode: ALL_ROWS
    Parsing user id: 202 (recursive depth: 1)
    Elapsed times include waiting on following events:
    Event waited on Times Max. Wait Total Waited
    ---------------------------------------- Waited ---------- ------------
    db file sequential read 32 0.01 0.22
    buffer busy waits 11091 0.02 0.29
    latch: cache buffers chains 30 0.00 0.00
    enq: TX - index contention 563 0.01 0.11
    log file switch completion 15 0.06 0.42
    latch: In memory undo latch 7 0.00 0.00
    buffer deadlock 88 0.00 0.00
    cursor: pin S 59 0.00 0.00
    cursor: pin S wait on X 1 0.00 0.00
    INSERT INTO RPM_CLEARANCE (CLEARANCE_ID, CLEARANCE_DISPLAY_ID, STATE,
    REASON_CODE, CLEARANCE_RESET_ID, RESET_IND, ITEM, ZONE_ID, LOCATION,
    ZONE_NODE_TYPE, EFFECTIVE_DATE, OUT_OF_STOCK_DATE, RESET_DATE, CHANGE_TYPE,
    CHANGE_AMOUNT, CHANGE_PERCENT, CHANGE_CURRENCY, VENDOR_FUNDED_IND,
    CREATE_DATE, CREATE_ID, APPROVAL_DATE, APPROVAL_ID, TSL_EVENT_REF,
    TSL_MARKDOWN_REF, TSL_EVENT_PHASE, TSL_COVER_GROUP, TSL_END_DATE,
    TSL_EVENT_POS_IND, TSL_EVENT_SEL_IND, TSL_HOPOS_TEMPLATE_ID )
    VALUES
    (:B23 , 'reset:'||:B22 , :B21 , :B20 , NULL, '1', :B19 , :B18 , :B17 , :B16 ,
    :B15 , :B15 - 1, NULL, :B14 , :B13 , :B12 , :B11 , '0', :B10 , :B9 , :B10 ,
    :B9 , :B8 , :B7 , :B6 , :B5 , :B4 , :B3 , :B2 , :B1 )
    call count cpu elapsed disk query current rows
    Parse 575 0.02 0.01 0 0 0 0
    Execute 1066687 1460.30 1478.77 0 121065 17556349 1066687
    Fetch 0 0.00 0.00 0 0 0 0
    total 1067262 1460.32 1478.79 0 121065 17556349 1066687
    Misses in library cache during parse: 0
    Optimizer mode: ALL_ROWS
    Parsing user id: 202 (recursive depth: 1)
    Elapsed times include waiting on following events:
    Event waited on Times Max. Wait Total Waited
    ---------------------------------------- Waited ---------- ------------
    latch: cache buffers chains 28 0.01 0.01
    buffer busy waits 13160 0.07 0.46
    enq: TX - index contention 522 0.00 0.09
    buffer deadlock 108 0.00 0.00
    latch: In memory undo latch 17 0.00 0.00
    cursor: pin S 8 0.00 0.00
    log file switch completion 2 0.05 0.07
    cursor: pin S wait on X 1 0.00 0.00
    ********************************************************************************

    For reducing buffer busy waits you need to increase DB_CACHE_SIZE parameter value.
    Try the query below
    14:40:23 SQL> select name,size_for_estimate,size_factor,estd_physical_reads,buffers_for_estimate fro
    m v$db_cache_advice order by 4;
    NAME    SIZE_FOR_ESTIMATE SIZE_FACTOR ESTD_PHYSICAL_READS BUFFERS_FOR_ESTIMATE
    DEFAULT               960       1.875              252983               119760
    DEFAULT               912      1.7813              268597               113772
    DEFAULT               864      1.6875              269415               107784
    DEFAULT               816      1.5938              269673               101796
    DEFAULT               768         1.5              270449                95808
    DEFAULT               720      1.4063              270923                89820
    DEFAULT               672      1.3125              272107                83832
    DEFAULT               624      1.2188              276651                77844
    DEFAULT               576       1.125              282272                71856
    DEFAULT               528      1.0313              308869                65868
    DEFAULT               512           1              334346                63872
    DEFAULT               480       .9375              411617                59880
    DEFAULT               432       .8438              467955                53892
    DEFAULT               384         .75              520223                47904
    DEFAULT               336       .6563              575829                41916
    DEFAULT               288       .5625              628226                35928
    DEFAULT               240       .4688              670286                29940
    DEFAULT               192        .375              725289                23952
    DEFAULT               144       .2813              784512                17964
    DEFAULT                96       .1875              921481                11976
    DEFAULT                48       .0938             1948144                 5988
    21 rows selected.You will get a result like above. SIZE_FOR_ESTIMATE column shows values in MB.
    Then from the result, choose a size for the buffer cache in which there is minimal physical reads but according to the avaliability of your physical memory and set it.

  • Slow query performance in Oracle 10.2.0.3

    Hi,
    We have Oracle 10.2.0.3 installed on RHEL 5(64 bit).We have two queries out of which one is a query using select while other query is using a insert.First we executed insert query which inserts 10000 rows in a table and then select query on this table.This works fine in one thread.But when we do samething in 10 threads, at that time INSERT is fine but select is taking very long time for 10 threads.Any bug related to parallel execution of queries for SELECT in 10.2.0.3?Any suggestion??
    Thanks in advance.
    Regards,
    RJ.

    Justin,
    We have a same queries for INSERT and Select in 10 manual sessions outof which select query is taking more time to execute.Please refer to WAITs given below.No there is no bottleneck as far as hardware is concerned because we tested it on different configuration of servers.
    Event                    Waits            Time(s)          Avg Wait(ms)     % Total Call Time     Wait Class
    CPU time                52                                    93.2     
    latch: cache buffers chains     45,542          6          0               10.7               Concurrency
    log file parallel write          2,107          3          1               5.2               System I/O
    log file sync               805          2          2               3.5               Commit
    latch: session allocation     5,116          1          0               2.6               Other Wait Events
    •     s - second
    •     cs - centisecond - 100th of a second
    •     ms - millisecond - 1000th of a second
    •     us - microsecond - 1000000th of a second
    •     ordered by wait time desc, waits desc (idle events last)
    Event                    Waits     %Time-outs     Total Wait Time (s)     Avg wait (ms)     Waits /txn
    latch: cache buffers chains     45,542     0.00          6                    0     22.99
    log file parallel write          2,107     0.00          3                    1     1.06
    log file sync               805     0.00          2                    2     0.41
    latch: session allocation     5,116     0.00          1                    0     2.58
    buffer busy waits          20,482     0.00          1                    0     10.34
    db file sequential read          157     0.00          1                    4     0.08
    control file parallel write     1,330     0.00          0                    0     0.67
    wait list latch free          39     0.00          0                    10     0.02
    enq: TX - index contention     632     0.00          0                    0     0.32
    latch free               996     0.00          0                    0     0.50
    SQL*Net break/reset to client     1,738     0.00          0                    0     0.88
    SQL*Net message to client     108,947     0.00          0                    0     55.00
    os thread startup          2     0.00          0                    19     0.00
    cursor: pin S wait on X          3     100.00          0                    11     0.00
    latch: In memory undo latch     136     0.00          0                    0     0.07
    log file switch completion     4     0.00          0                    7     0.00
    latch: shared pool          119     0.00          0                    0     0.06
    latch: undo global data          121     0.00          0                    0     0.06
    buffer deadlock               238     99.58          0                    0     0.12
    control file sequential read     1,735     0.00          0                    0     0.88
    SQL*Net more data to client     506     0.00          0                    0     0.26
    log file single write          2     0.00          0                    2     0.00
    SQL*Net more data from client     269     0.00          0                    0     0.14
    reliable message          12     0.00          0                    0     0.01
    LGWR wait for redo copy          26     0.00          0                    0     0.01
    rdbms ipc reply               6     0.00          0                    0     0.00
    latch: library cache          7     0.00          0                    0     0.00
    latch: redo allocation          2     0.00          0                    0     0.00
    enq: RO - fast object reuse     2     0.00          0                    0     0.00
    direct path write          21     0.00          0                    0     0.01
    cursor: pin S               1     0.00          0                    0     0.00
    log file sequential read     2     0.00          0                    0     0.00
    direct path read          8     0.00          0                    0     0.00
    SQL*Net message from client     108,949     0.00          43,397                    398     55.00
    jobq slave wait               14,527     49.56          35,159                    2420     7.33
    Streams AQ: qmn slave idle wait     246     0.00          3,524                    14326     0.12
    Streams AQ:qmn coordinator-
    idle wait                451     45.45          3,524                    7814     0.23
    wait for unread message on -
    broadcast channel          3,597     100.00          3,516                    978     1.82
    virtual circuit status          120     100.00          3,516                    29298     0.06
    class slave wait          2     0.00          0                    0     0.00 Message was edited by:
    RJiv
    Message was edited by:
    RJiv
    Message was edited by:
    RJiv
    Message was edited by:
    RJiv
    Message was edited by:
    RJiv
    Message was edited by:
    RJiv
    Message was edited by:
    RJiv
    Message was edited by:
    RJiv
    Message was edited by:
    RJiv

  • Cursor: pin S 事件导致系统性能严重下降

    环境:db:10.2.0.4.0 system:hp-ux 11.31
    近期系统在访问高峰时,性能严重下降,分析awr得到信息为
    Elapsed:          60.20 (mins)          
    DB Time:          1,916.95 (mins)     系统负担非常严重
    Library Hit %:     99.60     Soft Parse %:     99.44
    Execute to Parse %:     -22.89 重解析现象很严重
    Event     Waits     Time(s)     Avg Wait(ms)     % Total Call Time     Wait Class
    CPU time          26,661          23.2     
    cursor: pin S     5,346,339     11,727     2     10.2     Other
    latch: cache buffers chains     25,963     5,205     200     4.5     Concurrency
    db file scattered read     710,601     2,698     4     2.3     User I/O
    latch: library cache     6,743     2,087     309     1.8     Concurrenc
    cursor: pin S事件为等待时间第一, 一般判读为系统并发sql语句过多,由于没有之前正常状态的awr报告作为对比,所以对于这个系统问题的根源和解决办法不是很清楚,对于应用方面,减少并发sql语句,绑定变量等,在数据库方面调整ession_cached_cursors参数,增加sharepool空间,由于水平有限只能知道这么多,希望得到更加深入的分析。谢谢
    附部分awr(如何上传完整报告?)WORKLOAD REPOSITORY report for
    DB Name     DB Id     Instance     Inst num     Release     RAC     Host
    ORCL     1183953527     orcl     1     10.2.0.4.0     NO     HP-UX-1
    Snap Id     Snap Time     Sessions     Cursors/Session
    Begin Snap:     36825     14-8ÔÂ -12 09:00:16     1426      1.5
    End Snap:     36826     14-8ÔÂ -12 10:00:29     1630      1.4
    Elapsed:            60.20 (mins)           
    DB Time:            1,916.95 (mins)           
    Report Summary
    Cache Sizes
    Begin     End          
    Buffer Cache:      9,216M      9,216M     Std Block Size:      8K
    Shared Pool Size:      6,144M      6,144M     Log Buffer:      14,348K
    Load Profile
    Per Second     Per Transaction
    Redo size:      85,077.91      5,632.74
    Logical reads:      323,798.90      21,437.69
    Block changes:      489.98      32.44
    Physical reads:      4,190.38      277.43
    Physical writes:      36.21      2.40
    User calls:      7,273.13      481.53
    Parses:      1,499.45      99.27
    Hard parses:      8.39      0.56
    Sorts:      150.73      9.98
    Logons:      1.72      0.11
    Executes:      1,220.19      80.78
    Transactions:      15.10     
    % Blocks changed per Read:      0.15     Recursive Call %:      9.01
    Rollback per transaction %:      12.68     Rows per Sort:      118.78
    Instance Efficiency Percentages (Target 100%)
    Buffer Nowait %:      99.99     Redo NoWait %:      99.99
    Buffer Hit %:      98.71     In-memory Sort %:      100.00
    Library Hit %:      99.60     Soft Parse %:      99.44
    Execute to Parse %:      -22.89     Latch Hit %:      99.95
    Parse CPU to Parse Elapsd %:      6.61     % Non-Parse CPU:      94.79
    Shared Pool Statistics
    Begin     End
    Memory Usage %:      92.12      92.19
    % SQL with executions>1:      90.33      87.07
    % Memory for SQL w/exec>1:      90.37      88.32
    Top 5 Timed Events
    Event     Waits     Time(s)     Avg Wait(ms)     % Total Call Time     Wait Class
    CPU time            26,661            23.2     
    cursor: pin S      5,346,339      11,727      2      10.2     Other
    latch: cache buffers chains      25,963      5,205      200      4.5     Concurrency
    db file scattered read      710,601      2,698      4      2.3     User I/O
    latch: library cache      6,743      2,087      309      1.8     Concurrency
    Time Model Statistics
    Total time in database user-calls (DB Time): 115017.1s
    Statistics including the word "background" measure background process time, and so do not contribute to the DB time statistic
    Ordered by % or DB time desc, Statistic name
    Statistic Name     Time (s)     % of DB Time
    sql execute elapsed time     89,472.10     77.79
    DB CPU     26,660.94     23.18
    parse time elapsed     20,759.09     18.05
    PL/SQL execution elapsed time     322.11     0.28
    hard parse elapsed time     256.97     0.22
    connection management call elapsed time     90.04     0.08
    failed parse elapsed time     44.23     0.04
    hard parse (sharing criteria) elapsed time     27.70     0.02
    sequence load elapsed time     13.43     0.01
    PL/SQL compilation elapsed time     10.03     0.01
    hard parse (bind mismatch) elapsed time     2.82     0.00
    repeated bind elapsed time     0.10     0.00
    inbound PL/SQL rpc elapsed time     0.05     0.00
    DB time     115,017.09     
    background elapsed time     451.46     
    background cpu time     47.64     
    Back to Wait Events Statistics
    Back to Top
    Wait Class
    s - second
    cs - centisecond - 100th of a second
    ms - millisecond - 1000th of a second
    us - microsecond - 1000000th of a second
    ordered by wait time desc, waits desc
    Wait Class     Waits     %Time -outs     Total Wait Time (s)     Avg wait (ms)     Waits /txn
    Other     5,359,517     0.06     12,985     2     98.23
    Concurrency     41,787     7.97     7,899     189     0.77
    User I/O     1,541,553     0.00     4,385     3     28.25
    Network     28,263,892     0.00     515     0     518.03
    Commit     31,974     1.02     489     15     0.59
    Application     7,028     1.04     232     33     0.13
    System I/O     41,367     0.00     118     3     0.76
    Configuration     48     0.00     8     177     0.00
    Back to Wait Events Statistics
    Back to Top
    Wait Events
    s - second
    cs - centisecond - 100th of a second
    ms - millisecond - 1000th of a second
    us - microsecond - 1000000th of a second
    ordered by wait time desc, waits desc (idle events last)
    Event     Waits     %Time -outs     Total Wait Time (s)     Avg wait (ms)     Waits /txn
    cursor: pin S     5,346,339     0.00     11,727     2     97.99
    latch: cache buffers chains     25,963     0.00     5,205     200     0.48
    db file scattered read     710,601     0.00     2,698     4     13.02
    latch: library cache     6,743     0.00     2,087     309     0.12
    db file sequential read     657,512     0.00     1,191     2     12.05
    latch free     7,839     0.10     973     124     0.14
    read by other session     153,818     0.04     489     3     2.82
    log file sync     31,974     1.02     489     15     0.59
    SQL*Net more data to client     2,454,333     0.00     308     0     44.98
    enq: TX - row lock contention     82     89.02     227     2762     0.00
    latch: library cache lock     968     0.00     211     218     0.02
    latch: session allocation     1,369     0.00     196     143     0.03
    latch: shared pool     1,801     0.00     177     98     0.03
    SQL*Net message to client     25,709,776     0.00     130     0     471.22
    latch: row cache objects     2,772     0.00     113     41     0.05
    log file parallel write     33,654     0.00     105     3     0.62
    SQL*Net more data from client     99,783     0.00     77     1     1.83
    buffer busy waits     198     24.24     64     325     0.00
    latch: cache buffers lru chain     204     0.00     35     173     0.00
    cursor: pin S wait on X     3,284     99.97     32     10     0.06
    LGWR wait for redo copy     3,493     91.24     32     9     0.06
    latch: object queue header operation     91     0.00     8     91     0.00
    latch: In memory undo latch     43     0.00     8     191     0.00
    db file parallel read     530     0.00     7     14     0.01
    enq: SQ - contention     8     0.00     5     670     0.00
    log file sequential read     400     0.00     5     13     0.01
    SQL*Net break/reset to client     6,942     0.00     5     1     0.13
    Streams AQ: qmn coordinator waiting for slave to start     1     100.00     5     4889     0.00
    control file parallel write     1,844     0.00     5     2     0.03
    latch: enqueue hash chains     55     0.00     4     76     0.00
    Log archive I/O     344     0.00     3     9     0.01
    wait list latch free     104     0.00     2     20     0.00
    latch: undo global data     9     0.00     2     191     0.00
    log file switch (checkpoint incomplete)     11     0.00     2     150     0.00
    log file switch completion     28     0.00     1     53     0.00
    os thread startup     11     0.00     1     132     0.00
    control file sequential read     5,093     0.00     0     0     0.09
    latch: redo allocation     4     0.00     0     68     0.00
    enq: RO - fast object reuse     4     0.00     0     59     0.00
    local write wait     50     0.00     0     2     0.00
    latch: library cache pin     1     0.00     0     95     0.00
    latch: cache buffer handles     1     0.00     0     80     0.00
    enq: CF - contention     2     0.00     0     19     0.00
    direct path read temp     18,314     0.00     0     0     0.34
    log file single write     32     0.00     0     1     0.00
    enq: TX - index contention     3     0.00     0     5     0.00
    direct path write temp     25     0.00     0     0     0.00
    reliable message     4     0.00     0     0     0.00
    direct path read     350     0.00     0     0     0.01
    Background Wait Events
    ordered by wait time desc, waits desc (idle events last)
    Event     Waits     %Time -outs     Total Wait Time (s)     Avg wait (ms)     Waits /txn
    events in waitclass Other     2,441     130.93     140     57     0.04
    log file parallel write     33,654     0.00     105     3     0.62
    log file sequential read     400     0.00     5     13     0.01
    control file parallel write     1,844     0.00     5     2     0.03
    buffer busy waits     4     75.00     3     832     0.00
    Log archive I/O     344     0.00     3     9     0.01
    os thread startup     11     0.00     1     132     0.00
    latch: library cache     3     0.00     1     220     0.00
    control file sequential read     4,390     0.00     0     0     0.08
    latch: shared pool     1     0.00     0     309     0.00
    db file sequential read     82     0.00     0     4     0.00
    db file scattered read     56     0.00     0     4     0.00
    log file single write     32     0.00     0     1     0.00
    latch: library cache lock     1     0.00     0     26     0.00
    direct path read     344     0.00     0     0     0.01
    direct path write     343     0.00     0     0     0.01
    latch: In memory undo latch     1     0.00     0     0     0.00
    Back to Wait Events Statistics
    Back to Top
    Operating System Statistics
    Statistic     Total
    AVG_BUSY_TIME     355,404
    AVG_IDLE_TIME     5,624
    AVG_IOWAIT_TIME     1,350
    AVG_SYS_TIME     24,626
    AVG_USER_TIME     330,677
    BUSY_TIME     2,844,143
    IDLE_TIME     45,639
    IOWAIT_TIME     11,363
    SYS_TIME     197,893
    USER_TIME     2,646,250
    LOAD     8
    OS_CPU_WAIT_TIME     ###############
    RSRC_MGR_CPU_WAIT_TIME     0
    VM_IN_BYTES     0
    VM_OUT_BYTES     0
    PHYSICAL_MEMORY_BYTES     51,504,857,088
    NUM_CPUS     8
    NUM_CPU_SOCKETS     8
    Back to Wait Events Statistics
    Back to Top
    Service Statistics
    ordered by DB Time
    Service Name     DB Time (s)     DB CPU (s)     Physical Reads     Logical Reads
    orcl     108,264.60     24,956.90     12,796,071     1,110,466,840
    SYS$USERS     6,748.20     1,707.20     2,350,206     59,196,825
    SYS$BACKGROUND     0.00     0.00     1,188     41,361
    Back to Wait Events Statistics
    Back to Top
    Service Wait Class Stats
    Wait Class info for services in the Service Statistics section.
    Total Waits and Time Waited displayed for the following wait classes: User I/O, Concurrency, Administrative, Network
    Time Waited (Wt Time) in centisecond (100th of a second)
    Service Name     User I/O Total Wts     User I/O Wt Time     Concurcy Total Wts     Concurcy Wt Time     Admin Total Wts     Admin Wt Time     Network Total Wts     Network Wt Time
    orcl     1354220     373545     40097     757402     0     0     26777629     50042
    SYS$USERS     186115     64780     1563     27129     0     0     1456144     1238
    SYS$BACKGROUND     1221     183     23     584     0     0     0     0
    Back to Wait Events Statistics
    Back to Top
    SQL ordered by Elapsed Time
    Resources reported for PL/SQL code includes the resources used by all SQL statements called by the code.
    % Total DB Time is the Elapsed Time of the SQL statement divided into the Total Database Time multiplied by 100
    Elapsed Time (s)     CPU Time (s)     Executions     Elap per Exec (s)     % Total DB Time      SQL Id     SQL Module     SQL Text
    17,352     1,151     282,032     0.06     15.09     58nqcgq93qdpp      w3wp.exe     SELECT sysdate FROM dual
    13,412     3,344     3,516     3.81     11.66     gpz6zsfjvvgx7      obilling.exe     select sum ( costs ) , sum ( ...
    9,176     2,771     393     23.35     7.98     ajp712h079d57      dosage.exe     select DIAG_DESC from outp_mr ...
    8,351     2,411     3,163     2.64     7.26     9skkhx9qfmzjy      outpdoct.exe      SELECT "OUTP_TREAT_REC"."VIS...
    6,278     2,042     2,598     2.42     5.46     f9nfgywd1dzbc      outpdoct.exe      SELECT :"SYS_B_0" del_indica...
    3,406     1,070     729     4.67     2.96     3fgaurmna0hx9      obilling.exe      SELECT "OUTP_PRESC_T"."VISIT...
    3,228     867     9,221     0.35     2.81     3vy04kpknkqgk      w3wp.exe     select T1.REMIND_DATETIME, T1....
    2,802     514     214     13.09     2.44     f4q13jvt6n8r8      mzcx.exe      SELECT DISTINCT "OUTPBILL"."...
    2,406     533     725     3.32     2.09     87mazvgqfas1z      obilling.exe      SELECT "OUTP_TREAT_REC_T"."V...
    2,340     535     724     3.23     2.03     f4xrxypn2cpkm      obilling.exe      SELECT "OUTP_PRESC_T"."VISIT...
    2,303     781     15,143     0.15     2.00     89yw129xmhchc      w3wp.exe     select a.PATIENT_ID, a.VISIT_I...
    1,572     449     4,748     0.33     1.37     0nwg8cxxzcb0m      w3wp.exe     select T1.REMIND_DATETIME, T1....
    1,258     135     44     28.58     1.09     cnuj5dstf2559      outpdoct.exe      SELECT "DOCT_DRUG_PRESC_DE...
    Back to SQL Statistics
    Back to Top
    SQL ordered by CPU Time
    Resources reported for PL/SQL code includes the resources used by all SQL statements called by the code.
    % Total DB Time is the Elapsed Time of the SQL statement divided into the Total Database Time multiplied by 100
    CPU Time (s)     Elapsed Time (s)     Executions     CPU per Exec (s)     % Total DB Time      SQL Id     SQL Module     SQL Text
    3,344     13,412     3,516     0.95     11.66     gpz6zsfjvvgx7      obilling.exe     select sum ( costs ) , sum ( ...
    2,771     9,176     393     7.05     7.98     ajp712h079d57      dosage.exe     select DIAG_DESC from outp_mr ...
    2,411     8,351     3,163     0.76     7.26     9skkhx9qfmzjy      outpdoct.exe      SELECT "OUTP_TREAT_REC"."VIS...
    2,042     6,278     2,598     0.79     5.46     f9nfgywd1dzbc      outpdoct.exe      SELECT :"SYS_B_0" del_indica...
    1,151     17,352     282,032     0.00     15.09     58nqcgq93qdpp      w3wp.exe     SELECT sysdate FROM dual
    1,070     3,406     729     1.47     2.96     3fgaurmna0hx9      obilling.exe      SELECT "OUTP_PRESC_T"."VISIT...
    867     3,228     9,221     0.09     2.81     3vy04kpknkqgk      w3wp.exe     select T1.REMIND_DATETIME, T1....
    781     2,303     15,143     0.05     2.00     89yw129xmhchc      w3wp.exe     select a.PATIENT_ID, a.VISIT_I...
    540     825     1,790     0.30     0.72     cw74b4gm99xd5      ORACLE.EXE     SELECT "A4"."PATIENT_ID", TO_C...
    535     2,340     724     0.74     2.03     f4xrxypn2cpkm      obilling.exe      SELECT "OUTP_PRESC_T"."VISIT...
    533     2,406     725     0.73     2.09     87mazvgqfas1z      obilling.exe      SELECT "OUTP_TREAT_REC_T"."V...
    514     2,802     214     2.40     2.44     f4q13jvt6n8r8      mzcx.exe      SELECT DISTINCT "OUTPBILL"."...
    449     1,572     4,748     0.09     1.37     0nwg8cxxzcb0m      w3wp.exe     select T1.REMIND_DATETIME, T1....
    135     1,258     44     3.06     1.09     cnuj5dstf2559      outpdoct.exe      SELECT "DOCT_DRUG_PRESC_DE...
    Back to SQL Statistics
    Back to Top
    SQL ordered by Gets
    Resources reported for PL/SQL code includes the resources used by all SQL statements called by the code.
    Total Buffer Gets: 1,169,640,302
    Captured SQL account for 71.4% of Total
    Buffer Gets     Executions     Gets per Exec     %Total     CPU Time (s)     Elapsed Time (s)      SQL Id     SQL Module     SQL Text
    158,325,351     3,516     45,029.96     13.54     3344.37     13412.17     gpz6zsfjvvgx7      obilling.exe     select sum ( costs ) , sum ( ...
    142,486,001     3,163     45,047.74     12.18     2410.79     8351.24     9skkhx9qfmzjy      outpdoct.exe      SELECT "OUTP_TREAT_REC"."VIS...
    107,954,245     2,598     41,552.83     9.23     2041.61     6278.17     f9nfgywd1dzbc      outpdoct.exe      SELECT :"SYS_B_0" del_indica...
    54,640,906     729     74,953.23     4.67     1069.81     3405.85     3fgaurmna0hx9      obilling.exe      SELECT "OUTP_PRESC_T"."VISIT...
    54,464,191     15,143     3,596.66     4.66     781.43     2302.60     89yw129xmhchc      w3wp.exe     select a.PATIENT_ID, a.VISIT_I...
    52,300,799     9,221     5,671.92     4.47     866.73     3228.09     3vy04kpknkqgk      w3wp.exe     select T1.REMIND_DATETIME, T1....
    36,350,381     1,790     20,307.48     3.11     539.85     825.10     cw74b4gm99xd5      ORACLE.EXE     SELECT "A4"."PATIENT_ID", TO_C...
    28,097,571     1,132     24,821.18     2.40     105.62     304.33     44cpsg5hgwtv3      outpdoct.exe      SELECT "CLINIC_MASTER"."NAME...
    27,329,179     724     37,747.48     2.34     534.54     2339.79     f4xrxypn2cpkm      obilling.exe      SELECT "OUTP_PRESC_T"."VISIT...
    27,276,384     725     37,622.60     2.33     532.84     2405.61     87mazvgqfas1z      obilling.exe      SELECT "OUTP_TREAT_REC_T"."V...
    26,913,555     4,748     5,668.40     2.30     449.02     1571.69     0nwg8cxxzcb0m      w3wp.exe     select T1.REMIND_DATETIME, T1....
    25,335,084     214     118,388.24     2.17     514.25     2802.12     f4q13jvt6n8r8      mzcx.exe      SELECT DISTINCT "OUTPBILL"."...
    12,756,154     4     3,189,038.50     1.09     73.99     592.57     4v3jxnrj4duq0      ºǫ́ÅäÒ©ÐÂ2012.07.31.exe      SELECT "DRUG_PRESC_MASTER"....
    Back to SQL Statistics
    Back to Top
    SQL ordered by Reads
    Total Disk Reads: 15,136,657
    Captured SQL account for 81.4% of Total
    Physical Reads     Executions     Reads per Exec     %Total     CPU Time (s)     Elapsed Time (s)      SQL Id     SQL Module     SQL Text
    6,716,909     214     31,387.43     44.38     514.25     2802.12     f4q13jvt6n8r8      mzcx.exe      SELECT DISTINCT "OUTPBILL"."...
    3,642,877     44     82,792.66     24.07     134.64     1257.56     cnuj5dstf2559      outpdoct.exe      SELECT "DOCT_DRUG_PRESC_DE...
    1,707,395     12     142,282.92     11.28     56.42     470.82     bkkm26gd9ydjp      prescent.exe      SELECT "DRUG_PRESC_MASTER"....
    208,221     4     52,055.25     1.38     73.99     592.57     4v3jxnrj4duq0      ºǫ́ÅäÒ©ÐÂ2012.07.31.exe      SELECT "DRUG_PRESC_MASTER"....
    137,856     0           0.91     383.09     1041.12     b1y7hxw5s572b      ORACLE.EXE     SELECT /*+ OPAQUE_TRANSFORM */...
    45,210     1     45,210.00     0.30     136.24     617.92     9k2mtm738ffbk      ORACLE.EXE     SELECT "A1"."PATIENT_ID", "A1"...
    21,692     0           0.14     44.82     288.67     5w6u29xnk192r      ORACLE.EXE     SELECT "A1"."TEST_NO", "A1"."I...
    2,078     17,010     0.12     0.01     6.78     13.35     3ts97u1my91n4      testprn.exe      SELECT "LAB"."LAB_TEST_ITEM...
    539     1,732     0.31     0.00     124.85     289.44     0a96nnbzvb78j      sjcj.exe     select diag_desc , doctor fro...
    224     14,499     0.02     0.00     1.75     2.81     adr24vcznhjhz      C:\Documents and Settings\yj.ÐÄÄÚʵÑéÊÒ1.000\×À?     SELECT COSTS , BILLING_INDICA...
    Back to SQL Statistics
    Back to Top
    SQL ordered by Executions
    Total Executions: 4,407,615
    Captured SQL account for 43.0% of Total
    Executions     Rows Processed     Rows per Exec     CPU per Exec (s)     Elap per Exec (s)      SQL Id     SQL Module     SQL Text
    324,063     324,042     1.00     0.00     0.00     1jzhjx524wyfm      presdisp.exe     select TRADE_PRICE , retail_p...
    282,032     281,745     1.00     0.00     0.06     58nqcgq93qdpp      w3wp.exe     SELECT sysdate FROM dual
    197,067     197,006     1.00     0.00     0.00     f0wzs9nc663bn      obilling.exe     select sysdate from dual
    162,033     112,966     0.70     0.00     0.00     ak8wcxnbv0mq7      presdisp.exe     select location from drug_stoc...
    162,032     162,023     1.00     0.00     0.00     g982zswg09n71      presdisp.exe     SELECT sum ( DRUG_STOCK.QUANTI...
    152,651     1,296     0.01     0.00     0.00     ctc7t080mcrpr      PACSVR.exe     Select exam_master.*, exam_ma...
    89,187     89,186     1.00     0.00     0.00     054n8y7sfgsjc      anesmgr.exe     select 1 from dual
    53,088     24,076     0.45     0.00     0.00     fhzdwv4sgvurz      ORACLE.EXE     SELECT "A1"."END_DATE_TIME", S...
    43,074     991     0.02     0.00     0.00     8kj11spfu1w5k      ORACLE.EXE     SELECT DISTINCT "A1"."LOG_DATE...
    40,962     4,803     0.12     0.00     0.00     664crkjw8n47y      doctws.exe     select dept_code , ward_code ...
    Back to SQL Statistics
    Back to Top
    SQL ordered by Parse Calls
    Total Parse Calls: 5,416,394
    Captured SQL account for 41.3% of Total
    Parse Calls     Executions     % Total Parses      SQL Id     SQL Module     SQL Text
    457,941     152,651     8.45     ctc7t080mcrpr      PACSVR.exe     Select exam_master.*, exam_ma...
    324,064     324,063     5.98     1jzhjx524wyfm      presdisp.exe     select TRADE_PRICE , retail_p...
    281,992     282,032     5.21     58nqcgq93qdpp      w3wp.exe     SELECT sysdate FROM dual
    197,067     197,067     3.64     f0wzs9nc663bn      obilling.exe     select sysdate from dual
    162,033     162,033     2.99     ak8wcxnbv0mq7      presdisp.exe     select location from drug_stoc...
    162,032     162,032     2.99     g982zswg09n71      presdisp.exe     SELECT sum ( DRUG_STOCK.QUANTI...
    113,451     0     2.09     gbztx1wqkp1zw      ORACLE.EXE     SELECT * FROM "OPERATION_MASTE...
    109,344     0     2.02     1gk864rffswjx      ORACLE.EXE     SELECT * FROM "PAT_VISIT"
    104,957     0     1.94     dw9ggd0h86cw4      ORACLE.EXE     SELECT * FROM "ADT_LOG"
    89,187     89,187     1.65     054n8y7sfgsjc      anesmgr.exe     select 1 from dual
    Back to SQL Statistics
    Back to Top
    SQL ordered by Sharable Memory
    Only Statements with Sharable Memory greater than 1048576 are displayed
    Sharable Mem (b)     Executions     % Total      SQL Id     SQL Module     SQL Text
    14,285,976           0.22     998p6b7d1q9bz           ** SQL Text Not Available **
    14,265,528           0.22     dsxd21s5bw5h3           ** SQL Text Not Available **
    4,621,517     45     0.07     18h8s79av5v45      w3wp.exe     select T1.REMIND_DATETIME, T1....
    3,949,845     20     0.06     bn69cjprha24w      w3wp.exe     select T1.REMIND_DATETIME, T1....
    3,407,101     41     0.05     5syt1sugfkg7c      w3wp.exe     select T1.REMIND_DATETIME, T1....
    2,623,013     65     0.04     gh6391dujc994      w3wp.exe     select T1.REMIND_DATETIME, T1....
    2,314,741     30     0.04     49husy8txxw6h      w3wp.exe     select T1.REMIND_DATETIME, T1....
    2,184,421     87     0.03     5a77mhab8fmq4      w3wp.exe     select T1.REMIND_DATETIME, T1....
    2,029,197     24     0.03     57wp7yk0ajprt      w3wp.exe     select T1.REMIND_DATETIME, T1....
    2,028,837     5     0.03     csd4pjvrfpxry      w3wp.exe     select T1.REMIND_DATETIME, T1....
    2,024,581     44     0.03     890zsd9cju097      w3wp.exe     select T1.REMIND_DATETIME, T1....
    1,925,949     39     0.03     cw4y41z3vnz8y      w3wp.exe     select T1.REMIND_DATETIME, T1....
    1,793,389     71     0.03     gwx893aphkzw8      w3wp.exe     select T1.REMIND_DATETIME, T1....
    1,747,565     19     0.03     4p0vbj8qba0sj      w3wp.exe     select T1.REMIND_DATETIME, T1....
    1,747,253     75     0.03     6hhka0kpch5bq      w3wp.exe     select T1.REMIND_DATETIME, T1....
    1,746,061     60     0.03     cqbyt6m5m9f6g      w3wp.exe     select T1.REMIND_DATETIME, T1....
    1,745,909     47     0.03     aatwks74mtqkx      w3wp.exe     select T1.REMIND_DATETIME, T1....
    1,718,005     20     0.03     cmws54r2xv332      w3wp.exe     select T1.REMIND_DATETIME, T1....
    1,709,493     24     0.03     71zxrhv747290      w3wp.exe     select T1.REMIND_DATETIME, T1....
    1,265,905     1,585     0.02     a7c5z25vb0kvz      obilling.exe     SELECT COLUMN_NAME, DATA_TYPE...
    1,132,605     30     0.02     9s6btksh172dz      w3wp.exe     select T1.REMIND_DATETIME, T1....
    1,115,157     30     0.02     9s6btksh172dz      w3wp.exe     select T1.REMIND_DATETIME, T1....
    1,055,029     105     0.02     da0pcbq6qxxqx      w3wp.exe     select T1.REMIND_DATETIME, T1....
    Back to SQL Statistics
    Back to Top
    SQL ordered by Version Count
    Only Statements with Version Count greater than 20 are displayed
    Version Count     Executions      SQL Id     SQL Module     SQL Text
    698           998p6b7d1q9bz           ** SQL Text Not Available **
    697           dsxd21s5bw5h3           ** SQL Text Not Available **
    57     192     9mrhw217pc809      DOCTWS.EXE     select MR_CODE , mr_attr , m...
    43     1,307     53mkjqn5tdptj      examapt.exe     SELECT to_char ( last_updt_dat...
    32     125     399071urkdx7n      admit.exe     SELECT to_char ( last_updt_dat...
    26     564     14fz7s749tx24      prescent.exe     Select drug_spec , batch_no , ...
    Back to SQL Statistics
    Back to Top
    Complete List of SQL Text
    SQL Id     SQL Text
    054n8y7sfgsjc     select 1 from dual
    0a96nnbzvb78j     select diag_desc , doctor from outp_mr where patient_id =:1 and visit_date like visit_date and visit_no =:2
    0nwg8cxxzcb0m     select T1.REMIND_DATETIME, T1.START_DATETIME, T1.VISIT_DATE, T1.REMIND_TYPE, T1.REMIND_ID, T1.DELETE_FLAG, T1.REPEAT_INTERVAL, T1.VISIT_ID, T1.VISIT_NO, T1.DELETE_USER_ID, T1.DELETE_USER_NAME, T1.PATIENT_ID, T1.REMIND_DEPICT, T1.REMIND_INDEX, T2.REMIND_TYPE_NAME, T1.REMIND_USER_ID, T1.REMIND_USER_NAME from REMIND T1, REMIND_TYPE_DICT T2 where T1.REMIND_TYPE=T2.REMIND_TYPE_CODE(+) and T1.Remind_Type in (:"SYS_B_0") and T1.Delete_Flag=:"SYS_B_1" and T1.Remind_User_Name=:"SYS_B_2" order by T1.Remind_Id
    14fz7s749tx24     Select drug_spec , batch_no , quantity , supply_indicator from drug_stock where drug_code =:1 and firm_id =:2 and package_spec =:3 and package_units =:4 and storage =:5 order by batch_no DESC
    18h8s79av5v45     
    sql--省略
    Back to SQL Statistics
    Back to Top
    SGA breakdown difference
    ordered by Pool, Name
    N/A value for Begin MB or End MB indicates the size of that Pool/Name was insignificant, or zero in that snapshot
    Pool     Name     Begin MB     End MB     % Diff
    java     free memory     26.47     26.47     0.00
    java     joxlod exec hp     5.34     5.34     0.00
    large     free memory     32.00     32.00     0.00
    shared     CCursor     870.37     871.80     0.16
    shared     Cursor Stats     65.30     65.30     0.00
    shared     PCursor     506.48     505.60     -0.17
    shared     db_block_hash_buckets     90.00     90.00     0.00
    shared     free memory     484.07     480.10     -0.82
    shared     kglsim object batch     90.30     90.30     0.00
    shared     library cache     352.64     351.46     -0.33
    shared     sql area     3,290.12     3,291.57     0.04
    streams     free memory     16.00     16.00     0.00
         buffer_cache     9,216.00     9,216.00     0.00
         fixed_sga     1.99     1.99     0.00
         log_buffer     14.01     14.01     0.00
    Back to Memory Statistics
    Back to Top
    Resource Limit Stats
    only rows with Current or Maximum Utilization > 80% of Limit are shown
    ordered by resource name
    Resource Name     Current Utilization     Maximum Utilization      Initial Allocation      Limit
    processes     1,631     1,824     2000     2000
    sessions     1,639     1,884     2205     2205
    Back to Top
    init.ora Parameters
    Parameter Name     Begin value     End value (if different)
    audit_file_dest     /oracle/product/10.2.0/admin/orcl/adump      
    background_dump_dest     /oracle/product/10.2.0/admin/orcl/bdump      
    compatible     10.2.0.1.0      
    control_files     /oradata/orcl/control01.ctl, /oradata/orcl/control02.ctl, /oradata/orcl/control03.ctl, /oralog/orcl/control04.ctl      
    core_dump_dest     /oracle/product/10.2.0/admin/orcl/cdump      
    cursor_sharing     FORCE      
    db_block_size     8192      
    db_cache_size     9663676416      
    db_domain             
    db_file_multiblock_read_count     32      
    db_name     orcl      
    db_recovery_file_dest     /oracle/product/10.2.0/flash_recovery_area      
    db_recovery_file_dest_size     2147483648      
    java_pool_size     33554432      
    job_queue_processes     10      
    large_pool_size     33554432      
    log_archive_dest_1     LOCATION=/oralog/arch      
    nls_language     SIMPLIFIED CHINESE      
    nls_length_semantics     CHAR      
    nls_territory     CHINA      
    open_cursors     300      
    pga_aggregate_target     2147483648      
    processes     2000      
    remote_login_passwordfile     EXCLUSIVE      
    sessions     2205      
    shared_pool_size     6442450944      
    streams_pool_size     16777216      
    undo_management     AUTO      
    undo_tablespace     UNDOTBS1      
    user_dump_dest     /oracle/product/10.2.0/admin/orcl/udump      
    Back to Top
    End of Report

    焦点在于解析:
    Parses:      1,499.45      99.27
    Hard parses:      8.39      0.56
    Execute to Parse %:      -22.89每秒软解析1500次 ,Execute to Parse 解析是负值 , 这说明 执行解析比非常差。
    以下SQL parse次数频繁:
    SQL ordered by Parse Calls
    Total Parse Calls: 5,416,394
    Captured SQL account for 41.3% of Total
    Parse Calls     Executions     % Total Parses      SQL Id     SQL Module     SQL Text
    457,941     152,651     8.45     ctc7t080mcrpr      PACSVR.exe     Select exam_master.*, exam_ma...
    324,064     324,063     5.98     1jzhjx524wyfm      presdisp.exe     select TRADE_PRICE , retail_p...SQL ID ctc7t080mcrpr     1jzhjx524wyfm     
    ctc7t080mcrpr 实际执行只有 152,651 ,解析倒有457,941次
    建议分析以上语句为何产生了大量软解析
    Advice:
    1. 分析SQL语句,减少软解析的数量
    2. 利用open cursor、session cached cursors等技术减少软解析
    3. 考虑设置_kks_use_mutex_pin=FALSE 禁用10g中的MUTEX PIN CURSOR特性
    *<font color="red" size="2" face="courier">如果觉得本回复有意义,请点击本条回复右手边的Correct按钮,谢谢!</font>*
    Maclean Liu
    Oracle Database Administrator
    Oracle Certified 10g/11g Master     
    www.askmaclean.com
    Edited by: Liu Maclean on 2012-8-23 下午11:48

  • Latch problem.

    Hi All,
    I've severe latching problem in my database, when I look in AWR but confused what is causing it.
    Can someone guide me on this?
    NoWait Waiter
    Latch Name Where Misses Sleeps Sleeps
    In memory undo latch ktiFlush: child 0 5 0
    In memory undo latch kturbk 0 2 2
    cache buffers chains kcbchg: kslbegin: bufs not 0 573 69
    cache buffers chains kcbgtcr: fast path 0 380 554
    cache buffers chains kcbgtcr: kslbegin excl 0 298 547
    cache buffers chains kcbrls: kslbegin 0 183 153
    cache buffers chains kcbzgb: scan from tail. no 0 83 0
    cache buffers chains kcbibr 0 31 40
    cache buffers chains kcbchg: kslbegin: call CR 0 10 70
    cache buffers chains kcbgcur: kslbegin 0 4 13
    cache buffers chains kcbget: pin buffer 0 3 72
    cache buffers chains kcbnew: new latch again 0 3 18
    cache buffers chains kcbzwb 0 2 3
    cache buffers chains kcbbxsv 0 1 3
    cache buffers chains kcbcge 0 1 5
    cache buffers lru chain kcbzgws 0 45 0
    cache buffers lru chain kcbibr 0 43 97
    cache buffers lru chain kcbo_link_q 0 20 5
    cache buffers lru chain kcbw_quiesce_granule 0 5 0
    cache buffers lru chain kcbgtcr:CR Scan:KCBRSKIP 0 1 0
    enqueues ksqdel 0 1 0
    kks stats kks stats alloc/free 0 1 1
    library cache kglobpn: child: 0 2,978 4,471
    library cache kgldti: 2child 0 2,417 12
    library cache kglLockCursor 0 1,604 3,398
    library cache kglpin 0 1,426 538
    library cache kglpndl: child: after proc 0 1,185 7
    library cache kglhdgn: child: 0 766 1,485
    library cache kglpnp: child 0 618 8,902
    library cache kglhdgc: child: 0 331 0
    library cache kgldte: child 0 0 95 809
    library cache kglpndl: child: before pro 0 92 1,800
    library cache kglic 0 50 0
    library cache kglnti 0 30 0
    library cache kglati 0 28 0
    library cache kglobld 0 15 21
    library cache kglScanDependency 0 7 2
    library cache kglukp: child 0 6 5
    library cache kglhdbrnl: child 0 1 0
    library cache lock kgllkdl: child: no lock ha 0 10,017 217
    library cache lock kgllkdl: child: cleanup 0 87 119
    library cache lock kgllkal: child: multiinsta 0 80 48
    library cache lock alloc kgllkget 0 1 1
    library cache pin kglpndl 0 34 6
    library cache pin kglpnp: child 0 17 23
    library cache pin kglpnal: child: alloc spac 0 13 35
    object queue header oper kcbo_switch_cq 0 12 5
    object queue header oper kcbw_link_q 0 9 11
    object queue header oper kcbo_link_q:reget 0 5 0
    object queue header oper kcbw_unlink_q 0 2 10
    redo allocation kcrfw_redo_gen: redo alloc 0 13 0
    row cache objects kqreqd: reget 0 7 0
    row cache objects kqreqd 0 1 0
    session allocation ksuprc 0 44 7
    session allocation ksudlc 0 24 25
    session allocation ksuxds: not user session 0 16 1
    session allocation ksucri 0 9 62
    session allocation kspallmod 0 2 0
    shared pool kghalo 0 2,269 126
    shared pool kghupr1 0 690 3,122
    Thanks for the help.
    Thanks,
    Rana.

    DB Name         DB Id    Instance     Inst Num Release     RAC Host
    orvprd        3135597156 orvprd              1 10.2.0.3.0  NO  ix205
                  Snap Id      Snap Time      Sessions Curs/Sess
    Begin Snap:      8273 22-Jun-09 03:00:39       132      22.4
      End Snap:      8275 22-Jun-09 05:00:42       135      19.3
       Elapsed:              120.05 (mins)
       DB Time:              565.08 (mins)
    Cache Sizes
    ~~~~~~~~~~~                       Begin        End
                   Buffer Cache:     7,392M     7,264M  Std Block Size:         8K
               Shared Pool Size:     2,768M     2,896M      Log Buffer:    14,340K
    Load Profile
    ~~~~~~~~~~~~                            Per Second       Per Transaction
                      Redo size:             33,684.43              7,421.03
                  Logical reads:            133,813.99             29,480.60
                  Block changes:              4,946.72              1,089.81
                 Physical reads:                  2.37                  0.52
                Physical writes:                 12.92                  2.85
                     User calls:                154.34                 34.00
                         Parses:                438.57                 96.62
                    Hard parses:                  1.26                  0.28
                          Sorts:              5,130.10              1,130.21
                         Logons:                  0.07                  0.02
                       Executes:              5,604.35              1,234.70
                   Transactions:                  4.54
      % Blocks changed per Read:    3.70    Recursive Call %:    98.52
    Rollback per transaction %:    0.18       Rows per Sort:     2.85
    Instance Efficiency Percentages (Target 100%)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Buffer Nowait %:  100.00       Redo NoWait %:  100.00
                Buffer  Hit   %:  100.00    In-memory Sort %:  100.00
                Library Hit   %:  100.01        Soft Parse %:   99.71
             Execute to Parse %:   92.17         Latch Hit %:   99.32
    Parse CPU to Parse Elapsd %:    8.09     % Non-Parse CPU:   99.46
    Shared Pool Statistics        Begin    End
                 Memory Usage %:   84.14   84.89
        % SQL with executions>1:   97.57   97.11
      % Memory for SQL w/exec>1:   95.66   95.31
    Top 5 Timed Events                                         Avg %Total
    ~~~~~~~~~~~~~~~~~~                                        wait   Call
    Event                                 Waits    Time (s)   (ms)   Time Wait Class
    CPU time                                         14,995          44.2
    TCP Socket (KGAS)                   611,193       6,230     10   18.4    Network
    latch: library cache                 21,661       4,756    220   14.0 Concurrenc
    latch: shared pool                    3,333         320     96    0.9 Concurrenc
    db file sequential read              17,316         193     11    0.6   User I/O
              -------------------------------------------------------------

  • Insert Into Vs Insert through a Procedure

    Hi All,
    I want to make a comparison between insertion into a table through normal insert into staement and through a procedure.
    I create a table call test_table with 6 columns and a procedure to insert in that table.
    I opened a session that make normal insert for 32 times and another session that uses the procedure to insert for 39 times. My experience says that insertion through a procedure is faster than normal insert into statement. but when i make trace for both sessions here is the results:
    Session1(Normal Insert)
    INSERT INTO TEST_TABLE
    VALUES(:"SYS_B_0", :"SYS_B_1", :"SYS_B_2", :"SYS_B_3", :"SYS_B_4", :"SYS_B_5")
    call count cpu elapsed disk query current rows
    Parse 32 0.00 0.00 0 0 0 0
    Execute 32 0.01 0.02 0 32 129 32
    Fetch 0 0.00 0.00 0 0 0 0
    total 64 0.01 0.03 0 32 129 32
    Misses in library cache during parse: 1
    Optimizer mode: CHOOSE
    Parsing user id: 125 (CALLCNTR44)
    Rows Execution Plan
    0 INSERT STATEMENT MODE: CHOOSE
    OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
    call count cpu elapsed disk query current rows
    Parse 32 0.00 0.00 0 0 0 0
    Execute 32 0.01 0.02 0 32 129 32
    Fetch 0 0.00 0.00 0 0 0 0
    total 64 0.01 0.03 0 32 129 32
    Misses in library cache during parse: 1
    OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
    call count cpu elapsed disk query current rows
    Parse 0 0.00 0.00 0 0 0 0
    Execute 0 0.00 0.00 0 0 0 0
    Fetch 0 0.00 0.00 0 0 0 0
    total 0 0.00 0.00 0 0 0 0
    Misses in library cache during parse: 0
    32 user SQL statements in session.
    0 internal SQL statements in session.
    32 SQL statements in session.
    1 statement EXPLAINed in this session.
    Trace for Session2(Procedure Insert)
    BEGIN sec1.INSERT_INTO_TEXT_TABLE (); END;
    call count cpu elapsed disk query current rows
    Parse 39 0.04 0.00 0 0 0 0
    Execute 39 0.00 0.00 0 0 0 39
    Fetch 0 0.00 0.00 0 0 0 0
    total 78 0.04 0.01 0 0 0 39
    Misses in library cache during parse: 1
    Optimizer mode: CHOOSE
    Parsing user id: 125 (CALLCNTR44)
    INSERT INTO TEST_TABLE
    VALUES('TEST COL1', 'TEST COL2', 'TEST COL3', 'TEST COL4', 'TEST COL5', 'TEST COL6')
    call count cpu elapsed disk query current rows
    Parse 0 0.00 0.00 0 0 0 0
    Execute 39 0.03 0.00 0 0 80 39
    Fetch 0 0.00 0.00 0 0 0 0
    total 39 0.03 0.00 0 0 80 39
    Misses in library cache during parse: 0
    Optimizer mode: CHOOSE
    Parsing user id: 474 (SEC1) (recursive depth: 1)
    Rows Execution Plan
    0 INSERT STATEMENT MODE: CHOOSE
    OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
    call count cpu elapsed disk query current rows
    Parse 117 0.04 0.02 0 0 0 0
    Execute 117 0.03 0.03 0 0 0 78
    Fetch 39 0.03 0.00 0 312 0 0
    total 273 0.10 0.06 0 312 0 78
    Misses in library cache during parse: 3
    OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
    call count cpu elapsed disk query current rows
    Parse 2 0.00 0.00 0 0 0 0
    Execute 41 0.04 0.01 0 0 80 39
    Fetch 2 0.00 0.00 1 5 0 2
    total 45 0.04 0.02 1 5 80 41
    Misses in library cache during parse: 2
    118 user SQL statements in session.
    2 internal SQL statements in session.
    120 SQL statements in session.
    1 statement EXPLAINed in this session.
    So in session 2 parsing of the procedure it self is still done in each execution but for the statment inside it no and in session1 parsing for the insert is done for every execution. So does this proof my knowledge or it is wrong?
    I see that instead of parsing the insert in session1 there is parsing for the procedure in session2 is that correct?
    Thanks in advance

    Tomy3k_Bakr, you are doing the comparison wrong. Are you running these from sqlplus over and over again? Sqlplus parses every statement you send, this is not the correct way to compare.
    Try this instead.
    YAS@10G>create table t (a number,b number,c number);
    Table created.
    YAS@10G>r
      1  create or replace procedure insert_t(pa number,pb number,pc number) as
      2  begin
      3     insert into t values(pa,pb,pc);
      4* end;
    Procedure created.Run this and see the results.
    exec RUNSTATS_PKG.rs_start;
    begin
    for i in 1..10000 loop
         insert /*+ sql */ into t values(i,i,i);
    end loop;
    end;
    exec RUNSTATS_PKG.rs_middle;
    begin
    for i in 1..10000 loop
         insert_t(i,i,i);
    end loop;
    end;
    exec RUNSTATS_PKG.rs_stop;
    Run1 ran in 124 hsecs
    Run2 ran in 139 hsecs
    run 1 ran in 89.21% of the time
    Name                                  Run1        Run2        Diff
    LATCH.list of block allocation           1           0          -1
    STAT...redo ordering marks              92          91          -1
    STAT...table scans (short tabl           3           2          -1
    LATCH.session idle bit                  27          26          -1
    LATCH.In memory undo latch               0           1           1
    LATCH.file cache latch                   3           2          -1
    LATCH.redo allocation                    8           7          -1
    LATCH.archive process latch              1           0          -1
    LATCH.Consistent RBA                     2           3           1
    LATCH.lgwr LWN SCN                       3           2          -1
    LATCH.mostly latch-free SCN              3           2          -1
    LATCH.KMG MMAN ready and start           1           0          -1
    LATCH.compile environment latc           3           2          -1
    LATCH.session timer                      0           1           1
    LATCH.shared pool                        2           3           1
    STAT...change write time                11          12           1
    STAT...table scan blocks gotte           6           4          -2
    STAT...messages sent                     2           4           2
    STAT...data blocks consistent            3           1          -2
    LATCH.channel operations paren           8           6          -2
    STAT...rollbacks only - consis           3           1          -2
    STAT...cleanout - number of kt          27          31           4
    STAT...index fetch by key                6           2          -4
    STAT...cluster key scans                 6           2          -4
    STAT...active txn count during          27          31           4
    STAT...deferred (CURRENT) bloc           6           2          -4
    STAT...consistent changes               20          24           4
    STAT...consistent gets - exami          39          35          -4
    STAT...session cursor cache hi          11           7          -4
    STAT...CR blocks created                 6           2          -4
    STAT...commit cleanouts succes           6           2          -4
    STAT...commit cleanouts                  6           2          -4
    STAT...workarea memory allocat           5           1          -4
    STAT...calls to kcmgcs                  27          31           4
    STAT...execute count                10,014      10,009          -5
    LATCH.active checkpoint queue            5           0          -5
    STAT...opened cursors cumulati          14           8          -6
    STAT...db block changes             20,324      20,318          -6
    LATCH.redo writing                      14           8          -6
    STAT...parse count (total)              14           8          -6
    STAT...CPU used when call star         131         138           7
    STAT...calls to kcmgas                 101          94          -7
    LATCH.undo global data                  50          43          -7
    STAT...no work - consistent re          15           7          -8
    LATCH.dml lock allocation               12           4          -8
    STAT...DB time                         132         140           8
    STAT...cluster key scan block           12           4          -8
    LATCH.library cache lock                16           8          -8
    STAT...buffer is not pinned co          12           4          -8
    LATCH.messages                          27          17         -10
    STAT...db block gets from cach      10,484      10,473         -11
    STAT...db block gets                10,484      10,473         -11
    STAT...enqueue releases                 24          13         -11
    STAT...enqueue requests                 24          13         -11
    STAT...recursive cpu usage              96         107          11
    STAT...Elapsed Time                    127         140          13
    LATCH.enqueues                          35          22         -13
    STAT...redo entries                 10,181      10,195          14
    STAT...CPU used by this sessio         124         138          14
    LATCH.library cache pin             20,072      20,056         -16
    LATCH.cache buffers lru chain          135         153          18
    STAT...table scan rows gotten           54          36         -18
    STAT...calls to get snapshot s          37          19         -18
    LATCH.library cache                 20,094      20,073         -21
    LATCH.enqueue hash chains               50          26         -24
    STAT...bytes received via SQL*       1,210       1,185         -25
    STAT...consistent gets                  91          63         -28
    STAT...consistent gets from ca          91          63         -28
    STAT...free buffer requested           118         152          34
    STAT...session logical reads        10,575      10,536         -39
    LATCH.row cache objects                153         106         -47
    LATCH.checkpoint queue latch            57           0         -57
    LATCH.SQL memory manager worka          73           6         -67
    LATCH.cache buffer handles              16          84          68
    STAT...recursive calls              10,132      10,053         -79
    LATCH.object queue header oper         334         437         103
    LATCH.cache buffers chains          51,652      51,526        -126
    LATCH.session allocation               258         104        -154
    STAT...free buffer inspected           448         188        -260
    LATCH.simulator lru latch              440         865         425
    LATCH.simulator hash latch             458         892         434
    STAT...hot buffers moved to he       1,222         276        -946
    STAT...undo change vector size     684,064     682,932      -1,132
    STAT...redo size                 2,586,108   2,583,908      -2,200
    Run1 latches total versus runs -- difference and pct
    Run1        Run2        Diff       Pct
    94,017      94,489         472     99.50%For runstats see http://asktom.oracle.com/tkyte/runstats.html.

  • Grid Control tuning Advisors for 9i

    Hi all,
    I'm trying to manage a 9.2.0.1 database with Grid Control 10.2.0.2
    Everything is working fine except in the Performance tab, i can't find links for Memory Advisor and Undo Advisor.
    This Link below provides a Database Management Matrix and Memory/Undo advisor are listed in 9i version
    http://www.oracle.com/technology/products/oem/pdf/em_gc_8i-9i_4.pdf
    Any help ?

    Hi MarGur
    I try to help you. The OEM Grid Control in some times not discover the Oracle 9i databases and you must to add manually.
    First, check in your target machine that the user owner of agent software have rights of read and write in the ORACLE_HOME directories of Oracle 9i databases.
    Go to Grid Control Console
    Select Targets ---> Databases
    Press Add --> Select the host ebisdb2.oneshield.com and press continue
    The automatic discover process begin. When finished is very normal that nor discover the Oracle 9i databases. You press the Add manually button
    in the page that appear you must entry all the configuration information of database:
    ORACLE_HOME directory
    DBSNMP USER
    DBSNMP PASSWORD
    HOSTNAME
    PORT
    SID
    Check that the dbsnmp user in your database is unlock and press the TEST button to verify the connection work fine.
    After this many windows appear of configuration process, and you will put the sys password and host credentials.
    If you have problems, tell me
    Regards

  • Enq: TX - row lock contention wait event

    Hi,
    I would like to find which DML query has not given COMMIT or ROLLBACK after the execution. Because one of the development database have more table locks and developer reported that their session was hanging. I referred AWR report also and more timed waits occurred in the enq: TX - row lock contention. I need to trace which DML query has not commit or rollback.
    Please help me to solve the issue.
    Database version: 11.2.0.1.0
    Foreground Wait Events
    Event
    Waits
    %Time -outs
    Total Wait Time (s)
    Avg wait (ms)
    Waits /txn
    % DB time
    enq: TX - row lock contention
    320
    0
    72,047
    225147
    0.20
    99.53
    log file sync
    547
    0
    14
    26
    0.35
    0.02
    library cache lock
    13
    0
    11
    843
    0.01
    0.02
    SQL*Net break/reset to client
    1,080
    0
    2
    1
    0.69
    0.00
    SQL*Net message to client
    659,006
    0
    1
    0
    421.63
    0.00
    direct path sync
    3
    0
    1
    299
    0.00
    0.00
    SQL*Net more data from client
    5,541
    0
    1
    0
    3.55
    0.00
    db file scattered read
    554
    0
    0
    1
    0.35
    0.00
    SQL*Net more data to client
    14,975
    0
    0
    0
    9.58
    0.00
    db file sequential read
    2,817
    0
    0
    0
    1.80
    0.00
    ADR block file read
    4
    0
    0
    43
    0.00
    0.00
    enq: CR - block range reuse ckpt
    2
    0
    0
    71
    0.00
    0.00
    asynch descriptor resize
    38,073
    100
    0
    0
    24.36
    0.00
    latch: shared pool
    61
    0
    0
    1
    0.04
    0.00
    control file sequential read
    6,900
    0
    0
    0
    4.41
    0.00
    Disk file operations I/O
    550
    0
    0
    0
    0.35
    0.00
    cursor: pin S
    1
    0
    0
    8
    0.00
    0.00
    direct path write temp
    34
    0
    0
    0
    0.02
    0.00
    library cache: mutex X
    5
    0
    0
    1
    0.00
    0.00
    latch: In memory undo latch
    2
    0
    0
    1
    0.00
    0.00
    buffer busy waits
    14
    0
    0
    0
    0.01
    0.00
    SQL*Net message from client
    658,990
    0
    294,847
    447
    421.62
    jobq slave wait
    669
    99
    333
    497
    0.43
    PL/SQL lock timer
    1
    100
    1
    998
    0.00

    Oracle does not and cannot tell you from historical views (e.g. AWR) which DMLs have not COMMITed or ROLLBACKed. A Transaction ends with a COMMIT or ROLLBACK.  The transaction could have a million (or more) DML statements with a million (or more) SELECT statements between the first DML and the COMMIT / ROLLBACK.
    Even identifying such DMLs in real time is close to impossible.  Because the session holding the lock may have issued  a dozen or a million subsequent SQL statements while other sessions are waiting for the lock.  You can only identify the session that is the lock holder (the BLOCKING_SESSION in V$SESSION).
    If you have tracing enabled for all sessions, then you could review the trace file for the BLOCKING_SESSION to identify the DML(s) the session has executed.
    Hemant K Chitale

Maybe you are looking for

  • Can only Reconnect Media one file at a time

    I posted this problem almost two months ago, and was told at the time that it was a bug in FCP 6.0 and 6.0.1, but was fixed in 6.0.2. Well, I finally upgraded to 6.0.2, and the problem remains. What is strange, is that after I upgraded, the first two

  • Export Release Build - Problem with Server Settings

    Hello, i have a big problem and i searched the whole day yesterday but didnt found an answer... I started a Flex Mobile and PHP Project. The Project has a Data/Services included wich loads some customer from a database (sql)... for the first a used Z

  • NTFS permission problem keep showing when I try to repair the disk permissions

    Hi guys, After i install the Paragon NTFS 11.2.443, I found that the serial number I had failed to register it so I uninstall it. Now, I keep getting this when i repair my disk permissions. Repairing permissions for "Macintosh HD"Permissions differ o

  • Does JavaScript do Touch up Text?

    Hi, I'm a newbie in JavaScript. I have a requirement to find a string and replace with another string in a pdf file. I can use Touch up Text tool to update the text. However, as I need to do this "find and replace" action number of times, I thought o

  • System.log filling up with enqueueData rtn message

    I'm seeing the following log entry in system.log 7-9 times a second: Oct 22 11:00:00 pbair-paulywalnutz-com kernel[0]: ttyioss9000003: enqueueData rtn (e00002d8) I've tried googling and checked discussions here but can't find anything that seems to b