RMAN wait event

11.2.0.2.0, ASM, 2 node RAC, non-Exadata (youll see why I mention this)
getting really poor performance with RMAN backing up the FRA on ASM to an external NFS mount, 10gb fibre channels to the NFS. About 600gb in about 11 hours.
Checking the performance of the job while its running I see all 4 configured channels are experiencing really long waits on an event "ksv master wait", and as you go off to google and support that youll find this event is linked to a bug on 11.2.0.2.0 accessing ASM. Theres a note on it
High 'ksv master wait' And 'ASM File Metadata Operation' Waits In Non-Exadata 11g [ID 1308282.1]
But the patch mentioned in that note is applied to my grid and rdbms homes and cell_offload_processing access is set to false.
Its possible that the length of time is just how long it takes, but if it was IO causing it I would expect to see an RMAN IO wait event. Has anyone seen that ksv wait event before and can say its ok?

While Im getting the wait event during the RMAN backup Im more inclined to think after reading up on it that its only incidental that it did come up during RMAN and its a larger issue. I'll close the thread here shortly and open the call in there if nothing comes from here.
oops, already done! :)
Edited by: 961469 on Feb 28, 2013 6:54 AM

Similar Messages

  • Wait events

    Hi ! I have the following wait events in my top timed and I don't know who originated them:
    Wait Event Wait Time Summary Avg Wait Time (ms)
    I# Class Event Waits %Timeouts Total(s) Avg(ms) %DB time Avg Min Max Std Dev Cnt
    * DB CPU N/A N/A 59,651.48 N/A 45.87 2
    User I/O db file sequential read 4,369,213 0.0 20,831.46 4.8 16.02 4.72 4.29 5.14 0.60 2
    Other enq: CF - contention 155,822 3.9 10,390.74 66.7 7.99 68.62 60.31 76.94 11.76 2
    System I/O RMAN backup & recovery I/O 87,205 0.0 5,477.09 62.8 9.15 62.81 62.81 62.81 1
    Cluster gc current block 2-way 2,914,457 0.0 4,811.61 1.7 3.70 1.67 1.60 1.74 0.10 2
    System I/O control file sequential read 3,038,672 0.0 3,762.66 1.2 2.89 1.24 1.22 1.27 0.04 2
    Concurrenc os thread startup 2,842 0.0 3,695.14 1300.2 2.84 1311.83 1143.07 1480.59 238.66 2
    System I/O log file parallel write 1,341,907 0.0 2,530.17 1.9 1.95 1.88 1.88 1.89 0.01 2
    Other reliable message 471,495 0.1 2,388.01 5.1 1.84 5.08 4.12 6.03 1.35 2
    Concurrenc row cache lock 3,135,774 0.0 2,224.53 0.7 1.71 0.72 0.68 0.75 0.05 2
    1 DB CPU N/A N/A 22,584.30 N/A 37.75
    User I/O db file sequential read 2,451,215 0.0
    System I/O RMAN backup & recovery I/O 87,205 0.0
    Other enq: CF - contention 59,735 5.3
    Cluster gc current block 2-way 1,803,542 0.0
    System I/O control file sequential read 1,831,180 0.0
    Concurrenc os thread startup 1,323 0.0
    System I/O log file parallel write 727,883 0.0
    Cluster gc cr multi block request 523,744 0.0
    Concurrenc row cache lock 1,830,913 0.0
    2 DB CPU N/A N/A
    User I/O db file sequential read 1,917,998 0.0
    Other enq: CF - contention 96,087 3.0
    Cluster gc current block 2-way 1,110,915 0.0
    Concurrenc os thread startup 1,519 0.0
    System I/O control file sequential read 1,207,492 0.0
    User I/O direct path read 404,587 0.0
    Other reliable message 233,033 0.1
    System I/O log file parallel write 614,024 0.0
    System I/O control file parallel write 128,905 0.0
    Those are the most worrying events:
    enq: CF - contention
    I/O control file sequential read
    Concurrenc os thread startup
    I have been investigating and I wonder what is wrong to get Concurrenc os thread startup. According to one blog, os thread should be always related with network issues...
    The awr snapshot is from my production window day.
    Rac 11.2.0.3 two nodes on Solaris Sparc 10.

    I have studied the ASH in the problematic period and I have found that there are some full scans:
    Summary of All User Input
    Format         : TEXT
    DB Id          : 2752323407
    Inst num       : 1
    Begin time     : 08-Feb-13 09:30:00
    End time       : 08-Feb-13 10:20:00
    Slot width     : Default
    Report targets : 0
    Report name    : ashrpt_1_0208_1020.txt
    ASH Report For dbp/dbp1
    DB Name         DB Id    Instance     Inst Num Release     RAC Host
    dbp           2752323407 dbp1                1 11.2.0.3.0  YES host-dbp-1
    CPUs           SGA Size       Buffer Cache        Shared Pool    ASH Buffer Size
      16     12,651M (100%)    10,048M (79.4%)     1,921M (15.2%)       32.0M (0.3%)
              Analysis Begin Time:   08-Feb-13 09:30:00
                Analysis End Time:   08-Feb-13 10:20:00
                     Elapsed Time:        50.0 (mins)
                Begin Data Source:   DBA_HIST_ACTIVE_SESS_HISTORY
                                     in AWR snapshot 5100
                  End Data Source:   DBA_HIST_ACTIVE_SESS_HISTORY
                                     in AWR snapshot 5101
                                      + V$ACTIVE_SESSION_HISTORY
                     Sample Count:      10,069
          Average Active Sessions:       33.56
      Avg. Active Session per CPU:        2.10
                    Report Target:   None specified
    Top User Events                     DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
                                                                   Avg Active
    Event                               Event Class        % Event   Sessions
    library cache lock                  Concurrency          43.73      14.68
    cursor: pin S wait on X             Concurrency          18.61       6.25
    CPU + Wait for CPU                  CPU                  15.77       5.29
    reliable message                    Other                 5.88       1.97
    enq: KO - fast object checkpoint    Application           3.48       1.17
    Top Background Events               DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
                                                                   Avg Active
    Event                               Event Class     % Activity   Sessions
    CPU + Wait for CPU                  CPU                   1.25       0.42
    Top Cluster Events                  DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
    Event                          % Event Remote I % Activity
    gc current block 2-way            1.72        2       1.72
    gc cr grant 2-way                 1.58      N/A       1.07
    Top Event P1/P2/P3 Values           DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
    Event                          % Event  P1 Value, P2 Value, P3 Value % Activity
    Parameter 1                Parameter 2                Parameter 3
    library cache lock               43.75 "29115227816","29218763456","       1.22
    handle address             lock address               100*mode+namespace
                                           "29115227816","28694732944","       1.20
                                           "29115227816","28812373936","       1.17
    cursor: pin S wait on X          18.61 "1497800770","3934190043136",       1.54
    idn                        value                      where
                                           "1497800770","7773890805760",       1.15
    reliable message                  6.07 "30432532808","30354909248","       0.13
    channel context            channel handle             broadcast message
    enq: KO - fast object checkpoi    3.49      "1263468550","65640","1"       0.52
    name|mode                  2                          0
    db file sequential read           1.81               "1","25220","1"       0.01
    file#                      block#                     blocks
    Top Service/Module                  DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
    Service        Module                   % Activity Action               % Action
    dbp_DVEBMGS11  CL_SQL_STATEMENT========      86.80 383                     86.80
    dbp_D10_0066   CL_SQL_STATEMENT========       6.28 383                      3.34
                                                       104                      2.94
    dbp_D10_0064   CL_SQL_STATEMENT========       2.40 383                      2.39
    SYS$BACKGROUND UNNAMED                        1.51 UNNAMED                  1.51
    Top Client IDs                      DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
                      No data exists for this section of the report.
    Top SQL Command Types               DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
    -> 'Distinct SQLIDs' is the count of the distinct number of SQLIDs
          with the given SQL Command Type found over all the ASH samples
          in the analysis period
                                               Distinct            Avg Active
    SQL Command Type                             SQLIDs % Activity   Sessions
    SELECT                                          485      94.56      31.74
    ALTER TABLE                                     220       2.89       0.97
    Top Phases of Execution             DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
                                              Avg Active
    Phase of Execution             % Activity   Sessions
    Parse                               67.50      22.66
    SQL Execution                       30.46      10.22
    Hard Parse                           5.37       1.80
    Top Remote Instances                DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
    Wait Class           % Wait  Remote I % Activity
    Cluster                 5.22        2       3.90
                                      N/A       1.27
    Top SQL with Top Events     DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
                                                            Sampled #
                     SQL ID             Planhash        of Executions     % Activity
    Event                          % Event Top Row Source                    % RwSrc
              350v06jcnd822                  N/A                    0          18.03
    library cache lock                9.41 ** Row Source Not Available **       9.41
    SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
    "DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
    ___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
                                             N/A                    0          18.03
    cursor: pin S wait on X           8.62 ** Row Source Not Available **       8.62
    SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
    "DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
    ___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
              48unmtd983uz6                  N/A                    0          16.75
    library cache lock               12.87 ** Row Source Not Available **      12.87
    SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
    3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
    AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
                                             N/A                    0          16.75
    cursor: pin S wait on X           3.88 ** Row Source Not Available **       3.88
    SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
    3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
    AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
              350v06jcnd822           2426825131                    0          15.49
    library cache lock                9.74 ** Row Source Not Available **       9.74
    SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
    "DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
    ___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
    cursor: pin S wait on X           4.14 ** Row Source Not Available **       4.14
    CPU + Wait for CPU                1.61 SELECT STATEMENT                     1.58
              48unmtd983uz6           3511339786                    0          14.98
    library cache lock               11.50 ** Row Source Not Available **      11.50
    SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
    3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
    AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
    cursor: pin S wait on X           1.97 ** Row Source Not Available **       1.97
    CPU + Wait for CPU                1.51 SELECT STATEMENT                     1.42
              07tcvyb6frtkx           2929764020                    1           1.87
    gc cr grant 2-way                 0.80 TABLE ACCESS - BY USER ROWID         0.75
    SELECT "D3"."SID_0SHIP_TO" AS "SID" FROM "/BIC/FZ99IC035" "F" JOIN "/BIC/DZ99IC
    0352" "D2" ON "F" . "KEY_Z99IC0352" = "D2" . "DIMID" JOIN "/BI0/XMATERIAL" "X9"
    ON "D2" . "SID_0MATERIAL" = "X9" . "SID" JOIN "/BIC/DZ99IC0355" "D5" ON "F" .
    "KEY_Z99IC0355" = "D5" . "DIMID" JOIN "/BIC/DZ99IC0353" "D3" ON "F" . "KEY_Z99
    Top SQL with Top Events     DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
                                                            Sampled #
                     SQL ID             Planhash        of Executions     % Activity
    Event                          % Event Top Row Source                    % RwSrc
    Top SQL with Top Row Sources        DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
                                                            Sampled #
                     SQL ID             PlanHash        of Executions     % Activity
    Row Source                               % RwSrc Top Event               % Event
              350v06jcnd822                  N/A                    0          18.03
    ** Row Source Not Available **             18.03 library cache lock         9.41
    SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
    "DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
    ___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
              48unmtd983uz6                  N/A                    0          16.75
    ** Row Source Not Available **             16.75 library cache lock        12.87
    SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
    3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
    AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
              350v06jcnd822           2426825131                    0          15.49
    ** Row Source Not Available **             13.91 library cache lock         9.74
    SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
    "DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
    ___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
    SELECT STATEMENT                            1.58 CPU + Wait for CPU         1.58
              48unmtd983uz6           3511339786                    0          14.98
    ** Row Source Not Available **             13.56 library cache lock        11.50
    SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
    3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
    AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
    SELECT STATEMENT                            1.42 CPU + Wait for CPU         1.42
              07tcvyb6frtkx           2929764020                    1           1.87
    TABLE ACCESS - BY USER ROWID                1.59 gc cr grant 2-way          0.75
    SELECT "D3"."SID_0SHIP_TO" AS "SID" FROM "/BIC/FZ99IC035" "F" JOIN "/BIC/DZ99IC
    0352" "D2" ON "F" . "KEY_Z99IC0352" = "D2" . "DIMID" JOIN "/BI0/XMATERIAL" "X9"
    ON "D2" . "SID_0MATERIAL" = "X9" . "SID" JOIN "/BIC/DZ99IC0355" "D5" ON "F" .
    "KEY_Z99IC0355" = "D5" . "DIMID" JOIN "/BIC/DZ99IC0353" "D3" ON "F" . "KEY_Z99
    Top SQL using literals              DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
    -> FORCE_MATCHING_SIGNATURE is used to identify SQL statements that are
          identical except for their use of literals.
    -> Please refer to the Oracle Database Reference to understand how
          the FORCE_MATCHING_SIGNATURE is derived.
                                         # of Sampled
    FORCE_MATCHING_SIGNATURE % Activity SQL Versions
    Example SQL 1
    Example SQL 2
          1021017294885722791       2.89          218
    0htvt0stu1vtq
    SELECT COUNT(*) FROM "/BIC/FZ99IC003" WHERE "KEY_Z99IC003P" = :A0
    0htvt0stu1vtq
    Top Parsing Module/Action           DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
    Module                         Action                           % Activ Event
    CL_SQL_STATEMENT============== 383                                67.25 library
                                   383                                      cursor:
                                   383                                      CPU + Wa
    Top Sessions running PQs            DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
    -> This section aggregates all the work done by the PQ slaves into
          the session issuing the parallel query.
    Sid,Srl# (Inst) % Activity SQL ID        Event                          % Event
    User                 Program
      1506,   19(1)      33.57 350v06jcnd822 library cache lock               19.15
    UserID:
    SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
    "DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
    ___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
                                             cursor: pin S wait on X          12.76
    SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
    "DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
    ___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
                                             CPU + Wait for CPU                1.61
    SELECT "DT"."SID_0CALMONTH" AS "S____048" ,"D3"."SID_0MATERIAL" AS "S____006" ,
    "DU"."SID_0UNIT" AS "S____023" ,"DT"."SID_0CALDAY" AS "S____021" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" ,"X33"."S__Z99GRMAT" AS "S_
    ___4443" , SUM ( "F"."QUANTITY" ) AS "Z____1299" , COUNT( * ) AS "Z____016"
      2255, 1067(1)      31.78 48unmtd983uz6 library cache lock               24.37
    UserID:
    SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
    3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
    AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
                                             cursor: pin S wait on X           5.85
    SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
    3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
    AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
                                             CPU + Wait for CPU                1.51
    SELECT "DT"."SID_0CALDAY" AS "S____021" ,"DT"."SID_0CALMONTH" AS "S____048" ,"D
    3"."SID_0MATERIAL" AS "S____006" ,"DU"."SID_0UNIT" AS "S____023" ,"D2"."SID_0MET
    YPE" AS "S____1342" ,"D2"."SID_0VTYPE" AS "S____504" , SUM ( "F"."QUANTITY" )
    AS "Z____1299" , COUNT( * ) AS "Z____016" FROM "/BIC/FZ99IC114" "F" JOIN "/BIC
    Top DB Objects                      DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
    -> With respect to Application, Cluster, User I/O and buffer busy waits only.
          Object ID % Activity Event                             % Event
    Object Name (Type)                                    Tablespace
           13661539       2.45 gc buffer busy acquire               0.87
    SAPSR3./BIC/EZ99IC013 (TABLE)                         PSAPSR3SSD
    Top DB Files                        DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
    -> With respect to Cluster and User I/O events only.
            File ID % Activity Event                             % Event
    File Name                                             Tablespace
                 53       3.60 gc current block 2-way               0.98
    +dbp_DATA/dbp_2/datafile/psapsr3ssd.315.805562113     PSAPSR3SSD
    Top Latches                         DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
                      No data exists for this section of the report.
    Activity Over Time                  DB/Inst: dbp/dbp1  (Feb 08 09:30 to 10:20)
    -> Analysis period is divided into smaller time slots
    -> Top 3 events are reported in each of those slots
    -> 'Slot Count' shows the number of ASH samples in that slot
    -> 'Event Count' shows the number of ASH samples waiting for
       that event in that slot
    -> '% Event' is 'Event Count' over all ASH samples in the analysis period
                             Slot                                   Event
    Slot Time (Duration)    Count Event                             Count % Event
    09:30:00   (5.0 min)      260 gc buffer busy acquire               43    0.43
                                  reliable message                     34    0.34
                                  CPU + Wait for CPU                   29    0.29
    09:35:00   (5.0 min)      303 CPU + Wait for CPU                   76    0.75
                                  db file sequential read              40    0.40
                                  gc buffer busy acquire               39    0.39
    09:40:00   (5.0 min)      366 CPU + Wait for CPU                  209    2.08
                                  db file sequential read              26    0.26
                                  gc current block 2-way               22    0.22
    09:45:00   (5.0 min)      511 CPU + Wait for CPU                  249    2.47
                                  cursor: pin S wait on X              93    0.92
                                  reliable message                     45    0.45
    09:50:00   (5.0 min)    2,245 cursor: pin S wait on X           1,442   14.32
                                  library cache lock                  407    4.04
                                  reliable message                    112    1.11
    09:55:00   (5.0 min)    2,037 library cache lock                1,378   13.69
                                  cursor: pin S wait on X             297    2.95
                                  CPU + Wait for CPU                  125    1.24
    10:00:00   (5.0 min)    1,823 library cache lock                1,371   13.62
                                  CPU + Wait for CPU                  263    2.61
                                  reliable message                     72    0.72
    10:05:00   (5.0 min)    1,273 library cache lock                  866    8.60
                                  CPU + Wait for CPU                  155    1.54
                                  reliable message                     96    0.95
    10:10:00   (5.0 min)      798 library cache lock                  350    3.48
                                  CPU + Wait for CPU                  287    2.85
                                  reliable message                     54    0.54
    10:15:00   (5.0 min)      436 CPU + Wait for CPU                  200    1.99
                                  reliable message                     61    0.61
                                  enq: KO - fast object checkpoi       42    0.42
              -------------------------------------------------------------Problems are always on instance 1.
    The queries are different each day, the top sql with performance problem changes the sql_id and I cant attack them or apply a sql profile or tune them because they only execute during a period.
    Any idea?
    :(

  • Imm op in Top 5 wait event

    Hi,
    I found the event: "imm op" in the top of the "top 5 wait event" , at the awr report (10204 version) .
    Is it an idel event ? is related to the fact that the database is bening backedup at the same time (event Backup: sbtbackup ) ?
    Is there somthing i can do to reduce it value ?
    Thanks
    Top 5 Timed Events                                         Avg %Total
    ~~~~~~~~~~~~~~~~~~                                        wait   Call
    Event                                 Waits    Time (s)   (ms)   Time Wait Class
    imm op                                5,936       5,879    990   68.8      Other
    Backup: sbtbackup                         8       5,752 ######   67.3 Administra
    db file scattered read              200,642       3,335     17   39.0   User I/O
    CPU time                                          3,177          37.2
    log file parallel write             150,271       1,822     12   21.3 System I/O
              -------------------------------------------------------------

    Hi Werner,
    BACKUP_TAPE_IO_SLAVES is set to TRUE in the case of a RMAN backup if dedicated I/O slave processes are to copy backup write processes to tape and not the Oracle shadow processes. In this case, "imm op" waits only affect the backup runtime, but not the live system.My init.ora parameter are as followed:
    backup_tape_io_slaves TRUE
    dbwr_io_slaves 0
    db_writer_processes 1
    I dont know the reason why backup_tape_io_slaves is set to TRUE.
    Would you suggest me to set it to FALSE and test it again ?
    Thanks
    End value
    Parameter Name                Begin value                       (if different)
    aq_tm_processes               0
    background_dump_dest          C:\ORACLE\ADMIN\XXX\BDUMP
    backup_tape_io_slaves         TRUE
    compatible                    10.2.0
    control_files                 D:\ORADATA\XXX\CONTROL01.CTL, D:
    core_dump_dest                C:\ORACLE\ADMIN\XXX\CDUMP
    db_block_size                 8192
    db_cache_size                 8388608
    db_domain
    db_file_multiblock_read_count 16
    db_name                       xxxx
    db_recovery_file_dest         T:\oradata\XXX\archive
    db_recovery_file_dest_size    1073741824
    dispatchers                   (PROTOCOL=TCP) (SERVICE=XXXXDB)
    fast_start_mttr_target        300
    instance_name                 xxxx
    java_pool_size                0
    job_queue_processes           3
    large_pool_size               0
    local_listener                LISTENER_xxxx
    log_archive_dest_1            LOCATION=T:\oradata\XXX\archive
    log_archive_format            %t_%r_%s.dbf
    max_dump_file_size            UNLIMITED
    nls_length_semantics          BYTE
    open_cursors                  1200
    processes                     150
    query_rewrite_enabled         FALSE
    remote_login_passwordfile     EXCLUSIVE
    resource_limit                TRUE
    resource_manager_plan
    session_cached_cursors        300
    session_max_open_files        20
    sga_target                    838860800
    shared_pool_size              0
    sort_area_size                524288
    star_transformation_enabled   FALSE
    streams_pool_size             50331648
    timed_statistics              TRUE
    undo_management               AUTO
    undo_retention                3600
    undo_tablespace               UNDOTBS1
    user_dump_dest                C:\ORACLE\ADMIN\XXX\UDUMP
              -------------------------------------------------------------

  • Performance Issue: Wait event "log file sync" and "Execute to Parse %"

    In one of our test environments users are complaining about slow response.
    In statspack report folowing are the top-5 wait events
    Event Waits Time (cs) Wt Time
    log file parallel write 1,046 988 37.71
    log file sync 775 774 29.54
    db file scattered read 4,946 248 9.47
    db file parallel write 66 248 9.47
    control file parallel write 188 152 5.80
    And after runing the same application 4 times, we are geting Execute to Parse % = 0.10. Cursor sharing is forced and query rewrite is enabled
    When I view v$sql, following command is parsed frequently
    EXECUTIONS PARSE_CALLS
    SQL_TEXT
    93380 93380
    select SEQ_ORDO_PRC.nextval from DUAL
    Please suggest what should be the method to troubleshoot this and if I need to check some more information
    Regards,
    Sudhanshu Bhandari

    Well, of course, you probably can't eliminate this sort of thing entirely: a setup such as yours is inevitably a compromise. What you can do is make sure your log buffer is a good size (say 10MB or so); that your redo logs are large (at least 100MB each, and preferably large enough to hold one hour or so of redo produced at the busiest time for your database without filling up); and finally set ARCHIVE_LAG_TARGET to something like 1800 seconds or more to ensure a regular, routine, predictable log switch.
    It won't cure every ill, but that sort of setup often means the redo subsystem ceases to be a regular driver of foreground waits.

  • Need help to analysis "foreground and background wait events" on statspack report for oracle database 11.2.0.4 on AIX

    Hi: I'm analyzing this STATSPACK report: it is "volume test" on our UAT server, so most input is from 'bind variables'.  Our shared pool is well utilized in oracle.  Oracle redo logs is not appropriately configured on this server, as in 'Top 5 wait events' there are 2 for redos.
    I need to know what else information can be dig-out from 'foreground wait events' & 'background wait events', and what can assist us to better understanding, in combination of 'Top 5 wait event's, that how the server/test went?  it could be overwelming No. of wait events, so appreciate any helpful diagnostic or analysis.  Database is oracle 11.2.0.4 upgraded from 11.2.0.3, on IBM AIX power system 64bit, level 6.x
    STATSPACK report for
    Database    DB Id    Instance     Inst Num  Startup Time   Release     RAC
    ~~~~~~~~ ----------- ------------ -------- --------------- ----------- ---
    700000XXX   XXX              1 22-Apr-15 12:12 11.2.0.4.0  NO
    Host Name             Platform                CPUs Cores Sockets   Memory (G)
    ~~~~ ---------------- ---------------------- ----- ----- ------- ------------
         dXXXX_XXX    AIX-Based Systems (64-     2     1       0         16.0
    Snapshot       Snap Id     Snap Time      Sessions Curs/Sess Comment
    ~~~~~~~~    ---------- ------------------ -------- --------- ------------------
    Begin Snap:       5635 22-Apr-15 13:00:02      114       4.6
      End Snap:       5636 22-Apr-15 14:00:01      128       8.8
       Elapsed:      59.98 (mins) Av Act Sess:       0.6
       DB time:      35.98 (mins)      DB CPU:      19.43 (mins)
    Cache Sizes            Begin        End
    ~~~~~~~~~~~       ---------- ----------
        Buffer Cache:     2,064M              Std Block Size:         8K
         Shared Pool:     3,072M                  Log Buffer:    13,632K
    Load Profile              Per Second    Per Transaction    Per Exec    Per Call
    ~~~~~~~~~~~~      ------------------  ----------------- ----------- -----------
          DB time(s):                0.6                0.0        0.00        0.00
           DB CPU(s):                0.3                0.0        0.00        0.00
           Redo size:          458,720.6            8,755.7
       Logical reads:           12,874.2              245.7
       Block changes:            1,356.4               25.9
      Physical reads:                6.6                0.1
    Physical writes:               61.8                1.2
          User calls:            2,033.7               38.8
              Parses:              286.5                5.5
         Hard parses:                0.5                0.0
    W/A MB processed:                1.7                0.0
              Logons:                1.2                0.0
            Executes:              801.1               15.3
           Rollbacks:                6.1                0.1
        Transactions:               52.4
    Instance Efficiency Indicators
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                Buffer Nowait %:  100.00       Redo NoWait %:  100.00
                Buffer  Hit   %:   99.98  Optimal W/A Exec %:  100.00
                Library Hit   %:   99.77        Soft Parse %:   99.82
             Execute to Parse %:   64.24         Latch Hit %:   99.98
    Parse CPU to Parse Elapsd %:   53.15     % Non-Parse CPU:   98.03
    Shared Pool Statistics        Begin   End
                 Memory Usage %:   10.50   12.79
        % SQL with executions>1:   69.98   78.37
      % Memory for SQL w/exec>1:   70.22   81.96
    Top 5 Timed Events                                                    Avg %Total
    ~~~~~~~~~~~~~~~~~~                                                   wait   Call
    Event                                            Waits    Time (s)   (ms)   Time
    CPU time                                                       847          50.2
    enq: TX - row lock contention                    4,480         434     97   25.8
    log file sync                                  284,169         185      1   11.0
    log file parallel write                        299,537         164      1    9.7
    log file sequential read                           698          16     24    1.0
    Host CPU  (CPUs: 2  Cores: 1  Sockets: 0)
    ~~~~~~~~              Load Average
                          Begin     End      User  System    Idle     WIO     WCPU
                           1.16    1.84     19.28   14.51   66.21    1.20   82.01
    Instance CPU
    ~~~~~~~~~~~~                                       % Time (seconds)
                         Host: Total time (s):                  7,193.8
                      Host: Busy CPU time (s):                  2,430.7
                       % of time Host is Busy:      33.8
                 Instance: Total CPU time (s):                  1,203.1
              % of Busy CPU used for Instance:      49.5
            Instance: Total Database time (s):                  2,426.4
      %DB time waiting for CPU (Resource Mgr):       0.0
    Memory Statistics                       Begin          End
    ~~~~~~~~~~~~~~~~~                ------------ ------------
                      Host Mem (MB):     16,384.0     16,384.0
                       SGA use (MB):      7,136.0      7,136.0
                       PGA use (MB):        282.5        361.4
        % Host Mem used for SGA+PGA:         45.3         45.8
    Foreground Wait Events  DB/Inst: XXXXXs  Snaps: 5635-5636
    -> Only events with Total Wait Time (s) >= .001 are shown
    -> ordered by Total Wait Time desc, Waits desc (idle events last)
                                                                 Avg          %Total
                                              %Tim Total Wait   wait    Waits   Call
    Event                               Waits  out   Time (s)   (ms)     /txn   Time
    enq: TX - row lock contentio        4,480    0        434     97      0.0   25.8
    log file sync                     284,167    0        185      1      1.5   11.0
    Disk file operations I/O            8,741    0          4      0      0.0     .2
    direct path write                  13,247    0          3      0      0.1     .2
    db file sequential read             6,058    0          1      0      0.0     .1
    buffer busy waits                   1,800    0          1      1      0.0     .1
    SQL*Net more data to client        29,161    0          1      0      0.2     .1
    direct path read                    7,696    0          1      0      0.0     .0
    db file scattered read                316    0          1      2      0.0     .0
    latch: shared pool                    144    0          0      2      0.0     .0
    CSS initialization                     30    0          0      3      0.0     .0
    cursor: pin S                          10    0          0      9      0.0     .0
    row cache lock                         41    0          0      2      0.0     .0
    latch: row cache objects               19    0          0      3      0.0     .0
    log file switch (private str            8    0          0      7      0.0     .0
    library cache: mutex X                 28    0          0      2      0.0     .0
    latch: cache buffers chains            54    0          0      1      0.0     .0
    latch free                            290    0          0      0      0.0     .0
    control file sequential read        1,568    0          0      0      0.0     .0
    log file switch (checkpoint             4    0          0      6      0.0     .0
    direct path sync                        8    0          0      3      0.0     .0
    latch: redo allocation                 60    0          0      0      0.0     .0
    SQL*Net break/reset to clien           34    0          0      1      0.0     .0
    latch: enqueue hash chains             45    0          0      0      0.0     .0
    latch: cache buffers lru cha            7    0          0      2      0.0     .0
    latch: session allocation               5    0          0      1      0.0     .0
    latch: object queue header o            6    0          0      1      0.0     .0
    ASM file metadata operation            30    0          0      0      0.0     .0
    latch: In memory undo latch            15    0          0      0      0.0     .0
    latch: undo global data                 8    0          0      0      0.0     .0
    SQL*Net message from client     6,362,536    0    278,225     44     33.7
    jobq slave wait                     7,270  100      3,635    500      0.0
    SQL*Net more data from clien        7,976    0         15      2      0.0
    SQL*Net message to client       6,362,544    0          8      0     33.7
    Background Wait Events  DB/Inst: XXXXXs  Snaps: 5635-5636
    -> Only events with Total Wait Time (s) >= .001 are shown
    -> ordered by Total Wait Time desc, Waits desc (idle events last)
                                                                 Avg          %Total
                                              %Tim Total Wait   wait    Waits   Call
    Event                               Waits  out   Time (s)   (ms)     /txn   Time
    log file parallel write           299,537    0        164      1      1.6    9.7
    log file sequential read              698    0         16     24      0.0    1.0
    db file parallel write              9,556    0         13      1      0.1     .8
    os thread startup                     146    0         10     70      0.0     .6
    control file parallel write         2,037    0          2      1      0.0     .1
    Log archive I/O                        35    0          1     30      0.0     .1
    LGWR wait for redo copy             2,447    0          0      0      0.0     .0
    db file async I/O submit            9,556    0          0      0      0.1     .0
    db file sequential read               145    0          0      2      0.0     .0
    Disk file operations I/O              349    0          0      0      0.0     .0
    db file scattered read                 30    0          0      4      0.0     .0
    control file sequential read        5,837    0          0      0      0.0     .0
    ADR block file read                    19    0          0      4      0.0     .0
    ADR block file write                    5    0          0     15      0.0     .0
    direct path write                      14    0          0      2      0.0     .0
    direct path read                        3    0          0      7      0.0     .0
    latch: shared pool                      3    0          0      6      0.0     .0
    log file single write                  56    0          0      0      0.0     .0
    latch: redo allocation                 53    0          0      0      0.0     .0
    latch: active service list              1    0          0      3      0.0     .0
    latch free                             11    0          0      0      0.0     .0
    rdbms ipc message                 314,523    5     57,189    182      1.7
    Space Manager: slave idle wa        4,086   88     18,996   4649      0.0
    DIAG idle wait                      7,185  100      7,186   1000      0.0
    Streams AQ: waiting for time            2   50      4,909 ######      0.0
    Streams AQ: qmn slave idle w          129    0      3,612  28002      0.0
    Streams AQ: qmn coordinator           258   50      3,612  14001      0.0
    smon timer                             43    2      3,605  83839      0.0
    pmon timer                          1,199   99      3,596   2999      0.0
    SQL*Net message from client        17,019    0         31      2      0.1
    SQL*Net message to client          12,762    0          0      0      0.1
    class slave wait                       28    0          0      0      0.0
    thank you very much!

    Hi: just know it now: it is a large amount of 'concurrent transaction' designed in this "Volume Test" - to simulate large incoming transaction volme, so I guess wait in eq:TX - row is expected.
    The fact: (1) redo logs at uat server is known to not well-tune for configurations (2) volume test slow 5%, however data amount in its test is kept the same by each time import  production data, by the team. So why it slowed 5% this year?
    The wait histogram is pasted below, any one interest to take a look?  any ideas?
    Wait Event Histogram  DB/Inst: XXXX/XXXX  Snaps: 5635-5636
    -> Total Waits - units: K is 1000, M is 1000000, G is 1000000000
    -> % of Waits - column heading: <=1s is truly <1024ms, >1s is truly >=1024ms
    -> % of Waits - value: .0 indicates value was <.05%, null is truly 0
    -> Ordered by Event (idle events last)
                               Total ----------------- % of Waits ------------------
    Event                      Waits  <1ms  <2ms  <4ms  <8ms <16ms <32ms  <=1s   >1s
    ADR block file read          19   26.3   5.3  10.5  57.9
    ADR block file write          5                     40.0        60.0
    ADR file lock                 6  100.0
    ARCH wait for archivelog l   14  100.0
    ASM file metadata operatio   30  100.0
    CSS initialization           30              100.0
    Disk file operations I/O   9090   97.2   1.4    .6    .4    .2    .1    .1
    LGWR wait for redo copy    2447   98.5    .5    .4    .2    .2    .2    .1
    Log archive I/O              35   40.0         8.6  25.7   2.9        22.9
    SQL*Net break/reset to cli   34   85.3   8.8         5.9
    SQL*Net more data to clien   29K  99.9    .0    .0    .0          .0    .0
    buffer busy waits          1800   96.8    .7    .7    .6    .3    .4    .5
    control file parallel writ 2037   90.7   5.0   2.1    .8   1.0    .3    .1
    control file sequential re 7405  100.0                      .0
    cursor: pin S                10   10.0                    90.0
    db file async I/O submit   9556   99.9    .0                .0          .0
    db file parallel read         1  100.0
    db file parallel write     9556   62.0  32.4   1.7    .8   1.5   1.3    .1
    db file scattered read      345   72.8   3.8   2.3  11.6   9.0    .6
    db file sequential read    6199   97.2    .2    .3   1.6    .7    .0    .0
    direct path read           7699   99.1    .4    .2    .1    .1    .0
    direct path sync              8   25.0  37.5  12.5  25.0
    direct path write            13K  97.8    .9    .5    .4    .3    .1    .0
    enq: TX - row lock content 4480     .4    .7   1.3   3.0   6.8  12.3  75.4    .1
    latch free                  301   98.3    .3    .7    .7
    latch: In memory undo latc   15   93.3   6.7
    latch: active service list    1              100.0
    latch: cache buffers chain   55   94.5                     3.6   1.8
    latch: cache buffers lru c    9   88.9                    11.1
    latch: call allocation        6  100.0
    latch: checkpoint queue la    3  100.0
    latch: enqueue hash chains   45   97.8                     2.2
    latch: messages               4  100.0
    latch: object queue header    7   85.7        14.3
    latch: redo allocation      113   97.3               1.8    .9
    latch: row cache objects     19   89.5                           5.3   5.3
    latch: session allocation     5   80.0              20.0
    latch: shared pool          147   90.5   1.4   2.7   1.4    .7   1.4   2.0
    latch: undo global data       8  100.0
    library cache: mutex X       28   89.3         3.6         3.6         3.6
    log file parallel write     299K  95.6   2.6   1.0    .4    .3    .2    .0
    log file sequential read    698   29.5    .1               4.6  46.8  18.9
    log file single write        56  100.0
    log file switch (checkpoin    4               25.0  50.0  25.0
    log file switch (private s    8         12.5        37.5  50.0
    log file sync               284K  93.3   3.7   1.4    .7    .5    .3    .1
    os thread startup           146                                      100.0
    row cache lock               41   85.4   9.8               2.4         2.4
    DIAG idle wait             7184                                      100.0
    SQL*Net message from clien 6379K  86.6   5.1   2.9   1.3    .7    .3   2.8    .3
    SQL*Net message to client  6375K 100.0    .0    .0    .0    .0    .0    .0
    Wait Event Histogram  DB/Inst: XXXX/xxxx  Snaps: 5635-5636
    -> Total Waits - units: K is 1000, M is 1000000, G is 1000000000
    -> % of Waits - column heading: <=1s is truly <1024ms, >1s is truly >=1024ms
    -> % of Waits - value: .0 indicates value was <.05%, null is truly 0
    -> Ordered by Event (idle events last)
                               Total ----------------- % of Waits ------------------
    Event                      Waits  <1ms  <2ms  <4ms  <8ms <16ms <32ms  <=1s   >1s
    SQL*Net more data from cli 7976   99.7    .1    .1    .0                      .1
    Space Manager: slave idle  4086     .1    .2    .0    .0    .3         3.2  96.1
    Streams AQ: qmn coordinato  258   49.2                .8                    50.0
    Streams AQ: qmn slave idle  129                                            100.0
    Streams AQ: waiting for ti    2   50.0                                      50.0
    class slave wait             28   92.9   3.6   3.6
    jobq slave wait            7270     .0                               100.0
    pmon timer                 1199                                            100.0
    rdbms ipc message           314K  10.3   7.3  39.7  15.4  10.6   5.3   8.2   3.3
    smon timer                   43                                            100.0

  • Workflow error in fork step, process control, wait event

    I am using fork step in workflow which has 2 parallel branches. In 1st branch i have a user decision step followed by a task for posting PO document in case of approval. In the 2nd branch of fork step I have a wait step to wait for an event followed by the same task for posting document with a process control step after that in the end to cancel the workitem(workitem generated by user decision step in the 1st branch of fork). I created the event by using a custom BOR object.
    After the fork step is triggered, i have both a wait event running and workitem generated. When i raise the wait event from SWUE by entering the event, object key etc it works fine i.e., the workitem in the other branch is set of logically deleted and workflow ends.
    But if the wait event is triggered from the program i.e., using FM SWW_WI_CREATE_VIA_EVENT, both get an error message in workflow log(SWIA). The message is: Error when executing the binding between work item 000000XXXXXX and flow item 000000XXXXXX where workitem number is the workitem id of the posting document task and flow item id is the workflow parent id

    hi,
    message is self explanatory.
    Activate the event trace SWELS, then do the event with SWUE and within your program (als please use SAP_WAPI function modules).
    Now compare the 2 events in SWEL to see what the differences are .
    Kind regards, Rob Dielemans

  • Library Cache Pin Wait Event (within the context of APEX)

    Hello,
    Firstly -
    Oracle Version: 10.2.0.4.0
    Apex Version: 3.0.1.00.08
    Okay, my colleague (no really! This isn't one of those "Ahem ... A friend of mine has contracted something nasty +downstairs+..."-type questions) is having problems compiling a package (using TOAD incidentally, but it's the same in SQL Developer).
    I've searched the forum and the web for a bit of help on what's maybe happening here and it appears to be related to a concurrency conflict with the package definition - from what I can understand it's a case of the package is in use by another session, therefore another session cannot alter it at the same time (which makes sense)
    "What does this have to do with APEX?"... well, he is working on this package using the following methodology:
    1. Compile the package body/spec (as necessary - body more often obviously)
    2. run an apex page which uses the code in a process, which may or may not result in the error page being displayed
    3. Making changes to the package body/spec
    repeat steps 1-3 ad nauseum...
    He is the only user directly accessing the schema (and the only user accessing the page via APEX too, although I appreciate this isn't quite the same thing).
    I was wondering if, due to the architecture of APEX (the use of session pools etc), the state of a package might be being retained in some manner, thus resulting in this library cache pin wait event? If so, is there anything I can do to mitigate against this occurring?
    p.s. the only difference I can see between this particular package and any other package in the schema is that this one interacts with blobs (including making references to the wwv_flow_files view) - with blobs being passed as parameters between procedures (thus potentially creating temporary blobs which may or may not being closed).
    Any ideas?
    p.p.s. there are also no DBMS_SCHEDULER jobs or anything that might potentially be running the code incidentally...
    Edited by: Joel_C on 11-Nov-2011 11:58
    We got our DBAs to run a bit of code to identify the blocking session:
    select
             decode(lob.kglobtyp, 0, 'NEXT OBJECT', 1, 'INDEX', 2, 'TABLE', 3, 'CLUSTER',
                          4, 'VIEW', 5, 'SYNONYM', 6, 'SEQUENCE',
                          7, 'PROCEDURE', 8, 'FUNCTION', 9, 'PACKAGE',
                          11, 'PACKAGE BODY', 12, 'TRIGGER',
                          13, 'TYPE', 14, 'TYPE BODY',
                          19, 'TABLE PARTITION', 20, 'INDEX PARTITION', 21, 'LOB',
                          22, 'LIBRARY', 23, 'DIRECTORY', 24, 'QUEUE',
                          28, 'JAVA SOURCE', 29, 'JAVA CLASS', 30, 'JAVA RESOURCE',
                          32, 'INDEXTYPE', 33, 'OPERATOR',
                          34, 'TABLE SUBPARTITION', 35, 'INDEX SUBPARTITION',
                          40, 'LOB PARTITION', 41, 'LOB SUBPARTITION',
                          42, 'MATERIALIZED VIEW',
                          43, 'DIMENSION',
                          44, 'CONTEXT', 46, 'RULE SET', 47, 'RESOURCE PLAN',
                          48, 'CONSUMER GROUP',
                          51, 'SUBSCRIPTION', 52, 'LOCATION',
                          55, 'XML SCHEMA', 56, 'JAVA DATA',
                          57, 'SECURITY PROFILE', 59, 'RULE',
                          62, 'EVALUATION CONTEXT',
                         'UNDEFINED') object_type,
             lob.KGLNAOBJ object_name,
             pn.KGLPNMOD lock_mode_held,
             pn.KGLPNREQ lock_mode_requested,
             ses.sid,
             ses.serial#,
             ses.username
        FROM
           x$kglpn pn,
           v$session ses,
           x$kglob lob,
           v$session_wait vsw
      WHERE
       pn.KGLPNUSE = ses.saddr and
       pn.KGLPNHDL = lob.KGLHDADR
       and lob.kglhdadr = vsw.p1raw
       and vsw.event = 'library cache pin'
    order by lock_mode_held descresults as follows (I've changed some object names to protect the ignorant):
    OBJECT_TYP OBJECT_NAME                   LOCK_MODE_HELD LOCK_MODE_REQUESTED        SID    SERIAL# USERNAME
    PACKAGE    PKG_FOOBAR                             2                   0        356      21694 HTMLDB_PUBLIC_U
                                                                                                      SER
    PACKAGE    PKG_FOOBAR                             0                   3        463      22309 FOOHTMLDB_PUBLIC_USER is the apex user incidentally. The session is marked in the v$session table as "inactive", the last statement being
    Begin
       Dbms_session.reset_package;
    End;Edited by: Joel_C on 11-Nov-2011 14:39

    bump
    No-one?
    The problem seems to have 'resolved itself' over the weekend incidentally (although I don't believe anything truly resolves itself in this manner - something must have changed).

  • Problem identifying db object for "buffer busy waits" event.

    10.2.0.3
    AIX 64
    SELECT username, a.p1text, a.p1, a.p2text, a.p2, a.p3text, a.p3, event FROM v$session a WHERE
    a.status='ACTIVE'
    AND a.event = 'buffer busy waits'
    Query reports about 40 active sessions with this information:
    file# 3746
    block# 2
    class# 13
    select
    owner,
    segment_name,
    segment_type
    from
    dba_extents
    where
    file_id = 3746
    and
    2 between block_id and block_id + blocks -1;
    no rows returned
    SELECT MAX(a.file#) FROM sys.file$ a
    3535
    This was only a temporary situation when after couple of minutes(7) wait event "buffer busy waits" dissapeared completely.
    Any ideas?
    Thank you,
    Daniel.

    http://perfvision.com/papers/06_buffer_cache.ppt
    Slide 80-81 points at increasing the size of the initial and next extent for File Header Block buffer busy waits
    Side 85 points at high extent allocation for File Header Block buffer busy waits
    http://perfvision.com/ftp/hotsos/aas.ppt
    Side 55 points at extent allocation too small/too many extents being allocated for File Header Block buffer busy waits
    A couple hints from the documentation:
    http://download.oracle.com/docs/cd/B19306_01/server.102/b14211/instance_tune.htm
    "To determine the possible causes [of buffer busy waits], first query V$SESSION to identify the value of ROW_WAIT_OBJ# when the session waits for buffer busy waits."
    "To identify the object and object type contended for, query DBA_OBJECTS using the value for ROW_WAIT_OBJ# that is returned from V$SESSION."
    "V$SEGMENT_STATISTICS - This is a user-friendly view of statistic values. In addition to all the columns of V$SEGSTAT, it has information about such things as the segment owner and table space name. It makes the statistics easy to understand, but it is more costly."
    You may want to query DBA_TEMP_FILES for the specific FILE_ID identified by the V$SESSION. Taking a look at V$SEGMENT_STATISTICS might also be helpful.
    Are you using dictionary managed tablespaces, locally managed tablespaces with manual extent size management, ASSM with manual extent size management, or ASSM with automatic extent size management?
    Charles Hooper
    IT Manager/Oracle DBA
    K&M Machine-Fabricating, Inc.

  • Wait events - how to read it

    Hi frnds,
    As, I'm beginner to performance tuning I dont know
    What action do i need to take?
    I mean how to read the output which I given below.
    this is the output suffering buffer busy waits.
    Could anyone please tell me
    CLASS TOTAL_WAITS TOTAL_TIME
    data block 93303 58711
    unused 0 0
    system undo header 12 232
    undo header 7847 6636
    3rd level bmb 0 0
    save undo header 0 0
    bitmap index block 0 0
    file header block 0 0
    free list 0 0
    undo block 68 207
    segment header 422 399
    extent map 0 0
    2nd level bmb 0 0
    system undo block 0 0
    sort block 0 0
    save undo block 0 0
    1st level bmb 1 17
    bitmap block 0 0
    Thanks, Muhammed Thameem. S

    Hello,
    "Buffer busy waits" is contention for a buffer (representing a specific
    version of a database block) within the Buffer Cache. So, in essence
    it is block contention and thus it is most likely something to do with
    the design of the tables and indexes supporting the application. A
    built-in bottleneck. On indexes, it could be the age-old problem of
    insertions into an index on a column with a monotonically-ascending
    data value (i.e. timestamps or sequence numbers) which tends to cause
    contention on the highest leaf node of the index. On tables, it might
    have to do with many concurrent insertions into a table in a
    freelist-managed tablespace where the table has only one freelist. It
    could also be due to a home-grown implementation of sequence-number
    generators (i.e. small table with one row, one column in which contains
    the "last value" of a sequence, etc) which lots of people use to avoid
    not being "portable across databases" which they think means not using
    Oracle sequences (yadda yadda yadda).
    I'd look for any SQL statement in the "SQL sorted by Elapsed Time"
    section of the AWR report which exhibits high elapsed time but
    relatively low CPU time, indicating a lot of wait time. Of course,
    there are something like 800 possible wait events in current releases
    of Oracle, of which "buffer busy waits" is only one, so this is just
    inference and not a direct causal connection to your problem. But,
    once I find such statements I'd check to see if they are
    accessing/manipulating tables within the CUBS_DATA tablespace, and then
    use "select * from table(dbms_xplan.display_awr('sql-id'))" to
    get the execution plan(s), and then look for something ineffective
    within the execution plan. You might find the script "sqlhistory.sql" helpful
    here as well, to get a "historical perspective" on the execution of the
    SQL statements over time, in case the buffer busy waits peaked at some
    point in the past
    Please refer to:
    http://www.pubbs.net/201003/oracle/51925-understanding-awr-buffer-waits.html
    Also
    http://www.remote-dba.net/oracle_10g_tuning/t_buffer_busy_waits.htm
    kind regards
    Mohamed

  • Wait Time / Timeout Time in waiting events

    Hi,
    From manual, I came to know that some of waiting events having wait time or time out duration. I have not understood, what happens if it has reached timeout. What happens next ? ( Will you explain in more detail ). It saysl it will renew wait event. ( I have not understood what does mean "RENEW", Or what steps will be taken by Oracle, what happens if again reached timeout ) ?
    Thanks for clearing my doubt in detail.
    regards
    pjp

    Please read the FAQ and learn how to enclose your listings in tags so we can read them.
    I can't help you at this time because I can not read what you posted.                                                                                                                                                                                                                                                                                                                                               

  • Wait Events "log file parallel write" / "log file sync" during CREATE INDEX

    Hello guys,
    at my current project i am performing some performance tests for oracle data guard. The question is "How does a LGWR SYNC transfer influences the system performance?"
    To get some performance values, that i can compare i just built up a normal oracle database in the first step.
    Now i am performing different tests like creating "large" indexes, massive parallel inserts/commits, etc. to get the bench mark.
    My database is an oracle 10.2.0.4 with multiplexed redo log files on AIX.
    I am creating an index on a "normal" table .. i execute "dbms_workload_repository.create_snapshot()" before and after the CREATE INDEX to get an equivalent timeframe for the AWR report.
    After the index is built up (round about 9 GB) i perform an awrrpt.sql to get the AWR report.
    And now take a look at these values from the AWR
                                                                       Avg
                                                 %Time  Total Wait    wait     Waits
    Event                                 Waits  -outs    Time (s)    (ms)      /txn
    log file parallel write              10,019     .0         132      13      33.5
    log file sync                           293     .7           4      15       1.0
    ......How can this be possible?
    Regarding to the documentation
    -> log file sync: http://download.oracle.com/docs/cd/B19306_01/server.102/b14237/waitevents003.htm#sthref3120
    Wait Time: The wait time includes the writing of the log buffer and the post.-> log file parallel write: http://download.oracle.com/docs/cd/B19306_01/server.102/b14237/waitevents003.htm#sthref3104
    Wait Time: Time it takes for the I/Os to complete. Even though redo records are written in parallel, the parallel write is not complete until the last I/O is on disk.This was also my understanding .. the "log file sync" wait time should be higher than the "log file parallel write" wait time, because of it includes the I/O and the response time to the user session.
    I could accept it, if the values are close to each other (maybe round about 1 second in total) .. but the different between 132 seconds and 4 seconds is too noticeable.
    Is the behavior of the log file sync/write different when performing a DDL like CREATE INDEX (maybe async .. like you can influence it with the initialization parameter COMMIT_WRITE??)?
    Do you have any idea how these values come about?
    Any thoughts/ideas are welcome.
    Thanks and Regards

    Surachart Opun (HunterX) wrote:
    Thank you for Nice Idea.
    In this case, How can we reduce "log file parallel write" and "log file sync" waited time?
    CREATE INDEX with NOLOGGINGA NOLOGGING can help, can't it?Yes - if you create index nologging then you wouldn't be generating that 10GB of redo log, so the waits would disappear.
    Two points on nologging, though:
    <ul>
    it's "only" an index, so you could always rebuild it in the event of media corruption, but if you had lots of indexes created nologging this might cause an unreasonable delay before the system was usable again - so you should decide on a fallback option, such as taking a new backup of the tablespace as soon as all the nologging operatons had completed.
    If the database, or that tablespace, is in +"force logging"+ mode, the nologging will not work.
    </ul>
    Don't get too alarmed by the waits, though. My guess is that the +"log file sync"+ waits are mostly from other sessions, and since there aren't many of them the other sessions are probably not seeing a performance issue. The +"log file parallel write"+ waits are caused by your create index, but they are happeninng to lgwr in the background which is running concurrently with your session - so your session is not (directly) affected by them, so may not be seeing a performance issue.
    The other sessions are seeing relatively high sync times because their log file syncs have to wait for one of the large writes that you have triggered to complete, and then the logwriter includes their (little) writes with your next (large) write.
    There may be a performance impact, though, from the pure volume of I/O. Apart from the I/O to write the index you have LGWR writting (N copies) of the redo for the index and ARCH is reading and writing the completed log files caused by the index build. So the 9GB of index could easily be responsible for vastly more I/O than the initial 9GB.
    Regards
    Jonathan Lewis
    http://jonathanlewis.wordpress.com
    http://www.jlcomp.demon.co.uk
    To post code, statspack/AWR report, execution plans or trace files, start and end the section with the tag {noformat}{noformat} (lowercase, curly brackets, no spaces) so that the text appears in fixed format.
    "Science is more than a body of knowledge; it is a way of thinking"
    Carl Sagan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           

  • Wait events 'direct path write'  and 'direct path read'

    Hi,
    We have a query which is taking more that 2 min. It's a 9.2.0.7 database. We took the trace/tkprof of the query,and identified that there are so manay 'direct path write' and 'direct path read' wait events in the trace file.
    WAIT #3: nam='direct path write' ela= 5 p1=201 p2=70710 p3=15
    WAIT #3: nam='direct path read' ela= 170 p1=201 p2=71719 p3=15
    In the above, "p1=201" is a file_id, but we could not find any data file, temp file, control file with that id# 201.
    Can you please let us know what's "p1=201" here, how to identify the file which is causing the issue.
    Thanks
    Sravan

    What does:
    show parameter db_filesreturn? My guess, is that it returns 200.
    The direct file read and direct file write events are reads and writes to TEMP tablespace. In those wait events, the file# is reported as db_files+temp file id. So, 201 means temp file #1.
    Now, as to your actual performance problem.
    Without seeing the SQL and the corresponding execution plan, it's impossible to be sure. However, the most common causes of temp writes are sort operations and group by operations.
    If you decide to post your SQL and execution plan, please be sure to make it readable by formatting it. Information on how to do so can be found here.
    Hope that helps,
    -Mark
    Edited by: mbobak on May 1, 2011 1:50 AM

  • How to see the wait events info. after excute a select query

    Hi
    How to see the wait events info. after execute a select query. Are there any parameter to set for this option?
    And also wanna see the follwing info. in trace file. For this what are the parameters I have to set right?
    SELECT * FROM emp, dept
    WHERE emp.deptno = dept.deptno;
    call   count      cpu    elapsed     disk    query current    rows
    Parse      1     0.16      0.29         3       13       0       0
    Execute    1     0.00      0.00         0        0       0       0
    Fetch      1     0.03      0.26         2        2       4      14
    Misses in library cache during parse: 1
    Parsing user id: (8) SCOTT Regards
    Arpitha

    $ sqlplus scott/tiger
    SQL*Plus: Release 10.2.0.2.0 - Production on Wed Apr 20 15:29:33 2011
    Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.
    Connected to:
    Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
    With the Partitioning, OLAP and Data Mining options
    SQL> SHOW PARAMETER dump
    NAME                                 TYPE        VALUE
    background_core_dump                 string      partial
    background_dump_dest                 string      /user/oracle/app/oracle/admin/
                                                     orclsb/bdump
    core_dump_dest                       string      /user/oracle/app/oracle/admin/
                                                     orclsb/cdump
    max_dump_file_size                   string      UNLIMITED
    shadow_core_dump                     string      partial
    user_dump_dest                       string      /user/oracle/app/oracle/admin/
                                                     orclsb/udump
    SQL> ALTER SESSION SET EVENTS='10046 trace name context forever, level 12';
    Session altered.
    SQL> SELECT * FROM emp WHERE deptno=20;
         EMPNO ENAME      JOB              MGR HIREDATE         SAL       COMM
        DEPTNO
          7369 SMITH      CLERK           7902 17-DEC-80        800
            20
          7566 JONES      MANAGER         7839 02-APR-81       2975
            20
          7788 SCOTT      ANALYST         7566 19-APR-87       3000
            20
         EMPNO ENAME      JOB              MGR HIREDATE         SAL       COMM
        DEPTNO
          7876 ADAMS      CLERK           7788 23-MAY-87       1100
            20
          7902 FORD       ANALYST         7566 03-DEC-81       3000
            20Now
    $ pwd
    /user/oracle/app/oracle/admin/orclsb/udump
    $ ls -ltr|tail -5
    -rw-r-----   1 oracle   oinstall     622 Apr 20 11:35 orclsb_ora_949.trc
    -rw-r-----   1 oracle   oinstall     651 Apr 20 11:35 orclsb_ora_976.trc
    -rw-r-----   1 oracle   oinstall    1982 Apr 20 11:35 orclsb_ora_977.trc
    -rw-r-----   1 oracle   oinstall    1443 Apr 20 15:29 orclsb_ora_1251.trc
    -rw-r-----   1 oracle   oinstall  279719 Apr 20 15:30 orclsb_ora_1255.trc
    $ tkprof  orclsb_ora_1255.trc  orclsb_ora_1255.txt
    TKPROF: Release 10.2.0.2.0 - Production on Wed Apr 20 15:32:17 2011
    Copyright (c) 1982, 2005, Oracle.  All rights reserved.
    $ ls -ltr|tail -5
    -rw-r-----   1 oracle   oinstall     651 Apr 20 11:35 orclsb_ora_976.trc
    -rw-r-----   1 oracle   oinstall    1982 Apr 20 11:35 orclsb_ora_977.trc
    -rw-r-----   1 oracle   oinstall    1443 Apr 20 15:29 orclsb_ora_1251.trc
    -rw-r-----   1 oracle   oinstall  279719 Apr 20 15:30 orclsb_ora_1255.trc
    -rw-r--r--   1 oracle   oinstall   26872 Apr 20 15:32 orclsb_ora_1255.txtThis orclsb_ora_1255.txt contains the required information.

  • Hash join ending up in huge wait events

    Hi,
    We are experiencing huge wait events ( direct path read temp , direct path write temp) on our Materialized View refresh in 10.2.0.4 version of oracle 10g in linux rhel5 environment while monitoring the refresh session from db console. While checking the explain plan of the mv query there is a huge hash_join (due to self join nature of the query) is shown. As advised in some dba forums, i have increased my pga_aggregate_target to a value of 4 gb from 1800 mb. The PGA_HIT % is raised to 60% from 58% ( just 2% improvement). But still my direct path read temp and direct path write temp wait event have not reduced and a huge temp space is taken for hash join.
    Since we have some usage limit set by some hidden parameters for a each session on pga_aggregate_target, increase the size did not helped me much. The mv refresh is taking more than 5 hours ( sometimes it exceeds 5 hrs) to completes it refresh where as the same query in window (production) is completed less than two hours. Before a month, the refresh time in both environment was nearly close. But now it has changed and not able to figure it out.
    STATISTICS have been collected regularly using dbms_gather_stats in both environment. Both mv refresh are scheduled to run using dbms_scheduler (Manual refresh). SGA_TARGET and other memory parameters are almost same.
    Environment : Dataware house
    O/s : RHEL 5
    Oracle version : 10.2.0.4
    Work_policy=auto
    Is there any possibility to reduce this wait event and there by reducing the elapsed time? I am also interested to know changing the plan to use other sort will help? I don't know whether the details are sufficient to analyze this issue. If you need more details on this just let me know.
    I really appreciate your help and thanks in advance to all.

    Thans for your comments. Here is the code, explan plan and autotrace trace stat output.
    SELECT lasg.employee_number "EMPLOYEE_NUM",
    lasg.full_name "FULL_NAME",
    lasg.person_id "PERSON_ID",
    SUBSTR (lasg.organization, 1, 4) "DEPT",
    casg.assign_start_date "EFFECTIVE_START_DATE",
    casg.assign_end_date "EFFECTIVE_END_DATE",
    hasg.organization "PRIOR_ORG",
    casg.organization organization,
    hasg.supervisor "PRIOR_SUPERVISOR",
    casg.supervisor "SUPERVISOR_NAME",
    hasg.location "PRIOR_LOCATION",
    casg.location location,
    hasg.job_title "PRIOR_TITLE",
    casg.job_title job_name,
    CASE
    WHEN hasg.organization = casg.organization THEN 'No Change'
    ELSE 'Change'
    END
    org_change,
    CASE
    WHEN hasg.location = casg.location THEN 'No Change'
    ELSE 'Change'
    END
    loc_change,
    CASE
    WHEN hasg.supervisor = casg.supervisor THEN 'No Change'
    ELSE 'Change'
    END
    sup_change,
    CASE
    WHEN hasg.job_title = casg.job_title THEN 'No Change'
    ELSE 'Change'
    END
    job_change
    FROM panad.data_employ_details lasg,
    panad.data_employ_details casg,
    panad.data_employ_details hasg
    WHERE lasg.person_id = casg.person_id(+)
    AND lasg.assign_end_date = (SELECT MAX (lasg2.assign_end_date)
    FROM panad.data_employ_details lasg2
    WHERE lasg.person_id = lasg2.person_id)
    AND casg.person_id = hasg.person_id(+)
    AND hasg.assign_start_date =
    (SELECT MAX (hasg2.assign_start_date)
    FROM panad.data_employ_details hasg2
    WHERE hasg2.person_id = lasg.person_id
    AND hasg2.assign_end_date < casg.assign_start_date)
    | Id  | Operation                 | Name                       | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     |
    |   0 | SELECT STATEMENT          |                            |     1 |   303 |       | 10261  (91)| 00:02:04 |
    |*  1 |  FILTER                   |                            |       |       |       |            |          |
    |*  2 |   HASH JOIN               |                            |     1 |   303 |       | 10179  (91)| 00:02:03 |
    |*  3 |    HASH JOIN              |                            |     5 |  1060 |       | 10095  (92)| 00:02:02 |
    |*  4 |     HASH JOIN             |                            |  6786 |   960K|       | 10011  (93)| 00:02:01 |
    |   5 |      VIEW                 | VW_SQ_1                    |  6786 |   225K|       |  9927  (94)| 00:02:00 |
    |   6 |       HASH GROUP BY       |                            |  6786 |   384K|       |  9927  (94)| 00:02:00 |
    |   7 |        MERGE JOIN         |                            |    50M|  2820M|       |  1427  (53)| 00:00:18 |
    |   8 |         SORT JOIN         |                            | 31937 |   998K|  2776K|   367   (2)| 00:00:05 |
    |   9 |          TABLE ACCESS FULL| DATA_EMPLOY_DETAILS            | 31937 |   998K|       |    82   (2)| 00:00:01 |
    |* 10 |         SORT JOIN         |                            | 31937 |   810K|  2520K|   324   (2)| 00:00:04 |
    |  11 |          TABLE ACCESS FULL| DATA_EMPLOY_DETAILS        | 31937 |   810K|       |    82   (2)| 00:00:01 |
    |  12 |      TABLE ACCESS FULL    | DATA_EMPLOY_DETAILS        | 31937 |  3461K|       |    83   (3)| 00:00:01 |
    |  13 |     TABLE ACCESS FULL     | DATA_EMPLOY_DETAILS        | 31937 |  2089K|       |    83   (3)| 00:00:01 |
    |  14 |    TABLE ACCESS FULL      | DATA_EMPLOY_DETAILS        | 31937 |  2838K|       |    83   (3)| 00:00:01 |
    |  15 |   SORT AGGREGATE          |                            |     1 |    13 |       |            |          |
    |* 16 |    TABLE ACCESS FULL      | DATA_EMPLOY_DETAILS        |     5 |    65 |       |    82   (2)| 00:00:01 |
    Predicate Information (identified by operation id):
       1 - filter("LASG"."ASSIGN_END_DATE"= (SELECT MAX("LASG2"."ASSIGN_END_DATE") FROM
                  "PANAD"."DATA_EMPLOY_DETAILS" "LASG2" WHERE "LASG2"."PERSON_ID"=:B1))
       2 - access("CASG"."PERSON_ID"="HASG"."PERSON_ID" AND "HASG"."ASSIGN_START_DATE"="VW_COL_1")
       3 - access("LASG"."PERSON_ID"="CASG"."PERSON_ID" AND "PERSON_ID"="LASG"."PERSON_ID")
       4 - access("ROWID"=ROWID)
      10 - access(INTERNAL_FUNCTION("HASG2"."ASSIGN_END_DATE")<INTERNAL_FUNCTION("CASG"."ASSIGN_START_DATE")
           filter(INTERNAL_FUNCTION("HASG2"."ASSIGN_END_DATE")<INTERNAL_FUNCTION("CASG"."ASSIGN_START_DATE")
      16 - filter("LASG2"."PERSON_ID"=:B1)
    37 rows selected.
    - autot trace stat output -
    5070 rows selected.
    Statistics
          35203  recursive calls
              0  db block gets
        3675913  consistent gets
        4269882  physical reads
              0  redo size
        1046781  bytes sent via SQL*Net to client
           4107  bytes received via SQL*Net from client
            339  SQL*Net roundtrips to/from client
             69  sorts (memory)
              0  sorts (disk)
           5070  rows processed   I have tried running this query with paralell but not helped.
    I have read the links provided by both of you. Dictionary and fixed table stats are collected as a routine.
    From the link given byTaral, Greg Rahn has suggested that it is a bug as below.
    Its bug 9041800 and there is a 10.2.0.4 backport available as of 01/29/10.How can i get this bug fixed since there is no explanation of what need to be done? Do i need to contact oracle support for the 10.2.0.4 backport for RHEL5?
    Thanks in advance
    Edited by: Karthikambalav on Mar 9, 2010 2:43 AM

  • Tuning row lock contention wait events

    Hello everyone,
    Working on 10g/windows
    Top 5 events
    EVENT TOTAL_WAITS TIME_WAITED AVG_MS PERCENT
    CPU 9462339 48
    enq: TX - row lock contention 12531 3660728 2921.34 18
    control file parallel write 1300731 3088079 23.74 16
    log file parallel write 1510503 1264080 8.37 6
    log file sync 1072553 968007 9.03 5
    Distribution of row lock wait during the last 4 days in the database server
    END_INTERVAL_TIME TOTAL_WAITS TIME_WAITED_MICRO AVG_WAIT_MS
    2008-04-01 16:00:58 909 2721008230 2993.41
    2008-04-01 15:00:27 50 149941140 2998.82
    2008-03-31 12:00:42 193 575595397 2982.36
    2008-03-29 23:00:13 172 513058700 2982.9
    2008-03-29 22:00:37 164 483940046 2950.85
    2008-03-27 22:00:35 565 1667120838 2950.66
    2008-03-26 18:00:59 348 1042918982 2996.89
    My analysis:
    It's obvious that the row lock contention wait time is huge, and this direct me to find out SQL stmt, causing this.
    all the SQL statement was SELECT ....... FOR UPDATE stmt.
    I was also able to find out locked tables.
    My tuning idea:
    1. I'm thinking to reorganize hot tables as well as their indexes, but by instinct it seems to not give so much value to avoid the huge row lock wait time.
    2. I'm also seeing if I can reduce the number of rows per block, by increasing PCTFREE and diminishing PCTUSED, so the contention will spread over many blocks instead of one heavy block.
    Question
    As SQL stmt related to those locked tables are select ... for update, how could I tune this kind of stmt?
    Does someone have other idea to come up with this row lock contention?
    Tanks for your effort and help

    Taking another look at your suggested function based index, it depends on the data type of the DEV.POS_FOLIO_ID.POS_FOLIO_ID column. If the column is defined as a number, and it is a primary key, there will already be a usable index on that column.
    Yesterday, I wrote this: "Once I understood why or how the sessions were trying to insert duplicate primary key values, I would try to determine why the average number of seconds for the wait event is almost 3 seconds (maybe a timeout)."
    After fixing the formatting of the top 5 wait events (total duration unknown):
    EVENT                        TOTAL_WAITS  TIME_WAITED   AVG_MS PERCENT
    CPU                                         94,623.39             48
    enq: TX - row lock contention     12,531    36,607.28  2921.34    18
    control file parallel write    1,300,731    30,880.79    23.74    16
    log file parallel write        1,510,503    12,640.80     8.37     6
    log file sync                  1,072,553     9,680.07     9.03     512,531 * 3 second time out = 37,593 seconds = 10.44 hours.
    What if the reason for the 3 second average wait time is due to a timeout. I performed a little experiment... I changed a row in a test table and then made a pot of coffee.
    In session 1:
    CREATE TABLE T1 (
      C1 NUMBER(10),
      C2 NUMBER(10),
      PRIMARY KEY (C1));
    INSERT INTO T1
    SELECT
      ROWNUM,
      ROWNUM*10
    FROM
      DUAL
    CONNECT BY
      LEVEL<=1000000;
    COMMIT;I now have a test table with 1,000,000 rows. I start monitoring the changes in the wait events roughly every 60 seconds, and V$SESSION_WAIT and V$LOCK roughly 4 times per second.
    Back in session 1:
    UPDATE
      T1
    SET
      C1=-C1
    WHERE
      C1<=100;I have now modified the first 100 rows that were inserted into the table, time to make the pot of coffee.
    In session 2, I try to insert a row with a primary key value of -10:
    INSERT INTO T1 VALUES (
      -10,
      10);Session 2 hangs.
    If I take the third 60 second snap of the system wide wait events as the zero point, and the 11th snap as the end point. There were 149 waits on ENQ: TX - ROW LOCK CONTENTION, 148 time outs, 446.62 seconds of total time in the wait event, with an average wait time of 2.997450 seconds.
    Rolling down to the session level wait events, SID 208 (my session 2) had 149 waits on ENQ: TX - ROW LOCK CONTENTION, for a total time of 446.61 seconds with an average wait time of 2.997383 seconds. All of the 149 waits and the wait time was in this one session that was locked up for the full duration of this time period because session 1 was making a pot of coffee.
    Rolling down to V$SESSION_WAIT (sampled roughly 4 times per second): At the start of the third time interval, SID 208 has been in the ENQ: TX - ROW LOCK CONTENTION wait event for 39 seconds and is actively waiting trying to execute SQL with a hash value of 1001532423, the wait object is -1, wait file is 0, wait block is 0, wait row is 0, P1 is 1415053316, P2 is 196646, P3 is 4754.
    At the end of the 11th time interval: , SID 208 has been in the ENQ: TX - ROW LOCK CONTENTION wait event for 483 seconds and is actively waiting trying to execute SQL with a hash value of 1001532423, the wait object is -1, wait file is 0, wait block is 0, wait row is 0, P1 is 1415053316, P2 is 196646, P3 is 4754.
    Rolling down to V$LOCK (sampled roughly 4 times per second): I see that SID 214 (session 1) is blocking SID 208 (session 2). SID 214 has a TX lock in mode 6 with ID1 of 196646 and ID2 of 4754. SID 208 is requesting a TX lock in mode 4 with ID1 of 196646 and ID2 of 4754.
    So, it seems that I need a faster coffee pot rather than an additional index on my table. It could be that the above process would have found that the application associated with SID 214 was abandoned or crashed and for some reason the lock was not released for a long period of time, a little less than 10.44 hours in your case.
    Charles Hooper
    IT Manager/Oracle DBA
    K&M Machine-Fabricating, Inc.

Maybe you are looking for