PerfMon reporting dramatic disk access time increase on Oracle startup

Question

Hi,
My oracle 10g (10.2.0.4) database is hosted on a windows 2003 server.
The datafiles are stored on a RAID1 disk array, on a dedicated partition : currently 30 gigs free out of 180, wich should not be a concern unless i'm wrong, because the datafiles were created as 10 Go files with no autogrowth. I add a new datafile whenever i need more room for my tables (alerts when 80% used).
Since 2 days i experience a dramatic performance loss :
The EM console reports nothing special (no alarms related to storage) apart from the need for more paginated memory.
I issue a reorg when the segmentation advisor suggests it.
My optimizer statistics are calculated by the default scheduled job.
The weird thing I noticed is that as soon as I start the database, there's a huge increase in disk activity even though no query at all is submitted to the database.
PerfMon reports Current Disk Queue Length > 1000 and disk access time > 3000 ms
CPU is 2% activity on the 4-cpus server.
I have plenty of spare memory (currently 3 Go used out of 16).
This is only a dev server for ETL processes, it has very few concurrent connections.
Any suggestions welcome.
AWR report is available here
http://min.us/mqnXQhd5Z
Edited by: user10799939 on 22 mars 2012 09:30

Answer 1

Cache Sizes
~~~~~~~~~~~                       Begin        End
               Buffer Cache:     1,296M     1,296M  Std Block Size:         8K
           Shared Pool Size:       160M       160M      Log Buffer:    14,364K
Load Profile
~~~~~~~~~~~~                            Per Second       Per Transaction
                  Redo size:            460,955.72 ;         2,477,358.63
              Logical reads:              3,392.16 ;            18,230.80
              Block changes:              6,451.93 ;            34,675.22
             Physical reads:                  2.92 ;                15.67
            Physical writes:                394.52 ;             2,120.28
                 User calls:                  1.69 ;                 9.08
                     Parses:                  3.31 ;                17.81
                Hard parses:                  0.17 ;                 0.90
                      Sorts:                  1.32 ;                 7.09
                     Logons:                  0.06 ;                 0.31
                   Executes:                  7.01 ;                37.68
               Transactions:                  0.19
  % Blocks changed per Read:  190.20 ;   Recursive Call %:    96.23
Rollback per transaction %:    0.30 ;      Rows per Sort:    14.41
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            Buffer Nowait %:   99.98 ;      Redo NoWait %:   99.86
            Buffer  Hit   %:   99.92 ;   In-memory Sort %:  100.00
            Library Hit   %:   96.30 ;       Soft Parse %:   94.96
         Execute to Parse %:   52.74 ;        Latch Hit %:   99.07
Parse CPU to Parse Elapsd %:    0.35 ;    % Non-Parse CPU:   99.30
Shared Pool Statistics        Begin    End
             Memory Usage %:   75.48 ;  75.51
    % SQL with executions>1:   79.92 ;  85.03
  % Memory for SQL w/exec>1:   77.07 ;  70.09
Top 5 Timed Events                                         Avg %Total
~~~~~~~~~~~~~~~~~~                                        wait   Call
Event                                 Waits    Time (s)   (ms)   Time Wait Class
db file sequential read               9,052      17,688   1954   51.3 ;  User I/O
log file switch (checkpoint in        5,303       4,649    877   13.5 Configurat
log file switch completion            4,245       4,023    948   11.7 Configurat
wait for a undo record               32,393       3,531    109   10.3 ;     Other
db file parallel write               18,771       3,437    183   10.0 System I/O Havent seen this much wait on average. For example 877ms for "log file switch" is over threshold. And other wait events too..
Time Model Statistics                DB/Inst: MDMPRJ/MDMPRJ  Snaps: 2840-2841
-> Total time in database user-calls (DB Time): 34446.5s
-> Statistics including the word "background" measure background process
   time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name                                       Time (s) % of DB Time
sql execute elapsed time                              4,008.5 ;        11.6
parse time elapsed                                      352.9 ;         1.0
hard parse elapsed time                                 352.7 ;         1.0
PL/SQL compilation elapsed time                         120.1 ;          .3
DB CPU                                                   61.8 ;          .2
failed parse elapsed time                                21.3 ;          .1
PL/SQL execution elapsed time                             8.0 ;          .0
connection management call elapsed time                   0.0 ;          .0
hard parse (sharing criteria) elapsed time                0.0 ;          .0
repeated bind elapsed time                                0.0 ;          .0
hard parse (bind mismatch) elapsed time                   0.0 ;          .0
DB time                                              34,446.5 ;         N/A
background elapsed time                              14,889.7 ;         N/A
background cpu time                                      39.0 ;         N/A
Wait Class                            DB/Inst: MDMPRJ/MDMPRJ  Snaps: 2840-2841
-> s  - second
-> cs - centisecond -     100th of a second
-> ms - millisecond -    1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc
                                                                  Avg
                                       %Time       Total Wait    wait     Waits
Wait Class                      Waits  -outs         Time (s)    (ms)      /txn
User I/O                       10,515     .1           17,785    1691      15.8
Configuration                  10,186   79.5 ;           8,865     870      15.3
System I/O                     27,619     .0            8,774     318      41.6
Other                          57,768   98.3 ;           6,915     120      87.0
Commit                          2,634   88.6 ;           2,481     942       4.0
Concurrency                     2,847   75.4 ;           2,240     787       4.3
Application                       219    2.3 ;              23     105       0.3
Network                         4,790     .0                0       0       7.2
          ------------------------------------------------------------- again seen, there is very high wait on User IO
Wait Events                          DB/Inst: MDMPRJ/MDMPRJ  Snaps: 2840-2841
-> s  - second
-> cs - centisecond -     100th of a second
-> ms - millisecond -    1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
                                                                   Avg
                                             %Time  Total Wait    wait     Waits
Event                                 Waits  -outs    Time (s)    (ms)      /txn
db file sequential read               9,052     .0      17,688    1954      13.6
log file switch (checkpoint           5,303   78.0 ;      4,649     877       8.0
log file switch completion            4,245   89.2 ;      4,023     948       6.4
wait for a undo record               32,393   99.8 ;      3,531     109      48.8
db file parallel write               18,771     .0       3,437     183      28.3
wait for stopper event to be         24,203   99.8 ;      2,634     109      36.5
log file sync                         2,634   88.6 ;      2,481     942       4.0
control file sequential read          7,356     .0       2,431     330      11.1
buffer busy waits                     2,513   83.1 ;      2,173     865       3.8
log file parallel write                 520     .0       1,566    3012       0.8
control file parallel write             840     .0       1,334    1588       1.3
rdbms ipc reply                         172   91.3 ;        330    1916       0.3
enq: CF - contention                    309   23.0 ;        268     867       0.5
log buffer space                        638   28.5 ;        192     301       1.0
enq: PS - contention                     52   23.1 ;         71    1362       0.1
db file scattered read                  113     .0          67     590       0.2
os thread startup                        76   77.6 ;         63     834       0.1
reliable message                         57   78.9 ;         50     878       0.1
enq: RO - fast object reuse              22   22.7 ;         23    1038       0.0
latch free                              537     .0          16      30       0.8
Streams AQ: qmn coordinator               3  100.0 ;         15    5005       0.0 Overstepping
Background Wait Events               DB/Inst: MDMPRJ/MDMPRJ  Snaps: 2840-2841
-> ordered by wait time desc, waits desc (idle events last)
                                                                   Avg
                                             %Time  Total Wait    wait     Waits
Event                                 Waits  -outs    Time (s)    (ms)      /txn
db file parallel write               18,772     .0       3,437     183      28.3
events in waitclass Other            24,367   99.5 ;      3,010     124      36.7
control file sequential read          6,654     .0       2,333     351      10.0
log file parallel write                 520     .0       1,566    3012       0.8
control file parallel write             840     .0       1,334    1588       1.3
buffer busy waits                       899   94.2 ;        884     984       1.4
log file switch (checkpoint             206   82.0 ;        185     898       0.3
os thread startup                        76   77.6 ;         63     834       0.1
log file switch completion               46   93.5 ;         45     982       0.1
log buffer space                        158   31.0 ;         12      77       0.2
db file sequential read                  62     .0           7     111       0.1
db file scattered read                   20     .0           6     318       0.0
direct path read                        660     .0           5       7       1.0
log file sequential read                 66     .0           4      65       0.1
log file single write                    66     .0           1      16       0.1
enq: RO - fast object reuse               2     .0           0      38       0.0
latch: cache buffers chains               3     .0           0       6       0.0
direct path write                       660     .0          -5      -8       1.0
rdbms ipc message                     9,052   87.5 ;     21,399    2364      13.6
pmon timer                            1,318   90.4 ;      3,562    2703       2.0
Streams AQ: qmn coordinator             633   97.6 ;      3,546    5602       1.0
Streams AQ: waiting for time             77   61.0 ;      3,449   44795       0.1
PX Deq: Join ACK                         21     .0           0       0       0.0 Again overshooting
Tablespace IO Stats                  DB/Inst: MDMPRJ/MDMPRJ  Snaps: 2840-2841
-> ordered by IOs (Reads + Writes) desc
Tablespace
                 Av      Av     Av                       Av     Buffer Av Buf
         Reads Reads/s Rd(ms) Blks/Rd       Writes Writes/s      Waits Wt(ms)
UNDOTBS1
           914       0 ######     1.0 ;   1,368,515      383      2,534  863.2
MDMREF_INDICES
         6,918       2 ######     1.0 ;      11,086        3          0    0.0
SYSAUX
           626       0 ######     1.1 ;       1,804        1          0    0.0
SYSTEM
           850       0 ######     1.7 ;         296        0          0    0.0
MDMREF_DATA
           293       0  712.3 ;    1.0 ;         274        0          0    0.0
MDMPRJ_ODS
           198       0   72.1 ;    1.0 ;         198        0          0    0.0
FEU_VERT
            33       0   61.5 ;    1.0 ;          33        0          0    0.0
USERS
            33       0   31.5 ;    1.0 ;          33        0          0    0.0
          ------------------------------------------------------------- Now have a serious look at it. Av Rd(ms). Now for some tablespace value cannot event fit in window thats why its showing ##
According to oracle recommendation Av Rd(ms) shouldn't be greater then 20, if its goes over 20 then its considered to be an issue with IO subsystem. But as its seen that in your case its overshooting.
Now the question from my side
Have done any configuration changes?
I would suggest you to revert these changes asap and contact storage admin guys...
Hope this helps