Redo waits + I/O issue ?

Question

Hello All,
- Oracle 10.2.0.4.0 on HP-Unix
We are having performance issues during night batch cycles for past few days and we are observing following waits in top 5 wait events for past few days:
log file sync
log file parallel wait
Sometimes "log file parallel wait" has more waits as compared to "log file sync".
Question:
We want to check the performance of I/O system to confirm if this waits are due to slow I/O of redo files.
What is the command to get the "iostat" report and what columns should be watched to check the performance problems ?
Thanks in advance :)

Answer 1

+>>How often are your logs switching? How big are your logs? How big is the redo log buffer?+
- During the problem, the log switch occurs every 30-40 mins
- Redo Log Size = 300M (10 groups with 2 members each)
- Redo Log buffer = 13.9M
+>>Also let us know what kind of hardware you have - wouldn't be a RAID-5, would it?+
- I don't think so, but I am not sure. I will confirm this.
+>>What is your batch actually doing? Is it taking longer than before? How much longer?+
- Batch is doing Inserts, Updates selecting from many tables
- Yes, it is taking longer than before
- 2 hours more than what it used to be
+>>Could you share with us the top five from statspack?+
DB CPU
db file sequential read
log file parallel write
log file sync     
db file scattered read
+>>log file sync could be an indication that the batch is committing too often - has something changed in the code?+
- There is no change in the code anywhere (this is prod system)
+>>Are there messages in the alert log about slow log writes?+
- There are no errors, warnings or any kind of messages in alert log file.
I did some more analysis on the issue and based on following findings, I think that the I/O system could be the source of this issue. Please correct me if I am wrong:
1. The LGWR is giving out following messages in trace file (bdump):
Warning: log write time 1760ms, size 53KB (taking 1760ms to wring 53KB of data???)
There are lots and lots of messages of this kind all through the day, but increased frequency during the problem time at night.
Metalink:
"LGWR Is Generating Trace file with "Warning: Log Write Time 540ms, Size 5444kb" In 10.2.0.4 Database" Doc Id- 601316.1
2. During the problem times, the CPU consumption is max 40-50% (there are 4 CPUs), but almost all the sessions wait on event: "log file sync" or "log buffer space"
I checked the LGWR during that time and I found it to be waiting on the event: "log file parallel write".
This event has the highest wait time for the SID of LGWR process.
3. v$event_histogram - log file parallel write
EVENT                      WAIT_TIME_MILLI                WAIT_COUNT
log file parallel write            1                      44,460
log file parallel write            2                      59,134
log file parallel write            4                      43,951
log file parallel write            8                      22,438
log file parallel write            16                    17,838
log file parallel write            32                    12,336
log file parallel write            64                    8,545
log file parallel write            128                    5,435
log file parallel write            256                    2,308
log file parallel write            512                    1,392
log file parallel write            1,024                  1,497
log file parallel write            2,048                  476
log file parallel write            4,096                  271
log file parallel write            8,192                  120
log file parallel write            16,384                37
log file parallel write            32,768                34. vmstat
  procs           memory                 faults       cpu
r  b      avm    free   re   at    in     sy    cs  us sy id
3  1   782748  360923  483  175  2858  60521  2053  35  7 58
4  1   760830  373770  195  106  6488  12974  5070  53  5 42
1  2   774551  376473  338   68  2653  13045  1409  29  8 63
3  1   715546  376473  100   37  4025   4483  3072  20  5 76
1  5   749468  374585  165   63  3985  17626  4264  19  8 73
4  2   659704  392529  195   95  4075  18587  3917  41  7 52
3  4   660984  382799  290  158  4460  51595  4209  50 10 40
4  1   633560  383127  183   25  3184  16714  2556  42  7 50
3  2   679769  374495  189   40  4737  14745  3056  29  9 62
2  3   667695  373599  326  102 14890  25102  6666  11  9 79
4  2   730630  358697  414  101 17810  60920  7970  36 14 51I am looking at column "b" under procs, which shows blocked processes. There is lot of free memory and more than enough Idle CPU, but still there are >1 procs in blocked state.
5. CPU Consumption (sar -u 30 20)
%usr    %sys    %wio   %idle
  87       7       2       4
  74       5      14       7
  70       8      20       3
  66       5      16      12
  65       7      19       9
  60       8      27       5
  61      12      24       3
  68       6      24       2
  66      13      20       1
  86       6       7       0
  72       7      14       7
  78       6      10       6
  83       5      10       1
  92       4       3       0
  87       5       8       0
  73       6      18       3
  67       7      19       7
  68       6      19       7
  76      13       9       1
  76       6      16       2
  74       7      15       4   AVERAGE%wio is not very very high, but these numbers are still higher I think...
6. Disk activity (sar -d 30 20)
           device   %busy   avque   r+w/s  blks/s  avwait  avserv
Average    c9t6d0   13.50    1.74      33     289    4.55    8.49
Average    c0t6d0    9.22    2.44      18     175    6.60   10.80I am still to confirm what files are on these devices, but these are the ones that have highest number of waits and low service times
Conclusion:
For me, so far, these facts hints at the possible I/O issue.
Does anyone has any other interpretations ?
Thanks a lot :)