User I/O and db file parallel read is high in AWR report

Question

Hi,
We have one performance issue during a job execution.
From the awr report we have identified one query with a table having millions of records causing problems and then we had also fine tuned that query by changing it's code and by using the optmizer hints. It is being executed in plsql batches. After fine tuning, On the first batch execution(first 5000 records) the query is taking only 5 mins, but on the consecutive batches it is consuming more time( more than 30 mins).
From the awr report I got the statistics as
Release : 11.2.0.2.0
Instance Efficiency Percentages (Target 100%)
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 85.44 In-memory Sort %: 99.98
Library Hit %: 99.76 Soft Parse %: 99.15
Execute to Parse %: 88.91 Latch Hit %: 100.00
Parse CPU to Parse Elapsd %: 87.32 % Non-Parse CPU: 98.65
The buffer hit % is good. On each batch execution it is taking different set of records.
Top 5 Timed Foreground Events
Event Waits Time(s) Avg wait (ms) % DB time Wait Class
db file parallel read 120,485 42,540 353 89.60 User I/O
DB CPU 3,794 7.99
db file sequential read 145,074 606 4 1.28 User I/O
db file scattered read 70,030 556 8 1.17 User I/O
direct path write temp 12,423 21 2 0.04 User I/O
So the I/O is our main concern since that query contains one table with millions of records.
Host CPU (CPUs: 24 Cores: 24 Sockets: 4)
Load Average Begin Load Average End %User %System %WIO %Idle
1.40 1.45 0.6 0.3 3.7 99.0
Load is also normal.
From the Time model statistics , sql execute elapsed time is 98.27% of db time and only 7.99% is that of DB CPU.
Memory Statistics
Begin End
Host Mem (MB): 64,318.0 64,318.0
SGA use (MB): 30,720.0 30,720.0
PGA use (MB): 488.2 497.1
% Host Mem used for SGA+PGA: 48.52 48.54
Both the size of sga_max_size and sga_target are 32,212,254,720(32gb) bytes and that of
pga_aggregate target is 629,145,600(600mb)
from this it is evident that the memory is still available(so increase in memory size is not an option).
The sql statistics for that query shows like that
Elapsed Time (s) Executions Elapsed Time per Exec (s) %Total %CPU %IO SQL Id SQL Text
44,950.03 55 817.27 94.67 6.99 94.72 79dgmrxh4kv74 SELECT /*+ index(cdr_data cdr_...
I can't understand whether the problem is in the database side or with the query?
If the problem is with the query, then how it has been executed in 5 mins for the first batch ?
(all the batches are having 5000 records each).
And how can we reduce the db file parallel read ?
Your valuable advice will be greatly appreciated.
Thanks in advance
Manoj Kumar N

Answer 1

"db file parallel read" is likely to be associated with something like index prefetching.
See:
http://www.freelists.org/post/oracle-l/RE-Calculating-LIOs,11
http://aprakash.wordpress.com/2012/05/29/index-range-scan-and-db-file-scattered-read-as-session-wait-event/
http://jonathanlewis.wordpress.com/2006/12/15/index-operations/
Tune the SQL.
Review the execution plan.
Check whether the statistics are accurate.
Review whether the index hint (and others that we can't see) is appropriate.