Understanding Performance in Oracle 10.1.0.4.0

Question

I've been looking over my Oracle instance in response to complaints of performance issues.
Basic queries, say 700-1500 rows, which have taken 10 seconds take 30-300 seconds.
I've dropped the wait classes results and time usage below.
It's a quad processor DL380G3, however, the CPUs are pegged at 80-100% most all of the time.
The Idle wait class accounts for 99% of the time.
I increased the SGA size, doubled it from our previous level of 700M. And increased the PGA Agg target to 50M.
Can anyone (see below) give me advice or clarification on how I might determine if the idle time is legitimate? Have I misconfigured something?
WAIT_CLASS TIME_WAITED
Idle 46350690
User I/O 124480
Commit 106737
Concurrency 60048
System I/O 17271
Configuration 10104
Network 10050
Application 3794
Other 3128
STAT_NAME SECONDS
DB time 15822.0971
sql execute elapsed time 13732.1841
DB CPU 11336.9654
PL/SQL execution elapsed time 1171.44213
background elapsed time 808.668192
parse time elapsed 782.996948
hard parse elapsed time 464.313661
failed parse elapsed time 174.245089
connection management call elapsed time 134.020711
PL/SQL compilation elapsed time 103.442958
background cpu time 46.878018
sequence load elapsed time 15.442131
hard parse (sharing criteria) elapsed time 3.196418
hard parse (bind mismatch) elapsed time .280402
Java execution elapsed time .004433

Answer 1

834681 wrote:
Basic queries, say 700-1500 rows, which have taken 10 seconds take 30-300 seconds.
It's a quad processor DL380G3, however, the CPUs are pegged at 80-100% most all of the time.
I increased the SGA size, doubled it from our previous level of 700M. And increased the PGA Agg target to 50M.
STAT_NAME                                                           SECONDS
DB time                                                          15822.0971
sql execute elapsed time                                         13732.1841
DB CPU                                                           11336.9654
PL/SQL execution elapsed time                                    1171.44213
background elapsed time                                          808.668192
parse time elapsed                                               782.996948
hard parse elapsed time                                          464.313661
failed parse elapsed time                                        174.245089
connection management call elapsed time                          134.020711
PL/SQL compilation elapsed time                                  103.442958
background cpu time                                               46.878018
sequence load elapsed time                                        15.442131
hard parse (sharing criteria) elapsed time                         3.196418
hard parse (bind mismatch) elapsed time                             .280402
Java execution elapsed time                                         .004433
Assuming this is a standard 1 hour snapshot, you've got 14,400 CPU seconds available, and the DB CPU is taking 11,337 of those - so your 80% - 100% of CPU report is not surprising.
Since you've got a quad-core CPU we might guess that the machine has several GB of RAM - so it's a little surprising that your SGA was only 700MB, and the pga_aggregate_target only 50MB.
What is the total memory on the machine, how many users of this database, and does the machine have to support other applications or databases.
You haven't given us much go go on, but my first guess would be either that execution plans have changed to something is much more CPU intensive (e.g. use of wrong index in wrong join order); or you always had some very bad plans which have gradually got worse over time as the data size increased, but you were protected to some extent by the O/S memory outside the Oracle database.
I would like to see the Load Profile section and Top N timed event section of your AWR. I think you'll benefit from looking at the "SQL ordered by CPU" because that will tell you which statements are worth reviewing first, and I'll probably suggest looking at some other bits once you've posted the first two.
There are other problems hinted at by the time usage report - and your decision to increase the size of the SGA was probably appropriate, although we may find it could be much bigger still. (The timings on parsing suggest a shared pool problem, but the size of that problem might drop dramatically once we eliminate the source of the CPU load.)
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
A general reminder about "Forum Etiquette / Reward Points": http://forums.oracle.com/forums/ann.jspa?annID=718
If you never mark your questions as answered people will eventually decide that it's not worth trying to answer you because they will never know whether or not their answer has been of any use, or whether you even bothered to read it.
It is also important to mark answers that you thought helpful - again it lets other people know that you appreciate their help, but it also acts as a pointer for other people when they are researching the same question, moreover it means that when you mark a bad or wrong answer as helpful someone may be prompted to tell you (and the rest of the forum) what's so bad or wrong about the answer you found helpful.