Performance issue with high CPU and IO

Question

Hi guys,
I am encountering huge user response time on a production system and I don’t know how to solve it.
Doing some extra tests and using the instrumentation that we have in the code we concluded that the DB is the bottleneck.
We generated some AWR reports and noticed the CPU was in top wait events. Also noticed that in a random manner some simple sql take a long time to execute. We activated the sql trace on the system and noticed that for very simple SQLs (unique index access on one table) we have huge exec times. 9s
In the trace file the huge time we had it in fetch area: 9.1s cpu and elapsed 9.2.
And no or very small waits for this specific SQL.
it seems like the bottle neck is on the CPU but at that point there were very few processes running on the DB. Why can we have such a big cpu wait on a simple select? This is a machine with 128 cores. We have quicker responses on machines smaller/busier than this.
We noticed that we had a huge db_cache_size (12G) and after we scale it down we noticed some improvements but not enough. How can I prove that there is a link between high CPU and big cache_size? (there was not wait involved in SQL execution). what can we do in the case we need big DB cache size?
The second issue is that I tried to execute an sql on a big table (FTS on a big table. no join). Again on that smaller machine it runs in 30 seconds and on this machine it runs in 1038 seconds.
Also generated a trace for this SQL on the problematic machine:
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        1    402.08    1038.31    1842916    6174343          0           1
total        3    402.08    1038.32    1842916    6174343          0           1
  db file sequential read                     12419        0.21         40.02
  i/o slave wait                             135475        0.51        613.03
  db file scattered read                     135475        0.52        675.15
  log file switch completion                      5        0.06          0.18
  latch: In memory undo latch                     6        0.00          0.00
  latch: object queue header operation            1        0.00          0.00
********************************************************************************The high CPU is present here also but here I have huge wait on db file scattered read.
Looking at the session with the select the AWG_wait for db scattered read was 0.5. on the other machine it is like 0.07.
I though this is an IO issue. I did some IO tests at SO level and it seems like the read and writes operation are very fast…much faster than the machine that has the awg_wait smaller. Why the difference in waits?
One difference between these two DBs is that the problem one has the db block size = 16k and the other one has 8k.
I received some reports done at OS level on CPU and IO usage on the problematic machine (in normal operations). It seems like the CPU is very used and the IO stays very low.
On the other machine, the smaller and the faster one, it is other way around.
What is the problem here? How can I test further? Can I link the high CPU to low/slow IO?
we have 10G on sun os with ASM.
Thanks in advance.

Answer 1

Yes, there are many things you can and should do to isolate this. But first check MOS Poor Performance With Oracle9i and 10g Releases When Using Dynamic Intimate Shared Memory (DISM) [ID 1018855.1] isn't messing you up to start.
Also, be sure and post exact patch levels for both Oracle and OS.
Be sure and check all your I/O settings and see what MOS has to say about those.
Are you using ASSM? See Long running update
Since it got a little better with shrinking the SGA size, that might indicate (wild speculation here, something like) one of the problems is simply too much thrashing within the SGA, as oracle decides "small" objects being full scanned in memory is faster than range scans (or whatever) from disk, overloading the cpu, not allowing the cpu to ask for other full scans from I/O. Possibly made worse by row level locking, or some other app issue that just does too much cpu.
You probably have more than one thing wrong. High fetch count might mean you need to adjust the array size on the clients.
Now that that is all out of the way, if you still haven't found the problem, go through http://oracle-randolf.blogspot.com/2009/02/basic-sql-statement-performance.html
Edit: Oh, see Solaris 10 memory management conflicts with Automatic PGA Memory Management [ID 460424.1] too.
Edited by: jgarry on Nov 15, 2011 1:45 PM