Trace file analysis: High cpu timings for FETCH

Question

Hi!
I am doing a 10046 trace file analysis for a performance problem we see on a customer site, but we can't reproduce the problem on our local reference test system.
Basically, the cpu timings for FETCH calls for a SELECT statement are ten times as high on the customer system. There are no WAIT events in the trace file (for this statement), only a high cpu timing for the FETCHES:
Customer instance:
FETCH #19:c=140625,e=140189,p=0,cr=3358,cu=0,mis=0,r=1,dep=1,og=1,tim=1500177409
Reference system;
FETCH #4:c=15625,e=7984,p=0,cr=2262,cu=0,mis=0,r=1,dep=1,og=1,tim=5624714062
I see that the number of consistant reads is 50% higher, but I don't see why this result in 10 times the CPU time and about 20 times the elapsed time. This is does not seem to be a general problem with all statements, so the general timings are comparable to our reference system. The problem seems to be this Select statement only which only joins two tables.
I had the idea that a long running transaction that didn't commit would lead to this problem, since Oracle would need to reconstruct blocks from undo, but there are no such transactions on the system.
With no WAIT events emitted, where does the time go?
Thanks for your help,
Marcus

Answer 1

MMarcus wrote:
Here is the execution plan for the query. It is the same on both systems, with slightly different total costs (8 or 10).
| Id  | Operation                      | Name                | Rows  | Bytes | Cost (%CPU)| Time     |                                                                                                                                                                                                      
|   0 | SELECT STATEMENT               |                     |     1 |    35 |    10  (10)| 00:00:01 |                                                                                                                                                                                                      
|   1 |  SORT ORDER BY                 |                     |     1 |    35 |    10  (10)| 00:00:01 |                                                                                                                                                                                                      
|*  2 |   FILTER                       |                     |       |       |            |          |                                                                                                                                                                                                      
|*  3 |    TABLE ACCESS BY INDEX ROWID | PLANHIERARCHIE      |     1 |    35 |     5   (0)| 00:00:01 |                                                                                                                                                                                                      
|*  4 |     INDEX RANGE SCAN           | PLANHIERARCHIEINDEX |    21 |       |     2   (0)| 00:00:01 |                                                                                                                                                                                                      
|   5 |    NESTED LOOPS                |                     |     1 |    39 |     4   (0)| 00:00:01 |                                                                                                                                                                                                      
|*  6 |     TABLE ACCESS BY INDEX ROWID| PLANSCHRITTFOLGE    |     1 |    21 |     3   (0)| 00:00:01 |                                                                                                                                                                                                      
|*  7 |      INDEX RANGE SCAN          | PLANSCHRITTFOLGEPK  |    11 |       |     2   (0)| 00:00:01 |                                                                                                                                                                                                      
|*  8 |     TABLE ACCESS BY INDEX ROWID| PLANHIERARCHIE      |     1 |    18 |     1   (0)| 00:00:01 |                                                                                                                                                                                                      
|*  9 |      INDEX UNIQUE SCAN         | PLANHIERARCHIEPK    |     1 |       |     0   (0)| 00:00:01 |                                                                                                                                                                                                      
You have omitted the predicate section from the execution plan - and [+*there may be important clues*+|http://jonathanlewis.wordpress.com/2008/12/03/predicate-problems/] in the predicate section.
In your case the problem may +(*for example*+) simply be the way in which the predicates apply to the PLANSCHRITTFOLGEPK index range scan.
Imagine you have 250 index entries per leaf block, and the predicates that apply to the index identify an index range scan that spans 25 entries in one system but 25000 entries (100 leaf blocks) in the other. It is possible that filter predicates against the index eliminate most of the index entries before you go to the table. The CPU used to examine (filter) large numbers of index leaf entries can be significant - and since that index range scan is part of a filter subquery it could many several time - escalating the CPU usage. A check of the access and filter predicates may give you some clue about whether you are seeing that type of problem.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
To post code, statspack/AWR report, execution plans or trace files, start and end the section with the tag {noformat}{noformat} (lowercase, curly brackets, no spaces) so that the text appears in fixed format.
"Science is more than a body of knowledge; it is a way of thinking"
Carl Sagan