Help analyzing AWR report

Question

I need you help to analyze the results of the following awr report:
The following awr report is an one hour report of oracle EBS application, oracle version 10203,
on hpux with 20 cpus.
As you can see , the top wait event is on "CPU time".
I read an article by mr. lewis regarding the same issue:
http://jonathanlewis.wordpress.com/statspack-distractions/
on a machine with 20 CPUs , with one hour report , i have (3600*20)=72000 seconds to play with.
although for the one hour target we should be aiming to use less than (1*3600*20)=72,000 CPU seconds.
In my case I am using just 13,368 cpu second.
The Load averages (from hpux top command): 0.30, 0.29, 0.28 on the machine low.
Looking at "Top 5 Timed Events" shows that the last 3 wait events are not an issue.
I alreay know the reason for the db file scattered read , and fixed it.
Yet i still have a "nasty" huge plsql block (sqlid=32fwmj6fsaqm9) that run more than : 80973 sec (about 21 hour).
select last_call_et,status
from v$session
where sid= 97
LAST_CALL_ET     STATUS
80973             ACTIVEThe top 3 statment that appear at top of "SQL ordered by CPU Time" are the same as
the ones on the "SQL ordered by Gets ".
Soprisely the statment that run 21 hour is not at the top of any wait event.
It just at the 8 place of the "SQL ordered by Gets ".
Cache Sizes
~~~~~~~~~~~                       Begin        End
               Buffer Cache:     5,712M     5,712M  Std Block Size:         8K
           Shared Pool Size:     1,200M     1,200M      Log Buffer:    20,780K
Load Profile
~~~~~~~~~~~~                            Per Second       Per Transaction
                  Redo size:            131,028.28             44,647.67
              Logical reads:             46,663.75             15,900.60
              Block changes:                774.58                263.94
             Physical reads:                689.68                235.01
            Physical writes:                 45.77                 15.60
                 User calls:                197.04                 67.14
                     Parses:                 55.44                 18.89
                Hard parses:                  2.49                  0.85
                      Sorts:                 57.97                 19.75
                     Logons:                  0.32                  0.11
                   Executes:              1,538.69                524.31
               Transactions:                  2.93
  % Blocks changed per Read:    1.66    Recursive Call %:    90.83
Rollback per transaction %:   26.52       Rows per Sort:    67.02
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            Buffer Nowait %:  100.00       Redo NoWait %:  100.00
            Buffer  Hit   %:   98.53    In-memory Sort %:  100.00
            Library Hit   %:   99.73        Soft Parse %:   95.50
         Execute to Parse %:   96.40         Latch Hit %:   99.98
Parse CPU to Parse Elapsd %:   95.69     % Non-Parse CPU:   97.93
Shared Pool Statistics        Begin    End
             Memory Usage %:   74.33   69.54
    % SQL with executions>1:   78.32   78.17
  % Memory for SQL w/exec>1:   83.53   82.20
Top 5 Timed Events                                         Avg %Total
~~~~~~~~~~~~~~~~~~                                        wait   Call
Event                                 Waits    Time (s)   (ms)   Time Wait Class
CPU time                                         13,368          77.6
db file sequential read             844,095       3,578      4   20.8   User I/O
db file scattered read              212,830         452      2    2.6   User I/O
SQL*Net more data from client         3,116          92     30    0.5    Network
log file parallel write              10,807          74      7    0.4 System I/O
Time Model Statistics                 
Statistic Name                                       Time (s) % of DB Time
sql execute elapsed time                             16,785.3         97.4
DB CPU                                               13,367.6         77.6
DB time                                              17,232.8          N/A
                                                                  Avg
                                       %Time       Total Wait    wait     Waits
Wait Class                      Waits  -outs         Time (s)    (ms)      /txn
User I/O                    1,064,101     .0            4,032       4     100.7
System I/O                     38,328     .0              123       3       3.6
Network                       750,423     .0              102       0      71.0
Commit                          8,043     .0               59       7       0.8
perating System Statistics             DB/Inst:   Snaps: 13464-13465
Statistic                                       Total
AVG_BUSY_TIME                                 101,026
AVG_IDLE_TIME                                 258,945
AVG_IOWAIT_TIME                                26,203
AVG_SYS_TIME                                   22,092
AVG_USER_TIME                                  78,819
BUSY_TIME                                   2,022,842
IDLE_TIME                                   5,181,192
IOWAIT_TIME                                   526,327
SYS_TIME                                      444,162
USER_TIME                                   1,578,680
LOAD                                                0
OS_CPU_WAIT_TIME                      785,867,591,900
RSRC_MGR_CPU_WAIT_TIME                              0
VM_IN_BYTES                               158,449,660
VM_OUT_BYTES                                        0
PHYSICAL_MEMORY_BYTES                  51,271,172,096
NUM_CPUS                                           20
                                                                   Avg
                                             %Time  Total Wait    wait     Waits
Event                                 Waits  -outs    Time (s)    (ms)      /txn
db file sequential read             844,095     .0       3,578       4      79.9
db file scattered read              212,830     .0         452       2      20.1
Instance Activity Stats                DB/Inst:  
Statistic                                     Total     per Second     per Trans
CPU used by this session                    363,871          101.0          34.4
CPU used when call started                  519,575          144.2          49.2
SQL ordered by CPU Time               
    CPU      Elapsed                  CPU per  % Total
  Time (s)   Time (s)  Executions     Exec (s) DB Time    SQL Id
     9,993     10,180          470       21.26    59.1 g0g2sj2by3p75
Module: JDBC Thin Client
BEGIN WF_EVENT.LISTEN ( p_agent_name => :1, p_wait => :2, p_correlation => :3,
p_deq_condition => null, p_message_count => :4, p_max_error_count => :5 ); END
     9,981     10,159            0         N/A    59.0 86xp3qt4czvy5
Module: JDBC Thin Client
begin :v1 := AR_BUS_EVENT_SUB_PVT.AutoRcpt_Run(:v2, :v3); end;
     9,791      9,806            1     9791.15    56.9 bf8wy6bhr6nak
Module: JDBC Thin Client
SELECT C.CUSTOMER_ID CUSTOMER_ID, C.CUSTOMER_SITE_USE_ID SITE_USE_ID, C.CURRENCY
RCPT_CURRENCY, CR1.AMOUNT LAST_PAYMENT_AMOUNT, CR1.RECEIPT_DATE LAST_PAYMENT_DA
TE, CR1.RECEIPT_NUMBER LAST_PAYMENT_NUMBER FROM ( SELECT A.CUSTOMER_ID, A.CUSTOM
ER_SITE_USE_ID, A.CURRENCY, MAX(B.CASH_RECEIPT_ID) CASH_RECEIPT_ID FROM ( SELECT
SQL ordered by Gets                   
                                Gets              CPU     Elapsed
  Buffer Gets   Executions    per Exec   %Total Time (s)  Time (s)    SQL Id
    69,423,012          470    147,708.5   41.3  9992.60  10180.36 g0g2sj2by3p75
Module: JDBC Thin Client
BEGIN WF_EVENT.LISTEN ( p_agent_name => :1, p_wait => :2, p_correlation => :3,
p_deq_condition => null, p_message_count => :4, p_max_error_count => :5 ); END
    69,359,054            0          N/A   41.3  9980.60  10159.36 86xp3qt4czvy5
Module: JDBC Thin Client
begin :v1 := AR_BUS_EVENT_SUB_PVT.AutoRcpt_Run(:v2, :v3); end;
    60,043,663            1 ############   35.7  9791.15   9805.93 bf8wy6bhr6nak
Module: JDBC Thin Client
SELECT C.CUSTOMER_ID CUSTOMER_ID, C.CUSTOMER_SITE_USE_ID SITE_USE_ID, C.CURRENCY
RCPT_CURRENCY, CR1.AMOUNT LAST_PAYMENT_AMOUNT, CR1.RECEIPT_DATE LAST_PAYMENT_DA
TE, CR1.RECEIPT_NUMBER LAST_PAYMENT_NUMBER FROM ( SELECT A.CUSTOMER_ID, A.CUSTOM
ER_SITE_USE_ID, A.CURRENCY, MAX(B.CASH_RECEIPT_ID) CASH_RECEIPT_ID FROM ( SELECT
     9,346,430            0          N/A    5.6    59.53    133.04 32fwmj6fsaqm9
Module: 01@<elease11i/appl_top/bzqi/1.0/sql/BZAUTOCT.sql
DECLARE p_row_id VARCHAR2(30); p_batch_id ar_batches.batch_id%
TYPE; p_batch_date ar_batches.batch_date%TYPE; p_currency_code
ar_batches.currency_code%TYPE; p_name ar_batches.name%TYPE; p_
comments ar_batches.comments%TYPE; p_exchange_date ar_batches.exchangI also found that the object: FND_CONCURRENT_REQUE had 5,826 row lock waits
but i dnt know how much time its took.
Also it had 538 buffef bussy waits.
Could you suggest what to do with this information ?
Segments by Row Lock Waits            
                                                                     Row
           Tablespace                      Subobject  Obj.          Lock    % of
Owner         Name    Object Name            Name     Type         Waits Capture
APPLSYS    APPL_DATA  FND_CONCURRENT_REQUE            TABLE        5,826   79.28
APPLSYS    APPL_DATA  SYS_IOT_TOP_497864              INDEX          794   10.80
Segments by Buffer Busy Waits          
                                                                  Buffer
           Tablespace                      Subobject  Obj.          Busy    % of
Owner         Name    Object Name            Name     Type         Waits Capture
APPLSYS    APPL_DATA  FND_CONCURRENT_REQUE            TABLE          538   65.93
APPLSYS    APPL_DATA  SYS_IOT_TOP_11597965            INDEX          247   30.27
APPLSYS    APPL_INDEX FND_CONCURRENT_REQUE            INDEX            4     .49
          -------------------------------------------------------------The last interesting thing that i founs is that the shared pool may too small:
I have 32% miss.
Library Cache Activity
                         Get    Pct            Pin    Pct             Invali-
Namespace           Requests   Miss       Requests   Miss    Reloads  dations
SQL AREA              11,593   32.7      5,723,318    0.2      7,095    1,659
          -------------------------------------------------------------

Answer 1

user10634138 wrote:
Yet i still have a "nasty" huge plsql block (sqlid=32fwmj6fsaqm9) that run more than : 80973 sec (about 21 hour).
select last_call_et,status
from v$session
where sid= 97
LAST_CALL_ET     STATUS
80973             ACTIVE
The last_call_et would be from the moment the front-end code called the pl/sql block, and since then the pl/sql could have been running a loop with a lot of deliberate waits (dbms_lock.sleep), or a number of different SQL statements that individually didn't consume huge resources or lose a lot of time, yet totalled a lot of time.
Simple check before you start tracing, and if the session is still alive: select * from v$session_event where sid = 97, select name, value from V$sessstat where sid = 97 and value != 0, to see what clues this gives you about the time. You never know, you may find that a lot of the time was spent waiting on "PL/SQL lock timer".
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
To post code, statspack/AWR report, execution plans or trace files, start and end the section with the tag {noformat}{noformat} (lowercase, curly brackets, no spaces) so that the text appears in fixed format.
"Science is more than a body of knowledge; it is a way of thinking"
Carl Sagan