Dbms_lob , where did my time go ?

Question

Hi all
After using 10046 to identify the sql that is causing the slowness in a program “ less commits cause my program to go slower” i realised that i am missing something ,
There was a lot of time missing in the tkprof file , and no sql or wait event allocate the missing time , so i put the following test case together in an attempt to understand where the time is going .
Version of test database : 11.1.0.6.0
Name of test database: stdby ( :-) used my standby database)
Database non-default values
#     Parameter     Value1
1:     audit_file_dest     /u01/app/oracle/admin/stdby/adump
2:     audit_trail     DB
3:     compatible     11.1.0.0.0
4:     control_files     /u01/app/oracle/oradata/stdby/control01.ctl
5:     control_files     /u01/app/oracle/oradata/stdby/control02.ctl
6:     control_files     /u01/app/oracle/oradata/stdby/control03.ctl
7:     db_block_size     8192
8:     db_domain     
9:     db_name     stdby
10:     db_recovery_file_dest     /u01/app/oracle/flash_recovery_area
11:     db_recovery_file_dest_size     2147483648
12:     diagnostic_dest     /u01/app/oracle
13:     dispatchers     (PROTOCOL=TCP) (SERVICE=stdbyXDB)
14:     memory_target     314572800
15:     open_cursors     300
16:     processes     150
17:     remote_login_passwordfile     EXCLUSIVE
18:     undo_tablespace     UNDOTBS1More accurately I used existing example from http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:4084920819312
I hope Tom does not mind .
create table t ( x clob );
create or replace procedure p( p_open_close in boolean default false,
                                 p_iters in number default 100 )
  as
      l_clob clob;
  begin
      insert into t (x) values ( empty_clob() )
      returning x into l_clob;
      if ( p_open_close )
      then
          dbms_lob.open( l_clob, dbms_lob.lob_readwrite );
      end if;
      for i in 1 .. p_iters
      loop
          dbms_lob.WriteAppend( l_clob, 5, 'abcde' );
      end loop;
      if ( p_open_close )
      then
    dbms_lob.close( l_clob );
end if;
commit;
end;I did the tracing and the run of the pkg with this
alter session set timed_statistics = true;
alter session set max_dump_file_size = unlimited;
alter session set tracefile_identifier = 'test_clob_commit';
alter session set events '10046 trace name context forever, level 12';
exec p(TRUE,20000);
exitDid the tkprof of the 10046 trace file with
tkprof stdby_ora_3656_test_clob_commit.trc stdby_ora_3656_test_clob_commit.trc.tkp sort=(prsela,exeela,fchela) aggregate=yes waits=yes sys=yesWith output of
OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.02       0.02          0          0          0           0
Execute      1     46.89     147.81      38915     235267     492471           1
Fetch        0      0.00       0.00          0          0          0           0
total        2     46.92     147.83      38915     235267     492471           1
Misses in library cache during parse: 1
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                       2        0.00          0.00
  SQL*Net message from client                     2        0.00          0.00
  latch: shared pool                             24        0.05          0.07
  latch: row cache objects                        2        0.00          0.00
  log file sync                                   1        0.01          0.01
OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
call     count       cpu    elapsed       disk      query    current        rows
Parse      117      0.11       0.10          0          0          2           0
Execute    426      0.37       0.40          6          4          9           2
Fetch      645      0.17       0.51         63       1507          0        1952
total     1188      0.65       1.03         69       1511         11        1954
Misses in library cache during parse: 22
Misses in library cache during execute: 22
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file sequential read                     19778        1.12         30.31
  direct path write                           19209        0.00          0.44
  direct path read                            19206        0.00          0.37
  log file switch completion                      8        0.20          0.70
  latch: cache buffers lru chain                  5        0.01          0.02
    3  user  SQL statements in session.
  424  internal SQL statements in session.
  427  SQL statements in session.And it’s here where the time is being lost.The time of the main pkg p(TRUE,2000) takes 147.83 sec, which is correct , but what is making this time up.
From sorted trace file
SQL ID : catnjk0zv6jz1
BEGIN p(TRUE,20000); END;
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.02       0.02          0          0          0           0
Execute      1     46.89     147.81      38915     235267     492471           1
Fetch        0      0.00       0.00          0          0          0           0
total        2     46.92     147.83      38915     235267     492471           1
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 81
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  latch: shared pool                             24        0.05          0.07
  latch: row cache objects                        2        0.00          0.00
  log file sync                                   1        0.01          0.01
  SQL*Net message to client                       1        0.00          0.00
  SQL*Net message from client                     1        0.00          0.00
SQL ID : db78fxqxwxt7r
select /*+ rule */ bucket, endpoint, col#, epvalue
from
histgrm$ where obj#=:1 and intcol#=:2 and row#=:3 order by bucket
intresting , oracle is still using the rule hint in 11g ?
call     count       cpu    elapsed       disk      query    current        rows
Parse        3      0.00       0.00          0          0          0           0
Execute     98      0.05       0.05          0          0          0           0
Fetch       98      0.04       0.17         28        294          0        1538
total      199      0.10       0.22         28        294          0        1538
Misses in library cache during parse: 0
Optimizer mode: RULE
Parsing user id: SYS   (recursive depth: 3)
Rows     Row Source Operation
     20  SORT ORDER BY (cr=3 pr=1 pw=1 time=8 us cost=0 size=0 card=0)
     20   TABLE ACCESS CLUSTER HISTGRM$ (cr=3 pr=1 pw=1 time=11 us)
      1    INDEX UNIQUE SCAN I_OBJ#_INTCOL# (cr=2 pr=0 pw=0 time=0 us)(object id 408)
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file sequential read                        28        0.02          0.12
SQL ID : 5n1fs4m2n2y0r
select pos#,intcol#,col#,spare1,bo#,spare2,spare3
from
icol$ where obj#=:1
call     count       cpu    elapsed       disk      query    current        rows
Parse        2      0.00       0.00          0          0          0           0
Execute     19      0.03       0.03          0          0          0           0
Fetch       60      0.00       0.04          1        120          0          41
total       81      0.04       0.08          1        120          0          41
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: CHOOSE
Parsing user id: SYS   (recursive depth: 2)
Rows     Row Source Operation
      1  TABLE ACCESS BY INDEX ROWID ICOL$ (cr=4 pr=0 pw=0 time=0 us cost=2 size=54 card=2)
      1   INDEX RANGE SCAN I_ICOL1 (cr=3 pr=0 pw=0 time=0 us cost=1 size=0 card=2)(object id 42)
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file sequential read                         1        0.04          0.04None of the parse , execute ,fetch and wait times makes up the 147.83 seconds.
So i turned to oracles trcanlzr.sql that Carlos Sierra wrote and parsed the same trace file to find the offending sql .
And it starts getting intrestting
Trace Analyzer 11.2.6.2 Report: trcanlzr_75835.html
stdby_ora_3656_test_clob_commit.trc (6970486 bytes)
Total Trace Response Time: 148.901 secs.
2009-MAY-03 20:03:51.771 (start of first db call in trace).
2009-MAY-03 20:06:20.672 (end of last db call in trace).
RESPONSE TIME SUMMARY
~~~~~~~~~~~~~~~~~~~~~
                                          pct of                  pct of                  pct of
                                Time       total        Time       total        Time       total
Response Time Component    (in secs)   resp time   (in secs)   resp time   (in secs)   resp time
                    CPU:      47.579       32.0%
          Non-idle Wait:       0.467        0.3%
     ET Unaccounted-for:     100.825       67.7%
       Total Elapsed(1):                             148.871      100.0%
              Idle Wait:                               0.001        0.0%
     RT Unaccounted-for:                               0.029        0.0%
      Total Response(2):                                                     148.901      100.0%
(1) Total Elapsed = "CPU" + "Non-Idle Wait" + "ET Unaccounted-for".
(2) Total Response = "Total Elapsed Time" + "Idle Wait" + "RT Unaccounted-for".
Total Accounted-for = "CPU" + "Non-Idle Wait" + "Idle Wait" = 148.872 secs.
Total Unccounted-for = "ET Unaccounted-for" + "RT Unaccounted-for" = 100.854 secs.{font:Courier}
{color:red}
{size:19}100.825 seconds Wow , that is a lot 67.7 % of the time is not accounted for {size}
{color}
{font}
I even used TVD$XTAT TriVaDis eXtended Tracefile Analysis Tool with the same conclution .
{font:Courier}
{color:green}
{size:19}Looking at the raw trace file i see a lot of lines like this{size}
{color}
{font}
WAIT #7: nam='direct path read' ela= 11 file number=4 first dba=355935 block cnt=1 obj#=71067 tim=1241337833498756
WAIT #7: nam='direct path write' ela= 12 file number=4 first dba=355936 block cnt=1 obj#=71067 tim=1241337833499153
WAIT #7: nam='db file sequential read' ela= 1095 file#=4 block#=399 blocks=1 obj#=71067 tim=1241337833501366{font:Courier}
{color:green}
{size:19}
What is even more interesting is the sql for "PARSING IN CURSOR #7" is not in the trace file !
The question is where is the time going or is the parser of the 10046 trace file just not putting the detail in ? How do i fix this, without speculating, if I do not know where the problem is ?
I thought of doing a strace on the process . Where else can i look for my 100 sec
Please point me in a direction where i can look for my 100,825 seconds as this is a test case with a production system that is loosing the same amount of time but with a lot more sql arround its dbms_lob.writeappend.
{size}
{color}
{font}
Edited by: user5174849 on 2009/05/16 11:17 PM

Answer 1

user5174849 wrote:
After using 10046 to identify the sql that is causing the slowness in a program “ less commits cause my program to go slower” i realised that i am missing something ,
There was a lot of time missing in the tkprof file , and no sql or wait event allocate the missing time , so i put the following test case together in an attempt to understand where the time is going .
Version of test database : 11.1.0.6.0
What is even more interesting is the sql for "PARSING IN CURSOR #7" is not in the trace file !
The question is where is the time going or is the parser of the 10046 trace file just not putting the detail in ? How do i fix this, without speculating, if I do not know where the problem is ?
I thought of doing a strace on the process . Where else can i look for my 100 sec
Please point me in a direction where i can look for my 100,825 seconds as this is a test case with a production system that is loosing the same amount of time but with a lot more sql arround its dbms_lob.writeappend.I guess that the separate cursor that is opened for the LOB operation is where the time is spent, and unfortunately this part is not very well exposed via the usual interfaces (V$SQL, 10046 trace file etc).
You might want to read this post where Kerry identifies the offending SQL via V$OPEN_CURSOR: http://kerryosborne.oracle-guy.com/2009/04/hidden-sql-why-cant-i-find-my-sql-text/
The waits of this cursor #7 are quite likely rather relevant since they probably show you what the LOB operation is waiting for.
The LOB is created with the default NOCACHE attribute therefore it's read and written using direct path operations.
Regards,
Randolf
Oracle related stuff blog:
http://oracle-randolf.blogspot.com/
SQLTools++ for Oracle (Open source Oracle GUI for Windows):
http://www.sqltools-plusplus.org:7676/
http://sourceforge.net/projects/sqlt-pp/