Need help in understanding the trace file

Question

Hi,
I would need to understand
- the large number of fetch for the query
- SQL*Net message to client and SQL*Net message from clientbeing same
- latch: cache buffers chains
The issue I am experiencing is a 6x delay due to an unknown reason.
Can somebody assist me?
Thanks
D
SQL
SELECT R_RO.*
FROM
(SELECT b.ID,b.ANSWERS,b.C_CLASSID,b.C_ID, b.FCI,b.FROM_ID,b.TCI,b.TO_ID,b.FFU, b.TFU,b.NOU,b.ISN,b.SC_NAME,
b.DN_NAME,b.DN_ITEM_NUMBER,b.ISC, b.PRIORITY FROM VVP.RELATION b WHERE b.ID NOT IN (SELECT /*+ HASH_AJ */ SDE_DELETES_ROW_ID FROM VVP.D94 WHERE DELETED_AT IN (SELECT l.lineage_id FROM SDE.state_lineages l WHERE l.lineage_name = :source_lineage_name AND l.lineage_id <= :source_state_id) AND SDE_STATE_ID
= :"SYS_B_0") UNION ALL SELECT a.ID,a.ANSWERS,a.C_CLASSID, a.C_ID,a.FCI,a.FROM_ID,a.TCI, a.TO_ID,a.FFU,a.TFU,a.NOU, a.ISN,a.SC_NAME,a.DN_NAME, a.DN_ITEM_NUMBER,a.ISC,a.PRIORITY FROM VVP.A94 a, SDE.state_lineages SL WHERE (a.ID, a.SDE_STATE_ID) NOT IN (SELECT /*+ HASH_AJ */ SDE_DELETES_ROW_ID,SDE_STATE_ID FROM VVP.D94 WHERE DELETED_AT IN (SELECT l.lineage_id FROM SDE.state_lineages l WHERE l.lineage_name = :source_lineage_name AND l.lineage_id <= :source_state_id) AND SDE_STATE_ID > :"SYS_B_1") AND a.SDE_STATE_ID = SL.lineage_id AND SL.lineage_name = :source_lineage_name AND SL.lineage_id <= :source_state_id ) R_RO WHERE (ID = :"SYS_B_2")
call count cpu elapsed disk query current rows
Parse 3911 2.30 2.14 0 0 0 0
Execute 3911 2.47 2.43 0 0 0 0
Fetch 3911 268.96 270.60 28 15696558 0 3911
total 11733 273.73 275.18 28 15696558 0 3911
Misses in library cache during parse: 1
Optimizer mode: FIRST_ROWS
Parsing user id: 1031
Rows Row Source Operation
1 VIEW (cr=3966 pr=0 pw=0 time=84973 us)
1 UNION-ALL (cr=3966 pr=0 pw=0 time=84968 us)
1 HASH JOIN ANTI (cr=3963 pr=0 pw=0 time=84707 us)
1 TABLE ACCESS BY INDEX ROWID CONNECTION (cr=4 pr=0 pw=0 time=123 us)
1 INDEX UNIQUE SCAN R94_SDE_ROWID_UK (cr=3 pr=0 pw=0 time=88 us)(object id 7404)
8586 VIEW VW_NSO_2 (cr=3959 pr=0 pw=0 time=112274 us)
8586 NESTED LOOPS (cr=3959 pr=0 pw=0 time=103686 us)
3661 INDEX RANGE SCAN LINEAGES_PK (cr=7 pr=0 pw=0 time=67 us)(object id 307740)
8586 INDEX RANGE SCAN D94_PK1 (cr=3952 pr=0 pw=0 time=70624 us)(object id 1637355)
0 HASH JOIN ANTI (cr=3 pr=0 pw=0 time=248 us)
0 NESTED LOOPS (cr=3 pr=0 pw=0 time=71 us)
0 TABLE ACCESS BY INDEX ROWID A94 (cr=3 pr=0 pw=0 time=68 us)
0 INDEX RANGE SCAN A94_ROWID_IX1 (cr=3 pr=0 pw=0 time=65 us)(object id 129281)
0 INDEX UNIQUE SCAN LINEAGES_PK (cr=0 pr=0 pw=0 time=0 us)(object id 307740)
0 VIEW VW_NSO_1 (cr=0 pr=0 pw=0 time=0 us)
0 NESTED LOOPS (cr=0 pr=0 pw=0 time=0 us)
0 INDEX RANGE SCAN D94_PK1 (cr=0 pr=0 pw=0 time=0 us)(object id 1637355)
0 INDEX UNIQUE SCAN LINEAGES_PK (cr=0 pr=0 pw=0 time=0 us)(object id 307740)
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
SQL*Net message to client 3911 0.00 0.00
SQL*Net message from client 3911 0.49 72.42
latch: cache buffers chains 1434 0.00 0.07
latch: shared pool 1 0.00 0.00
db file sequential read 28 0.14 0.31
latch free 15 0.02 0.04
latch: row cache objects 1 0.00 0.00
log file switch completion 4 0.98 1.64

Answer 1

ssddgreg wrote:
Hi Randolf,
thank you for your excellent interpretation! I have Oracle DBMS 10.2.0.3 Ent Edition deployed on Sun Solaris (64-bit), FIRST_ROWS and CURSOR_SHARING = SIMILAR.
I also checked the table D94.SDE_DELETES_ROW_ID about indexes, it has 3 indexes on the same column:
D94_IX1, NONUNIQUE
D94_IX2, UNIQUE
D94_PK1, UNIQUE
This table is a system table from another middle-tier application and all the DXX tables are configured like that.Is this a third-party vendor application for that you don't have any control over the schema? Because your description of the indexes looks like a potential case of massive over-indexing, increasing the workload required to maintain all this indexes. Very likely some of these indexes are redundant and could be covered by a fewer number of indexes.
Besides that my comment regarding the execution plan was probably not clear enough - what I meant to say is that the HASH_AJ hint prevents the optimizer from doing the clever things with the predicates that I described.
So in principle the question is: What execution plan do you get if you omit the HASH_AJ hints? And how many consistent gets requires this new plan at execution time? You might need to add a NL_AJ hint instead to achieve what I've described, but it would be interesting to see in first place what execution plan is generated without any hints.
Some other comments:
FIRST_ROWS optimizer mode: Does this application require you to use the FIRST_ROWS optimizer mode? Because in principle, if you have an application that actually retrieves most of the time only the first few rows of a larger result set, then you should use the FIRST_ROWS(n) optimizer mode instead. The FIRST_ROWS optimizer mode is deprecated since Oracle 9i if I remember correctly and has some odd side effects on execution plans, in particular if the SQL contains an ORDER BY clause.
If your application usually processes all rows from a given result set, using the default optimizer mode ALL_ROWS is more appropriate - using FIRST_ROWS as a band-aid because with ALL_ROWS things are slower only shows that there is something wrong that should be addressed in a different way (by investigating why the ALL_ROWS mode doesn't arrive at a suitable execution plan as first activity).
CURSOR_SHARING=SIMILAR: Note that CURSOR_SHARING = SIMILAR has some other side effects (and bugs). Oracle has recently announced on My Oracle Support (see document 1169017.1) that CURSOR_SHARING=SIMILAR will be deprecated (no longer supported) in Oracle 12. See this note also for a description why this setting can be problematic.
Of course, if you don't have any control over a vendor application and it works fine and has been optimized for these settings (FIRST_ROWS, CURSOR_SHARING=SIMILAR) then there is not much you can do (or need to do) about that.
Regards,
Randolf
Oracle related stuff blog:
http://oracle-randolf.blogspot.com/
Co-author of the "OakTable Expert Oracle Practices" book:
http://www.apress.com/book/view/1430226684
http://www.amazon.com/Expert-Oracle-Practices-Database-Administration/dp/1430226684