Slow running query, plan attached

Question

Oracle Database 10g Release 10.2.0.1.0
A particular query is taking a rather long time to run. This was something which used to run rather quick until a couple of days back.
Indexes seem to be used and nothing else seem to have changed muck. The 5.38 sec cpu time is what i am trying to trace down.
Does the attached plan show anything that indicates an underlying issue?
SELECT NVL(SUM(NVL(TCR.EMPLOYEEHOURSWORKED, 0) + NVL(TCR.EMPLOYEEOTHOURSWORKED,0)), 0)
FROM   TYPE TYP, EQUIPMENT EQ, TIMECARD TC, TIMECARDRESOURCE TCR
WHERE  TYP.TYPEID =   EQ.TYPEID
AND    TCR.EQUIPMENTID = EQ.EQUIPMENTID
AND    EQ.TYPEID = :B2
AND    TCR.EMPLOYEEID = :B1
AND    TCR.TIMECARDID = TC.TIMECARDID         
call     count       cpu    elapsed       disk      query    current        rows
Parse        2      0.00       0.00          0          0          0           0
Execute    218      0.05       0.04          0          0          0           0
Fetch      218      5.33       5.23          0    2176948          0         218
total      438      5.38       5.28          0    2176948          0         218
Misses in library cache during parse: 1
Misses in library cache during execute: 2
Optimizer mode: ALL_ROWS
Parsing user id: 91     (recursive depth: 1)
Rows     Row Source Operation     
     68  SORT AGGREGATE (cr=679048 pr=0 pw=0 time=1580234 us)     
     56   TABLE ACCESS BY INDEX ROWID TIMECARDRESOURCE (cr=679048 pr=0 pw=0 time=1479170 us)
743036    NESTED LOOPS  (cr=9656 pr=0 pw=0 time=1507400 us)      
   2788     TABLE ACCESS BY INDEX ROWID EQUIPMENT (cr=1972 pr=0 pw=0 time=15059 us)
   2788      INDEX RANGE SCAN IDX_EQP_TYPEID (cr=204 pr=0 pw=0 time=6075 us)(object id 193877)
740180     INDEX RANGE SCAN IDX_TCR_EQPID (cr=7684 pr=0 pw=0 time=848397 us)(object id 193878)
********************************************************************************

Answer 1

997026 wrote:
Oracle Database 10g Release 10.2.0.1.0
A particular query is taking a rather long time to run. This was something which used to run rather quick until a couple of days back.
Indexes seem to be used and nothing else seem to have changed muck. The 5.38 sec cpu time is what i am trying to trace down.
Does the attached plan show anything that indicates an underlying issue?
SELECT NVL(SUM(NVL(TCR.EMPLOYEEHOURSWORKED, 0) + NVL(TCR.EMPLOYEEOTHOURSWORKED,0)), 0)
FROM   TYPE TYP, EQUIPMENT EQ, TIMECARD TC, TIMECARDRESOURCE TCR
WHERE  TYP.TYPEID =   EQ.TYPEID
AND    TCR.EQUIPMENTID = EQ.EQUIPMENTID
AND    EQ.TYPEID = :B2
AND    TCR.EMPLOYEEID = :B1
AND    TCR.TIMECARDID = TC.TIMECARDID         
call     count       cpu    elapsed       disk      query    current        rows
Parse        2      0.00       0.00          0          0          0           0
Execute    218      0.05       0.04          0          0          0           0
Fetch      218      5.33       5.23          0    2176948          0         218
total      438      5.38       5.28          0    2176948          0         218
Misses in library cache during parse: 1
Misses in library cache during execute: 2
Optimizer mode: ALL_ROWS
Parsing user id: 91     (recursive depth: 1)
Rows     Row Source Operation     
68  SORT AGGREGATE (cr=679048 pr=0 pw=0 time=1580234 us)     
56   TABLE ACCESS BY INDEX ROWID TIMECARDRESOURCE (cr=679048 pr=0 pw=0 time=1479170 us)
743036    NESTED LOOPS  (cr=9656 pr=0 pw=0 time=1507400 us)      
2788     TABLE ACCESS BY INDEX ROWID EQUIPMENT (cr=1972 pr=0 pw=0 time=15059 us)
2788      INDEX RANGE SCAN IDX_EQP_TYPEID (cr=204 pr=0 pw=0 time=6075 us)(object id 193877)
740180     INDEX RANGE SCAN IDX_TCR_EQPID (cr=7684 pr=0 pw=0 time=848397 us)(object id 193878)
Notice how smart the optimizer has been - you've written a query with 4 tables in it, but the optimizer has recognised that it can eliminate two of the tables because of the primary key and referential integrity constraints.
From the remaining two tables, you have a query that looks as if it is inherently NON-SCALABLE. You select all timecardresources for a given employee, and all pieces of equiplment of a given type, and join them on the equipment id.
I am going to guess that the number of timecardresources for an employee will always grow with time - which means the volume of work you need to do will grow with time; moreover that on some regular basis you add rows to timecardresources, which means the number of random block visits per employeeid will grow with time.
It's possible that hash join that you've engineered in your current solution was the original path, but the optimizer reached the breakpoint where the cost of random visits by employeeid became too great and the decided to use a nested loop from the equipment. (These things happen). However, the point may come when you current solution is also too expensive and the optimizer may then decide to do a tablescan on the timecradresources table.
This is a fairly classic type of problem - a (relatively) small result set arising from a join with a single predicate on each of the joined tables.
If my assumptions are correct, you need to find a way to do one of three things
a) reduce the cost of finding all the timecardresources for an employee - recreating the table as an IOT (index organized table) may be an appropriate mechanism
b) restructure you indexes to minimise random visits to tables - eliminating data as early as possible.
c) re-engineer your indexes and SQL so that you don't do random visits to tables unless you need the rows that you're going to find (Here's a link to my blog which let's you access a video made a couple of years ago of a presentation I did on the topic: https://jonathanlewis.wordpress.com/2011/06/23/video/ )
A couple of other notes:
a) Christian Antognini's book is probably the best practical book available at present on Oracle Tuning
b) The index you've dropped in your solution looks like it might be an index created to protect a foreign key constraint from the "foreign key locking" problem.
Keep an eye open for enqueue waits of type TM, and TM deadlocks.
Regards
Jonathan Lewis