SQL performance drops after 20 mins

Question

Hello,
I am using Oracle Version 9.2.0.5
I have an SQL query that starts off fast but the performance drops after running for 20 mins. It continues to drop till a point where it becomes painfully slow.
The query is as attached below:
SELECT      /*+ INDEX (abe ASR_I_BID_PRD_TOD_5FAX)
INDEX (aae ASR_I_AE_RC_BR)
INDEX (ab ASR_UK_BASES)
INDEX (aaac ASR_I_AFAC_SP)
+*/aae.f8_code
||','||     aae.sector
||','||     aae.ouc
||','||     aae.rule_type
||','||     aae.base_ref
||','||     aae.prodlist_ref
||','|| aaac.L101
||','|| aaac.L201
||','|| aaac.L301
||','|| aaac.L401
||','|| aaac.L501
||','|| aaac.L601
||','|| aaac.L701
||','|| aaac.L801
||','|| aaac.L802
||','|| aaac.L830
||','|| aaac.L850
||','|| aaac.L870
||','|| aaac.L901
||','|| aaac.L1001
||','||     LTRIM(TO_CHAR((aae.disaggregated_amount*abe.percentage*aaac.chain_perc)/10000,'999999999999999.99'))
FROM asr_owner.asr_account_entries aae
, asr_owner.asr_base_elements_5fax abe
, asr_ff_audit_chains aaac
, asr_owner.asr_bases ab
WHERE aae.run_code = '5fax'
AND ab.run_code = aae.run_code
AND ab.base_type = aaac.base_type
AND ab.base_id = abe.base_id
AND ab.base_ref = aae.base_ref
AND aaac.base_type = 'L'
AND aaac.business <> 'R'
AND aae.trans_type IN ('A','C','D')
AND abe.base_id = aae.base_id
AND 100*(abe.exhaust_level)+abe.exhaust_sub_level = 101
AND aaac.start_prod = abe.product
AND ABS((aae.disaggregated_amount*abe.percentage*aaac.chain_perc)/10000) >= 0.01
It has huge number of records. I have checked all join conditions and they are all fine.
asr_owner.asr_account_entries - 3.6 millions rows
asr_owner.asr_base_elements_5fax - 1.8 million rows
asr_ff_audit_chains - 5.8 million rows
asr_owner.asr_bases - 0.5 million rows
The autotrace and plan are as follows
Statistics
0 recursive calls
0 db block gets
5264 consistent gets
3522 physical reads
0 redo size
525 bytes sent via SQL*Net to client
456 bytes received via SQL*Net from client
1 SQL*Net roundtrips to/from client
5 sorts (memory)
0 sorts (disk)
0 rows processed
PLAN_TABLE_OUTPUT
| Id | Operation | Name | Rows | Bytes | Cost |
| 0 | SELECT STATEMENT | | 83 | 13529 | 12465 |
| 1 | TABLE ACCESS BY INDEX ROWID | ASR_FF_AUDIT_CHAINS | 83 | 6225 | 617 |
| 2 | NESTED LOOPS | | 83 | 13529 | 12465 |
| 3 | NESTED LOOPS | | 1 | 88 | 11848 |
| 4 | NESTED LOOPS | | 2201 | 113K| 3044 |
| 5 | TABLE ACCESS BY INDEX ROWID| ASR_BASES | 774 | 13158 | 722 |
| 6 | INDEX FULL SCAN | ASR_UK_BASES | 104K| | 104 |
| 7 | TABLE ACCESS BY INDEX ROWID| ASR_ACCOUNT_ENTRIES | 3 | 108 | 3 |
| 8 | INDEX RANGE SCAN | ASR_I_AE_RC_BR | 8 | | 1 |
| 9 | TABLE ACCESS BY INDEX ROWID | ASR_BASE_ELEMENTS_5FAX | 1 | 35 | 4 |
| 10 | INDEX RANGE SCAN | ASR_I_BID_PRD_TOD_5FAX | 1 | | 3 |
| 11 | INDEX RANGE SCAN | ASR_I_AFAC_SP | 8241 | | 9 |
Note: cpu costing is off, PLAN_TABLE' is old version
19 rows selected.
I have also tried without the hints and the plan looks like
| Id | Operation | Name | Rows | Bytes | Cost |
| 0 | SELECT STATEMENT | | 83 | 13529 | 679 |
| 1 | TABLE ACCESS BY INDEX ROWID| ASR_FF_AUDIT_CHAINS | 83 | 6225 | 617 |
| 2 | NESTED LOOPS | | 83 | 13529 | 679 |
| 3 | HASH JOIN | | 1 | 88 | 370 |
| 4 | HASH JOIN | | 2201 | 113K| 208 |
| 5 | TABLE ACCESS FULL | ASR_BASES | 774 | 13158 | 16 |
| 6 | TABLE ACCESS FULL | ASR_ACCOUNT_ENTRIES | 12125 | 426K| 191 |
| 7 | TABLE ACCESS FULL | ASR_BASE_ELEMENTS_5FAX | 18325 | 626K| 161 |
| 8 | INDEX RANGE SCAN | ASR_I_AFAC_SP | 8241 | | 9 |
Note: cpu costing is off, PLAN_TABLE' is old version
16 rows selected.
I have tried increasing the hash_area_size parameter to 20MB (it is currently 10MB).
I have tried setting the tuning parameters like optimizer_index_cost_adj but to no success.
Could you please help.

Answer 1

Thanks a lot for all the help.
Here is another query that is running quite slow. Any help on this might be helpful
EXPLAIN PLAN SET statement_id = 'auditextract' FOR
SELECT /*+NO_INDEX(aaac)
       NO_INDEX(aae)
       NO_INDEX(abe)
       NO_INDEX(ab)
       NO_INDEX(agp)+*/
     aae.run_code
,     aae.f8_code
,     aae.sector
,     aae.ouc
,     aae.rule_type
,     aae.base_ref
,     aae.prodlist_ref
,     aae.trans_type
,     aaac.base_type
,DECODE(NVL(aaac.exhaust_chain,'NULL'),'NULL',',',NULL)||aaac.exhaust_chain||
  DECODE(NVL(aaac.exhaust_chain,'NULL'),'NULL',NULL,',')||LTRIM(RTRIM(aaac.final_prod))||
            RPAD(',',14-DECODE(aaac.final_level,1001,14,901,13,870,12,850,11,830,10,802,9,801,8,701,7,601,6,501,5,401,4,301,3,201,2,101,1),',') exhaust_chain,
     aaac.final_level     ,
     aaac.business ,
     L101,
     L101_TOD,
     L201,
     L201_TOD,
     L301,
     L301_TOD,
     L401,
     L401_TOD,
     L501,
     L501_TOD,
     L601,
     L601_TOD,
     L701,
     L701_TOD,
     L801,
     L801_TOD,
     L802,
     L802_TOD,
     L830,
     L830_TOD,
     L850,
     L850_TOD,
     L870,
     L870_TOD,
     L901,
     L901_TOD,
     L1001,
     L1001_TOD,
     (aae.disaggregated_amount*abe.percentage*aaac.chain_perc)/10000 money
FROM   asr_owner.asr_account_entries aae
,      asr_owner.asr_base_elements_5fax abe
,      asr_owner.asr_bases ab
,      ASR_FF_AUDIT_CHAINS aaac
WHERE  aae.run_code    = '5fax'
AND    ab.run_code = aae.run_code
AND    abe.run_code = ab.run_code
AND    ab.base_type = aaac.base_type
AND    ab.base_id = abe.base_id
AND    aaac.base_type = 'L'
AND    aaac.business = 'R'
AND    ab.base_ref = aae.base_ref
AND    aae.trans_type IN ('A','C','D','G','J')
AND    abe.base_id     = aae.base_id
AND    abe.exhaust_level = 1
AND    abe.exhaust_sub_level = 1
AND    aaac.start_prod = abe.product
AND    NOT EXISTS (SELECT 1
                    FROM asr_group_prods agp
                    WHERE agp.product_group IN ('SW500','SR500','G176')
                    AND  (aaac.L101 =  agp.product OR
                              aaac.L201 =  agp.product OR
                                 aaac.L301 =  agp.product OR
                                 aaac.L401 =  agp.product OR
                                 aaac.L501 =  agp.product OR
                                 aaac.L601 =  agp.product OR
                                 aaac.L701 =  agp.product OR
                                 aaac.L801 =  agp.product OR
                                 aaac.L802 =  agp.product OR
                                 aaac.L830 =  agp.product OR
                                 aaac.L850 =  agp.product OR
                                 aaac.L870 =  agp.product OR
                                 aaac.L901 =  agp.product OR
                                 aaac.L1001 =  agp.product))
AND    abs((aae.disaggregated_amount*abe.percentage*aaac.chain_perc)) >= 100
PLAN_TABLE_OUTPUT
| Id  | Operation             |  Name                   | Rows  | Bytes | Cost  |
|   0 | SELECT STATEMENT      |                         |   571 |   112K|  5586 |
|*  1 |  FILTER               |                         |       |       |       |
|*  2 |   HASH JOIN           |                         |   571 |   112K|  5586 |
|*  3 |    HASH JOIN          |                         |    53 |  5406 |   370 |
|*  4 |     HASH JOIN         |                         |  2974 |   177K|   178 |
|*  5 |      TABLE ACCESS FULL| ASR_BASES               |   770 | 15400 |    16 |
|*  6 |      TABLE ACCESS FULL| ASR_BASE_ELEMENTS_5FAX  | 26178 |  1048K|   161 |
|*  7 |     TABLE ACCESS FULL | ASR_ACCOUNT_ENTRIES     | 17739 |   710K|   191 |
|*  8 |    TABLE ACCESS FULL  | ASR_FF_AUDIT_CHAINS     |   126K|    12M|  5215 |
|*  9 |   TABLE ACCESS FULL   | ASR_GROUP_PRODS         |     1 |    10 |     2 |
Predicate Information (identified by operation id):
   1 - filter( NOT EXISTS (SELECT /*+ */ 0 FROM "ASR_GROUP_PRODS" "AGP" WHERE
              ("AGP"."PRODUCT_GROUP"='G176' OR "AGP"."PRODUCT_GROUP"='SR500' OR
              "AGP"."PRODUCT_GROUP"='SW500') AND ("AGP"."PRODUCT"=:B1 OR "AGP"."PRODUCT"=:B2
              OR "AGP"."PRODUCT"=:B3 OR "AGP"."PRODUCT"=:B4 OR "AGP"."PRODUCT"=:B5 OR
              "AGP"."PRODUCT"=:B6 OR "AGP"."PRODUCT"=:B7 OR "AGP"."PRODUCT"=:B8 OR
              "AGP"."PRODUCT"=:B9 OR "AGP"."PRODUCT"=:B10 OR "AGP"."PRODUCT"=:B11 OR
              "AGP"."PRODUCT"=:B12 OR "AGP"."PRODUCT"=:B13 OR "AGP"."PRODUCT"=:B14)))
   2 - access("SYS_ALIAS_14"."START_PROD"="ABE"."PRODUCT")
       filter(ABS("AAE"."DISAGGREGATED_AMOUNT"*"ABE"."PERCENTAGE"*"SYS_ALIAS_14".
              "CHAIN_PERC")>=100)
   3 - access("AB"."BASE_REF"="AAE"."BASE_REF" AND
              "ABE"."BASE_ID"="AAE"."BASE_ID")
   4 - access("AB"."BASE_ID"="ABE"."BASE_ID")
   5 - filter("AB"."RUN_CODE"='5fax' AND "AB"."BASE_TYPE"='L')
   6 - filter("ABE"."RUN_CODE"='5fax' AND "ABE"."EXHAUST_LEVEL"=1 AND
              "ABE"."EXHAUST_SUB_LEVEL"=1)
   7 - filter("AAE"."RUN_CODE"='5fax' AND ("AAE"."TRANS_TYPE"='A' OR
              "AAE"."TRANS_TYPE"='C' OR "AAE"."TRANS_TYPE"='D' OR "AAE"."TRANS_TYPE"='G' OR
              "AAE"."TRANS_TYPE"='J'))
   8 - filter("SYS_ALIAS_14"."BASE_TYPE"='L' AND "SYS_ALIAS_14"."BUSINESS"='R')
   9 - filter(("AGP"."PRODUCT_GROUP"='G176' OR "AGP"."PRODUCT_GROUP"='SR500' OR
              "AGP"."PRODUCT_GROUP"='SW500') AND ("AGP"."PRODUCT"=:B1 OR "AGP"."PRODUCT"=:B2
              OR "AGP"."PRODUCT"=:B3 OR "AGP"."PRODUCT"=:B4 OR "AGP"."PRODUCT"=:B5 OR
              "AGP"."PRODUCT"=:B6 OR "AGP"."PRODUCT"=:B7 OR "AGP"."PRODUCT"=:B8 OR
              "AGP"."PRODUCT"=:B9 OR "AGP"."PRODUCT"=:B10 OR "AGP"."PRODUCT"=:B11 OR
              "AGP"."PRODUCT"=:B12 OR "AGP"."PRODUCT"=:B13 OR "AGP"."PRODUCT"=:B14))
Note: cpu costing is off
47 rows selected.