Query Degradation--Hash Join Degraded

Question

Hi All,
I found one query degradation issue.I am on 10.2.0.3.0 (Sun OS) with optimizer_mode=ALL_ROWS.
This is a dataware house db.
All 3 tables involved are parition tables (with daily partitions).Partitions are created in advance and ELT jobs loads bulk data into daily partitions.
I have checked that CBO is not using local indexes-created on them which i believe,is appropriate because when i used INDEX HINT, elapsed time increses.
I checked giving index hint for all tables one by one but dint get any performance improvement.
Partitions are daily loaded and after loading,partition-level stats are gathered with dbms_stats.
We are collecting stats at partition level(granularity=>'PARTITION').Even after collecting global stats,there is no change in access pattern.Stats gather command is given below.
PROCEDURE gather_table_part_stats(i_owner_name,i_table_name,i_part_name,i_estimate:= DBMS_STATS.AUTO_SAMPLE_SIZE, i_invalidate IN VARCHAR2 := 'Y',i_debug:= 'N')
Only SOT_KEYMAP.IPK_SOT_KEYMAP is GLOBAL.Rest all indexes are LOCAL.
Earlier,we were having BIND PEEKING issue,which i fixed but introducing NO_INVALIDATE=>FALSE in stats gather job.
Here,Partition_name (20090219) is being passed through bind variables.
SELECT a.sotrelstg_sot_ud sotcrct_sot_ud,
b.sotkey_ud sotcrct_orig_sot_ud, a.ROWID stage_rowid
FROM (SELECT sotrelstg_sot_ud, sotrelstg_sys_ud,
sotrelstg_orig_sys_ord_id, sotrelstg_orig_sys_ord_vseq
FROM sot_rel_stage
WHERE sotrelstg_trd_date_ymd_part = '20090219'
AND sotrelstg_crct_proc_stat_cd = 'N'
AND sotrelstg_sot_ud NOT IN(
SELECT sotcrct_sot_ud
FROM sot_correct
WHERE sotcrct_trd_date_ymd_part ='20090219')) a,
(SELECT MAX(sotkey_ud) sotkey_ud, sotkey_sys_ud,
sotkey_sys_ord_id, sotkey_sys_ord_vseq,
sotkey_trd_date_ymd_part
FROM sot_keymap
WHERE sotkey_trd_date_ymd_part = '20090219'
AND sotkey_iud_cd = 'I'
--not to select logical deleted rows
GROUP BY sotkey_trd_date_ymd_part,
sotkey_sys_ud,
sotkey_sys_ord_id,
sotkey_sys_ord_vseq) b
WHERE a.sotrelstg_sys_ud = b.sotkey_sys_ud
AND a.sotrelstg_orig_sys_ord_id = b.sotkey_sys_ord_id
AND NVL(a.sotrelstg_orig_sys_ord_vseq, 1) = NVL(b.sotkey_sys_ord_vseq, 1);
During normal business hr, i found that query takes 5-7 min(which is also not acceptable), but during high load business hr,it is taking 30-50 min.
I found that most of the time it is spending on HASH JOIN (direct path write temp).We have sufficient RAM (64 GB total/41 GB available).
Below is the execution plan i got during normal business hr.
| Id  | Operation                 | Name                | Starts | E-Rows | A-Rows |   A-Time   | Buffers | Reads  | Writes |  OMem |  1Mem | Used-Mem | Used-Tmp|
|   1 |  HASH GROUP BY            |                     |      1 |      1 |   7844K|00:05:28.78 |      16M|    217K|  35969 |       |       |          |         |
|*  2 |   HASH JOIN               |                     |      1 |      1 |   9977K|00:04:34.02 |      16M|    202K|  20779 |   580M|    10M|  563M (1)|     650K|
|   3 |    NESTED LOOPS ANTI      |                     |      1 |      6 |   7855K|00:01:26.41 |      16M|   1149 |      0 |       |       |          |         |
|   4 |     PARTITION RANGE SINGLE|                     |      1 |    258K|   8183K|00:00:16.37 |   25576 |   1149 |      0 |       |       |          |         |
|*  5 |      TABLE ACCESS FULL    | SOT_REL_STAGE       |      1 |    258K|   8183K|00:00:16.37 |   25576 |   1149 |      0 |       |       |          |         |
|   6 |     PARTITION RANGE SINGLE|                     |   8183K|    326K|    327K|00:01:10.53 |      16M|      0 |      0 |       |       |          |         |
|*  7 |      INDEX RANGE SCAN     | IDXL_SOTCRCT_SOT_UD |   8183K|    326K|    327K|00:00:53.37 |      16M|      0 |      0 |       |       |          |         |
|   8 |    PARTITION RANGE SINGLE |                     |      1 |    846K|     14M|00:02:06.36 |     289K|    180K|      0 |       |       |          |         |
|*  9 |     TABLE ACCESS FULL     | SOT_KEYMAP          |      1 |    846K|     14M|00:01:52.32 |     289K|    180K|      0 |       |       |          |         |
I will attached the same for high load business hr once query gives results.It is still executing for last 50 mins.
INDEX STATS (INDEXES ARE LOCAL INDEXES)
TABLE_NAME                          INDEX_NAME                          COLUMN_NAME        COLUMN_POSITION   NUM_ROWS DISTINCT_KEYS CLUSTERING_FACTOR
SOT_REL_STAGE                       IDXL_SOTRELSTG_SOT_UD               SOTRELSTG_SOT_UD                 1   25461560      25461560            184180
SOT_REL_STAGE                                                           SOTRELSTG_TRD_DATE               2   25461560      25461560            184180
                                                                        _YMD_PART
TABLE_NAME                          INDEX_NAME                          COLUMN_NAME        COLUMN_POSITION   NUM_ROWS DISTINCT_KEYS CLUSTERING_FACTOR
SOT_KEYMAP                          IDXL_SOTKEY_ENTORDSYS_UD            SOTKEY_ENTRY_ORD_S               1 1012306940             3          38308680
                                                                        YS_UD
SOT_KEYMAP                          IDXL_SOTKEY_HASH                    SOTKEY_HASH                      1 1049582320    1049582320        1049579520
SOT_KEYMAP                                                              SOTKEY_TRD_DATE_YM               2 1049582320    1049582320        1049579520
                                                                        D_PART
SOT_KEYMAP                          IDXL_SOTKEY_SOM_ORD                 SOTKEY_SOM_UD                    1 1023998560     268949136         559414840
SOT_KEYMAP                                                              SOTKEY_SYS_ORD_ID                2 1023998560     268949136         559414840
SOT_KEYMAP                          IPK_SOT_KEYMAP                      SOTKEY_UD                        1 1030369480    1015378900          24226580
TABLE_NAME                          INDEX_NAME                          COLUMN_NAME        COLUMN_POSITION   NUM_ROWS DISTINCT_KEYS CLUSTERING_FACTOR
SOT_CORRECT                         IDXL_SOTCRCT_SOT_UD                 SOTCRCT_SOT_UD                   1  412484756     412484756         411710982
SOT_CORRECT                                                             SOTCRCT_TRD_DATE_Y               2  412484756     412484756         411710982
                                                                        MD_PART
INDEX partiton stas (from dba_ind_partitions)
INDEX_NAME                     PARTITION_NAME       STATUS       BLEVEL LEAF_BLOCKS DISTINCT_KEYS CLUSTERING_FACTOR   NUM_ROWS SAMPLE_SIZE LAST_ANALYZ GLO
IDXL_SOTCRCT_SOT_UD            P20090219            USABLE            1         372        327879            216663     327879      327879 20-Feb-2009 YES
IDXL_SOTKEY_ENTORDSYS_UD       P20090219            USABLE            2        2910             3             36618     856229      856229 19-Feb-2009 YES
IDXL_SOTKEY_HASH               P20090219            USABLE            2        7783        853956            853914     853956      119705 19-Feb-2009 YES
IDXL_SOTKEY_SOM_ORD            P20090219            USABLE            2        6411        531492            157147     799758      132610 19-Feb-2009 YES
IDXL_SOTRELSTG_SOT_UD          P20090219            USABLE            2       13897       9682052             45867    9682052      794958 20-Feb-2009 YESThanks in advance.
Bhavik Desai

Answer 1

Hi Randolf,
Thanks for the time you spent on this issue.I appreciate it.
Please see my comments below:
1. You've mentioned several times that you're passing the partition name as bind variable, but you're obviously testing the statement with literals rather than bind
variables. So your tests obviously don't reflect what is going to happen in case of the actual execution. The cardinality estimates are potentially quite different when
using bind variables for the partition key.
Yes.I intentionaly used literals in my tests.I found couple of times that plan used by the application and plan generated by AUTOTRACE+EXPLAIN PLAN command...is same and
caused hrly elapsed time.
As i pointed out earlier,last month we solved couple of bind peeking issue by intproducing NO_VALIDATE=>FALSE in stats gather procedure,which we execute just after data
load into such daily partitions and before start of jobs which executes this query.
Execution plans From AWR (with parallelism on at table level DEGREE>1)-->This plan is one which CBO has used when degradation occured.This plan is used most of the times.
ELAPSED_TIME_DELTA BUFFER_GETS_DELTA DISK_READS_DELTA CURSOR(SELECT*FROMTA
        1918506000          46154275              918 CURSOR STATEMENT : 4
CURSOR STATEMENT : 4
PLAN_TABLE_OUTPUT
SQL_ID 39708a3azmks7
SELECT A.SOTRELSTG_SOT_UD SOTCRCT_SOT_UD, B.SOTKEY_UD SOTCRCT_ORIG_SOT_UD, A.ROWID STAGE_ROWID FROM (SELECT SOTRELSTG_SOT_UD,
SOTRELSTG_SYS_UD, SOTRELSTG_ORIG_SYS_ORD_ID, SOTRELSTG_ORIG_SYS_ORD_VSEQ FROM SOT_REL_STAGE WHERE SOTRELSTG_TRD_DATE_YMD_PART = :B1 AND
SOTRELSTG_CRCT_PROC_STAT_CD = 'N' AND SOTRELSTG_SOT_UD NOT IN( SELECT SOTCRCT_SOT_UD FROM SOT_CORRECT WHERE SOTCRCT_TRD_DATE_YMD_PART =
:B1 )) A, (SELECT MAX(SOTKEY_UD) SOTKEY_UD, SOTKEY_SYS_UD, SOTKEY_SYS_ORD_ID, SOTKEY_SYS_ORD_VSEQ, SOTKEY_TRD_DATE_YMD_PART FROM
SOT_KEYMAP WHERE SOTKEY_TRD_DATE_YMD_PART = :B1 AND SOTKEY_IUD_CD = 'I' GROUP BY SOTKEY_TRD_DATE_YMD_PART, SOTKEY_SYS_UD,
SOTKEY_SYS_ORD_ID, SOTKEY_SYS_ORD_VSEQ) B WHERE A.SOTRELSTG_SYS_UD = B.SOTKEY_SYS_UD AND A.SOTRELSTG_ORIG_SYS_ORD_ID =
B.SOTKEY_SYS_ORD_ID AND NVL(A.SOTRELSTG_ORIG_SYS_ORD_VSEQ, 1) = NVL(B.SOTKEY_SYS_ORD_VSEQ, 1)
Plan hash value: 1213870831
| Id  | Operation                     | Name                | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
|   0 | SELECT STATEMENT              |                     |       |       | 19655 (100)|          |       |       |        |      |            |
|   1 |  PX COORDINATOR               |                     |       |       |            |          |       |       |        |      |            |
|   2 |   PX SEND QC (RANDOM)         | :TQ10003            |     1 |   116 | 19655   (1)| 00:05:54 |       |       |  Q1,03 | P->S | QC (RAND)  |
|   3 |    HASH GROUP BY              |                     |     1 |   116 | 19655   (1)| 00:05:54 |       |       |  Q1,03 | PCWP |            |
|   4 |     PX RECEIVE                |                     |     1 |   116 | 19655   (1)| 00:05:54 |       |       |  Q1,03 | PCWP |            |
|   5 |      PX SEND HASH             | :TQ10002            |     1 |   116 | 19655   (1)| 00:05:54 |       |       |  Q1,02 | P->P | HASH       |
|   6 |       HASH GROUP BY           |                     |     1 |   116 | 19655   (1)| 00:05:54 |       |       |  Q1,02 | PCWP |            |
|   7 |        NESTED LOOPS ANTI      |                     |     1 |   116 | 19654   (1)| 00:05:54 |       |       |  Q1,02 | PCWP |            |
|   8 |         HASH JOIN             |                     |     1 |   102 | 19654   (1)| 00:05:54 |       |       |  Q1,02 | PCWP |            |
|   9 |          PX JOIN FILTER CREATE| :BF0000             |    13M|   664M|  2427   (3)| 00:00:44 |       |       |  Q1,02 | PCWP |            |
|  10 |           PX RECEIVE          |                     |    13M|   664M|  2427   (3)| 00:00:44 |       |       |  Q1,02 | PCWP |            |
|  11 |            PX SEND HASH       | :TQ10000            |    13M|   664M|  2427   (3)| 00:00:44 |       |       |  Q1,00 | P->P | HASH       |
|  12 |             PX BLOCK ITERATOR |                     |    13M|   664M|  2427   (3)| 00:00:44 |   KEY |   KEY |  Q1,00 | PCWC |            |
|  13 |              TABLE ACCESS FULL| SOT_REL_STAGE       |    13M|   664M|  2427   (3)| 00:00:44 |   KEY |   KEY |  Q1,00 | PCWP |            |
|  14 |          PX RECEIVE           |                     |    27M|  1270M| 17209   (1)| 00:05:10 |       |       |  Q1,02 | PCWP |            |
|  15 |           PX SEND HASH        | :TQ10001            |    27M|  1270M| 17209   (1)| 00:05:10 |       |       |  Q1,01 | P->P | HASH       |
|  16 |            PX JOIN FILTER USE | :BF0000             |    27M|  1270M| 17209   (1)| 00:05:10 |       |       |  Q1,01 | PCWP |            |
|  17 |             PX BLOCK ITERATOR |                     |    27M|  1270M| 17209   (1)| 00:05:10 |   KEY |   KEY |  Q1,01 | PCWC |            |
|  18 |              TABLE ACCESS FULL| SOT_KEYMAP          |    27M|  1270M| 17209   (1)| 00:05:10 |   KEY |   KEY |  Q1,01 | PCWP |            |
|  19 |         PARTITION RANGE SINGLE|                     | 16185 |   221K|     0   (0)|          |   KEY |   KEY |  Q1,02 | PCWP |            |
|  20 |          INDEX RANGE SCAN     | IDXL_SOTCRCT_SOT_UD | 16185 |   221K|     0   (0)|          |   KEY |   KEY |  Q1,02 | PCWP |            |
Other Execution plan from AWR
ELAPSED_TIME_DELTA BUFFER_GETS_DELTA DISK_READS_DELTA CURSOR(SELECT*FROMTA
        1053251381                 0             2925 CURSOR STATEMENT : 4
CURSOR STATEMENT : 4
PLAN_TABLE_OUTPUT
SQL_ID 39708a3azmks7
SELECT A.SOTRELSTG_SOT_UD SOTCRCT_SOT_UD, B.SOTKEY_UD SOTCRCT_ORIG_SOT_UD, A.ROWID STAGE_ROWID FROM (SELECT SOTRELSTG_SOT_UD,
SOTRELSTG_SYS_UD, SOTRELSTG_ORIG_SYS_ORD_ID, SOTRELSTG_ORIG_SYS_ORD_VSEQ FROM SOT_REL_STAGE WHERE SOTRELSTG_TRD_DATE_YMD_PART = :B1 AND
SOTRELSTG_CRCT_PROC_STAT_CD = 'N' AND SOTRELSTG_SOT_UD NOT IN( SELECT SOTCRCT_SOT_UD FROM SOT_CORRECT WHERE SOTCRCT_TRD_DATE_YMD_PART =
:B1 )) A, (SELECT MAX(SOTKEY_UD) SOTKEY_UD, SOTKEY_SYS_UD, SOTKEY_SYS_ORD_ID, SOTKEY_SYS_ORD_VSEQ, SOTKEY_TRD_DATE_YMD_PART FROM
SOT_KEYMAP WHERE SOTKEY_TRD_DATE_YMD_PART = :B1 AND SOTKEY_IUD_CD = 'I' GROUP BY SOTKEY_TRD_DATE_YMD_PART, SOTKEY_SYS_UD,
SOTKEY_SYS_ORD_ID, SOTKEY_SYS_ORD_VSEQ) B WHERE A.SOTRELSTG_SYS_UD = B.SOTKEY_SYS_UD AND A.SOTRELSTG_ORIG_SYS_ORD_ID =
B.SOTKEY_SYS_ORD_ID AND NVL(A.SOTRELSTG_ORIG_SYS_ORD_VSEQ, 1) = NVL(B.SOTKEY_SYS_ORD_VSEQ, 1)
Plan hash value: 3434900850
| Id  | Operation                     | Name                | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
|   0 | SELECT STATEMENT              |                     |       |       |  1830 (100)|          |       |       |        |      |            |
|   1 |  PX COORDINATOR               |                     |       |       |            |          |       |       |        |      |            |
|   2 |   PX SEND QC (RANDOM)         | :TQ10003            |     1 |   131 |  1830   (2)| 00:00:33 |       |       |  Q1,03 | P->S | QC (RAND)  |
|   3 |    HASH GROUP BY              |                     |     1 |   131 |  1830   (2)| 00:00:33 |       |       |  Q1,03 | PCWP |            |
|   4 |     PX RECEIVE                |                     |     1 |   131 |  1830   (2)| 00:00:33 |       |       |  Q1,03 | PCWP |            |
|   5 |      PX SEND HASH             | :TQ10002            |     1 |   131 |  1830   (2)| 00:00:33 |       |       |  Q1,02 | P->P | HASH       |
|   6 |       HASH GROUP BY           |                     |     1 |   131 |  1830   (2)| 00:00:33 |       |       |  Q1,02 | PCWP |            |
|   7 |        NESTED LOOPS ANTI      |                     |     1 |   131 |  1829   (2)| 00:00:33 |       |       |  Q1,02 | PCWP |            |
|   8 |         HASH JOIN             |                     |     1 |   117 |  1829   (2)| 00:00:33 |       |       |  Q1,02 | PCWP |            |
|   9 |          PX JOIN FILTER CREATE| :BF0000             |  1010K|    50M|   694   (1)| 00:00:13 |       |       |  Q1,02 | PCWP |            |
|  10 |           PX RECEIVE          |                     |  1010K|    50M|   694   (1)| 00:00:13 |       |       |  Q1,02 | PCWP |            |
|  11 |            PX SEND HASH       | :TQ10000            |  1010K|    50M|   694   (1)| 00:00:13 |       |       |  Q1,00 | P->P | HASH       |
|  12 |             PX BLOCK ITERATOR |                     |  1010K|    50M|   694   (1)| 00:00:13 |   KEY |   KEY |  Q1,00 | PCWC |            |
|  13 |              TABLE ACCESS FULL| SOT_KEYMAP          |  1010K|    50M|   694   (1)| 00:00:13 |   KEY |   KEY |  Q1,00 | PCWP |            |
|  14 |          PX RECEIVE           |                     |    11M|   688M|  1129   (3)| 00:00:21 |       |       |  Q1,02 | PCWP |            |
|  15 |           PX SEND HASH        | :TQ10001            |    11M|   688M|  1129   (3)| 00:00:21 |       |       |  Q1,01 | P->P | HASH       |
|  16 |            PX JOIN FILTER USE | :BF0000             |    11M|   688M|  1129   (3)| 00:00:21 |       |       |  Q1,01 | PCWP |            |
|  17 |             PX BLOCK ITERATOR |                     |    11M|   688M|  1129   (3)| 00:00:21 |   KEY |   KEY |  Q1,01 | PCWC |            |
|  18 |              TABLE ACCESS FULL| SOT_REL_STAGE       |    11M|   688M|  1129   (3)| 00:00:21 |   KEY |   KEY |  Q1,01 | PCWP |            |
|  19 |         PARTITION RANGE SINGLE|                     |  5209 | 72926 |     0   (0)|          |   KEY |   KEY |  Q1,02 | PCWP |            |
|  20 |          INDEX RANGE SCAN     | IDXL_SOTCRCT_SOT_UD |  5209 | 72926 |     0   (0)|          |   KEY |   KEY |  Q1,02 | PCWP |            |
EXECUTION PLAN AFTER SETTING DEGREE=1 (It was also degraded)
| Id  | Operation                 | Name                | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     | Pstart| Pstop |
|   0 | SELECT STATEMENT          |                     |     1 |   129 |       | 42336   (2)| 00:12:43 |       |       |
|   1 |  HASH GROUP BY            |                     |     1 |   129 |       | 42336   (2)| 00:12:43 |       |       |
|   2 |   NESTED LOOPS ANTI       |                     |     1 |   129 |       | 42335   (2)| 00:12:43 |       |       |
|*  3 |    HASH JOIN              |                     |     1 |   115 |    51M| 42334   (2)| 00:12:43 |       |       |
|   4 |     PARTITION RANGE SINGLE|                     |   846K|    41M|       |  8241   (1)| 00:02:29 |    81 |    81 |
|*  5 |      TABLE ACCESS FULL    | SOT_KEYMAP          |   846K|    41M|       |  8241   (1)| 00:02:29 |    81 |    81 |
|   6 |     PARTITION RANGE SINGLE|                     |  8161K|   490M|       | 12664   (3)| 00:03:48 |    81 |    81 |
|*  7 |      TABLE ACCESS FULL    | SOT_REL_STAGE       |  8161K|   490M|       | 12664   (3)| 00:03:48 |    81 |    81 |
|   8 |    PARTITION RANGE SINGLE |                     |  6525K|    87M|       |     1   (0)| 00:00:01 |    81 |    81 |
|*  9 |     INDEX RANGE SCAN      | IDXL_SOTCRCT_SOT_UD |  6525K|    87M|       |     1   (0)| 00:00:01 |    81 |    81 |
Predicate Information (identified by operation id):
   3 - access("SOTRELSTG_SYS_UD"="SOTKEY_SYS_UD" AND "SOTRELSTG_ORIG_SYS_ORD_ID"="SOTKEY_SYS_ORD_ID" AND
              NVL("SOTRELSTG_ORIG_SYS_ORD_VSEQ",1)=NVL("SOTKEY_SYS_ORD_VSEQ",1))
   5 - filter("SOTKEY_TRD_DATE_YMD_PART"=20090219 AND "SOTKEY_IUD_CD"='I')
   7 - filter("SOTRELSTG_CRCT_PROC_STAT_CD"='N' AND "SOTRELSTG_TRD_DATE_YMD_PART"=20090219)
   9 - access("SOTRELSTG_SOT_UD"="SOTCRCT_SOT_UD" AND "SOTCRCT_TRD_DATE_YMD_PART"=20090219)2. Why are you passing the partition name as bind variable? A statement executing 5 mins. best, > 2 hours worst obviously doesn't suffer from hard parsing issues and
doesn't need to (shouldn't) share execution plans therefore. So I strongly suggest to use literals instead of bind variables. This also solves any potential issues caused
by bind variable peeking.
This is a custom application which uses bind variables to extract data from daily partitions.So,daily automated data extract from daily paritions after load and ELT process.
Here,Value of bind variable is being passed through a procedure parameter.It would be bit difficult to use literals in such application.
3. All your posted plans suffer from bad cardinality estimates. The NO_MERGE hint suggested by Timur only caused a (significant) damage limitation by obviously reducing
the row source size by the group by operation before joining, but still the optimizer is way off, apart from the obviously wrong join order (larger row set first) in
particular the NESTED LOOP operation is causing the main troubles due to excessive logical I/O, as already pointed out by Timur.
Can i ask for alternatives to NESTED LOOP?
4. Your PLAN_TABLE seems to be old (you should see a corresponding note at the bottom of the DBMS_XPLAN.DISPLAY output), because none of the operations have a
filter/access predicate information attached. Since your main issue are the bad cardinality estimates, I strongly suggest to drop any existing PLAN_TABLEs in any non-Oracle
owned schemas because 10g already provides one in the SYS schema (GTT PLAN_TABLE$) exposed via a public synonym, so that the EXPLAIN PLAN information provides the
"Predicate Information" section below the plan covering the "Filter/Access" predicates.
Please post a revised explain plan output including this crucial information so that we get a clue why the cardinality estimates are way off.
I have dropped the old plan.Got above execution plan(listed above in first point) with PREDICATE information.
"As already mentioned the usage of bind variables for the partition name makes this issue potentially worse."
Is there any workaround without replacing bind variable.I am on 10g so 11g's feature will not help !!!
How are you gathering the statistics daily, can you post the exact command(s) used?
gather_table_part_stats(i_owner_name,i_table_name,i_part_name,i_estimate:= DBMS_STATS.AUTO_SAMPLE_SIZE, i_invalidate IN VARCHAR2 := 'Y',i_debug:= 'N')
Thanks & Regards,
Bhavik Desai