SQL PLAN MANAGEMENT - Locking in a more 'expensive', better plan.?

Question

Environment: 11.2.0.3 on HP-UX Itanium.
We're preparing for an upgrade of our main production database from 10.2.0.3 to 11.2.0.3. We're using SQL Tuning Sets to capture execution plans from the 10G environment and load them into the upgraded (test) environment as SQL Plan Baselines. We've been using SQL Performance Analyzer to identify plans which have regressed from 10G to 11G. For these regressed plans we wish to use SPM to lock in the better 10G plan.
We've been successful in locking in the 10G plans for those regressed queries where the 11g CBO "Cost" of the query is higher than it is in 10g, however we have some examples where SPA has shown a performance regression for queries (measured in terms of elapsed time, or buffer gets or CPU time), but the 11G explain plan has a lower "cost" as identified by the CBO. Ie: the 11G "cost" is lower, but the plan is worse.
For these small number of queries, I have not been able to get the 11G optimiser to choose the 10G plans even though they are "fixed, enabled and accepted" within SPM. The optimiser consistently chooses what it considers its "lower cost" plan, even though it is a worst plan. Am I missing something here or should it be possible, with the use of SPM to have the optimiser chose the 10G plan, regardless of what its internal cost says?
SQL> show parameter baseline
NAME                                 TYPE        VALUE
optimizer_capture_sql_plan_baselines boolean     FALSE
optimizer_use_sql_plan_baselines     boolean     TRUE

Answer 1

I'm at a bit of a loss now as to where to proceed without posting the entire SPM Trace file, so excuse me for taking up so much bandwidth.
But first some background. Our test databases should be identical in terms of schemas. That is we have a 10G prod database. From that we clone it to two test databases. One we leave at 10G, the other we upgrade to 11G. So there should be no schema differences between the 11G and 10G databases.
h5.
*** 2013-02-12 15:41:27.671
SPM: statement found in SMB
*** 2013-02-12 15:41:27.881
SPM: fixed planId's of plan baseline are: 386463207
SPM: using qksan to reproduce, cost and select accepted plan, sig = 17844158604836855983
SPM: plan reproducibility round 1 (plan outline + session OFE)
SPM: using qksan to reproduce accepted plan, planId = 386463207
SPM: plan reproducibility - session OFE = 11020003, hinted OFE = 10020003
SPM: planId in plan baseline = 386463207, planId of reproduced plan = 3080367404
------- START SPM Plan Dump -------
SPM: failed to reproduce the plan using the following info:
  parse_schema name        : CARADM
  plan_baseline signature  : 17844158604836855983
  plan_baseline plan_id    : 386463207
  plan_baseline hintset    :
    hint num  1 len 27 text: IGNORE_OPTIM_EMBEDDED_HINTS
    hint num  2 len 37 text: OPTIMIZER_FEATURES_ENABLE('10.2.0.3')
    hint num  3 len 36 text: OPT_PARAM('_always_anti_join' 'off')
    hint num  4 len 36 text: OPT_PARAM('_always_semi_join' 'off')
    hint num  5 len 40 text: OPT_PARAM('_index_join_enabled' 'false')
    hint num  6 len 43 text: OPT_PARAM('_optim_peek_user_binds' 'false')
    hint num  7 len 49 text: OPT_PARAM('_optimizer_skip_scan_enabled' 'false')
    hint num  8 len 39 text: OPT_PARAM('optimizer_index_cost_adj' 1)
    hint num  9 len 40 text: OPT_PARAM('optimizer_index_caching' 100)
    hint num 10 len 10 text: FIRST_ROWS
    hint num 11 len 29 text: OUTLINE_LEAF(@"SEL$EE94F965")
    hint num 12 len 22 text: MERGE(@"SEL$9E43CB6E")
    hint num 13 len 17 text: OUTLINE(@"SEL$4")
    hint num 14 len 24 text: OUTLINE(@"SEL$9E43CB6E")
    hint num 15 len 22 text: MERGE(@"SEL$58A6D7F6")
    hint num 16 len 17 text: OUTLINE(@"SEL$3")
    hint num 17 len 24 text: OUTLINE(@"SEL$58A6D7F6")
    hint num 18 len 15 text: MERGE(@"SEL$1")
    hint num 19 len 17 text: OUTLINE(@"SEL$2")
    hint num 20 len 17 text: OUTLINE(@"SEL$1")
    hint num 21 len 36 text: FULL(@"SEL$EE94F965" "CPGR"@"SEL$1")
    hint num 22 len 153 text: INDEX(@"SEL$EE94F965" "CPGP"@"SEL$1" ("CL_PROVIDER_GROUP_PERIOD"."CPGR_ID" "CL_PROVIDER_GROUP_PERIOD"."CPDE_ID" "CL_PRO
VIDER_GROUP_PERIOD"."START_DATE"))
    hint num 23 len 77 text: INDEX_RS_ASC(@"SEL$EE94F965" "CPDE"@"SEL$2" ("CL_PROVIDER_DETAIL"."CPDE_ID"))
    hint num 24 len 62 text: INDEX(@"SEL$EE94F965" "CFCO"@"SEL$3" ("FN_ACCOUNT"."CPDE_ID"))
    hint num 25 len 84 text: LEADING(@"SEL$EE94F965" "CPGR"@"SEL$1" "CPGP"@"SEL$1" "CPDE"@"SEL$2" "CFCO"@"SEL$3")
    hint num 26 len 38 text: USE_NL(@"SEL$EE94F965" "CPGP"@"SEL$1")
    hint num 27 len 38 text: USE_NL(@"SEL$EE94F965" "CPDE"@"SEL$2")
    hint num 28 len 38 text: USE_NL(@"SEL$EE94F965" "CFCO"@"SEL$3")
SPM: generated non-matching plan:
----- Explain Plan Dump -----
----- Plan Table -----
============
Plan Table
============
-------------------------------------------------------------+-----------------------------------+
| Id  | Operation                        | Name              | Rows  | Bytes | Cost  | Time      |
-------------------------------------------------------------+-----------------------------------+
| 0   | SELECT STATEMENT                 |                   |       |       |    22 |           |
| 1   |  SORT UNIQUE                     |                   |    41 |  3116 |    21 |  00:00:01 |
| 2   |   NESTED LOOPS                   |                   |       |       |       |           |
| 3   |    NESTED LOOPS                  |                   |    41 |  3116 |    20 |  00:00:01 |
| 4   |     NESTED LOOPS                 |                   |   826 |   50K |    11 |  00:00:01 |
| 5   |      NESTED LOOPS                |                   |   826 |   26K |     3 |  00:00:01 |
| 6   |       TABLE ACCESS FULL          | CL_PROVIDER_GROUP |   116 |  2668 |     2 |  00:00:01 |
| 7   |       INDEX RANGE SCAN           | CPGP_UK           |     7 |    63 |     1 |  00:00:01 |
| 8   |      TABLE ACCESS BY INDEX ROWID | CL_PROVIDER_DETAIL|     1 |    30 |     1 |  00:00:01 |
| 9   |       INDEX UNIQUE SCAN          | CPDE_PK           |     1 |       |     1 |  00:00:01 |
| 10  |     INDEX RANGE SCAN             | FACC_CPDE_FK_I    |     1 |       |     1 |  00:00:01 |
| 11  |    TABLE ACCESS BY INDEX ROWID   | FN_ACCOUNT        |     1 |    14 |     1 |  00:00:01 |
-------------------------------------------------------------+-----------------------------------+
Predicate Information:
7 - access("CPGR"."CPGR_ID"="CPGP"."CPGR_ID")
9 - access("CPDE"."CPDE_ID"="CPGP"."CPDE_ID")
10 - access("CPDE"."CPDE_ID"="CFCO"."CPDE_ID")
10 - filter("CFCO"."CPDE_ID" IS NOT NULL)
11 - filter((UPPER("CPGR"."NAME") LIKE UPPER('%'||:1||'%') OR UPPER("NAME") LIKE UPPER('%'||:2||'%') OR UPPER("ABN") LIKE UPPER(:3||'%') OR "CFCO"."A
CCT_NO" LIKE UPPER(:4||'%')))
Content of other_xml column
===========================
  db_version     : 11.2.0.3
  parse_schema   : CARADM
  plan_hash      : 524147661
  plan_hash_2    : 3080367404
  Outline Data:
  /*+
    BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.2.0.3')
      DB_VERSION('11.2.0.3')
      OPT_PARAM('_always_anti_join' 'off')
      OPT_PARAM('_always_semi_join' 'off')
      OPT_PARAM('_index_join_enabled' 'false')
      OPT_PARAM('_optim_peek_user_binds' 'false')
      OPT_PARAM('_optimizer_skip_scan_enabled' 'false')
      OPT_PARAM('optimizer_index_cost_adj' 1)
      OPT_PARAM('optimizer_index_caching' 100)
      FIRST_ROWS
      OUTLINE_LEAF(@"SEL$EE94F965")
      MERGE(@"SEL$9E43CB6E")
      OUTLINE(@"SEL$4")
      OUTLINE(@"SEL$9E43CB6E")
      MERGE(@"SEL$58A6D7F6")
      OUTLINE(@"SEL$3")
      OUTLINE(@"SEL$58A6D7F6")
      MERGE(@"SEL$1")
      OUTLINE(@"SEL$2")
      OUTLINE(@"SEL$1")
      FULL(@"SEL$EE94F965" "CPGR"@"SEL$1")
      INDEX(@"SEL$EE94F965" "CPGP"@"SEL$1" ("CL_PROVIDER_GROUP_PERIOD"."CPGR_ID" "CL_PROVIDER_GROUP_PERIOD"."CPDE_ID" "CL_PROVIDER_GROUP_PERIOD"."STA
RT_DATE"))
      INDEX_RS_ASC(@"SEL$EE94F965" "CPDE"@"SEL$2" ("CL_PROVIDER_DETAIL"."CPDE_ID"))
      INDEX(@"SEL$EE94F965" "CFCO"@"SEL$3" ("FN_ACCOUNT"."CPDE_ID"))
      LEADING(@"SEL$EE94F965" "CPGR"@"SEL$1" "CPGP"@"SEL$1" "CPDE"@"SEL$2" "CFCO"@"SEL$3")
      USE_NL(@"SEL$EE94F965" "CPGP"@"SEL$1")
      USE_NL(@"SEL$EE94F965" "CPDE"@"SEL$2")
      USE_NL(@"SEL$EE94F965" "CFCO"@"SEL$3")
      NLJ_BATCHING(@"SEL$EE94F965" "CFCO"@"SEL$3")
    END_OUTLINE_DATA
------- END SPM Plan Dump -------
SPM: plan reproducibility round 1 (plan outline only)
SPM: using qksan to reproduce accepted plan, planId = 386463207
SPM: planId in plan baseline = 386463207, planId of reproduced plan = 4009937738
------- START SPM Plan Dump -------
SPM: failed to reproduce the plan using the following info:
  parse_schema name        : CARADM
  plan_baseline signature  : 17844158604836855983
  plan_baseline plan_id    : 386463207
  plan_baseline hintset    :
    hint num  1 len 27 text: IGNORE_OPTIM_EMBEDDED_HINTS
    hint num  2 len 37 text: OPTIMIZER_FEATURES_ENABLE('10.2.0.3')
    hint num  3 len 36 text: OPT_PARAM('_always_anti_join' 'off')
    hint num  4 len 36 text: OPT_PARAM('_always_semi_join' 'off')
    hint num  5 len 40 text: OPT_PARAM('_index_join_enabled' 'false')
    hint num  6 len 43 text: OPT_PARAM('_optim_peek_user_binds' 'false')
    hint num  7 len 49 text: OPT_PARAM('_optimizer_skip_scan_enabled' 'false')
    hint num  8 len 39 text: OPT_PARAM('optimizer_index_cost_adj' 1)
    hint num  9 len 40 text: OPT_PARAM('optimizer_index_caching' 100)
    hint num 10 len 10 text: FIRST_ROWS
    hint num 11 len 29 text: OUTLINE_LEAF(@"SEL$EE94F965")
    hint num 12 len 22 text: MERGE(@"SEL$9E43CB6E")
    hint num 13 len 17 text: OUTLINE(@"SEL$4")
    hint num 14 len 24 text: OUTLINE(@"SEL$9E43CB6E")
    hint num 15 len 22 text: MERGE(@"SEL$58A6D7F6")
    hint num 16 len 17 text: OUTLINE(@"SEL$3")
    hint num 17 len 24 text: OUTLINE(@"SEL$58A6D7F6")
    hint num 18 len 15 text: MERGE(@"SEL$1")
    hint num 19 len 17 text: OUTLINE(@"SEL$2")
    hint num 20 len 17 text: OUTLINE(@"SEL$1")
    hint num 21 len 36 text: FULL(@"SEL$EE94F965" "CPGR"@"SEL$1")
    hint num 22 len 153 text: INDEX(@"SEL$EE94F965" "CPGP"@"SEL$1" ("CL_PROVIDER_GROUP_PERIOD"."CPGR_ID" "CL_PROVIDER_GROUP_PERIOD"."CPDE_ID" "CL_PRO
VIDER_GROUP_PERIOD"."START_DATE"))
    hint num 23 len 77 text: INDEX_RS_ASC(@"SEL$EE94F965" "CPDE"@"SEL$2" ("CL_PROVIDER_DETAIL"."CPDE_ID"))
    hint num 24 len 62 text: INDEX(@"SEL$EE94F965" "CFCO"@"SEL$3" ("FN_ACCOUNT"."CPDE_ID"))
    hint num 25 len 84 text: LEADING(@"SEL$EE94F965" "CPGR"@"SEL$1" "CPGP"@"SEL$1" "CPDE"@"SEL$2" "CFCO"@"SEL$3")
    hint num 26 len 38 text: USE_NL(@"SEL$EE94F965" "CPGP"@"SEL$1")
    hint num 27 len 38 text: USE_NL(@"SEL$EE94F965" "CPDE"@"SEL$2")
    hint num 28 len 38 text: USE_NL(@"SEL$EE94F965" "CFCO"@"SEL$3")
SPM: generated non-matching plan:
----- Explain Plan Dump -----
----- Plan Table -----
============
Plan Table
============
------------------------------------------------------------+-----------------------------------+
| Id  | Operation                       | Name              | Rows  | Bytes | Cost  | Time      |
------------------------------------------------------------+-----------------------------------+
| 0   | SELECT STATEMENT                |                   |       |       |   179 |           |
| 1   |  SORT UNIQUE                    |                   |   153 |   11K |   178 |  00:00:01 |
| 2   |   NESTED LOOPS                  |                   |   153 |   11K |   177 |  00:00:01 |
| 3   |    NESTED LOOPS                 |                   |   826 |   50K |    11 |  00:00:01 |
| 4   |     NESTED LOOPS                |                   |   826 |   26K |     3 |  00:00:01 |
| 5   |      TABLE ACCESS FULL          | CL_PROVIDER_GROUP |   116 |  2668 |     2 |  00:00:01 |
| 6   |      INDEX RANGE SCAN           | CPGP_UK           |     7 |    63 |     1 |  00:00:01 |
| 7   |     TABLE ACCESS BY INDEX ROWID | CL_PROVIDER_DETAIL|     1 |    30 |     1 |  00:00:01 |
| 8   |      INDEX UNIQUE SCAN          | CPDE_PK           |     1 |       |     1 |  00:00:01 |
| 9   |    TABLE ACCESS BY INDEX ROWID  | FN_ACCOUNT        |     1 |    14 |     1 |  00:00:01 |
| 10  |     INDEX RANGE SCAN            | FACC_CPDE_FK_I    |    27 |       |     1 |  00:00:01 |
------------------------------------------------------------+-----------------------------------+
Predicate Information:
6 - access("CPGR"."CPGR_ID"="CPGP"."CPGR_ID")
8 - access("CPDE"."CPDE_ID"="CPGP"."CPDE_ID")
9 - filter((UPPER("CPGR"."NAME") LIKE UPPER('%'||:1||'%') OR UPPER("CPDE"."NAME") LIKE UPPER('%'||:2||'%') OR UPPER("CPDE"."ABN") LIKE UPPER(:3||'%')
OR "CFCO"."ACCT_NO" LIKE UPPER(:4||'%')))
10 - access("CPDE"."CPDE_ID"="CFCO"."CPDE_ID")
10 - filter("CFCO"."CPDE_ID" IS NOT NULL)
Content of other_xml column
===========================
  db_version     : 11.2.0.3
  parse_schema   : CARADM
  plan_hash      : 3311216257
  plan_hash_2    : 4009937738
  Outline Data:
  /*+
    BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('10.2.0.3')
      DB_VERSION('11.2.0.3')
      OPT_PARAM('_always_anti_join' 'off')
      OPT_PARAM('_always_semi_join' 'off')
      OPT_PARAM('_index_join_enabled' 'false')
      OPT_PARAM('_optim_peek_user_binds' 'false')
      OPT_PARAM('_optimizer_skip_scan_enabled' 'false')
      OPT_PARAM('optimizer_index_cost_adj' 1)
      OPT_PARAM('optimizer_index_caching' 100)
      FIRST_ROWS
      OUTLINE_LEAF(@"SEL$EE94F965")
      MERGE(@"SEL$9E43CB6E")
      OUTLINE(@"SEL$4")
      OUTLINE(@"SEL$9E43CB6E")
      MERGE(@"SEL$58A6D7F6")
      OUTLINE(@"SEL$3")
      OUTLINE(@"SEL$58A6D7F6")
      MERGE(@"SEL$1")
      OUTLINE(@"SEL$2")
      OUTLINE(@"SEL$1")
      FULL(@"SEL$EE94F965" "CPGR"@"SEL$1")
      INDEX(@"SEL$EE94F965" "CPGP"@"SEL$1" ("CL_PROVIDER_GROUP_PERIOD"."CPGR_ID" "CL_PROVIDER_GROUP_PERIOD"."CPDE_ID" "CL_PROVIDER_GROUP_PERIOD"."STA
RT_DATE"))
      INDEX_RS_ASC(@"SEL$EE94F965" "CPDE"@"SEL$2" ("CL_PROVIDER_DETAIL"."CPDE_ID"))
      INDEX_RS_ASC(@"SEL$EE94F965" "CFCO"@"SEL$3" ("FN_ACCOUNT"."CPDE_ID"))
      LEADING(@"SEL$EE94F965" "CPGR"@"SEL$1" "CPGP"@"SEL$1" "CPDE"@"SEL$2" "CFCO"@"SEL$3")
      USE_NL(@"SEL$EE94F965" "CPGP"@"SEL$1")
      USE_NL(@"SEL$EE94F965" "CPDE"@"SEL$2")
      USE_NL(@"SEL$EE94F965" "CFCO"@"SEL$3")
    END_OUTLINE_DATA
------- END SPM Plan Dump -------
SPM: plan reproducibility round 2 (hinted OFE only)
SPM: using qksan to reproduce accepted plan, planId = 386463207
SPM: planId in plan baseline = 386463207, planId of reproduced plan = 4009937738
------- START SPM Plan Dump -------
SPM: failed to reproduce the plan using the following info:
  parse_schema name        : CARADM
  plan_baseline signature  : 17844158604836855983
  plan_baseline plan_id    : 386463207
  plan_baseline hintset    :
    hint num  1 len 37 text: OPTIMIZER_FEATURES_ENABLE('10.2.0.3')
SPM: generated non-matching plan:
----- Explain Plan Dump -----
----- Plan Table -----
============
Plan Table
============
------------------------------------------------------------+-----------------------------------+
| Id  | Operation                       | Name              | Rows  | Bytes | Cost  | Time      |
------------------------------------------------------------+-----------------------------------+
| 0   | SELECT STATEMENT                |                   |       |       |   179 |           |
| 1   |  SORT UNIQUE                    |                   |   153 |   11K |   178 |  00:00:01 |
| 2   |   NESTED LOOPS                  |                   |   153 |   11K |   177 |  00:00:01 |
| 3   |    NESTED LOOPS                 |                   |   826 |   50K |    11 |  00:00:01 |
| 4   |     NESTED LOOPS                |                   |   826 |   26K |     3 |  00:00:01 |
| 5   |      TABLE ACCESS FULL          | CL_PROVIDER_GROUP |   116 |  2668 |     2 |  00:00:01 |
| 6   |      INDEX RANGE SCAN           | CPGP_UK           |     7 |    63 |     1 |  00:00:01 |
| 7   |     TABLE ACCESS BY INDEX ROWID | CL_PROVIDER_DETAIL|     1 |    30 |     1 |  00:00:01 |
| 8   |      INDEX UNIQUE SCAN          | CPDE_PK           |     1 |       |     1 |  00:00:01 |
| 9   |    TABLE ACCESS BY INDEX ROWID  | FN_ACCOUNT        |     1 |    14 |     1 |  00:00:01 |
| 10  |     INDEX RANGE SCAN            | FACC_CPDE_FK_I    |    27 |       |     1 |  00:00:01 |
------------------------------------------------------------+-----------------------------------+
Predicate Information:
6 - access("CPGR"."CPGR_ID"="CPGP"."CPGR_ID")
8 - access("CPDE"."CPDE_ID"="CPGP"."CPDE_ID")
9 - filter((UPPER("CPGR"."NAME") LIKE UPPER('%'||:1||'%') OR UPPER("CPDE"."NAME") LIKE UPPER('%'||:2||'%') OR UPPER("CPDE"."ABN") LIKE UPPER(:3||'%')
OR "CFCO"."ACCT_NO" LIKE UPPER(:4||'%')))
10 - access("CPDE"."CPDE_ID"="CFCO"."CPDE_ID")
10 - filter("CFCO"."CPDE_ID" IS NOT NULL)
Content of other_xml column
===========================
  db_version     : 11.2.0.3
  parse_schema   : CARADM
  plan_hash      : 3311216257
  plan_hash_2    : 4009937738
  Outline Data:
  /*+
    BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('10.2.0.3')
      DB_VERSION('11.2.0.3')
      OPT_PARAM('optimizer_index_cost_adj' 1)
      OPT_PARAM('optimizer_index_caching' 100)
      FIRST_ROWS
      OUTLINE_LEAF(@"SEL$EE94F965")
      MERGE(@"SEL$9E43CB6E")
      OUTLINE(@"SEL$4")
      OUTLINE(@"SEL$9E43CB6E")
      MERGE(@"SEL$58A6D7F6")
      OUTLINE(@"SEL$3")
      OUTLINE(@"SEL$58A6D7F6")
      MERGE(@"SEL$1")
      OUTLINE(@"SEL$2")
      OUTLINE(@"SEL$1")
      FULL(@"SEL$EE94F965" "CPGR"@"SEL$1")
      INDEX(@"SEL$EE94F965" "CPGP"@"SEL$1" ("CL_PROVIDER_GROUP_PERIOD"."CPGR_ID" "CL_PROVIDER_GROUP_PERIOD"."CPDE_ID" "CL_PROVIDER_GROUP_PERIOD"."STA
RT_DATE"))
      INDEX_RS_ASC(@"SEL$EE94F965" "CPDE"@"SEL$2" ("CL_PROVIDER_DETAIL"."CPDE_ID"))
      INDEX_RS_ASC(@"SEL$EE94F965" "CFCO"@"SEL$3" ("FN_ACCOUNT"."CPDE_ID"))
      LEADING(@"SEL$EE94F965" "CPGR"@"SEL$1" "CPGP"@"SEL$1" "CPDE"@"SEL$2" "CFCO"@"SEL$3")
      USE_NL(@"SEL$EE94F965" "CPGP"@"SEL$1")
      USE_NL(@"SEL$EE94F965" "CPDE"@"SEL$2")
      USE_NL(@"SEL$EE94F965" "CFCO"@"SEL$3")
    END_OUTLINE_DATA
------- END SPM Plan Dump -------
SPM: couldn't reproduce any enabled+accepted plan so using the cost-based plan, planId = 3838063866
SPM: kkopmCheckSmbUpdate (enter) xscP=0x9fffffff7f3c8cc8, pmExCtx=0xc000000c8bbade48, ciP=0xc000000c872f95c0, dtCtx=0x6000000000054c30apologies once again for the huge post.