Same query, same dataset, same ddl setup, but wildly different explain plan

Question

Hello o fountains of oracle knowledge!
We have a problem that caused us a full stop when rolling out a new version of our system to a customer and a whole Sunday to boot.
The scenario is as follows:
1. An previous version database schema
2. The current version database schema
3. A migration script to migrate the old schema to the new
So we perform the following migration:
1. Export the previous version database schema
2. Import into a new schema called schema_old
3. Create a new schema called schema_new
4. Run migration script which creates objects, copies data, creates indexes etc etc in schema_new
The migration runs fine in all environments (development, test and production)
In our development and test environments performance is stellar, on the customer production server the performance is terrible.
This using the exact same export file (from the production environment) and performing the exact same steps with the exact same migration script.
Database version is 10.2.0.1.0 EE on all databases. OS is Microsoft Windows Server 2003 EE SP2 on all servers.
The system is not in any sense under a heavy load (we have tested with no other load than ourselves).
Looking at the explain plan for a query that is run frequently and does not use bind variables we see wildly different explain plans.
The explain plan cost on our development and test servers is estimated to *7* for this query and there are no full table scans.
On the production server the cost is *8433* and there are two full table scans of which one is on the largest table.
We have tried to run analyse on all objects with very little effect. The plan changed very slightly, but still includes the two full table scans on the problem server and the cost is still the same.
All tables and indexes are identical (including storage options), created from the same migration script.
I am currently at loss for where to look? What can be causing this? I assume this could be caused by some parameter that is set on the server, but I don't know what to look for.
I would be very grateful for any pointers.
Thanks,
Håkon

Answer 1

Thank you for your answer.
We collected statistics only after we determined that the production server where not behaving according to expectations.
In this case we used TOAD and the tool within to collect statistics for all objects. We used 'Analyze' and 'Compute Statistics' options.
I am not an expert, so sorry if this is too naive an approach.
Here is the query:SELECT count(0)  
FROM score_result sr, web_scorecard sc, product p
WHERE sr.score_final_decision like 'VENT%'  
AND sc.CREDIT_APPLICATION_ID = sr.CREDIT_APPLICATION_ID  
AND sc.application_complete='Y'   
AND p.product = sc.web_product   
AND p.inactive_product = '2' ;I use this as an example, but the problem exists for virtually all queries.
The output from the 'good' server:
| Id  | Operation                      | Name                  | Rows  | Bytes | Cost (%CPU)|
|   0 | SELECT STATEMENT               |                       |     1 |    39 |     7   (0)|
|   1 |  SORT AGGREGATE                |                       |     1 |    39 |            |
|   2 |   NESTED LOOPS                 |                       |     1 |    39 |     7   (0)|
|   3 |    NESTED LOOPS                |                       |     1 |    30 |     6   (0)|
|   4 |     TABLE ACCESS BY INDEX ROWID| SCORE_RESULT          |     1 |    17 |     4   (0)|
|   5 |      INDEX RANGE SCAN          | SR_FINAL_DECISION_IDX |     1 |       |     3   (0)|
|   6 |     TABLE ACCESS BY INDEX ROWID| WEB_SCORECARD         |     1 |    13 |     2   (0)|
|   7 |      INDEX UNIQUE SCAN         | WEB_SCORECARD_PK      |     1 |       |     1   (0)|
|   8 |    TABLE ACCESS BY INDEX ROWID | PRODUCT               |     1 |     9 |     1   (0)|
|   9 |     INDEX UNIQUE SCAN          | PK_PRODUCT            |     1 |       |     0   (0)|
---------------------------------------------------------------------------------------------The output from the 'bad' server:
| Id  | Operation                 | Name                  | Rows  | Bytes | Cost (%CPU)|
|   0 | SELECT STATEMENT          |                       |     1 |    32 |  8344   (3)|
|   1 |  SORT AGGREGATE           |                       |     1 |    32 |            |
|   2 |   HASH JOIN               |                       | 10887 |   340K|  8344   (3)|
|   3 |    TABLE ACCESS FULL      | PRODUCT               |     6 |    42 |     3   (0)|
|   4 |    HASH JOIN              |                       | 34381 |   839K|  8340   (3)|
|   5 |     VIEW                  | index$_join$_001      | 34381 |   503K|  2193   (3)|
|   6 |      HASH JOIN            |                       |       |       |            |
|   7 |       INDEX RANGE SCAN    | SR_FINAL_DECISION_IDX | 34381 |   503K|   280   (3)|
|   8 |       INDEX FAST FULL SCAN| SCORE_RESULT_PK       | 34381 |   503K|  1371   (2)|
|   9 |     TABLE ACCESS FULL     | WEB_SCORECARD         |   489K|  4782K|  6137   (4)|
----------------------------------------------------------------------------------------I hope the formatting makes this readable.
Stats (from SQL Developer), good table:NUM_ROWS     489716
BLOCKS     27198
AVG_ROW_LEN     312
SAMPLE_SIZE     489716
LAST_ANALYZED     15.12.2009
LAST_ANALYZED_SINCE     15.12.2009Stats (from SQL Developer), bad table:
NUM_ROWS     489716
BLOCKS     27199
AVG_ROW_LEN     395
SAMPLE_SIZE     489716
LAST_ANALYZED     17.12.2009
LAST_ANALYZED_SINCE     17.12.2009I'm unsure what would cause the difference in average row length.
I could obviously try to tune our sql-statements to work on the server not behaving better, but I would rather understand why they are different and make sure that we can expect similar behaviour between environments.
Thank you again for trying to help me.
Håkon
Edited by: ergates on 17.des.2009 05:57
Edited by: ergates on 17.des.2009 06:02