CBO not picking correct indexes or doing Full Scans

Question

Database version - 10.2.0.4
OS : Solaris 5.8
Storage: SAN
Application : PeopleSoft Financials
DB size : 450 gb
DB server : 12 CPU ( 900 Mghz each ), 36 GB RAM
ASMM - sga_target_size = 5 gb
Locally managed tablespaces - MANUAL
- db_file_multiblock_read_count - not set explicitly in spfile ( implicitly defaulted to 128 )
- other optimizer related parameters are set to default
- system_statistics - CPUSPEEDNW=456.282722513089, IOSEEKTIM =10, IOTFRSPEED=4096     
- dictionary object system stats were last gather in Nov 09
- stats on schema objs are gathered every night using custom script, but I have to say there are some histograms on some tables which were gathered by PS admins
begin
dbms_stats.gather_schema_stats(
ownname=> 'SCHEMANM' ,
cascade=> DBMS_STATS.AUTO_CASCADE,
estimate_percent=> DBMS_STATS.AUTO_SAMPLE_SIZE,
degree=> 10,
no_invalidate=> DBMS_STATS.AUTO_INVALIDATE,
granularity=> 'AUTO',
method_opt=> 'FOR ALL COLUMNS SIZE 1',
options=> 'GATHER STALE');
end;
Details :
We are experiencing erratic database performance. It was upgraded from 9i to 10g along with PS software upgrade. A job that runs on one day in 12 hrs(that is high in itself) would take more than 24 hrs another day.
We have Test and Staging envs on other servers which do not have performance issues of production magnitude. The test, staging dbs are clones of prod, created by ftp'ing files from one location to another, not sure if that makes any difference but just pointing out.
On Prod db some symptoms which baffle me are :
1) sql executing for over 40 minutes, check the plan and it is using an "incorrect" index, idx1. Hint sql with "correct" index, idx2, and it runs in few seconds. result set <400 rows. Cost with hint with idx2 is HIGHER. This scenario is still understandable as CBO is likely to pick a lower cost.
- But why is there so much discrepancy in cost and response time?
- what is influencing the CBO to pick an index which is obviously not the fastest response time?
2) sql plan shows FTS , execution time , runs forever . But a hint with index in this case shows the cost is LOWER and response time is in seconds, so why is CBO not even evaluating this index path? Because as I understand the CBO, it will always pick the lower cost plan.
What is influencing the CBO. Is it system stats? or any other database parameter? or the hardware ? the large SGA?? histogram ?? Stats gathering?? is there is any known issue with DBMS_STATS.AUTO_SAMPLE_SIZE and cardinaility?
Where should I put my focus?
What am I looking for ?
I do have Jonathan Lewis's Cost-Based Oracle Fundamentals which I have just started so perhaps I will be enlightened but while I read it, it would be of immense help to get some advice from forum gurus.
At this time I am not posting any exec plan or code, but I can do so if required.
Thanks,
JC

Answer 1

Picking up just a couple of points - the ones about: why can performance change so much from one day to the next, and how can the optimizer miss a plan which (when hinted) shows a lower cost.
NOTE: These are only suggestions that may allow you to look at critical code and recognise the pattern of behaviour
Performance change:
You are using "gather stale" which means tables won't get new statistics is the volume of change since the last collection is "small". But peoplesoft tables can be big, so some tables may need a lot of data to change (which might take several days or weeks) before they get new stats. The 10g optimizer is able to compare the high-values stored with the predicate values supplied in queries, and decide that since the predicate is higher than the known highest value, it should scale down its estimates of data returned. After a few days when the stats haven't changed, the optimizer suddenly gets to the point where the estimated data size is much too small and the plan changes to something that you can see is bad. The typical example here is a date column that increases with time, and a query on "recent data" - the optimizer basically says: "you want 9th Feb - the high value says 28th Jan, there's virtually no data". Keeping date and sequence-based high-values up to date is important. (See dbms_stats.set_column_stats).
Lower cost plans available:
Sometimes this is a bug. Sometimes it's an unlucky side effect. When a query has a large number of tables in a single query block, the optimizer can only examine a limited subset of the possible join orders (for example, a 10-table join has over 3million join orders, and the default limit is 2,000 checked). Because of this limit, Oracle starts working through a few joins orders in a methodical manner, then does a "big jump" to a different set of join orders where it checks a few more join orders, then does another "big jump". It may be that it never gets to your nice plan with the lower cost because it doesn't have time. However, by putting in the hint, you may make some of the orders the optimizer would have examined drop out of the "methodical list" - so it examines a few more in one sequence, or possibly "jumps" to a different sequence that it would not othewise have reached. (I wrote a related blog item some time ago that might make this clearer, even though it's not about exactly the same phenomenon: http://jonathanlewis.wordpress.com/2006/11/03/table-order/ ).
Your recent comment about dynamic sampling is correct, by the way - by default (which for your version is is level 2 as a system parameter) it will only apply for tables which don't have any statistics. It's only for higher levels, or when explicitly hinted for tables, that it can apply to tables with existing statistics.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
"Science is more than a body of knowledge; it is a way of thinking"
Carl Sagan
To post code, statspack/AWR report, execution plans or trace files, start and end the section with the tag {noformat}{noformat} (lowercase, curly brackets, no spaces) so that the text appears in fixed format.