Testing Process for Gathering Single Object stats.

Question

Hello Oracle Experts,
I work a critical system and due to some high stakes all and every change is very heavily scrutinized here whatever the level is. And one of such changes which is currently under scrutiny is gathering object stats for single objects. Just to give you a background its an Oracle eBusiness site so fnd_stats is used instead of usual dbms_stats and we've an inhouse job that depending on the staleness of the objects gather stats on them using FND_STATS. (RDBMS : 10.2.0.4 Apps Release 12i).
Now, we've seen that occasionally it leaves some of the objects that should ideally be gathered so they need to be gathered individually and our senior technical management wants a process around it - for gathering this single object stats (I know!). I think I need to explicitly mention here that this need to gather stale object stats has emerged becs one of the plans has gone pretty poor (from 2 ms to 90 mins) and sql tuning task states that stats are stale and in our PROD copy env (where the issue exists) gathering stats reverts to original good plan! So we are not gathering just because they are stale but instead because that staleness is actually causing a realtime problem!
Anyway, my point is that it has been gathered multiple times in the past on that object and also it might get gathered anytime by that automatic job (run nightly). There arguments are:
i. There may be several hundred sql plans depending on that object and we never know how many, and to what, those plan change and it can change for worse causing unexpected issues in the service!
ii. There may be related objects whose objects have gone stale as well (for example sales and inventory tables both see related amount of changes on column stock_level) and if we gather stats only on one of them and since those 2 cud be highly related (in queries etc.) that may mess up the join cardinality etc. messing up the plans etc.
Now, you see they know Oracle as well !
My Oracle (and optimizer knowledge) clearly suggests me that these arguments are baseless BUT want to keep an open mind. So my questions are :
i. Do the risks highlighted above stand any ground or what probably do you think is there of happening any of the above?
ii. Any other point that I can make to convince the management.
iii. Or if those guys are right, Do you guys use or recommend any testing strategy/process that you can suggest to us pls?
Another interesting point is that, they are not even very clear at this stage how they are gonna 'test' this whole thing as the 'cost' option like RAT (Real Application Testing) is out of question and developing an inhouse testing tool still need analyzing in terms of efforts, worth and reliability.
In the end, Can I request top experts from the 'Oak Table' network to make a comment so that I can take their backings!? Well I am hoping here they'll back me up but that may not necessarily the case and I obviously want an honest expert assessment of the situation and not merely my backing.
Thanks so much in advance!

Answer 1

>
I work a critical system and due to some high stakes all and every change is very heavily scrutinized here whatever the level is.
Another interesting point is that, they are not even very clear at this stage how they are gonna 'test' this whole thing as the 'cost' option like RAT (Real Application Testing) is out of question and developing an inhouse testing tool still need analyzing in terms of efforts, worth and reliability.Unfortunately your management's opinion of their system as expressed in the first paragraph is not consistent with the opinion expressed in the second paragraph.
Getting a stable strategy for statistics is not easy, requires careful analysis, and takes a lot of effort for complex systems.
>
In the end, Can I request top experts from the 'Oak Table' network to make a comment so that I can take their backings!? Well I am hoping here they'll back me up but that may not necessarily the case and I obviously want an honest expert assessment of the situation and not merely my backing.
The ideal with stats collection is to do something simple to start with, and then build on the complex bits that are needed - something along the lines suggested by Dan Morgan works: a table driven approach to deal with the special cases which are usually: the extreme indexes, the flag columns, the time-based/sequential columns, the occasional histogram, and new partitions. Unfortunately you can't get from where you are to where you need to be without some risk (after all, you don't know which bits of your current strategy are causing problems).
You may have to progress by letting mistakes happen - in other words, when some very bad plans show up, work out WHY they were bad (missing histogram, excess histogram, out of date high values) to work out the minimum necessary fix. Put a defensive measure in place (add it to the table of special cases) and run with it.
As a direction to aim at - I avoid histograms unless really necessary, I like introducing function-based indexes where possible, and I'm perfectly happy to write small programs to fix columns stats (low/high/distinct) or index stats (clustering_factor/blevel/distinct_keys) and create static histograms.
Remember that Oracle saves old statistics when you create new ones, so any new stats that cause problems can be reversed out very promptly.
Regards
Jonathan Lewis