Estimating range of performance for a query

Question

I wonder if this is an intractable problem:
Using information available from the data dictionary and from the tables referenced in a query, estimate actual minimum and maximum run times for a given query, predict performance for various combinations of realistic values for the variables that influence performance.
Variables would include:
what kinds of IO are used
how much of each type of IO is used
atpr - average time per read for various types of IO
data relationships - min/max/avg #child records per parent record
server caching - how many gets require IO
clustering factor of indexes
I think a related item would be client caching
continued rows - how many per N primary rows
Type of plan - initally I think perhaps all NL and all hash joins are simple starting points
Some of the variables are observable from target systems ( atpr, data relationships, clustering factor, .. ).
I know, the optimizer already does this.
I know, it's faster to just run test queries.
Repeated work with such ideas would cause refinement of the method and
eventually might result in reasonably accurate estimates and improved understanding
of the variables involved in query performance, the latter being more important
than the former.
Please tell me why this is a bad or good idea. I already mentioned a couple of counter-arguments above,
please don't bother elaborating on them unless you're shedding light and not flaming. I think this would be
somewhat like the index evaluation methods in Lahdenmaki/Leach's book, but more Oracle-centric.
Or maybe I'm kidding myself..

Answer 1

Justin Cave wrote:
Could you possibly invert the problem a bit and use some statistical process control techniques to set some baselines and to identify statements that are moving outside of their control limits?
For example, a control chart could be built for each SQL statement based on actual execution performance data in the AWR-- you just need to know the average and the standard deviation execution time for that which should be relatively easy to derive. You can then gather the performance data for every snapshot interval, add a new data point to the chart. There are a number of different sets of rules for determining a "signal" from this data as opposed to normal random variation. That would generally be a reasonable way for everyone to agree on what performance should really be expected for a SQL statement and it would be a good early warning system that "something has changed" when you see a particular query start to run consistently slower (or faster) than it had been. That, in turn, might lead to better discussions, i.e. "sql_id Y is starting to run more slowly than we were expecting based on prior testing because we just introduced Process X that is generating a pile of I/O in the same window your query runs in. We can adjust Y's baseline to incorporate this new reality. Or we can move when Y runs so that it isn't competing with X. Or we can try to optimize Y further. Or we can get the folks that own X and Y into a room and determine how to meet everyone's requirements". Particularly if your performance testing can identify issues in test before the new Process X code goes into prod.
JustinThose are interesting ideas. Better discussions would be a good thing.
Re inverting the problem from prediction to reaction:
I have done some work with the script at http://kerryosborne.oracle-guy.com/2008/10/unstable-plans/ which of course work only for as much AWR data as you keep. I've had mixed results. I haven't tried to set it up to alert me about problems or to monitor a specific set of sql_ids. I've found it to be useful when users/developers are complaining about general slowness but won't give you any useful details about what is slow.
Here are a few complicating factors re identifying significant divergences in query performance or resource use - There are queries that accept different inputs that rightly generate completely different workloads for the same query, e.g., a product/location/time query whose inputs allow wide variations in the selectivity for each of the dimensions. There are applications that never heard of a bind variable, and there are queries that rightly do not use bind variables ( yes, even in the age of sql injection ).
In general , aside from the usual Grid Control and Nagios alerts re CPU/Memory/IO thresholds, and some blocking-locks alert programs, it's up to our users/developers to report performance problems.
Re my original question - I'll admit I was pretty sleep deprived when I wrote it. Sleep deprivation isn't usually conducive to clear thinking, so it will be interesting to see how this all looks in a month. Still, given that so much testing is affected by previous test runs ( caching ), I thought it made sense to try to understand the worst-performing case for a given execution plan. It's not a good thing to find that the big query that was tested to death and gave users a certain expectation of run time runs in production on a system where the caches have been dominated for the last N hours by completely unrelated workloads, and that when the query runs every single block must be read from a spindle, giving the users a much slower query than was seen in test.
Maybe the best way to predict worst-case performance is to work from the v$ metrics from test, manipulating the metrics to simulate different amounts of caching and different IO rates, thus generating different estimates for run-time.