Why was RBO used?

Question

Oracle 11.1.0.7.
spfile - optimizer_mode = ALL_ROWS. optimizer_features_enable = 11.1.0.7.
Auto stats gathering jobs are all enabled. So everything should be all CBO all the way.
Some queries were mysteriously slow on Friday. Digging into AWR stats finally revealed that a SQL statement (sql_id) had spawned a plan_hash_value that was the culprit. It was doing a full table scan on a large table and consuming a lot of CPU/IO. Dumping out everything AWR knows about this SQL using
select * from table(dbms_xplan.display_awr('45knuy45ha5f2',NULL,NULL,'ADVANCED'))Pored over the output and sure enough, the "bad" plan_hash_value was using the FTS and the Note in the output said "rule based optimizer used (consider using cbo)".
I couldn't believe my eyes. Why in the world did that one child cursor use the RBO?
Any ideas?

Answer 1

Karthick - No. The query is coming from a application so the query and session parameters are fixed. As AWR shows, the query was executed dozens of times and all the plan_hash_values for the same sql_id use CBO, just this one bad apple used the RBO. Why?
Hoek - Yes, I read all of Kerry Osborne's excellent blog entries, that's how I narrowed down the problem to this particular sql_id. But it doesn't talk about WHY a particular query would use the RBO. The most common culprit, bind variable peeking, may cause the plan to flip between 2 CBO plans but it shouldn't cause the RBO to be used.
Martin - That's what I thought too but all objects used in the query do have stats. Since the Oracle auto-stats job/task is running daily, stats are always up-to-date. Here are all the optimizer* parameters in the spfile.
This is really frustrating.
optimizer_features_enable     11.1.0.7
optimizer_mode     ALL_ROWS
optimizer_index_cost_adj     100
optimizer_index_caching     0
optimizer_dynamic_sampling     2
optimizer_secure_view_merging     TRUE
optimizer_use_pending_statistics     FALSE
optimizer_capture_sql_plan_baselines     FALSE
optimizer_use_sql_plan_baselines     TRUE
optimizer_use_invisible_indexes     FALSE[Oracle Support is looking into it but they usually ask for a 10046 trace and we can't provide that because AWR doesn't capture that and the issue is not reproducible at-will]