Strange CBO choice

Question

Hi guys,
I'm runing some tests to check weather processed_flag should have NULL for already processed values or not. I thought that having a smaller index on the processed_flag column should be better.
Check the following example:
CREATE TABLE processed_flag_not_null AS
SELECT LEVEL id,
       lpad('A', 10, 'A') VALUE,
       CASE
         WHEN MOD(LEVEL, 100) = 0 THEN
          0
         ELSE
          1
       END processed_flag
  FROM dual
CONNECT BY LEVEL <= 1000000;
CREATE INDEX IDX_PROCESSED_FLAG_NOT_NULL ON PROCESSED_FLAG_NOT_NULL (PROCESSED_FLAG);
CREATE TABLE processed_flag_null AS
SELECT LEVEL id,
       lpad('A', 10, 'A') VALUE,
       CASE
         WHEN MOD(LEVEL, 100) = 0 THEN
          0
       END processed_flag
  FROM dual
CONNECT BY LEVEL <= 1000000;
CREATE INDEX IDX_PROCESSED_FLAG_NULL ON PROCESSED_FLAG_NULL (PROCESSED_FLAG);
BEGIN
  dbms_stats.gather_table_stats(USER, 'PROCESSED_FLAG_NOT_NULL', cascade => TRUE);
  dbms_stats.gather_table_stats(USER, 'PROCESSED_FLAG_NULL', cascade => TRUE);
END;The first strange thing I've found was in all_tab_histograms:
SELECT *
  FROM all_tab_histograms
WHERE table_name IN ('PROCESSED_FLAG_NOT_NULL', 'PROCESSED_FLAG_NULL')
   AND column_name = 'PROCESSED_FLAG'
ORDER BY 2,4;
TABLE_NAME     COLUMN_NAME     ENDPOINT_NUMBER     ENDPOINT_VALUE
PROCESSED_FLAG_NOT_NULL     PROCESSED_FLAG     0     0
PROCESSED_FLAG_NOT_NULL     PROCESSED_FLAG     1     1
PROCESSED_FLAG_NULL     PROCESSED_FLAG     0     0
PROCESSED_FLAG_NULL     PROCESSED_FLAG     1     0Only after running queries against both tables with ("where processed_flag = 0") and gathering table stats again I get the following histograms:
TABLE_NAME                     COLUMN_NAME     ENDPOINT_NUMBER     ENDPOINT_VALUE
PROCESSED_FLAG_NOT_NULL     PROCESSED_FLAG     59                     0
PROCESSED_FLAG_NOT_NULL     PROCESSED_FLAG     5598                     1
PROCESSED_FLAG_NULL     PROCESSED_FLAG     10000                     0Can someone explain to me this behaviour?
Now for the main question, I don't understand why the CBO chooses an Index Range Scan for the PROCESSED_FLAG_NOT_NULL table, and a Full Table Scan for the PROCESSED_FLAG_NULL:
SQL> set timing on
SQL> set autotrace traceonly
SQL> SELECT *
  2    FROM processed_flag_not_null
  3   WHERE processed_flag = 0;
10000 rows selected.
Elapsed: 00:00:00.20
Execution Plan
Plan hash value: 3652560023
| Id  | Operation                   | Name                        | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT            |                             | 10539 |   195K|    93   (0)| 00:00:02 |
|   1 |  TABLE ACCESS BY INDEX ROWID| PROCESSED_FLAG_NOT_NULL     | 10539 |   195K|    93   (0)| 00:00:02 |
|*  2 |   INDEX RANGE SCAN          | IDX_PROCESSED_FLAG_NOT_NULL | 10539 |  |    23   (0)| 00:00:01 |
Predicate Information (identified by operation id):
   2 - access("PROCESSED_FLAG"=0)
Statistics
          1  recursive calls
          0  db block gets
       4444  consistent gets
          0  physical reads
          0  redo size
     306954  bytes sent via SQL*Net to client
       7745  bytes received via SQL*Net from client
        668  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
      10000  rows processed
SQL> SELECT *
  2    FROM processed_flag_null
  3   WHERE processed_flag = 0;
10000 rows selected.
Elapsed: 00:00:00.22
Execution Plan
Plan hash value: 1150676937
| Id  | Operation         | Name                | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT  |                     | 10000 |   166K|   802   (2)| 00:00:10 |
|*  1 |  TABLE ACCESS FULL| PROCESSED_FLAG_NULL | 10000 |   166K|   802   (2)| 00:00:10 |
Predicate Information (identified by operation id):
   1 - filter("PROCESSED_FLAG"=0)
Statistics
          1  recursive calls
          0  db block gets
       3571  consistent gets
          0  physical reads
          0  redo size
     174974  bytes sent via SQL*Net to client
       7745  bytes received via SQL*Net from client
        668  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
      10000  rows processedIf I compare both aproaches using Runstats:
BEGIN
  runstats_pkg.rs_start;
  BEGIN
    FOR i IN 1 .. 1000
    LOOP
      FOR x IN (SELECT *
                  FROM processed_flag_not_null
                 WHERE processed_flag = 0)
      LOOP
        NULL;
      END LOOP;
    END LOOP;
  END;
  runstats_pkg.rs_middle;
  BEGIN
    FOR i IN 1 .. 1000
    LOOP
      FOR x IN (SELECT *
                  FROM processed_flag_null
                 WHERE processed_flag = 0)
      LOOP
        NULL;
      END LOOP;
    END LOOP;
  END;
  runstats_pkg.rs_stop;
END;
-- Output
Run1 ran in 4295 hsecs
Run2 ran in 5123 hsecs
run 1 ran in 83.84% of the timeIf I compare the processed_flag_null without hints versus processed_flag_null with index hint this are the results:
BEGIN
  runstats_pkg.rs_start;
  BEGIN
    FOR i IN 1 .. 1000
    LOOP
      FOR x IN (SELECT *
                  FROM processed_flag_null
                 WHERE processed_flag = 0)
      LOOP
        NULL;
      END LOOP;
    END LOOP;
  END;
  runstats_pkg.rs_middle;
  BEGIN
    FOR i IN 1 .. 1000
    LOOP
      FOR x IN (SELECT /*+ index(processed_flag_null IDX_PROCESSED_FLAG_NULL) */
                  FROM processed_flag_null
                 WHERE processed_flag = 0)
      LOOP
        NULL;
      END LOOP;
    END LOOP;
  END;
  runstats_pkg.rs_stop;
END;
-- Output
Run1 ran in 5017 hsecs
Run2 ran in 2212 hsecs
run 1 ran in 226.81% of the timeAs I expected using the hint is more than twices fast of not using the index, why doesn't the CBO choose the index aproach? Can I tune the stats to give the CBO more information?
Thanks in advance,
Manuel Vidigal
EDIT:
Forgot to mention the tests were done using my laptop with Oracle 11.2.0.1.0.
Edited by: Manuel Vidigal on 16/Set/2010 11:53

Answer 1

I don't have runstats installed on my 11.2 instance so i can't reproduce your findings there, however with a simple SET TIMING ON and observing the results i get no noticeable difference between the INDEX hinted query and the not hinted query (i have an isolated sandbox where i am on the only user on the server so this is a reasonably safe method in my opinion).
TUBBY_TUBBZ?
BEGIN
FOR i IN 1 .. 1000
LOOP
   FOR x IN (SELECT *
               FROM processed_flag_null
              WHERE processed_flag = 0)
   LOOP
     NULL;
   END LOOP;
END LOOP;
END;
12  /
PL/SQL procedure successfully completed.
Elapsed: 00:00:51.26
BEGIN
FOR i IN 1 .. 1000
LOOP
   FOR x IN (SELECT /*+ index(processed_flag_null IDX_PROCESSED_FLAG_NULL) */
               FROM processed_flag_null
              WHERE processed_flag = 0)
   LOOP
     NULL;
   END LOOP;
END LOOP;
END;
13  /
PL/SQL procedure successfully completed.
Elapsed: 00:00:50.53The results you are seeing are a consequence of the tiny nature of your table. The way you have set it up you should be seeing roughly 300 rows fitting on to a single data block (i observed 325 on my instance running on ASM) and you are only interested in returning every 100th row (again the way you set up your data) but to get every 100th row means you will have to visit EVERY block in the table.
select
   blocks
from dba_segments
where segment_name = 'PROCESSED_FLAG_NULL';
PL/SQL procedure successfully completed.
Elapsed: 00:00:04.51
TUBBY_TUBBZ?
            BLOCKS
              3072
1 row selected.
Elapsed: 00:00:00.13Since you haven't ordered your data it's going to be inserted as the rows are generated from DUAL (so ordered by LEVEL asc). That means that you will need to get every block from the table (getting roughly 3 results per block).
In order to visit every block in the table via an index access you cannot utilize multi-block IO, so that would be the reason the optimizer took the FULL table scan as it's choice.
So in terms of 'fixing' this you have a couple of options.
1) change the order of the data so it's not so evenly distributed across the blocks in the table (or possibly create this table as an index organized one so the values are sorted upon insertion and kept physically close to each other)
2) change the size of a row such that less rows will fit on a single data block meaning that you will not actually have to return every data block for this table via your query
lpad('A', 1000, 'A') VALUE,Would be a sufficient change to shift things in favor of index access in your example.