Why does not my query use an index?

Question

I have a table with some processed rows (state: 9) and some unprocessed rows (states: 0,1,2,3,4).
This table has over 120000 rows, but this number will grow.
Most of the rows are processed and most of them also contain a group id. Number of groups is relatively small (let's assume 20).
I would like to obtain the oldest some_date for every group. This values has to be outer joined to a on-line report (contains one row for each group).
Here is my set-up:
Tested on: 10.2.0.4 (Solaris), 10.2.0.1 (WinXp)
drop table t purge;
create table t(
  id number not null primary key,
  grp_id number,
  state number,
  some_date date,
  pad char(200)
insert into t(id, grp_id, state, some_date, pad)
select level,
     trunc(dbms_random.value(0,20)),
        9,
        sysdate+dbms_random.value(1,100),
        'x'
from dual
connect by level <= 120000;
insert into t(id, grp_id, state, some_date, pad)
select level + 120000,
     trunc(dbms_random.value(0,20)),
        trunc(dbms_random.value(0,5)),
        sysdate+dbms_random.value(1,100),
        'x'
from dual
connect by level <= 2000;
commit;
exec dbms_stats.gather_table_stats(user, 'T', estimate_percent=>100, method_opt=>'FOR ALL COLUMNS SIZE 1');
Tom Kyte's printtab
==============================
TABLE_NAME                    : T
NUM_ROWS                      : 122000
BLOCKS                        : 3834I know, this could be easily solved by fast refresh on commit materialized view like this:
select
  grp_id,
  min(some_date),
from
  t
where
  state in (0,1,2,3,4)
grpup by
  grp_id;+ I have to create log on (grp_id, some_date, state)
Number of rows with active state will be always relatively small. Let's assume 1000-2000.
So my another idea was to create a selective index. An index which would contain only data for rows with an active state.
Something like this:
create index fidx_active on t ( 
  case state 
    when 0 then grp_id
    when 1 then grp_id
    when 2 then grp_id
    when 3 then grp_id
    when 4 then grp_id
  end,
  case state
    when 0 then some_date
    when 1 then some_date
    when 2 then some_date
    when 3 then some_date
    when 4 then some_date
  end) compress 1; so a tuple (group_id, some_date) is projected to tuple (null, null) when the state is not an active state and therefore it is not indexed.
We can save even more space by compressing 1st expression.
analyze index idx_grp_state_date validate structure;
select * from index_stats
@pr
Tom Kyte's printtab
==============================
HEIGHT                        : 2
BLOCKS                        : 16
NAME                          : FIDX_ACTIV
LF_ROWS                       : 2000 <-- we're indexing only active rows
LF_BLKS                       : 6 <-- small index: 1 root block with 6 leaf blocks
BR_ROWS                       : 5
BR_BLKS                       : 1
DISTINCT_KEYS                 : 2000
PCT_USED                      : 69
PRE_ROWS                      : 25
PRE_ROWS_LEN                  : 224
OPT_CMPR_COUNT                : 1
OPT_CMPR_PCTSAVE              : 0Note: @pr is a Tom Kyte's print table script adopted by Tanel Poder (I'm using Tanel's library) .
Then I created a query to be outer joined to the report (report contains a row for every group).
I want to achieve a full scan of the index.
select
  case state -- 1st expression
    when 0 then grp_id
    when 1 then grp_id
    when 2 then grp_id
    when 3 then grp_id
    when 4 then grp_id
  end grp_id,
  min(case state --second expression
        when 0 then some_date
        when 1 then some_date
        when 2 then some_date
        when 3 then some_date
        when 4 then some_date
      end) as mintime
from t 
where
  case state --1st expression: at least one index column has to be not null
    when 0 then grp_id
    when 1 then grp_id
    when 2 then grp_id
    when 3 then grp_id
    when 4 then grp_id
  end is not null
group by
  case state --1st expression
    when 0 then grp_id
    when 1 then grp_id
    when 2 then grp_id
    when 3 then grp_id
    when 4 then grp_id
  end;-------------
Doc's snippet:
13.5.3.6 Full Scans
A full scan is available if a predicate references one of the columns in the index. The predicate does not need to be an index driver. A full scan is also available when there is no predicate, if both the following conditions are met:
All of the columns in the table referenced in the query are included in the index.
At least one of the index columns is not null.
A full scan can be used to eliminate a sort operation, because the data is ordered by the index key. It reads the blocks singly.
13.5.3.7 Fast Full Index Scans
Fast full index scans are an alternative to a full table scan when the index contains all the columns that are needed for the query, and at least one column in the index key has the NOT NULL constraint. A fast full scan accesses the data in the index itself, without accessing the table. It cannot be used to eliminate a sort operation, because the data is not ordered by the index key. It reads the entire index using multiblock reads, unlike a full index scan, and can be parallelized.
You can specify fast full index scans with the initialization parameter OPTIMIZER_FEATURES_ENABLE or the INDEX_FFS hint. Fast full index scans cannot be performed against bitmap indexes.
A fast full scan is faster than a normal full index scan in that it can use multiblock I/O and can be parallelized just like a table scan.
So the question is: Why does oracle do a full table scan?
Everything needed is in the index and one expression is not null, but index (fast) full scan is not even considered by CBO (I did a 10053 trace)
| Id  | Operation          | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
|   1 |  HASH GROUP BY     |      |      1 |     85 |     20 |00:00:00.11 |    3841 |
|*  2 |   TABLE ACCESS FULL| T    |      1 |   6100 |   2000 |00:00:00.10 |    3841 |
Predicate Information (identified by operation id):
   2 - filter(CASE "STATE" WHEN 0 THEN "GRP_ID" WHEN 1 THEN "GRP_ID" WHEN 2
              THEN "GRP_ID" WHEN 3 THEN "GRP_ID" WHEN 4 THEN "GRP_ID" END  IS NOT NULL)Let's try some minimalistic examples. Firstly with no FBI.
create index idx_grp_id on t(grp_id);
select grp_id,
       min(grp_id) min
from t
where grp_id is not null
group by grp_id;
| Id  | Operation             | Name       | Starts | E-Rows | A-Rows |   A-Time   | Buffers | Reads  |
|   1 |  HASH GROUP BY        |            |      1 |     20 |     20 |00:00:01.00 |     244 |    237 |
|*  2 |   INDEX FAST FULL SCAN| IDX_GRP_ID |      1 |    122K|    122K|00:00:00.54 |     244 |    237 |
Predicate Information (identified by operation id):
   2 - filter("GRP_ID" IS NOT NULL)This kind of output I was expected to see with FBI. Index FFS was used although grp_id has no NOT NULL constraint.
Let's try a simple FBI.
create index fidx_grp_id on t(trunc(grp_id));
select trunc(grp_id),
       min(trunc(grp_id)) min
from t
where trunc(grp_id) is not null
group by trunc(grp_id);
| Id  | Operation          | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
|   1 |  HASH GROUP BY     |      |      1 |     20 |     20 |00:00:00.94 |    3841 |
|*  2 |   TABLE ACCESS FULL| T    |      1 |   6100 |    122K|00:00:00.49 |    3841 |
Predicate Information (identified by operation id):
   2 - filter(TRUNC("GRP_ID") IS NOT NULL)Again, index (fast) full scan not even considered by CBO.
I tried:
alter table t modify grp_id not null;
alter table t add constraint trunc_not_null check (trunc(grp_id) is not null);I even tried to set table hidden column (SYS_NC00008$) to NOT NULL
It has no effect, FTS is still used..
Let's try another query:
select distinct trunc(grp_id)
from t
where trunc(grp_id) is not null
| Id  | Operation             | Name        | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
|   1 |  HASH UNIQUE          |             |      1 |     20 |     20 |00:00:00.85 |     244 |
|*  2 |   INDEX FAST FULL SCAN| FIDX_GRP_ID |      1 |    122K|    122K|00:00:00.49 |     244 |
Predicate Information (identified by operation id):
   2 - filter("T"."SYS_NC00008$" IS NOT NULL)Here the index FFS is used..
Let's try one more query, very similar to the above query:
select trunc(grp_id)
from t
where trunc(grp_id) is not null
group by trunc(grp_id)
| Id  | Operation          | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
|   1 |  HASH GROUP BY     |      |      1 |     20 |     20 |00:00:00.86 |    3841 |
|*  2 |   TABLE ACCESS FULL| T    |      1 |    122K|    122K|00:00:00.49 |    3841 |
Predicate Information (identified by operation id):
   2 - filter(TRUNC("GRP_ID") IS NOT NULL)And again no index full scan..
So my next question is:
What are the restrictions which prevent index (fast) fullscan to be used in these scenarios?
Thank you very much for your answers.
Edited by: user1175494 on 16.11.2010 15:23
Edited by: user1175494 on 16.11.2010 15:25

Answer 1

I'll start off with the caveat that i'm no Johnathan Lewis so hopefully someone will be able to come along and give you a more coherent explanation than i'm going to attempt here.
It looks like the application of the MIN function against the case statement is confusing the optimizer and disallowing the usage of your FBI. I tested this against my 11.2.0.1 instance and your query chooses the fast full scan without being nudged in the right direction.
That being said, i was able to get this to use a fast full scan on my 10 instance, but i had to jiggle the wires a bit. I modified your original query slightly, just to make it easier to do my fiddling.
original (in the sense that it still takes the full table scan) query
with data as
  select
    case state -- 1st expression
      when 0 then grp_id
      when 1 then grp_id
      when 2 then grp_id
      when 3 then grp_id
      when 4 then grp_id
    end as grp_id,
    case state --second expression
          when 0 then some_date
          when 1 then some_date
          when 2 then some_date
          when 3 then some_date
          when 4 then some_date
    end as mintime
  from t
  where
    case state --1st expression: at least one index column has to be not null
      when 0 then grp_id
      when 1 then grp_id
      when 2 then grp_id
      when 3 then grp_id
      when 4 then grp_id
    end is not null
  and
    case state --second expression
          when 0 then some_date
          when 1 then some_date
          when 2 then some_date
          when 3 then some_date
          when 4 then some_date
    end is not null 
select--+ GATHER_PLAN_STATISTICS
  grp_id,
  min(mintime)
from data
group by grp_id;
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL, NULL, 'allstats  +peeked_binds'));
| Id  | Operation        | Name | Starts | E-Rows | A-Rows |      A-Time   | Buffers |
|   1 |  HASH GROUP BY        |       |      2 |      33 |       40 |00:00:00.07 |    7646 |
|*  2 |   TABLE ACCESS FULL| T       |      2 |      33 |     4000 |00:00:00.08 |    7646 |
Predicate Information (identified by operation id):
   2 - filter((CASE "STATE" WHEN 0 THEN "GRP_ID" WHEN 1 THEN "GRP_ID" WHEN 2
           THEN "GRP_ID" WHEN 3 THEN "GRP_ID" WHEN 4 THEN "GRP_ID" END  IS NOT NULL AND
           CASE "STATE" WHEN 0 THEN "SOME_DATE" WHEN 1 THEN "SOME_DATE" WHEN 2 THEN
           "SOME_DATE" WHEN 3 THEN "SOME_DATE" WHEN 4 THEN "SOME_DATE" END  IS NOT
           NULL))
modified version where we prevent the MIN function from being applied too early, by using ROWNUM
with data as
  select
    case state -- 1st expression
      when 0 then grp_id
      when 1 then grp_id
      when 2 then grp_id
      when 3 then grp_id
      when 4 then grp_id
    end as grp_id,
    case state --second expression
          when 0 then some_date
          when 1 then some_date
          when 2 then some_date
          when 3 then some_date
          when 4 then some_date
    end as mintime
  from t
  where
    case state --1st expression: at least one index column has to be not null
      when 0 then grp_id
      when 1 then grp_id
      when 2 then grp_id
      when 3 then grp_id
      when 4 then grp_id
    end is not null
  and
    case state --second expression
          when 0 then some_date
          when 1 then some_date
          when 2 then some_date
          when 3 then some_date
          when 4 then some_date
    end is not null 
  and rownum > 0
select--+ GATHER_PLAN_STATISTICS
  grp_id,
  min(mintime)
from data
group by grp_id;
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL, NULL, 'allstats  +peeked_binds'));
| Id  | Operation           | Name        | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
|   1 |  HASH GROUP BY           |            |      2 |     20 |     40 |00:00:00.01 |      18 |
|   2 |   VIEW                |            |      2 |     33 |   4000 |00:00:00.07 |      18 |
|   3 |    COUNT           |            |      2 |      |   4000 |00:00:00.05 |      18 |
|*  4 |     FILTER           |            |      2 |      |   4000 |00:00:00.03 |      18 |
|*  5 |      INDEX FAST FULL SCAN| FIDX_ACTIVE |      2 |     33 |   4000 |00:00:00.01 |      18 |
Predicate Information (identified by operation id):
   4 - filter(ROWNUM>0)
   5 - filter(("T"."SYS_NC00006$" IS NOT NULL AND "T"."SYS_NC00007$" IS NOT NULL))