Index (or not) for excluding NULL values in a query

Question

Hello,
I have table that can become very large. The table has a varchar2 column (let's call it TEXT) that can contain NULL values. I want to process only the records that have a value (NOT NULL). Also, the table is continuously expanded with newly inserted records. The inserts should suffer as little performance loss as possible.
My question: should I use an index on the column and if so, what kind of index?
I have done a little test with a function based index (inspired by this Tom Kyte article: http://tkyte.blogspot.com/2006/01/something-about-nothing.html):
create index text_isnull_idx on my_table(text,0);
I notice that if I use the clause WHERE TEXT IS NULL, the index is used. But if I use a clause WHERE TEXT IS NOT NULL (which is the clause I want to use), a full table scan is performed. Is this bad? Can I somehow improve the speed of this selection?
Thanks in advance,
Frans

Answer 1

I build a test case with very simple table with 2 columns and it shows that FTS is better than index access even when above ratio is <= 0.01 (1%):
DROP TABLE T1;
CREATE TABLE T1
           C1 VARCHAR2(100)
          ,C2 NUMBER
INSERT INTO T1 (SELECT TO_CHAR(OBJECT_ID), ROWNUM FROM USER_OBJECTS);
BEGIN
     FOR I IN 1..100 LOOP
          INSERT INTO T1 (SELECT NULL, ROWNUM FROM USER_OBJECTS);
     END LOOP;
END;
CREATE INDEX T1_IDX ON T1(C1);
ANALYZE TABLE T1 COMPUTE STATISTICS
     FOR TABLE
     FOR ALL INDEXES
     FOR ALL INDEXED COLUMNS
SET AUTOTRACE TRACEONLY
SELECT
          C1, C2
     FROM T1 WHERE C1 IS NOT NULL;
3864 rows selected.
real: 1344
Execution Plan
   0      SELECT STATEMENT Optimizer=CHOOSE (Cost=59 Card=3864 Bytes=30912)
   1    0   TABLE ACCESS (FULL) OF 'T1' (Cost=59 Card=3864 Bytes=30912)
Statistics
          0  recursive calls
          0  db block gets
       2527 consistent gets
       3864 rows processed
BUT
SELECT
     --+ FIRST_ROWS
          C1, C2
     FROM T1 WHERE C1 IS NOT NULL;
3864 rows selected.
real: 1296
Execution Plan
   0      SELECT STATEMENT Optimizer=HINT: FIRST_ROWS (Cost=35 Card=3864 Bytes=30912)
   1    0   TABLE ACCESS (BY INDEX ROWID) OF 'T1' (Cost=35 Card=3864 Bytes=30912)
   2    1     INDEX (FULL SCAN) OF 'T1_IDX' (NON-UNIQUE) (Cost=11 Card=3864)
Statistics
          0  recursive calls
          0  db block gets
       5052 consistent gets
       3864 rows processed
and just for comparison:
SELECT * FROM T1 WHERE C1 IS NULL;
386501 rows selected.
real: 117878
Execution Plan
   0      SELECT STATEMENT Optimizer=CHOOSE (Cost=59 Card=386501 Bytes=3092008)
   1    0   TABLE ACCESS (FULL) OF 'T1' (Cost=59 Card=386501 Bytes=3092008)
Statistics
          0  recursive calls
          0  db block gets
     193850 consistent gets
     386501 rows processedHence you have to benchmark you queries with and w/o index[es]