Pagination query help needed for large table - force a different index

Question

I'm using a slight modification of the pagination query from over at Ask Tom's: [http://www.oracle.com/technology/oramag/oracle/07-jan/o17asktom.html]
Mine looks like this when fetching the first 100 rows of all members with last name Smith, ordered by join date:
SELECT members.*
FROM members,
    SELECT RID, rownum rnum
    FROM
        SELECT rowid as RID
        FROM members
        WHERE last_name = 'Smith'
        ORDER BY joindate
    WHERE rownum <= 100
WHERE rnum >= 1
         and RID = members.rowidThe difference between this and the one at Ask Tom's is that my innermost query just returns the ROWID. Then in the outermost query we join the ROWIDs returned to the members table, after we have pruned the ROWIDs down to only the chunk of 100 we want. This makes it MUCH faster (verifiably) on our large tables, as it is able to use the index on the innermost query (well... read on).
The problem I have is this:
SELECT rowid as RID
FROM members
WHERE last_name = 'Smith'
ORDER BY joindateThis will use the index for the predicate column (last_name) instead of the unique index I have defined for the joindate column (joindate, sequence). (Verifiable with explain plan). It is much slower this way on a large table. So I can hint it using either of the following methods:
SELECT /*+ index(members, joindate_idx) */ rowid as RID
FROM members
WHERE last_name = 'Smith'
ORDER BY joindate
SELECT /*+ first_rows(100) */ rowid as RID
FROM members
WHERE last_name = 'Smith'
ORDER BY joindateEither way, it now uses the index of the ORDER BY column (joindate_idx), so now it is much faster as it does not have to do a sort (remember, VERY large table, millions of records). So that seems good. But now, on my outermost query, I join the rowid with the meaningful columns of data from the members table, as commented below:
SELECT members.*      -- Select all data from members table
FROM members,           -- members table added to FROM clause
    SELECT RID, rownum rnum
    FROM
        SELECT /*+ index(members, joindate_idx) */ rowid as RID   -- Hint is ignored now that I am joining in the outer query
        FROM members
        WHERE last_name = 'Smith'
        ORDER BY joindate
    WHERE rownum <= 100
WHERE rnum >= 1
        and RID = members.rowid           -- Merge the members table on the rowid we pulled from the inner queriesOnce I do this join, it goes back to using the predicate index (last_name) and has to perform the sort once it finds all matching values (which can be a lot in this table, there is high cardinality on some columns).
So my question is, in the full query above, is there any way I can get it to use the ORDER BY column for indexing to prevent it from having to do a sort? The join is what causes it to revert back to using the predicate index, even with hints. Remove the join and just return the ROWIDs for those 100 records and it flies, even on 10 million records.
It'd be great if there was some generic hint that could accomplish this, such that if we change the table/columns/indexes, we don't need to change the hint (the FIRST_ROWS hint is a good example of this, while the INDEX hint is the opposite), but any help would be appreciated. I can provide explain plans for any of the above if needed.
Thanks!

Answer 1

Lakmal Rajapakse wrote:
OK here is an example to illustrate the advantage:
SQL> set autot traceonly
SQL> select * from (
2  select a.*, rownum x  from
3  (
4  select a.* from aoswf.events a
5  order by EVENT_DATETIME
6  ) a
7  where rownum <= 1200
8  )
9  where x >= 1100
10  /
101 rows selected.
Execution Plan
Plan hash value: 3711662397
| Id  | Operation                      | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT               |            |  1200 |   521K|   192   (0)| 00:00:03 |
|*  1 |  VIEW                          |            |  1200 |   521K|   192   (0)| 00:00:03 |
|*  2 |   COUNT STOPKEY                |            |       |       |            |          |
|   3 |    VIEW                        |            |  1200 |   506K|   192   (0)| 00:00:03 |
|   4 |     TABLE ACCESS BY INDEX ROWID| EVENTS     |   253M|    34G|   192   (0)| 00:00:03 |
|   5 |      INDEX FULL SCAN           | EVEN_IDX02 |  1200 |       |     2   (0)| 00:00:01 |
Predicate Information (identified by operation id):
1 - filter("X">=1100)
2 - filter(ROWNUM<=1200)
Statistics
0  recursive calls
0  db block gets
443  consistent gets
0  physical reads
0  redo size
25203  bytes sent via SQL*Net to client
281  bytes received via SQL*Net from client
8  SQL*Net roundtrips to/from client
0  sorts (memory)
0  sorts (disk)
101  rows processed
SQL>
SQL>
SQL> select * from aoswf.events a, (
2  select rid, rownum x  from
3  (
4  select rowid rid from aoswf.events a
5  order by EVENT_DATETIME
6  ) a
7  where rownum <= 1200
8  ) b
9  where x >= 1100
10  and a.rowid = rid
11  /
101 rows selected.
Execution Plan
Plan hash value: 2308864810
| Id  | Operation                   | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT            |            |  1200 |   201K|   261K  (1)| 00:52:21 |
|   1 |  NESTED LOOPS               |            |  1200 |   201K|   261K  (1)| 00:52:21 |
|*  2 |   VIEW                      |            |  1200 | 30000 |   260K  (1)| 00:52:06 |
|*  3 |    COUNT STOPKEY            |            |       |       |            |          |
|   4 |     VIEW                    |            |   253M|  2895M|   260K  (1)| 00:52:06 |
|   5 |      INDEX FULL SCAN        | EVEN_IDX02 |   253M|  4826M|   260K  (1)| 00:52:06 |
|   6 |   TABLE ACCESS BY USER ROWID| EVENTS     |     1 |   147 |     1   (0)| 00:00:01 |
Predicate Information (identified by operation id):
2 - filter("X">=1100)
3 - filter(ROWNUM<=1200)
Statistics
8  recursive calls
0  db block gets
117  consistent gets
0  physical reads
0  redo size
27539  bytes sent via SQL*Net to client
281  bytes received via SQL*Net from client
8  SQL*Net roundtrips to/from client
0  sorts (memory)
0  sorts (disk)
101  rows processed
Lakmal (and OP),
Not sure what advantage you are trying to show here. But considering that we are talking about pagination query here and order of records is important, your 2 queries will not always generate output in same order. Here is the test case:
SQL> select * from v$version ;
BANNER
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Prod
PL/SQL Release 10.2.0.1.0 - Production
CORE     10.2.0.1.0     Production
TNS for Linux: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 - Production
SQL> show parameter optimizer
NAME                                 TYPE        VALUE
optimizer_dynamic_sampling           integer     2
optimizer_features_enable            string      10.2.0.1
optimizer_index_caching              integer     0
optimizer_index_cost_adj             integer     100
optimizer_mode                       string      ALL_ROWS
optimizer_secure_view_merging        boolean     TRUE
SQL> show parameter pga
NAME                                 TYPE        VALUE
pga_aggregate_target                 big integer 103M
SQL> create table t nologging as select * from all_objects where 1 = 2 ;
Table created.
SQL> create index t_idx on t(last_ddl_time) nologging ;
Index created.
SQL> insert /*+ APPEND */ into t (owner, object_name, object_id, created, last_ddl_time) select owner, object_name, object_id, created, sysdate - dbms_random.value(1, 100) from all_objects order by dbms_random.random;
40617 rows created.
SQL> commit ;
Commit complete.
SQL> exec dbms_stats.gather_table_stats(user, 'T', cascade=>true);
PL/SQL procedure successfully completed.
SQL> select object_id, object_name, created from t, (select rid, rownum rn from (select rowid rid from t order by created desc) where rownum <= 1200) t1 where rn >= 1190 and t.rowid = t1.rid ;
OBJECT_ID OBJECT_NAME                    CREATED
     47686 ALL$OLAP2_JOIN_KEY_COLUMN_USES 28-JUL-2009 08:08:39
     47672 ALL$OLAP2_CUBE_DIM_USES        28-JUL-2009 08:08:39
     47681 ALL$OLAP2_CUBE_MEASURE_MAPS    28-JUL-2009 08:08:39
     47682 ALL$OLAP2_FACT_LEVEL_USES      28-JUL-2009 08:08:39
     47685 ALL$OLAP2_AGGREGATION_USES     28-JUL-2009 08:08:39
     47692 ALL$OLAP2_CATALOGS             28-JUL-2009 08:08:39
     47665 ALL$OLAPMR_FACTTBLKEYMAPS      28-JUL-2009 08:08:39
     47688 ALL$OLAP2_DIM_LEVEL_ATTR_MAPS  28-JUL-2009 08:08:39
     47689 ALL$OLAP2_DIM_LEVELS_KEYMAPS   28-JUL-2009 08:08:39
     47669 ALL$OLAP9I2_HIER_DIMENSIONS    28-JUL-2009 08:08:39
     47666 ALL$OLAP9I1_HIER_DIMENSIONS    28-JUL-2009 08:08:39
11 rows selected.
SQL> select object_id, object_name, last_ddl_time from t, (select rid, rownum rn from (select rowid rid from t order by last_ddl_time desc) where rownum <= 1200) t1 where rn >= 1190 and t.rowid = t1.rid ;
OBJECT_ID OBJECT_NAME                    LAST_DDL_TIME
     11749 /b9fe5b99_OraRTStatementComman 06-FEB-2010 03:43:49
     13133 oracle/jdbc/driver/OracleLog$3 06-FEB-2010 03:45:44
     37534 com/sun/mail/smtp/SMTPMessage  06-FEB-2010 03:46:14
     36145 /4e492b6f_SerProfileToClassErr 06-FEB-2010 03:11:09
     26815 /7a628fb8_DefaultHSBChooserPan 06-FEB-2010 03:26:55
     16695 /2940a364_RepIdDelegator_1_3   06-FEB-2010 03:38:17
     36539 sun/io/ByteToCharMacHebrew     06-FEB-2010 03:28:57
     14044 /d29b81e1_OldHeaders           06-FEB-2010 03:12:12
     12920 /25f8f3a5_BasicSplitPaneUI     06-FEB-2010 03:11:06
     42266 SI_GETCLRHSTGRFTR              06-FEB-2010 03:40:20
     15752 /2f494dce_JDWPThreadReference  06-FEB-2010 03:09:31
11 rows selected.
SQL> select object_id, object_name, last_ddl_time from (select t1.*, rownum rn from (select * from t order by last_ddl_time desc) t1 where rownum <= 1200) where rn >= 1190 ;
OBJECT_ID OBJECT_NAME                    LAST_DDL_TIME
     37534 com/sun/mail/smtp/SMTPMessage  06-FEB-2010 03:46:14
     13133 oracle/jdbc/driver/OracleLog$3 06-FEB-2010 03:45:44
     11749 /b9fe5b99_OraRTStatementComman 06-FEB-2010 03:43:49
     42266 SI_GETCLRHSTGRFTR              06-FEB-2010 03:40:20
     16695 /2940a364_RepIdDelegator_1_3   06-FEB-2010 03:38:17
     36539 sun/io/ByteToCharMacHebrew     06-FEB-2010 03:28:57
     26815 /7a628fb8_DefaultHSBChooserPan 06-FEB-2010 03:26:55
     14044 /d29b81e1_OldHeaders           06-FEB-2010 03:12:12
     36145 /4e492b6f_SerProfileToClassErr 06-FEB-2010 03:11:09
     12920 /25f8f3a5_BasicSplitPaneUI     06-FEB-2010 03:11:06
     15752 /2f494dce_JDWPThreadReference  06-FEB-2010 03:09:31
11 rows selected.
SQL> select object_id, object_name, last_ddl_time from t, (select rid, rownum rn from (select rowid rid from t order by last_ddl_time desc) where rownum <= 1200) t1 where rn >= 1190 and t.rowid = t1.rid order by last_ddl_time desc ;
OBJECT_ID OBJECT_NAME                    LAST_DDL_TIME
     37534 com/sun/mail/smtp/SMTPMessage  06-FEB-2010 03:46:14
     13133 oracle/jdbc/driver/OracleLog$3 06-FEB-2010 03:45:44
     11749 /b9fe5b99_OraRTStatementComman 06-FEB-2010 03:43:49
     42266 SI_GETCLRHSTGRFTR              06-FEB-2010 03:40:20
     16695 /2940a364_RepIdDelegator_1_3   06-FEB-2010 03:38:17
     36539 sun/io/ByteToCharMacHebrew     06-FEB-2010 03:28:57
     26815 /7a628fb8_DefaultHSBChooserPan 06-FEB-2010 03:26:55
     14044 /d29b81e1_OldHeaders           06-FEB-2010 03:12:12
     36145 /4e492b6f_SerProfileToClassErr 06-FEB-2010 03:11:09
     12920 /25f8f3a5_BasicSplitPaneUI     06-FEB-2010 03:11:06
     15752 /2f494dce_JDWPThreadReference  06-FEB-2010 03:09:31
11 rows selected.
SQL> set autotrace traceonly
SQL> select object_id, object_name, last_ddl_time from t, (select rid, rownum rn from (select rowid rid from t order by last_ddl_time desc) where rownum <= 1200) t1 where rn >= 1190 and t.rowid = t1.rid order by last_ddl_time desc
  2  ;
11 rows selected.
Execution Plan
Plan hash value: 44968669
| Id  | Operation                       | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT                |       |  1200 | 91200 |   180   (2)| 00:00:03 |
|   1 |  SORT ORDER BY                  |       |  1200 | 91200 |   180   (2)| 00:00:03 |
|*  2 |   HASH JOIN                     |       |  1200 | 91200 |   179   (2)| 00:00:03 |
|*  3 |    VIEW                         |       |  1200 | 30000 |    98   (0)| 00:00:02 |
|*  4 |     COUNT STOPKEY               |       |       |       |            |          |
|   5 |      VIEW                       |       | 40617 |   475K|    98   (0)| 00:00:02 |
|   6 |       INDEX FULL SCAN DESCENDING| T_IDX | 40617 |   793K|    98   (0)| 00:00:02 |
|   7 |    TABLE ACCESS FULL            | T     | 40617 |  2022K|    80   (2)| 00:00:01 |
Predicate Information (identified by operation id):
   2 - access("T".ROWID="T1"."RID")
   3 - filter("RN">=1190)
   4 - filter(ROWNUM<=1200)
Statistics
          1  recursive calls
          0  db block gets
        348  consistent gets
          0  physical reads
          0  redo size
       1063  bytes sent via SQL*Net to client
        385  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
         11  rows processed
SQL> select object_id, object_name, last_ddl_time from (select t1.*, rownum rn from (select * from t order by last_ddl_time desc) t1 where rownum <= 1200) where rn >= 1190 ;
11 rows selected.
Execution Plan
Plan hash value: 882605040
| Id  | Operation                | Name | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT         |      |  1200 | 62400 |    80   (2)| 00:00:01 |
|*  1 |  VIEW                    |      |  1200 | 62400 |    80   (2)| 00:00:01 |
|*  2 |   COUNT STOPKEY          |      |       |       |            |          |
|   3 |    VIEW                  |      | 40617 |  1546K|    80   (2)| 00:00:01 |
|*  4 |     SORT ORDER BY STOPKEY|      | 40617 |  2062K|    80   (2)| 00:00:01 |
|   5 |      TABLE ACCESS FULL   | T    | 40617 |  2062K|    80   (2)| 00:00:01 |
Predicate Information (identified by operation id):
   1 - filter("RN">=1190)
   2 - filter(ROWNUM<=1200)
   4 - filter(ROWNUM<=1200)
Statistics
          0  recursive calls
          0  db block gets
        343  consistent gets
          0  physical reads
          0  redo size
       1063  bytes sent via SQL*Net to client
        385  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
         11  rows processed
SQL> select object_id, object_name, last_ddl_time from t, (select rid, rownum rn from (select rowid rid from t order by last_ddl_time desc) where rownum <= 1200) t1 where rn >= 1190 and t.rowid = t1.rid ;
11 rows selected.
Execution Plan
Plan hash value: 168880862
| Id  | Operation                      | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT               |       |  1200 | 91200 |   179   (2)| 00:00:03 |
|*  1 |  HASH JOIN                     |       |  1200 | 91200 |   179   (2)| 00:00:03 |
|*  2 |   VIEW                         |       |  1200 | 30000 |    98   (0)| 00:00:02 |
|*  3 |    COUNT STOPKEY               |       |       |       |            |          |
|   4 |     VIEW                       |       | 40617 |   475K|    98   (0)| 00:00:02 |
|   5 |      INDEX FULL SCAN DESCENDING| T_IDX | 40617 |   793K|    98   (0)| 00:00:02 |
|   6 |   TABLE ACCESS FULL            | T     | 40617 |  2022K|    80   (2)| 00:00:01 |
Predicate Information (identified by operation id):
   1 - access("T".ROWID="T1"."RID")
   2 - filter("RN">=1190)
   3 - filter(ROWNUM<=1200)
Statistics
          0  recursive calls
          0  db block gets
        349  consistent gets
          0  physical reads
          0  redo size
       1063  bytes sent via SQL*Net to client
        385  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
         11  rows processed
SQL> select object_id, object_name, last_ddl_time from (select t1.*, rownum rn from (select * from t order by last_ddl_time desc) t1 where rownum <= 1200) where rn >= 1190 order by last_ddl_time desc ;
11 rows selected.
Execution Plan
Plan hash value: 882605040
| Id  | Operation           | Name | Rows     | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT      |     |  1200 | 62400 |    80   (2)| 00:00:01 |
|*  1 |  VIEW                |     |  1200 | 62400 |    80   (2)| 00:00:01 |
|*  2 |   COUNT STOPKEY       |     |     |     |          |          |
|   3 |    VIEW            |     | 40617 |  1546K|    80   (2)| 00:00:01 |
|*  4 |     SORT ORDER BY STOPKEY|     | 40617 |  2062K|    80   (2)| 00:00:01 |
|   5 |      TABLE ACCESS FULL      | T     | 40617 |  2062K|    80   (2)| 00:00:01 |
Predicate Information (identified by operation id):
   1 - filter("RN">=1190)
   2 - filter(ROWNUM<=1200)
   4 - filter(ROWNUM<=1200)
Statistics
     175  recursive calls
       0  db block gets
     388  consistent gets
       0  physical reads
       0  redo size
       1063  bytes sent via SQL*Net to client
     385  bytes received via SQL*Net from client
       2  SQL*Net roundtrips to/from client
       4  sorts (memory)
       0  sorts (disk)
      11  rows processed
SQL> set autotrace off
SQL> spool offAs you will see, the join query here has to have an ORDER BY clause at the end to ensure that records are correctly sorted. You can not rely on optimizer choosing NESTED LOOP join method and, as above example shows, when optimizer chooses HASH JOIN, oracle is free to return rows in no particular order.
The query that does not involve join always returns rows in the desired order. Adding an ORDER BY does add a step in the plan for the query using join but does not affect the other query.