TOP-N query optimization

Question

Hi,
We have a 3rd party GUI component which provides data paging capabilities by using the "ROW_NUMBER" analytic function. Sadly it does that in an inefficient way, because it forces Oracle to perform a large sort only to later discard a large portion of data. By slightly modifying the query, it is possible to fix that (demonstrated below), but the problem is that I can't change the SQL the component issues.
I'd appreciate if anyone has any suggestion whether it's possible to fix that in some way.
Test case:
SQL> create table test (
  2     id           number (10) not null,
  3     c1           number (2) not null,
  4     other_data   varchar2 (200)
  5  );
Table created.
SQL> insert into test
  2  with generator as (
  3      select  --+ materialize
  4         rownum id
  5      from dual
  6      connect by
  7          rownum <= 10000
  8  )
  9  select
10      rownum     id,
11      mod(rownum, 20),
12      rpad('x',200)
13  from
14      generator   v1,
15      generator   v2
16  where   rownum <= 1000000
17  ;
1000000 rows created.
SQL> commit;
Commit complete.
SQL> create index i_1 on test(id, c1);
Index created.
SQL> exec dbms_stats.gather_table_stats(user,'TEST',cascade=>true);
PL/SQL procedure successfully completed.This is the SQL the 3rd party component issues:
SQL> SELECT *
  2    FROM (  SELECT t2.*, ROW_NUMBER () OVER (ORDER BY t2.id) AS "rnum"
  3              FROM (SELECT *
  4                      FROM test
  5                     WHERE c1 != 5) t2
  6          ORDER BY t2.id) t1
  7   WHERE t1."rnum" <= 10;Here you can see that in line 3 of the execution plan, the whole table is read and in line 2 it's sorted to find the required 10 rows:
| Id  | Operation                | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers | Reads  |  OMem |  1Mem | Used-Mem |
|   0 | SELECT STATEMENT         |      |      1 |        |     10 |00:00:00.93 |   30502 |  30497 |    |          |          |
|*  1 |  VIEW                    |      |      1 |    950K|     10 |00:00:00.93 |   30502 |  30497 |    |          |          |
|*  2 |   WINDOW SORT PUSHED RANK|      |      1 |    950K|     11 |00:00:00.93 |   30502 |  30497 |  4096 |  4096 | 4096  (0)|
|*  3 |    TABLE ACCESS FULL     | TEST |      1 |    950K|    950K|00:00:00.49 |   30502 |  30497 |    |          |          |
Predicate Information (identified by operation id):
   1 - filter("T1"."rnum"<=10)
   2 - filter(ROW_NUMBER() OVER ( ORDER BY "TEST"."ID")<=10)
   3 - filter("C1"<>5)However, by moving the "ORDER BY" to the outermost part of the query, Oracle can use the index I_1 to read only 10 rows from it and stop as soon as it reaches that count. And what's more important, no sorting takes place:
SQL>   SELECT *
  2      FROM (SELECT t2.*, ROW_NUMBER () OVER (ORDER BY t2.id) AS "rnum"
  3              FROM (SELECT *
  4                      FROM test
  5                     WHERE c1 != 5) t2
  6           )
  7     WHERE "rnum" <= 10
  8  ORDER BY id;
| Id  | Operation                     | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers | Reads  |  OMem |  1Mem | Used-Mem |
|   0 | SELECT STATEMENT              |      |      1 |        |     10 |00:00:00.01 |       6 |      2 |       |       |          |
|*  1 |  VIEW                         |      |      1 |    950K|     10 |00:00:00.01 |       6 |      2 |       |       |          |
|*  2 |   WINDOW NOSORT STOPKEY       |      |      1 |    950K|     10 |00:00:00.01 |       6 |      2 |   213M|  4884K|          |
|   3 |    TABLE ACCESS BY INDEX ROWID| TEST |      1 |    950K|     11 |00:00:00.01 |       6 |      2 |       |       |          |
|*  4 |     INDEX FULL SCAN           | I_1  |      1 |    950K|     11 |00:00:00.01 |       4 |      1 |       |       |          |
Predicate Information (identified by operation id):
   1 - filter("rnum"<=10)
   2 - filter(ROW_NUMBER() OVER ( ORDER BY "TEST"."ID")<=10)
   4 - filter("C1"<>5)Is there any way to "fix" the first query without changing it (maybe using an outline, but I don't see how Oracle can avoid the whole table sort)?
Thanks in advance and regards,
Jure

Answer 1

Thank you very much for the link to DBMS_ADVANCED_REWRITE. I didn't know of its existence and it sure seems a useful tool for situations similar to mine (I'm on 11.2.0.1).
However, I encountered two problems when trying to rewrite my query. Maybe I did something obviously wrong, since that's the first time I use DBMS_ADVANCED_REWRITE:
1) The rewrite doesn't seem to work if the source statement contains an ORDER BY clause, which is an essential part of the query I'm trying to rewrite. I reproduced it here such that if the rewrite takes place, I get back the "rewritten query" text (plus some dummy columns to match the resultset structure of the original query), otherwise 10 rows are returned.
First I setup the rewrite equivalence:
SQL> begin
  2    sys.dbms_advanced_rewrite.declare_rewrite_equivalence (
  3      name             => 'DEMO',
  4      source_stmt      =>
  5      'SELECT *
  6         FROM (  SELECT t2.*, ROW_NUMBER () OVER (ORDER BY t2.id) AS "rnum"
  7                   FROM (SELECT *
  8                           FROM test
  9                          WHERE c1 != 5) t2
10               ORDER BY t2.id) t1
11        WHERE t1."rnum" <= 10',
12      destination_stmt => 'select null dummy_a, null dummy_b, ''rewritten query'' dummy_c, 1 dummy_rownum from dual',
13      validate         => FALSE,
14      rewrite_mode     => 'GENERAL'
15    );
16  end;
17  /
PL/SQL procedure successfully completed.Now I try to run the query with the exact same query text as defined in "source_stmt". No rewrite takes place:
SQL> SELECT *
  2    FROM (  SELECT t2.*, ROW_NUMBER () OVER (ORDER BY t2.id) AS "rnum"
  3              FROM (SELECT *
  4                      FROM test
  5                     WHERE c1 != 5) t2
  6          ORDER BY t2.id) t1
  7   WHERE t1."rnum" <= 10
  8  /
<10 rows are retireved>However, if I remove the "ORDER BY t2.id" part from the "t1" query block, the rewrite works:
SQL> exec sys.dbms_advanced_rewrite.drop_rewrite_equivalence('DEMO');
PL/SQL procedure successfully completed.
SQL> begin
  2    sys.dbms_advanced_rewrite.declare_rewrite_equivalence (
  3      name             => 'DEMO',
  4      source_stmt      =>
  5      'SELECT *
  6         FROM (  SELECT t2.*, ROW_NUMBER () OVER (ORDER BY t2.id) AS "rnum"
  7                   FROM (SELECT *
  8                           FROM test
  9                          WHERE c1 != 5) t2
10               ) t1
11        WHERE t1."rnum" <= 10',
12      destination_stmt => 'select null dummy_a, null dummy_b, ''rewritten query'' dummy_c, 1 dummy_rownum from dual',
13      validate         => FALSE,
14      rewrite_mode     => 'GENERAL'
15    );
16  end;
17  /
PL/SQL procedure successfully completed.
SQL> SELECT *
  2    FROM (  SELECT t2.*, ROW_NUMBER () OVER (ORDER BY t2.id) AS "rnum"
  3              FROM (SELECT *
  4                      FROM test
  5                     WHERE c1 != 5) t2
  6          ) t1
  7   WHERE t1."rnum" <= 10
  8  /
I C OTHER_DATA            rnum
    rewritten query          1But the problem is of course that the rewrite should take place also when the "ORDER BY" part is provided in the query, since this is what the application issues:
SQL> SELECT *
  2    FROM (  SELECT t2.*, ROW_NUMBER () OVER (ORDER BY t2.id) AS "rnum"
  3              FROM (SELECT *
  4                      FROM test
  5                     WHERE c1 != 5) t2
  6          ORDER BY t2.id) t1
  7   WHERE t1."rnum" <= 10
  8  /
<10 rows are retrieved, so no rewrite took place>Maybe the reason for such a behaviour (I don't know how to prove it) is that materialized views don't store the ORDER BY clause as part of their definition, and the same rewrite engine that is used for materialized views is used for equivalence query rewrite:
http://download.oracle.com/docs/cd/E11882_01/appdev.112/e16760/d_advrwr.htm#sthref154
Query rewrite using equivalence declarations occurs simultaneously and in concert with query rewrite using materialized views. The same query rewrite engine is used for both.
http://download.oracle.com/docs/cd/E11882_01/server.112/e16579/basicmv.htm#DWHSG8208
The ORDER BY clause is not considered part of the materialized view definition
Maybe I'm completely wrong, so any suggestion is welcome.
2) I didn't mention that "WHERE "rnum" <= 10" is actually "WHERE "rnum" <= :b", and bind variables aren't supported for rewrite. I checked the MOS document "Using DBMS_ADVANCED_REWRITE When Binds Are Present (Avoiding ORA-30353) [ID 392214.1]" and maybe I could remove the WHERE clause, but I'd have to further test which "similar" queries would be rewritten to this one, since REWRITE_MODE is 'GENERAL' instead of 'TEXT_MATCH'. But I'd first have to resolve problem number 1) above :-)
Thanks in advance for any answers.
Regards,
Jure