Poor query performance when joining CONTAINS to another table

Question

We just recently began evaluating Oracle Text for a search solution. We need to be able to search a table that can have over 20+ million rows. Each user may only have visibility to a tiny fraction of those rows. The goal is to have a single Oracle Text index that represents all of the searchable columns in the table (multi column datastore) and provide a score for each search result so that we can sort the search results in descending order by score. What we're seeing is that query performance from TOAD is extremely fast when we write a simple CONTAINS query against the Oracle Text indexed table. However, when we attempt to first reduce the rows the CONTAINS query needs to search by using a WITH we find that the query performance degrades significantly.
For example, we can find all the records a user has access to from our base table by the following query:
SELECT d.duns_loc
FROM duns d
JOIN primary_contact pc
ON d.duns_loc = pc.duns_loc
AND pc.emp_id = :employeeID;
This query can execute in <100 ms. In the working example, this query returns around 1200 rows of the primary key duns_loc.
Our search query looks like this:
SELECT score(1), d.*
FROM duns d
WHERE CONTAINS(TEXT_KEY, :search,1) > 0
ORDER BY score(1) DESC;
The :search value in this example will be 'highway'. The query can return 246k rows in around 2 seconds.
2 seconds is good, but we should be able to have a much faster response if the search query did not have to search the entire table, right? Since each user can only "view" records they are assigned to we reckon that if the search operation only had to scan a tiny tiny percent of the TEXT index we should see faster (and more relevant) results. If we now write the following query:
WITH subset
AS
(SELECT d.duns_loc
FROM duns d
JOIN primary_contact pc
ON d.duns_loc = pc.duns_loc
AND pc.emp_id = :employeeID
SELECT score(1), d.*
FROM duns d
JOIN subset s
ON d.duns_loc = s.duns_loc
WHERE CONTAINS(TEXT_KEY, :search,1) > 0
ORDER BY score(1) DESC;
For reasons we have not been able to identify this query actually takes longer to execute than the sum of the durations of the contributing parts. This query takes over 6 seconds to run. We nor our DBA can seem to figure out why this query performs worse than a wide open search. The wide open search is not ideal as the query would end up returning records to the user they don't have access to view.
Has anyone ever ran into something like this? Any suggestions on what to look at or where to go? If anyone would like more information to help in diagnosis than let me know and i'll be happy to produce it here.
Thanks!!

Answer 1

Sometimes it can be good to separate the tables into separate sub-query factoring (with) clauses or inline views in the from clause or an in clause as a where condition. Although there are some differences, using a sub-query factoring (with) clause is similar to using an inline view in the from clause. However, you should avoid duplication. You should not have the same table in two different places, as in your original query. You should have indexes on any columns that the tables are joined on, your statistics should be current, and your domain index should have regular synchronization, optimization, and periodically rebuild or drop and recreate to keep it performing with maximum efficiency. The following demonstration uses a composite domain index (cdi) with filter by, as suggested by Roger, then shows the explained plans for your original query, and various others. Your original query has nested loops. All of the others have the same plan without the nested loops. You could also add index hints.
SCOTT@orcl_11gR2> -- tables:
SCOTT@orcl_11gR2> CREATE TABLE duns
  2    (duns_loc  NUMBER,
  3       text_key  VARCHAR2 (30))
  4  /
Table created.
SCOTT@orcl_11gR2> CREATE TABLE primary_contact
  2    (duns_loc  NUMBER,
  3       emp_id       NUMBER)
  4  /
Table created.
SCOTT@orcl_11gR2> -- data:
SCOTT@orcl_11gR2> INSERT INTO duns VALUES (1, 'highway')
  2  /
1 row created.
SCOTT@orcl_11gR2> INSERT INTO primary_contact VALUES (1, 1)
  2  /
1 row created.
SCOTT@orcl_11gR2> INSERT INTO duns
  2  SELECT object_id, object_name
  3  FROM   all_objects
  4  WHERE  object_id > 1
  5  /
76027 rows created.
SCOTT@orcl_11gR2> INSERT INTO primary_contact
  2  SELECT object_id, namespace
  3  FROM   all_objects
  4  WHERE  object_id > 1
  5  /
76027 rows created.
SCOTT@orcl_11gR2> -- indexes:
SCOTT@orcl_11gR2> CREATE INDEX duns_duns_loc_idx
  2  ON duns (duns_loc)
  3  /
Index created.
SCOTT@orcl_11gR2> CREATE INDEX primary_contact_duns_loc_idx
  2  ON primary_contact (duns_loc)
  3  /
Index created.
SCOTT@orcl_11gR2> -- composite domain index (cdi) with filter by clause
SCOTT@orcl_11gR2> -- as suggested by Roger:
SCOTT@orcl_11gR2> CREATE INDEX duns_text_key_idx
  2  ON duns (text_key)
  3  INDEXTYPE IS CTXSYS.CONTEXT
  4  FILTER BY duns_loc
  5  /
Index created.
SCOTT@orcl_11gR2> -- gather statistics:
SCOTT@orcl_11gR2> EXEC DBMS_STATS.GATHER_TABLE_STATS (USER, 'DUNS')
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> EXEC DBMS_STATS.GATHER_TABLE_STATS (USER, 'PRIMARY_CONTACT')
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> -- variables:
SCOTT@orcl_11gR2> VARIABLE employeeid NUMBER
SCOTT@orcl_11gR2> EXEC :employeeid := 1
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> VARIABLE search VARCHAR2(100)
SCOTT@orcl_11gR2> EXEC :search := 'highway'
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> -- original query:
SCOTT@orcl_11gR2> SET AUTOTRACE ON EXPLAIN
SCOTT@orcl_11gR2> WITH
  2    subset AS
  3        (SELECT d.duns_loc
  4         FROM      duns d
  5         JOIN      primary_contact pc
  6         ON      d.duns_loc = pc.duns_loc
  7         AND      pc.emp_id = :employeeID)
  8  SELECT score(1), d.*
  9  FROM   duns d
10  JOIN   subset s
11  ON     d.duns_loc = s.duns_loc
12  WHERE  CONTAINS (TEXT_KEY, :search,1) > 0
13  ORDER  BY score(1) DESC
14  /
  SCORE(1)   DUNS_LOC TEXT_KEY
        18          1 highway
1 row selected.
Execution Plan
Plan hash value: 4228563783
| Id  | Operation                      | Name              | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT               |                   |     2 |    84 |   121   (4)| 00:00:02 |
|   1 |  SORT ORDER BY                 |                   |     2 |    84 |   121   (4)| 00:00:02 |
|*  2 |   HASH JOIN                    |                   |     2 |    84 |   120   (3)| 00:00:02 |
|   3 |    NESTED LOOPS                |                   |    38 |  1292 |    50   (2)| 00:00:01 |
|   4 |     TABLE ACCESS BY INDEX ROWID| DUNS              |    38 |  1102 |    11   (0)| 00:00:01 |
|*  5 |      DOMAIN INDEX              | DUNS_TEXT_KEY_IDX |       |       |     4   (0)| 00:00:01 |
|*  6 |     INDEX RANGE SCAN           | DUNS_DUNS_LOC_IDX |     1 |     5 |     1   (0)| 00:00:01 |
|*  7 |    TABLE ACCESS FULL           | PRIMARY_CONTACT   |  4224 | 33792 |    70   (3)| 00:00:01 |
Predicate Information (identified by operation id):
   2 - access("D"."DUNS_LOC"="PC"."DUNS_LOC")
   5 - access("CTXSYS"."CONTAINS"("D"."TEXT_KEY",:SEARCH,1)>0)
   6 - access("D"."DUNS_LOC"="D"."DUNS_LOC")
   7 - filter("PC"."EMP_ID"=TO_NUMBER(:EMPLOYEEID))
SCOTT@orcl_11gR2> -- queries with better plans (no nested loops):
SCOTT@orcl_11gR2> -- subquery factoring (with) clauses:
SCOTT@orcl_11gR2> WITH
  2    subset1 AS
  3        (SELECT pc.duns_loc
  4         FROM      primary_contact pc
  5         WHERE  pc.emp_id = :employeeID),
  6    subset2 AS
  7        (SELECT score(1), d.*
  8         FROM      duns d
  9         WHERE  CONTAINS (TEXT_KEY, :search,1) > 0)
10  SELECT subset2.*
11  FROM   subset1, subset2
12  WHERE  subset1.duns_loc = subset2.duns_loc
13  ORDER  BY score(1) DESC
14  /
  SCORE(1)   DUNS_LOC TEXT_KEY
        18          1 highway
1 row selected.
Execution Plan
Plan hash value: 153618227
| Id  | Operation                     | Name              | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT              |                   |    38 |  1406 |    83   (5)| 00:00:01 |
|   1 |  SORT ORDER BY                |                   |    38 |  1406 |    83   (5)| 00:00:01 |
|*  2 |   HASH JOIN                   |                   |    38 |  1406 |    82   (4)| 00:00:01 |
|   3 |    TABLE ACCESS BY INDEX ROWID| DUNS              |    38 |  1102 |    11   (0)| 00:00:01 |
|*  4 |     DOMAIN INDEX              | DUNS_TEXT_KEY_IDX |       |       |     4   (0)| 00:00:01 |
|*  5 |    TABLE ACCESS FULL          | PRIMARY_CONTACT   |  4224 | 33792 |    70   (3)| 00:00:01 |
Predicate Information (identified by operation id):
   2 - access("PC"."DUNS_LOC"="D"."DUNS_LOC")
   4 - access("CTXSYS"."CONTAINS"("TEXT_KEY",:SEARCH,1)>0)
   5 - filter("PC"."EMP_ID"=TO_NUMBER(:EMPLOYEEID))
SCOTT@orcl_11gR2> -- inline views (sub-queries in the from clause):
SCOTT@orcl_11gR2> SELECT subset2.*
  2  FROM   (SELECT pc.duns_loc
  3            FROM   primary_contact pc
  4            WHERE  pc.emp_id = :employeeID) subset1,
  5           (SELECT score(1), d.*
  6            FROM   duns d
  7            WHERE  CONTAINS (TEXT_KEY, :search,1) > 0) subset2
  8  WHERE  subset1.duns_loc = subset2.duns_loc
  9  ORDER  BY score(1) DESC
10  /
  SCORE(1)   DUNS_LOC TEXT_KEY
        18          1 highway
1 row selected.
Execution Plan
Plan hash value: 153618227
| Id  | Operation                     | Name              | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT              |                   |    38 |  1406 |    83   (5)| 00:00:01 |
|   1 |  SORT ORDER BY                |                   |    38 |  1406 |    83   (5)| 00:00:01 |
|*  2 |   HASH JOIN                   |                   |    38 |  1406 |    82   (4)| 00:00:01 |
|   3 |    TABLE ACCESS BY INDEX ROWID| DUNS              |    38 |  1102 |    11   (0)| 00:00:01 |
|*  4 |     DOMAIN INDEX              | DUNS_TEXT_KEY_IDX |       |       |     4   (0)| 00:00:01 |
|*  5 |    TABLE ACCESS FULL          | PRIMARY_CONTACT   |  4224 | 33792 |    70   (3)| 00:00:01 |
Predicate Information (identified by operation id):
   2 - access("PC"."DUNS_LOC"="D"."DUNS_LOC")
   4 - access("CTXSYS"."CONTAINS"("TEXT_KEY",:SEARCH,1)>0)
   5 - filter("PC"."EMP_ID"=TO_NUMBER(:EMPLOYEEID))
SCOTT@orcl_11gR2> -- ansi join:
SCOTT@orcl_11gR2> SELECT SCORE(1), duns.*
  2  FROM   duns
  3  JOIN   primary_contact
  4  ON     duns.duns_loc = primary_contact.duns_loc
  5  WHERE  CONTAINS (duns.text_key, :search, 1) > 0
  6  AND    primary_contact.emp_id = :employeeid
  7  ORDER  BY SCORE(1) DESC
  8  /
  SCORE(1)   DUNS_LOC TEXT_KEY
        18          1 highway
1 row selected.
Execution Plan
Plan hash value: 153618227
| Id  | Operation                     | Name              | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT              |                   |    38 |  1406 |    83   (5)| 00:00:01 |
|   1 |  SORT ORDER BY                |                   |    38 |  1406 |    83   (5)| 00:00:01 |
|*  2 |   HASH JOIN                   |                   |    38 |  1406 |    82   (4)| 00:00:01 |
|   3 |    TABLE ACCESS BY INDEX ROWID| DUNS              |    38 |  1102 |    11   (0)| 00:00:01 |
|*  4 |     DOMAIN INDEX              | DUNS_TEXT_KEY_IDX |       |       |     4   (0)| 00:00:01 |
|*  5 |    TABLE ACCESS FULL          | PRIMARY_CONTACT   |  4224 | 33792 |    70   (3)| 00:00:01 |
Predicate Information (identified by operation id):
   2 - access("DUNS"."DUNS_LOC"="PRIMARY_CONTACT"."DUNS_LOC")
   4 - access("CTXSYS"."CONTAINS"("DUNS"."TEXT_KEY",:SEARCH,1)>0)
   5 - filter("PRIMARY_CONTACT"."EMP_ID"=TO_NUMBER(:EMPLOYEEID))
SCOTT@orcl_11gR2> -- old join:
SCOTT@orcl_11gR2> SELECT SCORE(1), duns.*
  2  FROM   duns, primary_contact
  3  WHERE  CONTAINS (duns.text_key, :search, 1) > 0
  4  AND    duns.duns_loc = primary_contact.duns_loc
  5  AND    primary_contact.emp_id = :employeeid
  6  ORDER  BY SCORE(1) DESC
  7  /
  SCORE(1)   DUNS_LOC TEXT_KEY
        18          1 highway
1 row selected.
Execution Plan
Plan hash value: 153618227
| Id  | Operation                     | Name              | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT              |                   |    38 |  1406 |    83   (5)| 00:00:01 |
|   1 |  SORT ORDER BY                |                   |    38 |  1406 |    83   (5)| 00:00:01 |
|*  2 |   HASH JOIN                   |                   |    38 |  1406 |    82   (4)| 00:00:01 |
|   3 |    TABLE ACCESS BY INDEX ROWID| DUNS              |    38 |  1102 |    11   (0)| 00:00:01 |
|*  4 |     DOMAIN INDEX              | DUNS_TEXT_KEY_IDX |       |       |     4   (0)| 00:00:01 |
|*  5 |    TABLE ACCESS FULL          | PRIMARY_CONTACT   |  4224 | 33792 |    70   (3)| 00:00:01 |
Predicate Information (identified by operation id):
   2 - access("DUNS"."DUNS_LOC"="PRIMARY_CONTACT"."DUNS_LOC")
   4 - access("CTXSYS"."CONTAINS"("DUNS"."TEXT_KEY",:SEARCH,1)>0)
   5 - filter("PRIMARY_CONTACT"."EMP_ID"=TO_NUMBER(:EMPLOYEEID))
SCOTT@orcl_11gR2> -- in clause:
SCOTT@orcl_11gR2> SELECT SCORE(1), duns.*
  2  FROM   duns
  3  WHERE  CONTAINS (duns.text_key, :search, 1) > 0
  4  AND    duns.duns_loc IN
  5           (SELECT primary_contact.duns_loc
  6            FROM   primary_contact
  7            WHERE  primary_contact.emp_id = :employeeid)
  8  ORDER  BY SCORE(1) DESC
  9  /
  SCORE(1)   DUNS_LOC TEXT_KEY
        18          1 highway
1 row selected.
Execution Plan
Plan hash value: 3825821668
| Id  | Operation                     | Name              | Rows  | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT              |                   |    38 |  1406 |    83   (5)| 00:00:01 |
|   1 |  SORT ORDER BY                |                   |    38 |  1406 |    83   (5)| 00:00:01 |
|*  2 |   HASH JOIN SEMI              |                   |    38 |  1406 |    82   (4)| 00:00:01 |
|   3 |    TABLE ACCESS BY INDEX ROWID| DUNS              |    38 |  1102 |    11   (0)| 00:00:01 |
|*  4 |     DOMAIN INDEX              | DUNS_TEXT_KEY_IDX |       |       |     4   (0)| 00:00:01 |
|*  5 |    TABLE ACCESS FULL          | PRIMARY_CONTACT   |  4224 | 33792 |    70   (3)| 00:00:01 |
Predicate Information (identified by operation id):
   2 - access("DUNS"."DUNS_LOC"="PRIMARY_CONTACT"."DUNS_LOC")
   4 - access("CTXSYS"."CONTAINS"("DUNS"."TEXT_KEY",:SEARCH,1)>0)
   5 - filter("PRIMARY_CONTACT"."EMP_ID"=TO_NUMBER(:EMPLOYEEID))
SCOTT@orcl_11gR2>