Using fuzzy operator in contains

Question

I am new to all of this and would like to get more comfortable that the approach I've taken is correct. Our problem is taking user input which may not be 100% correct or slighty off and attempting to search the database with that input. The idea is the user provided '201 Wilshire Blvd' when the databse contains '201 Wiltshire Ave'. Can we find this customer in the system even if they left the 't' out of Wiltshire and not searching for Ave, Blvd, St, etc? Oracle Text fuzzy operators seems the right solution. We are using Oracle 9i. The app is written in Java/JDBC (1.3 jdk).
I tried to keep it as simple as possible but here is what I did:
1- create a table specifically for searching containing an id, varchar2(360), varchar2(666)
2- create a CONTEXT index on the first varchar
3- create a CONTEXT index on the second varchar
4- the varchar2(666) contains the address information in the following format: 'zip state city addr1 addr2'
5- triggers are defined to keep the search table in sync with its source tables
6- the indexes are re-sync'ed nightly
I created a separate search table because I was concerned over performance if I were to create indexes on the source tables. The select statements I construct look like the following:
Where CONTAINS(address, ‘24032 & MD & Frostburg’, 10) > 0 AND
CONTAINS(address, ‘1616’, 20) > 0 AND
CONTAINS(address, ‘fuzzy(Pullman, 60, 30, weight)’, 30) > 0 AND
CONTAINS(customer_name, ‘fuzzy(Acme, 60, 30, weight)’, 40) > 0
So, zip, state and city must match and street address must match. Terms extracted from the address and name are searched for using the fuzzy operator.
My concerns are:
1- performance: My search table has over 2.1M records. What can I do to improve lookups.
2- ignorance: Like I said, I'm new to all of this; am I correct that using the fuzzy operator for numbers makes no sense? So if they type '30' but meant '300' too bad?
3- accuracy: How can I use the input parameters to improve my hit rate? Given that I am indexing varchars and not a document set, does it make sense to change the min score from the conatins clause?
Sorry this is so long but I appreciate any comments/suggestions. Hope I haven't left out anything important.
David

Answer 1

Hi,
Here's a start - taking your questions one at a time:
(1) "Can we find this customer in the system even if they left the 't' out of Wiltshire..."
create table z_test (col1 varchar2(100));
insert into z_test values ('wiltshire');
commit;
create index z_test_idx on z_test(col1)
indextype is ctxsys.context;
-- SQL> column col1 format a20
-- SQL> select score(1), col1
-- 2 from z_test
-- 3 where contains(col1, '!wilshire', 1) > 0;
-- SCORE(1) COL1
-- 3 wiltshire
(2) "...and not searching for Ave, Blvd, St, etc"
truncate table z_test;
insert into z_test values ('wiltshire blvd');
insert into z_test values ('wiltshire ave');
insert into z_test values ('wilshire ave');
commit;
exec ctx_ddl.sync_index('Z_TEST_IDX')
-- SQL> COLUMN COL1 FORMAT A20
-- SQL> select score(1), col1
-- 2 from z_test
-- 3 where contains(col1, '!wilshire, ave', 1) > 0
-- 4 order by 1 desc;
-- SCORE(1) COL1
-- 52 wiltshire ave
-- 52 wilshire ave
-- 2 wiltshire blvd
-- wiltshire blvd is returned even though the search was
-- for ave - see the comma separating tokens in this case.
-- Note also the difference in score as a result.
(3) "5- triggers are defined to keep the search table in sync with its source tables"
That has to be expensive. More on this to come.
(4) "I created a separate search table because I was concerned over
performance if I were to create indexes on the source tables. "
Please explain where you anticipate performance problems that prompted
the separate table. If search, do a trace on the search and see
where Oracle spends its time. You'd be better off creating a storage
preference, storing the DR$ tables in a separate tablespace.
(5) "CONTAINS(address, ‘24032 & MD & Frostburg’, 10) > 0 AND
CONTAINS(address, ‘1616’, 20) > 0 AND
CONTAINS(address, ‘fuzzy(Pullman, 60, 30, weight)’, 30) > 0 AND
CONTAINS(customer_name, ‘fuzzy(Acme, 60, 30, weight)’, 40) > 0 "
You can (and should) simplify this query. There are three
passes at the same column and it isn't necessary. Read up on
searching using contains some more.
-Ron