Full text search for web ? Yes or no ?

Question

Hi,
I have a DB that has more then 1.8mil records in a single table .... and would like to implement full text search or some sort of caching for quicker Web search ....
Let me describe you what I have .... The table that holds 1.8mil records is made out of 30 clob columns ... each holding text .... actually these are alphabetic columns ... so words that start with char 'A' are in the first clob ... 'B' in second 'C' in third and so forth ....
Searching is always done first by customerID and CreateDate which are both indexed columns , and then clobs are searched using instr ...
Execution plan was good ... but searching times started to increase ....
So therefor I would like to improve the search ... by implementing some sort of caching mechanism ....
I read a lot about this and found an example where I would create a table containing unique words and table for occurrences of the words ... but this would then
make like 1.8mil articles containing approc 500 words , which would then repeat through articles ... so ok there would be less then 50.000 unique words (in our language ) , but the occurrences would dramatically increase cause every word inside article has to have a link in occurrences table ... so this would then be like 900mil records inside table ..
Is this at all possible to have so many records inside single table ? And still make it quick ?
Is the Oracle Full text search the only right way in this situation ?
Any suggestions ? Did anyone implement anything like this ?
Thanks,
Kris

Answer 1

Let's start with your Oracle version. Please specify which version you run because Text capabilities vary dramatically between releases.
>
I tried using Oracle Text as suggested ... now if I understand correctly ....
CTXCAT - would be great because when new records are added, index is updated automatically .... but doesn't support CLOBs ... so no go
>
CTXCAT is a concatenated transactional index that is supposed to optimize combined searches on text and other columns. No go for you as it indeed does not support CLOB columns.
>
CONTEXT - supports CLObs , but I need to explicitly synchronize index ....
There are like 4000 inserts per day ..... and they all need to be indexed in a real-time ...
>
Not true, at least since 10g: SYNC(ON COMMIT) parameter makes this index type transactional (it's synchronized automatically on commit with this parameter set.)
>
If CTX_DLL.SYNC_INDEX procedure synchronize the whole table which is now 1.8mil records, this can take a while ... so it can't be run after inserts ....
>
It does not, it only synchronizes changed data since last sync operation.
So CONTEXT is actually perfectly suited for your needs (just redesign those 30 columns into one document column and index it.) Note that you need to regularly maintain CONTEXT indexes by scheduling CTX_DDL.OPTIMIZE_INDEX to run at off-hours and purge stale/removed data and rebuild its own internal index bitmaps for better performance. Otherwise you will see performance degrade as changes to the indexed data accumulate. You might also want to tweak initial indexing parameters, especially MEMORY parameter, as it greatly affects resulting index fragmentation - the more memory you give for initial indexing or optimization, the less fragmented and the more performant the index will be all other things equal.