Internal knowlege base and highlighted abstract (like google)

Question

Two questions about searching ...
1.) Is there a way to browse dump or explore the build-in knowledge bases?
2.) We programmed stored procedures that returns a table with hits, containing among other things also a short abstract of the documents with highlighted search expressions (like google).
We perform roughly the following sequence of queries:
NumHits := CTXSYS.CTX_QUERY.count_hits(idx,QueryString,TRUE);
OPEN SearchCursor FOR SELECT /*+ FIRST_ROWS */ score(1),title,url,summary,content_length,
TO_CHAR(date_last_modified,'DD. Mon YYYY'), content_type, pk FROM directives
WHERE CONTAINS (url, QueryString,1)>0 ORDER BY SCORE(1) DESC;
... and then for each hit line
ctx_doc.highlight( index_name => indx, textkey => key, text_query => query,
restab => res, -- out-parameter
plaintext => false -- html
Is there a more elegant and faster way to get documents and the positions of the query expression therein with one function call? Or is there a way to directly access the indices?
With highlighting the abstract the queries take 10x longer!

Answer 1

I had this issue myself, and implemented a work around that produces much faster results. I created a new CLOB column in my document database which contains the indexed BLOB and indexed it, so I have 2 indexes one on the blob, one on the clob. Upon insertion, deletion etc of a document I execute the ctx_doc.filter procedure on the blob and transfer the output to the CLOB column. To keep the indexes in sync, i conduct parallel syncs etc. While this approach may not seem ideal as it requires more space in the table, consumes more resources when altering the document table and requires extra steps to ensure proper data integrity it actually places less strain on the server when the application is run. Instead of executing the highlight, filter or markup procedure on the blob for every document returned in the query I simply execute it on the clob column which reduced the time to almost nothing. Why is this the case? When I created the index on the clob, I chose to use ctxsys.null_filter which bypasses the INSO filter and saves processing time. Now, instead of performing on the fly filtering(which is fine for one document at a time) I filter all documents when they are altered and simply execute the markup procedure on the filtered document. To trim the document, you can either pass the temporary clob to your application and trim it there or trim it in your procedure(the preferable option)