Problem indexings hyphenized words in PDFs

Question

Hello everyone on this forum
In the new site we are building, I am using Oracle Text to implement the search functionality.
I have problems indexings hyphenized words in PDFs.
The code I used to create the content table and the Oracle Text index, is like follows
CREATE TABLE JMMC_TST_OracleText( article_id NUMBER PRIMARY KEY
, desc VARCHAR2(30)
, doc BLOB DEFAULT empty_blob()
COMMIT ;
I populated the doc column from a database column in our CMS, containing a PDF document. Just for testing, also populated it from a PDF file, using TOAD for Oracle 8.6.
EXEC CTX_DDL.create_preference( 'jmmc_BSJC_lexer2', 'BASIC_LEXER' );
EXEC CTX_DDL.SET_ATTRIBUTE( 'jmmc_BSJC_lexer2', 'SKIPJOINS', '-' );
EXEC CTX_DDL.SET_ATTRIBUTE( 'jmmc_BSJC_lexer2', 'CONTINUATION', '-' );
CREATE INDEX JMMC_TST_INDEX
ON JMMC_TST_OracleText( doc )
INDEXTYPE IS CTXSYS.CONTEXT
PARAMETERS ( 'LEXER jmmc_BSJC_lexer2
STOPLIST CTXSYS.EMPTY_STOPLIST' );
COMMIT ;
The following sql
select ctx_report.describe_index('JMMC_TST_INDEX') from dual ;
SELECT err_timestamp, err_text
FROM ctx_user_index_errors
ORDER BY err_timestamp DESC;
shows that indexing went without errors, and index was correctly created.
The word: processo
(that in the PDF is hyphenized visually as
........... pro-
cesso .....
) is indexed as 2 tokens instead of just one token, as the following sql shows
select token_text
from dr$JMMC_TST_INDEX$i
where UPPER(token_text) = UPPER('CESSO')
or UPPER(token_text) = UPPER('PRO') ;
The following query returns 1 result
SELECT SCORE(1), article_id , doc
FROM JMMC_TST_OracleText
WHERE CONTAINS( doc, 'pro cesso', 1) > 0 ;
The following query returns 0 results
SELECT SCORE(1), article_id , doc
FROM JMMC_TST_OracleText
WHERE CONTAINS( doc, 'processo', 1) > 0 ;
Strange thing is, several months ago I tried this test with the same PDF, and everything went by without any problem.
The tests were done on different machines, and on both occasions I used Oracle 10.1.0.5.0.
Looks like I'm overlooking something or maybe some obscure setting (of DB, server or system) is causing the problem.
Suddenly hyphenized words in PDFs, stopped being indexed correctly.
Searched in the manuals and in this forum, and could not find a solution. Any help from anyone in this forum ?
Thanks in advance.

Answer 1

Hello everybody on this forum
As the initiator of this thread, I am glad that after some months, someone else is looking at this issue.
To add/clear to the confusion, I have followed Roger Ford suggestion.
Here’s the test I ran
1) Created a minimal test file (using Windows Notepad) with the following content:
ABC-
DEF
Hex view of above file is:
41 42 43 2D 0D 0A 44 45 46 00
A B C - . . D E F .
2) Created test table
CREATE TABLE JMMC_TST_OracleText(
article_id NUMBER PRIMARY KEY
, fmt VARCHAR2(30)
, doc BLOB DEFAULT empty_blob()
The main difference to Roger Ford test case is: my content column is a BLOB instead of a VARCHAR2.
The reason why my doc column is a BLOB, is because the site I’m building, content come from our CMS, and has different types both text and binary (eg Word, PDFs, etc), that I need to index together.
So I use a mixed-content column in a materialized view, to prepare/consolidate/hold all contents I index.
3) I inserted 1 row in above table (using TOAD for Oracle 8.6.), putting my minimal test file in the doc column.
4) Create Preferences and Index
EXEC CTX_DDL.create_preference( 'jmmc_BSJC_lexer2', 'BASIC_LEXER' );
EXEC CTX_DDL.SET_ATTRIBUTE( 'jmmc_BSJC_lexer2', 'SKIPJOINS', chr(45) );
EXEC CTX_DDL.SET_ATTRIBUTE( 'jmmc_BSJC_lexer2', 'CONTINUATION', chr(45) );
COMMIT;
CREATE INDEX JMMC_TST_INDEX
ON JMMC_TST_OracleText( doc )
INDEXTYPE IS CTXSYS.CONTEXT
PARAMETERS ( 'LEXER jmmc_BSJC_lexer2
FILTER CTXSYS.AUTO_FILTER
STOPLIST CTXSYS.EMPTY_STOPLIST
FORMAT COLUMN fmt' );
COMMIT;
Note: the basic lexer SKIPJOINS and CONTINUATION characters, were set the same as the hyphen character used in test file.
5) Tokens indexed:
select token_text from dr$JMMC_TST_INDEX$i
Shows:
ABC
DEF
6) Filter indexed content and generate a plaintext version:
create table JMMC_filtertab (
query_id number
, document clob
commit ;
begin
ctx_doc.filter( 'JMMC_TST_INDEX', '1', 'JMMC_filtertab', '11', TRUE);
end;
Hex view of plaintext version is:
41 42 43 2D 20 20 44 45 46 00
A B C - D E F .
Note that the original end-of-line chars (0D 0A) were replaced by 2 SPACES.
It looks like the filter replaces end-of-line chars by SPACES, and feeds the lexer, with something like:
ABC- DEF (instead of: ABC-DEF) ;
So the poor lexer, sees 2 tokens and has no clue they originally were only one hyphenised token.
This is consistent with what Meta Link Note 124624.1 - Intermedia Text & Continuation Character ('-') in PDF says.
7) Just for comparison the result of Roger Ford test (using a VARCHAR column instead of a BLOB) is:
Hex view of the filtered plaintext version is:
61 62 63 2D 0D 0A 64 65 66 00
a b c - . . d e f .
So the main difference seems to be different filtering behaviour for BLOB or VARCHAR columns, in dealing with end-of-line chars.
I have also tried other combinations of INDEX/LEXER preferences (i.e. SKIPJOINS/CONTINUATION/FILTER/NEWLINE,etc), and different file types (WORD,PDF) which means I also tested with “true binary content” and different end-of-line chars.
No matter what I tried, results were all the same: if I index a BLOB column, I’m not able to index hyphenized lines correctly.
According to the manuals, the CTXSYS.AUTO_FILTER were supposed to deal correctly with mixed-content columns if given the correct information (ie FORMAT COLUMN)
Hope this triggers a response from someone.
Thanks to all the people that took time to look at this problem.