Auto_lexer (stemming) not working for other languages(other than english)

Question

Hi All,
We are planning to use AUTO_LEXER instead of MULTI_LEXER and observed that stemming is not working for German where as it is working for English when I use AUTO_LEXER. I even used query templates but of no use.
But when i use MULTI_LEXER, creating sub lexers for German and Enlgish, Stemming is working for German also after changing the session language and setting stemmer of wod list to German (Which is expected).
Following is the code for AUTO_LEXER creation:
CREATE TABLE test_auto_lexer(pkey NUMBER(5,0),
lang_index VARCHAR2(3),
content_text CLOB,
CONSTRAINT test_auto_lexer_pk primary key (pkey))
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (1,'eng','I drive a bike')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (2,'eng','I drove a bike')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (3,'eng','I have driven a bike')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (4,'eng','Just check for other things')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (5,'eng','always drive a car')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (6,'eng','it is nothing')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (7,'ger','Ich fahre ein Fahrrad')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (8,'ger','Ich fuhr ein Fahrrad')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (9,'ger','Ich habe ein Fahrrad gefahren')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (10,'ger','Aktivieren Sie einfach für andere Dinge')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (11,'ger','immer Auto fahren')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (12,'ger','es ist nichts')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (13,'ger','sprechen')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (14,'ger','sprach')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (15,'ger','gesprochen')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (16,'ger','tale')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (17,'ger','Indlæg')
INSERT INTO test_auto_lexer(pkey,lang_index,content_text)
VALUES (18,'ger','tales')
BEGIN
ctx_ddl.create_preference('auto_lexer1', 'auto_lexer');
END;
BEGIN
ctx_ddl.set_attribute('auto_lexer1','index_stems','yes');
END;
BEGIN
ctx_ddl.set_attribute('auto_lexer1','language','danish english german');
END;
BEGIN
ctx_ddl.set_attribute('auto_lexer1','alternate_spelling','german');
END;
CREATE INDEX test_auto_lexer_ix1
ON test_auto_lexer(content_text)
INDEXTYPE IS ctxsys.context
PARAMETERS ('datastore ctxsys.direct_datastore filter ctxsys.null_filter lexer auto_lexer1')
select to_char(content_Text) from test_auto_lexer where contains(content_text,
'<query>
<textquery lang="ger" grammar="context">
$fahren
</textquery>
<score datatype="integer"/>
</query>') > 0
this doesnt work even after changing the session language. My default Sesson language is AMERICAN.
Could any one help me with this, in case I am missing something here. I would be really thankful.
Edited by: Nagendra Prasad on Jan 23, 2010 12:05 AM

Answer 1

In the future, please begin a new thread for a new question and provide a link to a related question.
According to the online documentation, Oracle only supports stemming for English, Dutch, French, German, Italian, Spanish, and Japanese. Oracle does not support Russian stemming, although you may have gotten some accidental results, due to identifying it as a different language and applying stemming rules for that language. The following manual sections show the languages available for the index_stems attribute of the basic_lexer and stemmer attribute of the basic_wordlist:
Basic_lexer attributes:
http://docs.oracle.com/cd/E11882_01/text.112/e24436/cdatadic.htm#CIHHBAAD
Basic_wordlist attributes:
http://docs.oracle.com/cd/E11882_01/text.112/e24436/cdatadic.htm#g1019326
Also, please see the following note:
Features not available in 11.2.0.3:
http://docs.oracle.com/cd/E11882_01/readmes.112/e22488/toc.htm#BABDGGDB
"Certain Oracle Text functionality based on third-party technologies, including AUTO_LEXER and CTX_ENTITY, have been disabled in this release (reference Bug 12618046). For BASIC_LEXER, the usage of the INDEX_STEMS attribute values that depend on third-party technologies, is also affected. If this impacts an existing application, contact Oracle Support Services for guidance."
Also, the auto_lexer has been replaced by the world_lexer, which only has one attribute:
http://docs.oracle.com/cd/E11882_01/text.112/e24436/cdatadic.htm#BHCJEGFJ
I don't work for Oracle, so I don't know why they do things the way they do.
So, you need to stick with the basic_wordlist for stemming, not the lexers. However, you may get better performance by combining the multi_lexer with the basic_wordlist, as the multi_lexer eliminates time spent on automatic language detection. But, you need to remember not to use the index_stems with your lexer, since it will conflict with the wordlist. I am not experiencing any time problems as you have. Please see the demo below. I have provided the whole script below that, so that you can reproduce it on your system. If you still have problems, then it may be due to differences in versions.
-- demo:
SCOTT@orcl_11gR2> select * from v$version
  2  /
BANNER
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
PL/SQL Release 11.2.0.1.0 - Production
CORE     11.2.0.1.0     Production
TNS for 64-bit Windows: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production
5 rows selected.
SCOTT@orcl_11gR2> select value from v$nls_parameters
  2  where  parameter = 'NLS_CHARACTERSET'
  3  /
VALUE
AL32UTF8
1 row selected.
SCOTT@orcl_11gR2> DROP TABLE news
  2  /
Table dropped.
SCOTT@orcl_11gR2> EXEC CTX_DDL.DROP_PREFERENCE ('my_wordlist')
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> EXEC CTX_DDL.DROP_PREFERENCE ('global_lexer')
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> EXEC CTX_DDL.DROP_PREFERENCE ('english_LEXER')
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> EXEC CTX_DDL.DROP_PREFERENCE ('german_LEXER')
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> CREATE TABLE news
  2    (pkey            NUMBER,
  3       lang            VARCHAR2 (2),
  4       short_content  CLOB)
  5  /
Table created.
SCOTT@orcl_11gR2> INSERT ALL
  2  INTO news (pkey, lang, short_content)
  3    VALUES  (1, 'en', 'I drive a bike.')
  4  INTO news (pkey, lang, short_content)
  5    VALUES  (2, 'en', 'I drove a bike.')
  6  INTO news (pkey, lang, short_content)
  7    VALUES  (3, 'en', 'I have driven a bike.')
  8  INTO news (pkey, lang, short_content)
  9    VALUES  (4, 'en', 'I always drive a car')
10  INTO news (pkey, lang, short_content)
11    VALUES  (5, 'en', 'This is nothing')
12  INTO news (pkey, lang, short_content)
13    VALUES  (6, 'de', 'Ich fahre ein Fahrrad.')
14  INTO news (pkey, lang, short_content)
15    VALUES  (7, 'de', 'Ich fuhr ein Fahrrad.')
16  INTO news (pkey, lang, short_content)
17    VALUES  (8, 'de', 'Ich habe ein Fahrrad gefahren.')
18  INTO news (pkey, lang, short_content)
19    VALUES  (9, 'de', 'Ich habe immer ein Auto fahren.')
20  INTO news (pkey, lang, short_content)
21    VALUES  (10, 'de', 'Es ist nichts.')
22  SELECT * FROM DUAL
23  /
10 rows created.
SCOTT@orcl_11gR2> BEGIN
  2    -- word list:
  3    ctx_ddl.create_preference ('my_wordlist',   'basic_wordlist');
  4    ctx_ddl.set_attribute      ('my_wordlist',   'stemmer',           'auto');
  5    -- english lexer:
  6    ctx_ddl.create_preference ('english_lexer', 'basic_lexer');
  7    -- german lexer:
  8    ctx_ddl.create_preference ('german_lexer',  'basic_lexer');
  9    ctx_ddl.set_attribute      ('german_lexer',  'composite',       'german');
10    ctx_ddl.set_attribute      ('german_lexer',  'alternate_spelling', 'german');
11    ctx_ddl.set_attribute      ('german_lexer',  'mixed_case',       'no');
12    ctx_ddl.set_attribute      ('german_lexer',  'base_letter',       'yes');
13    -- multi_lexer:
14    ctx_ddl.create_preference ('global_lexer',  'multi_lexer');
15    ctx_ddl.add_sub_lexer      ('global_lexer',  'default',            'english_lexer');
16    ctx_ddl.add_sub_lexer      ('global_lexer',  'german',            'german_lexer');
17  END;
18  /
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> create index search_news
  2  on news (short_content)
  3  indextype is ctxsys.context
  4  parameters
  5    ('lexer            global_lexer
  6        language column  lang
  7        wordlist       my_wordlist')
  8  /
Index created.
SCOTT@orcl_11gR2> EXEC DBMS_STATS.GATHER_TABLE_STATS (USER, 'NEWS')
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> COLUMN short_content FORMAT A30
SCOTT@orcl_11gR2> ALTER SESSION SET NLS_LANGUAGE = 'AMERICAN'
  2  /
Session altered.
SCOTT@orcl_11gR2> set timing on
SCOTT@orcl_11gR2> select * from news
  2  where  contains (short_content, '$drive') > 0
  3  /
      PKEY LA SHORT_CONTENT
         1 en I drive a bike.
         2 en I drove a bike.
         3 en I have driven a bike.
         4 en I always drive a car
4 rows selected.
Elapsed: 00:00:00.05
SCOTT@orcl_11gR2> select * from news
  2  where  contains (short_content, '$drove') > 0
  3  /
      PKEY LA SHORT_CONTENT
         1 en I drive a bike.
         2 en I drove a bike.
         3 en I have driven a bike.
         4 en I always drive a car
4 rows selected.
Elapsed: 00:00:00.05
SCOTT@orcl_11gR2> ALTER SESSION SET NLS_LANGUAGE = 'GERMAN'
  2  /
Session altered.
Elapsed: 00:00:00.01
SCOTT@orcl_11gR2> select * from news
  2  where  contains (short_content, '$fahr') > 0
  3  /
      PKEY LA SHORT_CONTENT
         6 de Ich fahre ein Fahrrad.
         7 de Ich fuhr ein Fahrrad.
         8 de Ich habe ein Fahrrad gefahren.
         9 de Ich habe immer ein Auto fahren
4 rows selected.
Elapsed: 00:00:00.07
SCOTT@orcl_11gR2> select * from news
  2  where  contains (short_content, '$fuhr') > 0
  3  /
      PKEY LA SHORT_CONTENT
         6 de Ich fahre ein Fahrrad.
         7 de Ich fuhr ein Fahrrad.
         8 de Ich habe ein Fahrrad gefahren.
         9 de Ich habe immer ein Auto fahren
4 rows selected.
Elapsed: 00:00:00.16-- script for you to reproduce the demo on your system:
select * from v$version
select value from v$nls_parameters
where  parameter = 'NLS_CHARACTERSET'
DROP TABLE news
EXEC CTX_DDL.DROP_PREFERENCE ('my_wordlist')
EXEC CTX_DDL.DROP_PREFERENCE ('global_lexer')
EXEC CTX_DDL.DROP_PREFERENCE ('english_LEXER')
EXEC CTX_DDL.DROP_PREFERENCE ('german_LEXER')
CREATE TABLE news
  (pkey           NUMBER,
   lang           VARCHAR2 (2),
   short_content  CLOB)
INSERT ALL
INTO news (pkey, lang, short_content)
  VALUES  (1, 'en', 'I drive a bike.')
INTO news (pkey, lang, short_content)
  VALUES  (2, 'en', 'I drove a bike.')
INTO news (pkey, lang, short_content)
  VALUES  (3, 'en', 'I have driven a bike.')
INTO news (pkey, lang, short_content)
  VALUES  (4, 'en', 'I always drive a car')
INTO news (pkey, lang, short_content)
  VALUES  (5, 'en', 'This is nothing')
INTO news (pkey, lang, short_content)
  VALUES  (6, 'de', 'Ich fahre ein Fahrrad.')
INTO news (pkey, lang, short_content)
  VALUES  (7, 'de', 'Ich fuhr ein Fahrrad.')
INTO news (pkey, lang, short_content)
  VALUES  (8, 'de', 'Ich habe ein Fahrrad gefahren.')
INTO news (pkey, lang, short_content)
  VALUES  (9, 'de', 'Ich habe immer ein Auto fahren.')
INTO news (pkey, lang, short_content)
  VALUES  (10, 'de', 'Es ist nichts.')
SELECT * FROM DUAL
BEGIN
  -- word list:
  ctx_ddl.create_preference ('my_wordlist',   'basic_wordlist');
  ctx_ddl.set_attribute     ('my_wordlist',   'stemmer',            'auto');
  -- english lexer:
  ctx_ddl.create_preference ('english_lexer', 'basic_lexer');
  -- german lexer:
  ctx_ddl.create_preference ('german_lexer',  'basic_lexer');
  ctx_ddl.set_attribute     ('german_lexer',  'composite',          'german');
  ctx_ddl.set_attribute     ('german_lexer',  'alternate_spelling', 'german');
  ctx_ddl.set_attribute     ('german_lexer',  'mixed_case',          'no');
  ctx_ddl.set_attribute     ('german_lexer',  'base_letter',         'yes');
  -- multi_lexer:
  ctx_ddl.create_preference ('global_lexer',  'multi_lexer');
  ctx_ddl.add_sub_lexer     ('global_lexer',  'default',             'english_lexer');
  ctx_ddl.add_sub_lexer     ('global_lexer',  'german',              'german_lexer');
END;
create index search_news
on news (short_content)
indextype is ctxsys.context
parameters
  ('lexer            global_lexer
    language column  lang
    wordlist         my_wordlist')
EXEC DBMS_STATS.GATHER_TABLE_STATS (USER, 'NEWS')
COLUMN short_content FORMAT A30
ALTER SESSION SET NLS_LANGUAGE = 'AMERICAN'
set timing on
select * from news
where  contains (short_content, '$drive') > 0
select * from news
where  contains (short_content, '$drove') > 0
ALTER SESSION SET NLS_LANGUAGE = 'GERMAN'
select * from news
where  contains (short_content, '$fahr') > 0
select * from news
where  contains (short_content, '$fuhr') > 0
/