Stemmer generating tokens off stop words - why??

Question

Apparently when stemming is enabled, the stemmer generates tokens from the stop words which IMHO is contrary to the intent and purpose of the stop words (and goes against common sense).
In my case this results in the index getting bloated with tokens like 'BE' (despite BE, AM, IS, ARE, etc. being stop words) and 'INDIANA' (coming from "IN", despite IN being a stop word).
This bloating contributes (IMHO) to performance problems, so I decided to raise the question.
Here's a short example: (DB is 10.2.0.4)
Creating the test case:
create table book_table (
  book_id               integer                primary key,
  book_title     varchar2(100) not null
begin
  ctx_ddl.create_stoplist('TEST_STOPLIST', 'BASIC_STOPLIST');
  ctx_ddl.add_stopword('TEST_STOPLIST', 'be');
  ctx_ddl.add_stopword('TEST_STOPLIST', 'is');
  ctx_ddl.add_stopword('TEST_STOPLIST', 'are');
  ctx_ddl.add_stopword('TEST_STOPLIST', 'this');
  ctx_ddl.add_stopword('TEST_STOPLIST', 'these');
  ctx_ddl.add_stopword('TEST_STOPLIST', 'in');
end;
begin
  ctx_ddl.create_preference('TEST_LEXER','BASIC_LEXER');
  ctx_ddl.set_attribute('TEST_LEXER','INDEX_THEMES','NO');
  ctx_ddl.set_attribute('TEST_LEXER','INDEX_STEMS','ENGLISH');
end;
begin
  ctx_ddl.create_preference('TEST_WORDLIST','BASIC_WORDLIST');
  ctx_ddl.set_attribute('TEST_WORDLIST','STEMMER','ENGLISH');
  ctx_ddl.set_attribute('TEST_WORDLIST','FUZZY_MATCH','GENERIC');
end;
insert into book_table values (1, 'be this');
insert into book_table values (2, 'is this');
insert into book_table values (3, 'these are');
insert into book_table values (4, 'in these');
commit;
create index book_title_ix
  on book_table (book_title)
  indextype is ctxsys.context
  parameters('
    lexer          TEST_LEXER
    wordlist          TEST_WORDLIST
    stoplist        TEST_STOPLIST
    sync (on commit)
/Now let's look at the tokens:
select token_text, token_type, token_count from dr$book_title_ix$i;yields
TOKEN_TEXT                                                       TOKEN_TYPE TOKEN_COUNT
BE                                                                        9           2
INDIANA                                                                   9           1
THIS                                                                      9           2So... is this a bug like I think or a "feature"? If it is a feature, is there any way to suppress the behavior? E.g. have stemming enabled but not generate tokens off of the stop words?
Thanks!

Answer 1

This issue is reported in defect 10168916, which is still in progress.