Characters (multi-byte) in printjoin ignored in context index

Question

Hi everybody,
I have one problem with a challenging text search in our company. The paragraph-sign should be found in multiple searches so we need to index tokens containing it.
To make clear what I found out so far I constructed a test case. Seems to be important that the database character set is AL32UTF8 (edit: Database Version is 11.2.0.3).
So at first I built a test-table and filled it with some rows:
CREATE TABLE test_tab (test_col VARCHAR2 (30));
INSERT ALL
  INTO test_tab VALUES ('print§join')
  INTO test_tab VALUES ('print#join')
  INTO test_tab VALUES ('print*join')
  INTO test_tab VALUES ('print\join')
  INTO test_tab VALUES ('print-join')
  INTO test_tab VALUES ('print_join')
  INTO test_tab VALUES ('print~join')
  INTO test_tab VALUES ('print;join')
  INTO test_tab VALUES ('print:join')
  INTO test_tab VALUES ('print,join')
  INTO test_tab VALUES ('print.join')
  INTO test_tab VALUES ('print/join')
  INTO test_tab VALUES ('print+join')
SELECT * FROM DUAL;
After that I created index preferences and built an index:
BEGIN
  ctx_ddl.create_preference(
    preference_name => 'auf_auftrag_lexer'
   ,object_name     => 'BASIC_LEXER'
  ctx_ddl.set_attribute(
    preference_name => 'auf_auftrag_lexer'
   ,attribute_name  => 'printjoins'
   ,attribute_value => '§-./+~#_\,;:*'
  ctx_ddl.set_attribute(
    preference_name => 'auf_auftrag_lexer'
   ,attribute_name  => 'punctuations'
   ,attribute_value => '!:.;,?'
  ctx_ddl.set_attribute(
    preference_name => 'auf_auftrag_lexer'
   ,attribute_name  => 'numjoin'
   ,attribute_value => ','
  ctx_ddl.set_attribute('auf_auftrag_lexer'
   ,attribute_name  => 'alternate_spelling'
   ,attribute_value => 'german'
END;
CREATE INDEX i_test_tab_ctx ON test_tab(test_col)
  INDEXTYPE IS ctxsys.CONTEXT PARAMETERS ('LEXER auf_auftrag_lexer
                                          WORDLIST auft_fuzzy_pref
                                          STOPLIST ctxsys.empty_stoplist
                                          SYNC (EVERY "SYSDATE+1/24/30")');
So printjoins contains all the special characters used in the rows of the table.
But when we examine the results of the indexed tokens:
SELECT token_text FROM dr$i_test_tab_ctx$i;
TOKEN_TEXT
JOIN
PRINT
PRINT#JOIN
PRINT*JOIN
PRINT+JOIN
PRINT,JOIN
PRINT-JOIN
PRINT.JOIN
PRINT/JOIN
PRINT:JOIN
PRINT;JOIN
PRINT\JOIN
PRINT_JOIN
PRINT~JOIN
14 rows selected
All rows are indexed correctly cos all the special characters are defined as printjoins. One row is missing - the row containing the paragraph sign is splitted in two tokens ("JOIN" and "PRINT"). The paragraph sign is treated as not existent in the printjoins attribute value (but it is defined with the correct codepoint in the ctx-View).
The paragraph sign differs from the other special characters so far, that it is stored as a multi(2-)-byte-character in the db. Content of the test_tab table:
DUMP(TEST_COL)
Typ=1 Len=11: 112,114,105,110,116,194,167,106,111,105,110 §-Sign
Typ=1 Len=10: 112,114,105,110,116,35,106,111,105,110
Typ=1 Len=10: 112,114,105,110,116,42,106,111,105,110
Typ=1 Len=10: 112,114,105,110,116,92,106,111,105,110
Typ=1 Len=10: 112,114,105,110,116,45,106,111,105,110
Typ=1 Len=10: 112,114,105,110,116,95,106,111,105,110
Typ=1 Len=10: 112,114,105,110,116,126,106,111,105,110
Typ=1 Len=10: 112,114,105,110,116,59,106,111,105,110
Typ=1 Len=10: 112,114,105,110,116,58,106,111,105,110
Typ=1 Len=10: 112,114,105,110,116,44,106,111,105,110
Typ=1 Len=10: 112,114,105,110,116,46,106,111,105,110
Typ=1 Len=10: 112,114,105,110,116,47,106,111,105,110
Typ=1 Len=10: 112,114,105,110,116,43,106,111,105,110
13 rows selected
So I suspect the multy-byte character set as a cause of the problem. But documentation says BASIC_LEXER supports the character set. I can't determine whether the problem has to do with storing the index preferences or appears while indexing the data.
Seems to be a non-common problem - ideas anybody
Thx in Advance
Stephan
Edit:
Missed one thing - the mentioned auft_fuzzy_pref for index creation is just set for configure the index as a substring index:
EXEC ctx_ddl.create_preference('AUFT_FUZZY_PREF', 'BASIC_WORDLIST');
EXEC ctx_ddl.set_attribute('AUFT_FUZZY_PREF','SUBSTRING_INDEX','TRUE');

Answer 1

Hi Teresa,
You can test to send the XML-Mesage in "ISO-8859-1" encoding.
You Chance the encoding on XML Header:
<?xml version="1.0" encoding="ISO-8859-1"?>
maybe this works.
Regards,
Robin