Creating of the word-frequency histogram from the Oracle Text

Question

I need make from the Oracle Text index of the "word-frequency histogram", this is list of the tokens in this index, where each token contains the list of documents that contain that token and frequency this token in the every document. Don´t anybody know how to get this data from Oracle Text index so that result will save to the table or to the text file?

Answer 1

You can use ctx_report.token_info to decipher the token_info column, but I don't think the report format that it produces is what you want. You can use a query template and specify algorithm=count to obtain the number of times a token appears in the indexed column. You can do that for every token by using the dr$...$i table, as shown below. Formatting is preserved by prefacing the code with pre enclosed in square brackets on the line above all of the code and /pre in square brackets on the line below all of the code.
SCOTT@10gXE> create table otntest
  2    (doc_id       number primary key,
  3       document  varchar2(100))
  4  /
Table created.
SCOTT@10gXE> insert all
  2  into otntest values (1, 'This is a test for generating a histogram')
  3  into otntest values (2, 'Histogram shows the list of documents that contain that token and frequency')
  4  into otntest values (3, 'frequency histogram frequency histogram frequency')
  5  select * from dual
  6  /
3 rows created.
SCOTT@10gXE> create index otntest_ctx_idx
  2  on otntest(document)
  3  indextype is ctxsys.context
  4  /
Index created.
SCOTT@10gXE> column token_text format a30
SCOTT@10gXE> select t.doc_id, i.token_text, score (1) as token_count
  2  from   otntest t,
  3           (select distinct token_text
  4            from   dr$otntest_ctx_idx$i) i
  5            where  contains
  6                  (document,
  7                   '<query>
  8                   <textquery grammar="CONTEXT">'
  9                   || i.token_text ||
10                   '</textquery>
11                   <score datatype="INTEGER" algorithm="COUNT"/>
12                   </query>',
13                   1) > 0
14  order  by doc_id, token_text
15  /
    DOC_ID TOKEN_TEXT                     TOKEN_COUNT
         1 GENERATING                               1
         1 HISTOGRAM                                1
         1 TEST                                     1
         2 CONTAIN                                  1
         2 DOCUMENTS                                1
         2 FREQUENCY                                1
         2 HISTOGRAM                                1
         2 LIST                                     1
         2 SHOWS                                    1
         2 TOKEN                                    1
         3 FREQUENCY                                3
         3 HISTOGRAM                                2
12 rows selected.
SCOTT@10gXE>