Context Index and performance

Hi,
I want to create a context index on one column which contains large text. And the table contains millions of records and daily inserts happen into the same table. My question is
1.Do we need to run any procedures after inserting the records daily?
2.Is there any problem from performace point of view creating context index on the table
Thanks,
Sri

sri333 wrote:
Hi,
I want to create a context index on one column which contains large text. And the table contains millions of records and daily inserts happen into the same table. My question is
1.Do we need to run any procedures after inserting the records daily?Not for what you describe. But you didn't describe much. I guess you will do something with this table data later. It depends from that. But since you only mentioned that you insert. Then no there is nothing to do after that.
2.Is there any problem from performace point of view creating context index on the tableSure. Creating the index takes time. If the index is there new inserts will take more time.
Edited by: Sven W. on Oct 10, 2012 12:02 PM

Similar Messages

Trying to understand context indexes and contains-help

Hi
i am using
Achieving functionality of many preferences using one context index
to understand context indexes and contains
and i get the following
Error starting at line 1 in command:
begin
ctx_ddl.create_preference ('nd_mcds', 'multi_column_datastore');
ctx_ddl.set_attribute ('nd_mcds', 'columns', 'text nd, text text');
ctx_ddl.create_section_group ('nd_sg', 'basic_section_group');
ctx_ddl.add_ndata_section ('nd_sg', 'nd', 'nd');
ctx_ddl.create_preference ('test_lex', 'basic_lexer');
ctx_ddl.set_attribute ('test_lex', 'whitespace', '/\|-_+');
end;
Error report:
ORA-06550: line 5, column 15:
PLS-00302: component 'ADD_NDATA_SECTION' must be declared
ORA-06550: line 5, column 7:
PL/SQL: Statement ignored
06550. 00000 - "line %s, column %s:\n%s"
*Cause: Usually a PL/SQL compilation error.
*Action:
so i am using the following to check for the error
http://docs.oracle.com/cd/E18283_01/text.112/e16593/cddlpkg.htm#BABCBFCB
plus
oracle text application developer's guide
plus
oracle text reference
but these have not listed that error (i have even googled this in vain)
background::we were actually using catsearch but because of its downsides i want to implement this
Is Achieving functionality of many preferences using one context index a good place to start when one does not know about
context and contains??
please post any other useful link for contains and context index that even explains
1) fuzzy
2) stem
3) synonym
4) near
5) soundex
6)ndata
7)lexer
thanks in advance

Ndata is new to Oracle 11g. Your other posts indicate that you are using Oracle 10g, so you don't have ndata, so you get an error when you try to use it. If you want to use the 11g features that enable context indexes with contains to do all of the things that ctxcat indexes with catsearch do, then you need to upgrade to 11g.
The online documentation is searchable. Most things regarding Oracle Text are contained in either the Oracle Text Reference or the Oracle Text Application Developer's guide.
I suggest that you start with something very simple, then build from there.
The following is similar to your other post that used catsearch:
SCOTT@orcl_11gR2> CREATE TABLE mv_cat_seg_reg_prod
2 (cat_ids VARCHAR2 ( 7),
3 act_status VARCHAR2 (10),
4 name VARCHAR2 ( 1),
5 email VARCHAR2 ( 1),
6 address1 VARCHAR2 ( 1),
7 address2 VARCHAR2 ( 1),
8 contact_name VARCHAR2 ( 1),
9 mobile VARCHAR2 ( 1),
10 telephone VARCHAR2 ( 1))
11 /
Table created.
SCOTT@orcl_11gR2> INSERT ALL
2 INTO mv_cat_seg_reg_prod VALUES
3 ('1', 'Y', 'A', 'B', 'C', 'D', 'E', 'F', 'G')
4 INTO mv_cat_seg_reg_prod VALUES
5 ('2', 'N', 'H', 'I', 'J', 'K', 'L', 'M', 'N')
6 SELECT * FROM DUAL
7 /
2 rows created.
SCOTT@orcl_11gR2> CREATE INDEX mv_cat_seg_reg_prod_idx
2 ON mv_cat_seg_reg_prod (cat_ids)
3 INDEXTYPE IS CTXSYS.CONTEXT
4 /
Index created.
SCOTT@orcl_11gR2> SELECT token_text FROM dr$mv_cat_seg_reg_prod_idx$i
2 /
TOKEN_TEXT
1
2
2 rows selected.
SCOTT@orcl_11gR2> SELECT *
2 FROM (SELECT SCORE (1), name, email, address1, address2, contact_name, mobile, telephone
3 FROM mv_cat_seg_reg_prod
4 WHERE CONTAINS (cat_ids, '1', 1) > 0
5 AND act_status = 'Y'
6 ORDER BY DBMS_RANDOM.VALUE)
7 WHERE ROWNUM < 8
8 /
SCORE(1) N E A A C M T
 4 A B C D E F G
1 row selected.

Regarding Indexes and performance tuning.

Hi Everyone,
I need some elaborate explanation about indexes and performance tuning.
1. How do you find out whether the select query which I write is utilizing the indexes .
2. Is it true that the sequence in which the indexes are defined in se11
for eg: MANDT
KNUMH
KOPOS
your select query should also have the same sequence in the where clause else the indexes are not utilized well .
3. Is there any precautions/ special method to write select queries for proper utilization of indexes.
Thanks to all reading and answering in advance.
Rgds,
Anu.

Hi
You will find like this
If your select like this
select matnr mtart from mara into<itab>
where matnr = <>..
Go to table and see mara ..if matnr is checked as primary key,then you are using primary index.
Check secondary index tab and see if any fields,if those fields you are using in select then you are using secondary index.
2.Not sequence,check how many fields are checked tick as primary key,all thsoe are index.
3.Always try to use proper primary index in select statment and avoid nested select statements.
Thanks

Portal Context Index Creation Performance issue

Recreating Portal Context Indexes takes around 36 hours at our site (after portal upgrade from 3.0.9.8.2 to 3.0.9.8.5 as per release notes). I was following the Note:158368.1 to rebuild the indexes. Is there anything that i can do to tune this ?
thanks
subu

Unfortunately indexing is generally a fairly intensive operation and can be time consuming.
There are some things that you can do to optimize the performance of your database as a whole which may in turn help the performance of your indexing operation. Look at the Performance Guide and Reference book in the database documentation.
Much of the time spent indexing is taken up by filtering binary documents and fetch content identified by URL attributes. In the case of the later, it might be worth checking in the ctx_user_index_errors view to ensure that you don't have a lot of URL requests that are timing out. The timeout is set to 30 seconds and if there are a lot or URLs where the host cannot be resolved or the fetch times out it might be costing a lot of time during the indexing operation. This is often the case if a proxy is required to reach the URLs but the proxy has not be configured correctly.

Context index and contains operator syntax how it works ?

Hi
I create a context index on four collumns (text_prof, text_gest, text_citizen, text)
of the same table content.
When i have more than one collumn being queryed using the contains syntax, oracle display the ora 29907 error saying found duplicated labels in primary invocations .
This query works:
SELECT * FROM content WHERE cod_type = '1'
AND (UPPER(title) LIKE UPPER('%tabagismo%')
OR contains (text, 'tabagismo',1)>0
This not works:
SELECT * FROM content
WHERE cod_type = '1' AND (
UPPER(title) LIKE UPPER('%tabagismo%')
OR contains (text, 'tabagismo',1)>0
OR contains (text_citizen,'tabagismo',1)>0
OR contains (text_gest,'tabagismo',1)>0
OR contains (text_prof,'tabagismo',1)>0
How can i fix it ?
I need to query all these colluns !
Does the contains operator can be used only in one collumn?
Thank´s in advance

Hi
I create a context index on four collumns (text_prof, text_gest, text_citizen, text)
of the same table content.
When i have more than one collumn being queryed using the contains syntax, oracle display the ora 29907 error saying found duplicated labels in primary invocations .
This query works:
SELECT * FROM content WHERE cod_type = '1'
AND (UPPER(title) LIKE UPPER('%tabagismo%')
OR contains (text, 'tabagismo',1)>0
This not works:
SELECT * FROM content
WHERE cod_type = '1' AND (
UPPER(title) LIKE UPPER('%tabagismo%')
OR contains (text, 'tabagismo',1)>0
OR contains (text_citizen,'tabagismo',1)>0
OR contains (text_gest,'tabagismo',1)>0
OR contains (text_prof,'tabagismo',1)>0
How can i fix it ?
I need to query all these colluns !
Does the contains operator can be used only in one collumn?
Thank´s in advance

Context indexing and PDFs

I recently used application express to create an upload system to house word docs, Excel files and PDFs in a BLOB. I used a context index to index them and a CONTAINS query to search through the BLOB and have it find which documents contained certain words.
My Problem is some PDFs are not indexing correctly, and I wondered what limitations there were to this and if we could fix them. Has anyone else encountered this?

I am logged in as the user who owns the index.
I set a filter "token_text = 'oracle' " when I am viewing the data in the DR$TEMP_INDEX$I table and I don't get anything returned(oracle is obviously listed in the 10g product guide, but it isn't in anything else). It returns other keywords in other documents.
I am thinking it may be the embedded fonts issue? thets the only difference in the files I can see.
Message was edited by:
in3d

Context Indexes and ignoring characters

So, we're trying to get a text index to ignore apostrophe's.
insert into table values ('Arby''s');We want the above entry to be located with either of the following queries:
select *
from table
where contains(field, 'Arbys')>0and also
select *
from table
where contains(field, 'Arby''s')>0The second SQL works already, it's the former search that is finding no records. I tried adding an apostrophe to the STOPLIST, but it didn't seem to make a difference. Is there another tweak I can make, so that this works? Or am I going to need to create 2 columns, one without special characters which actually has the context index on it?
Thanks,
--=Chuck

So this is what I've setup:
SQL> begin
2
3
4
5    CTX_DDL.CREATE_STOPLIST(stoplist_name => 'TEST_APOSTROPHE',
6                            stoplist_type => 'BASIC_STOPLIST');
7
8    CTX_DDL.ADD_STOPWORD(stoplist_name => 'TEST_APOSTROPHE',
9                         stopword      => '''');
10
11
12 end;
13 /
PL/SQL procedure successfully completed.
SQL> create table test_apos (name varchar2(100));
Table created.
SQL> CREATE INDEX TEST_APOS_NAME_CTX ON TEST_APOS
2 (NAME)
3 INDEXTYPE IS CTXSYS.CONTEXT
4 PARAMETERS('STOPLIST TEST_APOSTROPHE SYNC(ON COMMIT)');
Index created.
SQL> insert into test_apos values ('Arby''s');
1 row created.
SQL> commit;
Commit complete.The following usages of CONTAINS( ) will find the record, but, they either explicitly mention the apostrophe, or, they look for the string prior to the apostrophe. I expect all of these to work regardless of the STOPLIST entry:
SQL> select * from test_apos where contains (name, 'Arby') > 0;
NAME
Arby's
SQL> select * from test_apos where contains (name, 'Arby%') > 0;
NAME
Arby's
SQL> select * from test_apos where contains (name, 'Arby''s') > 0;
NAME
Arby's
SQL> select * from test_apos where contains (name, '$(Arby)') > 0;
NAME
Arby'sNone of the following work (incl your suggestion, which I greatly appreciate, btw):
SQL> select * from test_apos where contains (name, 'Arbys') > 0;
no rows selected
SQL> select * from test_apos where contains (name, 'Arby%s') > 0;
no rows selected
SQL> select * from test_apos where contains (name, 'Arbys%') > 0;
no rows selected
SQL> select * from test_apos where contains (name, '$(Arbys)') > 0;
no rows selected--=cf

CONTEXT index creation - performance!

Hi,
I have a table with about 5Million rows. The content that needs to be indexed is of RAW datatype. The average size (length) of this field is about 50 characters (it could be more).
I am trying to index this column to perform a keyword search. DEtails are furnished below.
table:
SQL> desc kwtai
Name Null? Type
TSD_HH24 DATE
COUNTRY_CODE_ALPHA_2 VARCHAR2(2)
ONETWORK NUMBER(6)
OADDRESS VARCHAR2(25)
DNETWORK NUMBER(6)
DADDRESS VARCHAR2(25)
MESSAGE_LENGTH NUMBER
MESSAGE_CONTENT RAW(2000)
Preferences:-
begin
Ctx_Ddl.Create_Preference('mc_storage', 'BASIC_STORAGE');
ctx_ddl.set_attribute('mc_storage','I_TABLE_CLAUSE',
'tablespace large_index storage (initial 10M next 10M)');
ctx_ddl.set_attribute('mc_storage', 'K_TABLE_CLAUSE',
'tablespace large_index storage (initial 10M next 10M)');
ctx_ddl.set_attribute('mc_storage', 'R_TABLE_CLAUSE',
'tablespace large_index storage (initial 1M) lob (data) store as (cache)');
ctx_ddl.set_attribute('mc_storage', 'N_TABLE_CLAUSE',
'tablespace large_index storage (initial 1M)');
ctx_ddl.set_attribute('mc_storage', 'I_INDEX_CLAUSE',
'tablespace large_index storage (initial 1M) compress 2');
ctx_ddl.create_preference('mc_lex', 'BASIC_LEXER');
ctx_ddl.set_attribute('mc_lex', 'skipjoins', '_-"''`~!@#$%^&*()+=|}{[]\:;<>?/.,');
ctx_ddl.set_attribute('mc_lex', 'INDEX_STEMS','NONE');
end;
create index kwtaidx on kwtai (message_content) indextype is ctxsys.context
parameters (' lexer mc_lex storage mc_storage memory 500M ')
parallel 16;
This create index takes about 4 hours to complete on a 8CPU dual core machine.
This is on Oracle 10g (10.2.0.4)
The reason i am creating the index as opposed to syncing it is because the data gets loaded into this table only once a day and it gets cleared once my keyword analysis is done.
Any pointers to speed up the index creation will be really appreciated! Thanks in advance!

My base table has the text that needs to be indexed stored in the "MESSAGE_CONTENT" column which for now is RAW data type. The data stored in this table are in hex representation.
Some examples -
MESSAGE_CONTENT
616C70686120626574612067616D6D612064656C746120657073696C6F6E207A657461206E69F16F
616C70686120626574612067616D6D612064656C746120657073696C6F6E207A657461
616C70686120626574612067616D6D612064656C746120657073696C6F6E207A657461206E69C3B16F
6865792E2C2C77686174277320676F696E67206F6E2E2E2E7066206368616E67277320697320736F6D652072657374617572616E742E2074686579206172652070736564756F2D636F6F6C
54686520477265656B20616C7068616265742069732074686520736372697074207468617420686173206265656E
54686520477265656B20616C7068616265742069732074686520736372697074207468617420686173206265656E20706F73742D64617461
Now with your suggestion i tried to bypass this. So what i did was added a format column to my base table and updated it to "TEXT". My database is in UTF8.
Now when i create the index with the following preferences it takes less than a minute.
begin
Ctx_Ddl.Create_Preference('kwta_storage', 'BASIC_STORAGE');
ctx_ddl.set_attribute('kwta_storage','I_TABLE_CLAUSE',
'tablespace TEXT_INDEX storage (initial 10M next 10M)');
ctx_ddl.set_attribute('kwta_storage', 'K_TABLE_CLAUSE',
'tablespace TEXT_INDEX storage (initial 10M next 10M)');
ctx_ddl.set_attribute('kwta_storage', 'R_TABLE_CLAUSE',
'tablespace TEXT_INDEX storage (initial 1M) lob (data) store as (cache)');
ctx_ddl.set_attribute('kwta_storage', 'N_TABLE_CLAUSE',
'tablespace TEXT_INDEX storage (initial 1M)');
ctx_ddl.set_attribute('kwta_storage', 'I_INDEX_CLAUSE',
'tablespace TEXT_INDEX storage (initial 1M) compress 2');
ctx_ddl.create_preference('mylex', 'BASIC_LEXER');
ctx_ddl.set_attribute('mylex', 'skipjoins', '_-"''`~!@#$%^&*()+=|}{[]\:;<>?/,');
ctx_ddl.set_attribute('mylex','punctuations','.?!');
ctx_ddl.set_attribute('mylex', 'INDEX_STEMS','NONE');
ctx_ddl.set_attribute('mylex', 'continuation','\-');
Ctx_Ddl.Create_Stoplist ( 'mystop' );
Ctx_Ddl.Add_Stopword ( 'mystop', 'is' );
Ctx_Ddl.Add_Stopword ( 'mystop', 'has' );
Ctx_Ddl.Add_Stopword ( 'mystop', 'the' );
Ctx_Ddl.Add_Stopword ( 'mystop', 'that' );
end;
create index kwtaidx on kwtai (message_content) indextype is ctxsys.context
parameters ('filter ctxsys.auto_filter format column fmt stoplist mystop lexer mylex storage kwta_storage memory 500M')
parallel 16;
When i select distinct tokens from the $I table i get the following
TOKEN_TEXT
RESTAURANT
GREEK
WHATS
ARE
DELTA
ZETA
ALPHA
ALPHABET
EPSILON
PF
PSEDUOCOOL
SOME
CHANGS
NIÃO
ON
POSTDATA
SCRIPT
BEEN
GAMMA
GOING
HEY
NI
O
THEY
BETA
Now what i am also wondering is if the text (message_content column) is being converted to UTF8 (database characterset) by using AUTO_FILTER. Is my assumption correct? Not sure how to validate this?
And, would you kindly share of why RAW must not be used in this case?
Thanks for all your pointers!

Substring search with Oracle context indexes

Hi,
i would like to know if it is possibile to do a substring search with one of the obtion offer with the context indexes.
(ctxcat,ctxrule,context)
example:
i would like to search the word 'berub' in a column A in table_example.
the value in the column a are :
The betther
berube
A.berube
berub
Berub
BERUB
R berube
S tartif
Y Thibeault
the rows return should be :
berube
A.berube
berub
Berub
BERUB
R berube
A simple sql could be
select * from table_example where upper(a) like upper('%berub%' );
How i can do this same action with the context indexes and a select (catsearch, contains, matches), if it is possible?
A example will be welcome
Thanks

I know how to do explain plan.
my point is not the query i post, it's just a example.
I have many query on my production we optimize many times (they past from 3min to 15 sec with optimisation, but we want to have better result). At this point we are looking to implant the context indexes to make them more efficient.
Do make this sql more efficient we have to deal with like '%xxxxxx%' and the context indexes like to be a option, but we have to be able to do some substring search with context option.
Is it possible to do it and how?
This is my question and why i post it here. The query is just a simple example to illsutrate what i want.
Thanks to anyone who can answer my question.

Optimizing context index

I have created a context index and want to optimize it regularly. I issue the following statement:
ctx_ddl.optimize_index('MY_INDEX','FAST');
In the Enterprise Manager I can see, that there are a lot of Sort/Merge-operations, but the optimize process doesn't finish.
(During optimize there are inserts into the table and the index is updated by the ctx_ddl.sync_index procedure).
Can anyone tell me why the optimize doesn't finish? How can I avoid this situation?

How long did you leave it running?Optimizing ran more than 10 hours. So I cancelled it, because this is not acceptable.
Optimization is an intensive process, and can sometimes take longer than the original index creation.
If you want it to complete in less time, consider using FULL optimize, but setting a time limit on it. That way it will do as much as it can within the time limit, and then start again next time from where it finished last time.During the night, I start a FULL optimize with MAXTIME = 60 minutes. But this seems not to be enough. If I don't start FAST optimizing during the day, the index get to much fragmentated.
Alternatively, if optimization is taking a lot longer than creating the index, consider dropping the index and recreating it from scratch. This will require downtime on the system, unless you get clever and use the USER_DATASTORE to create two indexes on different, dummy, columns - and switch searching between them when you want to rebuild the index. Make sure you use a generous setting for INDEX_MEMORY to avoid fragmentation.
- Roger

Performance of context index with sorting

Dear All,
I've got a problem and don't know how to solve this.
there has a table which have a XMLTYPE field to store the unstructred xml, and created with context index.
When I try to select a record from it by using contains (res, '[searchingfield]')>0, the response time is quick, but when I try to order by another field which in the same table, the response time is drop down slightly. (ex. select id, path, res, update_date from testingtbl where contains(res, 'shopper')>0 order by update_date desc.
Actually there is a context index build for field 'res', any other index build for field 'update_date', when sql without 'order by update_date', the context index will use, but the update_date index will not be used even have ordering criteria.
Is there any expect can tell how to solve this? how to keep the performance even doing the sorting process?
Thanks and Regards
Raymond

Thanks for your quick reply.
The mentions information provide after back to office, actually I just want to know if there is any method(s) which can use the context index (with contains keyword) and sorting without slow down the performance.
Thanks and Regards
Raymond

Slow performance for context index

Hi, I'm just a newbie here in forum and I would like ask for your expertise about oracle context index. I have my sql and I'm using wild character for searching '%%' .
I used the sql below with a context index (ctxsys.context) in order to avoid full table scan for wild character searching.
SELECT BODY_ID
 TITLE, trim(upper(title)) as title_sort,
 SUM(JAN) as JAN,
 SUM(FEB) as FEB,
 SUM(MAR) as MAR,
 SUM(APR) as APR,
 SUM(MAY) as MAY,
 SUM(JUN) as JUN,
 SUM(JUL) as JUL,
 SUM(AUG) as AUG,
 SUM(SEP) as SEP,
 SUM(OCT) as OCT,
 SUM(NOV) as NOV,
 SUM(DEC) AS DEC
 FROM APP_REPCBO.CBO_TURNAWAY_REPORT
 WHERE contains (BODY_ID,'%240103%') >0 and
PERIOD BETWEEN '1201' AND '1212'
 GROUP BY BODY_ID, trim(upper(title))
But i was surprised that performance was very slow, and when I try this on explain plan time of performance almost consume 2 hours.
plan FOR succeeded.
PLAN_TABLE_OUTPUT
Plan hash value: 814472363
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 1052K| 97M| | 805K (1)| 02:41:12 |
| 1 | HASH GROUP BY | | 1052K| 97M| 137M| 805K (1)| 02:41:12 |
|* 2 | TABLE ACCESS BY INDEX ROWID| CBO_TURNAWAY_REPORT | 1052K| 97M| | 782K (1)| 02:36:32 |
|* 3 | DOMAIN INDEX | CBO_REPORT_BID_IDX | | | | 663K (0)| 02:12:41 |
Predicate Information (identified by operation id):
2 - filter("PERIOD">='1201' AND "PERIOD"<='1212')
3 - access("CTXSYS"."CONTAINS"("BODY_ID",'%240103%')>0)
16 rows selected
oracle version: Oracle Database 11g Release 11.1.0.7.0 - 64bit Production
Thanks,
Zack

Hi Rod,
Thanks for the reply, yes I already made gather stats on that table, including rebuild index.
but its so strange when I use another body_id the performance will vary.
SQL> EXPLAIN PLAN FOR
2 SELECT BODY_ID
3 TITLE, trim(upper(title)) as title_sort,
4 SUM(JAN) as JAN,
5 SUM(FEB) as FEB,
6 SUM(MAR) as MAR,
7 SUM(APR) as APR,
8 SUM(MAY) as MAY,
9 SUM(JUN) as JUN,
10 SUM(JUL) as JUL,
11 SUM(AUG) as AUG,
12 SUM(SEP) as SEP,
13 SUM(OCT) as OCT,
14 SUM(NOV) as NOV,
15 SUM(DEC) as DEC
16 FROM WEB_REPCBO.CBO_TURNAWAY_REPORT
17 WHERE contains (BODY_ID,'%119915311%')> 0 and
18 PERIOD BETWEEN '1201' AND '1212'
19 GROUP BY BODY_ID, trim(upper(title));
SELECT * FROM TABLE (dbms_xplan.display);
Explained.
SQL>
Explained.
SQL>
PLAN_TABLE_OUTPUT
Plan hash value: 814472363
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 990 | 96030 | 1477 (1)| 00:00:18 |
| 1 | HASH GROUP BY | | 990 | 96030 | 1477 (1)| 00:00:18 |
|* 2 | TABLE ACCESS BY INDEX ROWID| CBO_TURNAWAY_REPORT | 990 | 96030 | 1475 (0)| 00:00:18 |
|* 3 | DOMAIN INDEX | CBO_REPORT_BID_IDX | | | 647 (0)| 00:00:08 |
Predicate Information (identified by operation id):
2 - filter("PERIOD">='1201' AND "PERIOD"<='1212')
3 - access("CTXSYS"."CONTAINS"("BODY_ID",'%119915311%')>0)
16 rows selected.

Oracle 10g – Performance with BIG CONTEXT indexes

I would like to use Oracle XE 10.2.0.1.0 only for the full-text searching of the files residing outside the database on the FTP server.
Recently I have found out that size of the files to be indexed is 5GB.
As I have read somewhere on this forum before size of the index should be 30-40% of the indexed text files (so with formatted documents like PDF or DOC even less).
Lets say that the CONTEXT index size over these files will be 1.5-2GB.
Number of the concurrent user will be max. 5.
I can not easily test it my self yet.
Does anybody have any experience with Oracle XE or other Oracle Database edition performance with the CONTEXT index this BIG?
Will Oracle XE hardware resources license limitation be sufficient to handle one CONTEXT indexe this BIG?
(Oracle XE license limitations: 1 GB RAM and 1 CPU)
Regards.

That depends on at least three things:
(1) what is the range of words that will appear in the document set (wide range of documents = smaller resultsets = better performance)
(2) how precise are the user's queries likely to be (more precise = smaller resultsets = better performance)
(3) how many milliseconds are your users willing to wait for results
So, unfortunately, you'll probably have to experiment a bit before you'll know...

Performance Problems - Index and Statistics

Dear Gurus,
I am having problems lossing indexes and statistics on cubes ,it seems my indexes are too old which in fact are not too old just created a month back and we check indexes daily and it returns us RED on the manage TAB.
please help

Dear Mr Syed ,
Solution steps I mentioned in my previous reply itself explains so called RE-ORG of tables;however to clarify more on that issue.
Occasionally,ORACLE Cost-Based Optimizer may calculate the estimated costs for a Full Table Scan lower than those for an Index Scan, although the actual runtime of an access via an index would be considerably lower than the runtime of the Full Table Scan,Some Imperative points to be considered in order to perk up the performance and improve on quandary areas such as extensive running times for Change runs & Aggregate activate & fill ups.
Performance problems based on a wrong optimizer decision would show that there is something serious missing at Database level and we need to RE_ORG the degenerated indexes in order to perk up the overall performance and avoid daily manual (RSRV+RSNAORA)activities on almost similar indexes.
For Re-organizing degenerated indexes 3 options are available-
1) DROP INDEX ..., and CREATE INDEX 
2)ALTER INDEX <index name> REBUILD (ONLINE PARALLEL x NOLOGGING)
3) ALTER INDEX <index name> COALESCE [as of Oracle 8i (8.1) only]
Each option has its Pros & Cons ,option 2 seems to be having lot of advantages to
Advantages- option 2
1)Fast storage in a different table space possible
2)Creates a new index tree
3)Gives the option to change storage parameters without deleting the index
4)As of Oracle 8i (8.1), you can avoid a lock on the table by specifying the ONLINE option. In this case, Oracle waits until the resource has been released, and then starts the rebuild. The "resource busy" error no longer occurs.
I would still leave the Database tech team be the best to judge and take a call on these.
These modus operandi could be institutionalized for all fretful cubes & its indexes as well.
However,I leave the thoughts with you.
Hope it Helps
Chetan
@CP..

Creating a single context index on a one-to-many and lookup table

Hello,
I've been successfully setting up text indexes on multiple columns on the same table (using MULTI_COLUMN_DATASTORE preferences), but now I have a situation with a one-to-many data collection table (with a FK to a lookup table), and I need to search columns across both of these tables. Sample code below, more of my chattering after the code block:
CREATE TABLE SUBMISSION
( SUBMISSION_ID NUMBER(10) NOT NULL,
SUBMISSION_NAME VARCHAR2(100) NOT NULL
CREATE TABLE ADVISOR_TYPE
( ADVISOR_TYPE_ID NUMBER(10) NOT NULL,
ADVISOR_TYPE_NAME VARCHAR2(50) NOT NULL
CREATE TABLE SUBMISSION_ADVISORS
( SUBMISSION_ADVISORS_ID NUMBER(10) NOT NULL,
SUBMISSION_ID NUMBER(10) NOT NULL,
ADVISOR_TYPE_ID NUMBER(10) NOT NULL,
FIRST_NAME VARCHAR(50) NULL,
LAST_NAME VARCHAR(50) NULL,
SUFFIX VARCHAR(20) NULL
INSERT INTO SUBMISSION (SUBMISSION_ID, SUBMISSION_NAME) VALUES (1, 'Some Research Paper');
INSERT INTO SUBMISSION (SUBMISSION_ID, SUBMISSION_NAME) VALUES (2, 'Thesis on 17th Century Weather Patterns');
INSERT INTO SUBMISSION (SUBMISSION_ID, SUBMISSION_NAME) VALUES (3, 'Statistical Analysis on Sunny Days in March');
INSERT INTO ADVISOR_TYPE (ADVISOR_TYPE_ID, ADVISOR_TYPE_NAME) VALUES (1, 'Department Chair');
INSERT INTO ADVISOR_TYPE (ADVISOR_TYPE_ID, ADVISOR_TYPE_NAME) VALUES (2, 'Department Co-Chair');
INSERT INTO ADVISOR_TYPE (ADVISOR_TYPE_ID, ADVISOR_TYPE_NAME) VALUES (3, 'Professor');
INSERT INTO ADVISOR_TYPE (ADVISOR_TYPE_ID, ADVISOR_TYPE_NAME) VALUES (4, 'Associate Professor');
INSERT INTO ADVISOR_TYPE (ADVISOR_TYPE_ID, ADVISOR_TYPE_NAME) VALUES (5, 'Scientist');
INSERT INTO SUBMISSION_ADVISORS (SUBMISSION_ADVISORS_ID, SUBMISSION_ID, ADVISOR_TYPE_ID, FIRST_NAME, LAST_NAME, SUFFIX) VALUES (1,1,2,'John', 'Doe', 'PhD');
INSERT INTO SUBMISSION_ADVISORS (SUBMISSION_ADVISORS_ID, SUBMISSION_ID, ADVISOR_TYPE_ID, FIRST_NAME, LAST_NAME, SUFFIX) VALUES (2,1,2,'Jane', 'Doe', 'PhD');
INSERT INTO SUBMISSION_ADVISORS (SUBMISSION_ADVISORS_ID, SUBMISSION_ID, ADVISOR_TYPE_ID, FIRST_NAME, LAST_NAME, SUFFIX) VALUES (3,2,3,'Johan', 'Smith', NULL);
INSERT INTO SUBMISSION_ADVISORS (SUBMISSION_ADVISORS_ID, SUBMISSION_ID, ADVISOR_TYPE_ID, FIRST_NAME, LAST_NAME, SUFFIX) VALUES (4,2,4,'Magnus', 'Jackson', 'MS');
INSERT INTO SUBMISSION_ADVISORS (SUBMISSION_ADVISORS_ID, SUBMISSION_ID, ADVISOR_TYPE_ID, FIRST_NAME, LAST_NAME, SUFFIX) VALUES (5,3,5,'Williard', 'Forsberg', 'AMS');
COMMIT;I want to be able to create a text index to lump these fields together:
SUBMISSION_ADVISORS.FIRST_NAME
SUBMISSION_ADVISORS.LAST_NAME
SUBMISSION_ADVISORS.SUFFIX
ADVISOR_TYPE.ADVISOR_TYPE_NAME
I've looked at DETAIL_DATASTORE and USER_DATASTORE, but the examples in Oracle Docs for DETAIL_DATASTORE leave me a little bit perplexed. It seems like this should be pretty straightforward.
Ideally, I'm trying to avoid creating new columns, and keeping the trigger adjustments to a minimum. But I'm open to any and all suggestions. Thanks for for your time and thoughts.
-Jamie

I would create a procedure that creates a virtual document with tags, which is what the multi_column_datatstore does behind the scenes. Then I would use that procedure in a user_datastore, so the result is the same for multiple tables as what a multi_column_datastore does for one table. I would also use either auto_section_group or some other type of section group, so that you can search using WITHIN as with the multi_column_datastore. Please see the demonstration below.
SCOTT@orcl_11gR2> -- tables and data that you provided:
SCOTT@orcl_11gR2> CREATE TABLE SUBMISSION
2 ( SUBMISSION_ID NUMBER(10) NOT NULL,
3 SUBMISSION_NAME VARCHAR2(100) NOT NULL
4 )
5 /
Table created.
SCOTT@orcl_11gR2> CREATE TABLE ADVISOR_TYPE
2 ( ADVISOR_TYPE_ID NUMBER(10) NOT NULL,
3 ADVISOR_TYPE_NAME VARCHAR2(50) NOT NULL
4 )
5 /
Table created.
SCOTT@orcl_11gR2> CREATE TABLE SUBMISSION_ADVISORS
2 ( SUBMISSION_ADVISORS_ID NUMBER(10) NOT NULL,
3 SUBMISSION_ID NUMBER(10) NOT NULL,
4 ADVISOR_TYPE_ID NUMBER(10) NOT NULL,
5 FIRST_NAME VARCHAR(50) NULL,
6 LAST_NAME VARCHAR(50) NULL,
7 SUFFIX VARCHAR(20) NULL
8 )
9 /
Table created.
SCOTT@orcl_11gR2> INSERT ALL
2 INTO SUBMISSION (SUBMISSION_ID, SUBMISSION_NAME)
3 VALUES (1, 'Some Research Paper')
4 INTO SUBMISSION (SUBMISSION_ID, SUBMISSION_NAME)
5 VALUES (2, 'Thesis on 17th Century Weather Patterns')
6 INTO SUBMISSION (SUBMISSION_ID, SUBMISSION_NAME)
7 VALUES (3, 'Statistical Analysis on Sunny Days in March')
8 SELECT * FROM DUAL
9 /
3 rows created.
SCOTT@orcl_11gR2> INSERT ALL
2 INTO ADVISOR_TYPE (ADVISOR_TYPE_ID, ADVISOR_TYPE_NAME)
3 VALUES (1, 'Department Chair')
4 INTO ADVISOR_TYPE (ADVISOR_TYPE_ID, ADVISOR_TYPE_NAME)
5 VALUES (2, 'Department Co-Chair')
6 INTO ADVISOR_TYPE (ADVISOR_TYPE_ID, ADVISOR_TYPE_NAME)
7 VALUES (3, 'Professor')
8 INTO ADVISOR_TYPE (ADVISOR_TYPE_ID, ADVISOR_TYPE_NAME)
9 VALUES (4, 'Associate Professor')
10 INTO ADVISOR_TYPE (ADVISOR_TYPE_ID, ADVISOR_TYPE_NAME)
11 VALUES (5, 'Scientist')
12 SELECT * FROM DUAL
13 /
5 rows created.
SCOTT@orcl_11gR2> INSERT ALL
2 INTO SUBMISSION_ADVISORS (SUBMISSION_ADVISORS_ID, SUBMISSION_ID, ADVISOR_TYPE_ID, FIRST_NAME, LAST_NAME, SUFFIX)
3 VALUES (1,1,2,'John', 'Doe', 'PhD')
4 INTO SUBMISSION_ADVISORS (SUBMISSION_ADVISORS_ID, SUBMISSION_ID, ADVISOR_TYPE_ID, FIRST_NAME, LAST_NAME, SUFFIX)
5 VALUES (2,1,2,'Jane', 'Doe', 'PhD')
6 INTO SUBMISSION_ADVISORS (SUBMISSION_ADVISORS_ID, SUBMISSION_ID, ADVISOR_TYPE_ID, FIRST_NAME, LAST_NAME, SUFFIX)
7 VALUES (3,2,3,'Johan', 'Smith', NULL)
8 INTO SUBMISSION_ADVISORS (SUBMISSION_ADVISORS_ID, SUBMISSION_ID, ADVISOR_TYPE_ID, FIRST_NAME, LAST_NAME, SUFFIX)
9 VALUES (4,2,4,'Magnus', 'Jackson', 'MS')
10 INTO SUBMISSION_ADVISORS (SUBMISSION_ADVISORS_ID, SUBMISSION_ID, ADVISOR_TYPE_ID, FIRST_NAME, LAST_NAME, SUFFIX)
11 VALUES (5,3,5,'Williard', 'Forsberg', 'AMS')
12 SELECT * FROM DUAL
13 /
5 rows created.
SCOTT@orcl_11gR2> -- constraints presumed based on your description:
SCOTT@orcl_11gR2> ALTER TABLE submission ADD CONSTRAINT submission_id_pk
2 PRIMARY KEY (submission_id)
3 /
Table altered.
SCOTT@orcl_11gR2> ALTER TABLE advisor_type ADD CONSTRAINT advisor_type_id_pk
2 PRIMARY KEY (advisor_type_id)
3 /
Table altered.
SCOTT@orcl_11gR2> ALTER TABLE submission_advisors ADD CONSTRAINT submission_advisors_id_pk
2 PRIMARY KEY (submission_advisors_id)
3 /
Table altered.
SCOTT@orcl_11gR2> ALTER TABLE submission_advisors ADD CONSTRAINT submission_id_fk
2 FOREIGN KEY (submission_id) REFERENCES submission (submission_id)
3 /
Table altered.
SCOTT@orcl_11gR2> ALTER TABLE submission_advisors ADD CONSTRAINT advisor_type_id_fk
2 FOREIGN KEY (advisor_type_id) REFERENCES advisor_type (advisor_type_id)
3 /
Table altered.
SCOTT@orcl_11gR2> -- resulting data:
SCOTT@orcl_11gR2> COLUMN submission_name FORMAT A45
SCOTT@orcl_11gR2> COLUMN advisor FORMAT A40
SCOTT@orcl_11gR2> SELECT s.submission_name,
2 a.advisor_type_name || ' ' ||
3 sa.first_name || ' ' ||
4 sa.last_name || ' ' ||
5 sa.suffix AS advisor
6 FROM submission_advisors sa,
7 submission s,
8 advisor_type a
9 WHERE sa.advisor_type_id = a.advisor_type_id
10 AND sa.submission_id = s.submission_id
11 /
SUBMISSION_NAME ADVISOR
Some Research Paper Department Co-Chair John Doe PhD
Some Research Paper Department Co-Chair Jane Doe PhD
Thesis on 17th Century Weather Patterns Professor Johan Smith
Thesis on 17th Century Weather Patterns Associate Professor Magnus Jackson MS
Statistical Analysis on Sunny Days in March Scientist Williard Forsberg AMS
5 rows selected.
SCOTT@orcl_11gR2> -- procedure to create virtual documents:
SCOTT@orcl_11gR2> CREATE OR REPLACE PROCEDURE submission_advisors_proc
2 (p_rowid IN ROWID,
3 p_clob IN OUT NOCOPY CLOB)
4 AS
5 BEGIN
6 FOR r1 IN
7 (SELECT *
8 FROM submission_advisors
9 WHERE ROWID = p_rowid)
10 LOOP
11 IF r1.first_name IS NOT NULL THEN
12 DBMS_LOB.WRITEAPPEND (p_clob, 12, '<first_name>');
13 DBMS_LOB.WRITEAPPEND (p_clob, LENGTH (r1.first_name), r1.first_name);
14 DBMS_LOB.WRITEAPPEND (p_clob, 13, '</first_name>');
15 END IF;
16 IF r1.last_name IS NOT NULL THEN
17 DBMS_LOB.WRITEAPPEND (p_clob, 11, '<last_name>');
18 DBMS_LOB.WRITEAPPEND (p_clob, LENGTH (r1.last_name), r1.last_name);
19 DBMS_LOB.WRITEAPPEND (p_clob, 12, '</last_name>');
20 END IF;
21 IF r1.suffix IS NOT NULL THEN
22 DBMS_LOB.WRITEAPPEND (p_clob, 8, '<suffix>');
23 DBMS_LOB.WRITEAPPEND (p_clob, LENGTH (r1.suffix), r1.suffix);
24 DBMS_LOB.WRITEAPPEND (p_clob, 9, '</suffix>');
25 END IF;
26 FOR r2 IN
27 (SELECT *
28 FROM submission
29 WHERE submission_id = r1.submission_id)
30 LOOP
31 DBMS_LOB.WRITEAPPEND (p_clob, 17, '<submission_name>');
32 DBMS_LOB.WRITEAPPEND (p_clob, LENGTH (r2.submission_name), r2.submission_name);
33 DBMS_LOB.WRITEAPPEND (p_clob, 18, '</submission_name>');
34 END LOOP;
35 FOR r3 IN
36 (SELECT *
37 FROM advisor_type
38 WHERE advisor_type_id = r1.advisor_type_id)
39 LOOP
40 DBMS_LOB.WRITEAPPEND (p_clob, 19, '<advisor_type_name>');
41 DBMS_LOB.WRITEAPPEND (p_clob, LENGTH (r3.advisor_type_name), r3.advisor_type_name);
42 DBMS_LOB.WRITEAPPEND (p_clob, 20, '</advisor_type_name>');
43 END LOOP;
44 END LOOP;
45 END submission_advisors_proc;
46 /
Procedure created.
SCOTT@orcl_11gR2> SHOW ERRORS
No errors.
SCOTT@orcl_11gR2> -- examples of virtual documents that procedure creates:
SCOTT@orcl_11gR2> DECLARE
2 v_clob CLOB := EMPTY_CLOB();
3 BEGIN
4 FOR r IN
5 (SELECT ROWID rid FROM submission_advisors)
6 LOOP
7 DBMS_LOB.CREATETEMPORARY (v_clob, TRUE);
8 submission_advisors_proc (r.rid, v_clob);
9 DBMS_OUTPUT.PUT_LINE (v_clob);
10 DBMS_LOB.FREETEMPORARY (v_clob);
11 END LOOP;
12 END;
13 /
<first_name>John</first_name><last_name>Doe</last_name><suffix>PhD</suffix><submission_name>Some
Research Paper</submission_name><advisor_type_name>Department Co-Chair</advisor_type_name>
<first_name>Jane</first_name><last_name>Doe</last_name><suffix>PhD</suffix><submission_name>Some
Research Paper</submission_name><advisor_type_name>Department Co-Chair</advisor_type_name>
<first_name>Johan</first_name><last_name>Smith</last_name><submission_name>Thesis on 17th Century
Weather Patterns</submission_name><advisor_type_name>Professor</advisor_type_name>
<first_name>Magnus</first_name><last_name>Jackson</last_name><suffix>MS</suffix><submission_name>The
sis on 17th Century Weather Patterns</submission_name><advisor_type_name>Associate
Professor</advisor_type_name>
<first_name>Williard</first_name><last_name>Forsberg</last_name><suffix>AMS</suffix><submission_name
Statistical Analysis on Sunny Days inMarch</submission_name><advisor_type_name>Scientist</advisor_type_name>
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> -- user_datastore that uses procedure:
SCOTT@orcl_11gR2> BEGIN
2 CTX_DDL.CREATE_PREFERENCE ('sa_datastore', 'USER_DATASTORE');
3 CTX_DDL.SET_ATTRIBUTE ('sa_datastore', 'PROCEDURE', 'submission_advisors_proc');
4 END;
5 /
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> -- index (on optional extra column) that uses user_datastore and section group:
SCOTT@orcl_11gR2> ALTER TABLE submission_advisors ADD (any_column VARCHAR2(1))
2 /
Table altered.
SCOTT@orcl_11gR2> CREATE INDEX submission_advisors_idx
2 ON submission_advisors (any_column)
3 INDEXTYPE IS CTXSYS.CONTEXT
4 PARAMETERS
5 ('DATASTORE sa_datastore
6 SECTION GROUP CTXSYS.AUTO_SECTION_GROUP')
7 /
Index created.
SCOTT@orcl_11gR2> -- what is tokenized, indexed, and searchable:
SCOTT@orcl_11gR2> SELECT token_text FROM dr$submission_advisors_idx$i
2 /
TOKEN_TEXT
17TH
ADVISOR_TYPE_NAME
AMS
ANALYSIS
ASSOCIATE
CENTURY
CHAIR
CO
DAYS
DEPARTMENT
DOE
FIRST_NAME
FORSBERG
JACKSON
JANE
JOHAN
JOHN
LAST_NAME
MAGNUS
MARCH
PAPER
PATTERNS
PHD
PROFESSOR
RESEARCH
SCIENTIST
SMITH
STATISTICAL
SUBMISSION_NAME
SUFFIX
SUNNY
THESIS
WEATHER
WILLIARD
34 rows selected.
SCOTT@orcl_11gR2> -- sample searches across all data:
SCOTT@orcl_11gR2> VARIABLE search_string VARCHAR2(100)
SCOTT@orcl_11gR2> EXEC :search_string := 'professor'
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> SELECT s.submission_name,
2 a.advisor_type_name || ' ' ||
3 sa.first_name || ' ' ||
4 sa.last_name || ' ' ||
5 sa.suffix AS advisor
6 FROM submission_advisors sa,
7 submission s,
8 advisor_type a
9 WHERE CONTAINS (sa.any_column, :search_string) > 0
10 AND sa.advisor_type_id = a.advisor_type_id
11 AND sa.submission_id = s.submission_id
12 /
SUBMISSION_NAME ADVISOR
Thesis on 17th Century Weather Patterns Professor Johan Smith
Thesis on 17th Century Weather Patterns Associate Professor Magnus Jackson MS
2 rows selected.
SCOTT@orcl_11gR2> EXEC :search_string := 'doe'
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> /
SUBMISSION_NAME ADVISOR
Some Research Paper Department Co-Chair John Doe PhD
Some Research Paper Department Co-Chair Jane Doe PhD
2 rows selected.
SCOTT@orcl_11gR2> EXEC :search_string := 'paper'
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> /
SUBMISSION_NAME ADVISOR
Some Research Paper Department Co-Chair John Doe PhD
Some Research Paper Department Co-Chair Jane Doe PhD
2 rows selected.
SCOTT@orcl_11gR2> -- sample searches within specific columns:
SCOTT@orcl_11gR2> EXEC :search_string := 'chair'
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> SELECT s.submission_name,
2 a.advisor_type_name || ' ' ||
3 sa.first_name || ' ' ||
4 sa.last_name || ' ' ||
5 sa.suffix AS advisor
6 FROM submission_advisors sa,
7 submission s,
8 advisor_type a
9 WHERE CONTAINS (sa.any_column, :search_string || ' WITHIN advisor_type_name') > 0
10 AND sa.advisor_type_id = a.advisor_type_id
11 AND sa.submission_id = s.submission_id
12 /
SUBMISSION_NAME ADVISOR
Some Research Paper Department Co-Chair John Doe PhD
Some Research Paper Department Co-Chair Jane Doe PhD
2 rows selected.
SCOTT@orcl_11gR2> EXEC :search_string := 'phd'
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> SELECT s.submission_name,
2 a.advisor_type_name || ' ' ||
3 sa.first_name || ' ' ||
4 sa.last_name || ' ' ||
5 sa.suffix AS advisor
6 FROM submission_advisors sa,
7 submission s,
8 advisor_type a
9 WHERE CONTAINS (sa.any_column, :search_string || ' WITHIN suffix') > 0
10 AND sa.advisor_type_id = a.advisor_type_id
11 AND sa.submission_id = s.submission_id
12 /
SUBMISSION_NAME ADVISOR
Some Research Paper Department Co-Chair John Doe PhD
Some Research Paper Department Co-Chair Jane Doe PhD
2 rows selected.
SCOTT@orcl_11gR2> EXEC :search_string := 'weather'
PL/SQL procedure successfully completed.
SCOTT@orcl_11gR2> SELECT s.submission_name,
2 a.advisor_type_name || ' ' ||
3 sa.first_name || ' ' ||
4 sa.last_name || ' ' ||
5 sa.suffix AS advisor
6 FROM submission_advisors sa,
7 submission s,
8 advisor_type a
9 WHERE CONTAINS (sa.any_column, :search_string || ' WITHIN submission_name') > 0
10 AND sa.advisor_type_id = a.advisor_type_id
11 AND sa.submission_id = s.submission_id
12 /
SUBMISSION_NAME ADVISOR
Thesis on 17th Century Weather Patterns Professor Johan Smith
Thesis on 17th Century Weather Patterns Associate Professor Magnus Jackson MS
2 rows selected.

Context Index and performance

Similar Messages

Maybe you are looking for